An In-Depth Guide: Baivaru Viber Bot Skeleton

Project Developed by: @eyaadh
Guide by: @eyaadh
Proof read and fixes by: @izaya and @tiashe

Introduction

What is a bot you ask? A bot is a software application that is programmed to do certain tasks. That doesn’t sound enough of an explanation does it? Bots are automated, which means they run according to their instructions without a human user needing to start them up. Bots often imitate or replace a human user's behavior. Typically, they do repetitive tasks and they can do them much faster than human users could.

Now that we understand what bots are, let me also tell you that Viber’s bot platform is open and free for developers to create bots in their ecosystem. As social media and messaging apps have long become our main way to keep in touch with one another and, as such, we count on the service providers and brands on these platforms to offer us the most personal connections possible. Brands have found the answer to this in chat bots – a way for them to simulate conversation with a human through what is essentially a computer program that operates under a specific set of rules.

We are going to use Python for this project since that is all that I know in the world of computer programming. Let’s also not forget that Viber already has an official Python library which is synchronous, however my interests are in doing something asynchronous. Let me pause for a minute once again, before jumping into the fun part of  explaining you the difference between synchronous and asynchronous execution.

Synchronous vs Asynchronous Execution

Basically, when you execute a task synchronously - it executes one step at a time. Even with conditional branching, loops and function calls, you can still think about the code in terms of taking one execution step at a time. When each step is complete, the program moves on to the next one.

And when you execute a task asynchronously, it still takes one execution step at a time. The difference is that the system may not wait for an execution step to be completed before moving on to the next one. This means that the program will move on to future execution steps even though a previous step hasn’t yet finished and is still running elsewhere. This also means that the program knows what to do when a previous step does finish running.

Why do we want to write asynchronous programs you say? — because it could increase the performance of your program. Let’s say you have a single core machine on which you are running your app. The app receives a request, and you need to make two database queries to fulfill that request. Each query takes 50ms of time. With a synchronous program, you would make the second request only after completing the first — total time 100ms. With an asynchronous program, you could fire off both the queries one after the other — total time 50ms. Now it’s your turn to use your mighty brain to understand why my interests are in doing this project asynchronously rather than using the already existing synchronous library.

Viber REST API

In order to develop our Viber bot asynchronously we are going to use the Viber REST API which requires the following:

1.    An Active Viber accounton a platform which supports bots (iOS/Android). This account will automatically be set as the account administrator during the account creation process.
2.    Active bot - Create the bot from here.
3.    Account authentication token - unique account identifier used to validate your account in all API requests. Once your account is created your authentication token will appear in the account’s “edit info” screen (for admins only). Each request posted to Viber by the account will need to contain the token.
4.    Setup account webhook– I hear you ask what a webhook is, let me explain you that too. A webhook (also called a web callback or HTTP push API) is a way for an application to provide other applications with real-time information. A webhook delivers data to other applications as it happens, meaning you get data immediately. Unlike typical APIs where you would need to poll for data very frequently in order to get it real-time.

Environment Setup

Well those are what you need to begin with this development and let me also remind you again that I am using Python 3.8.1 for this demonstration. At this point I would also like to emphasize on the importance on using Python Virtual Environments as the main purpose of virtual environments is to manage settings and dependencies of a particular project regardless of other Python projects you are doing. If you are using an IDE such as PyCharm you need not worry about Virtual Environments much as virtualenv tool comes bundled with PyCharm, so the user doesn't need to install it and typically would create the virtual environment for you as you create a new project within it. (A huge thank you to Mr. Athphane who taught me these stuffs which made my life so easy at debugging since I started using them, and now it’s your turn to listen and learn).

Viber message flow

The Project

For the ease of explanation, I am going to refer the modules on the repository that I had publicized for the very purpose of this documentation. To begin with let me give you a brief explanation of the directory structure it contains, and the contents each directory has.

Directory Structure

utils - The name utils define exactly what this directory is for. All the utilities that helps the application to work through is kept in this directory.

api - api consists of modules that deals with viber API.

database - This directory contains the modules that concerns the DB/RDBMS the application uses. For the purpose of this demonstration I am using TinyDB which is a lightweight document-oriented database just like MongoDB (which would be my preferred DBMS for larger scale projects) however you are free to use whichever you prefer.

helpers - Helpers directory consists of modules that connects to 3rd party APIs/sources to help the application gather the data it requires. For this demonstration we scrape data and the scraper module is present in this directory.

webserver - This directory consists of the modules that are required for the webhook server that the application uses.

working_dir - This directory basically consists of the config.ini file/s. Also, if there are temporary files that are generated by the application, we tend to leave them within this directory at the course of their existence.

Modules

Commonly Shared Variables
Now that you have a good idea of the directory structures let’s begin with the module ViberCommon() defined on common.py located under /viber/utils/.

The purpose of this module is for sharing commonly used variables across the application such as the API keys, webhook URL and etc. You might have already noticed that I am loading the values these variables share from an external source. We load these variables from external - as a best practice since we do not want to hard code them to our program no do, we want to share our secret keys, passwords, etc. with the source as a best practice.

The configuration file - config.ini(above mentioned external source) located under the viber/working_dir/consists of sections, each led by a [section] header, followed by key/value entries separated by a specific string (= or : by default). By default, section names are case sensitive, but keys are not. Leading and trailing whitespace is removed from keys and values. Values can be omitted; in which case the key/value delimiter may also be left out. Values can also span multiple lines, as long as they are indented deeper than the first line of the value.

For this portion of the document we will only concentrate on keys which are under first section - [viber].

These are the keys that are required for setting up our application for receiving updates i.e. callback data from Viber.      

auth_token - as explained earlier unique account identifier used to validate your account in all API requests.
webhook_uri - the URL for webhook.
name - the name you would like to give your bot.
avatar - in simpler terms is the bloody profile picture for the bot.

To parse the configurations from this .ini file we use the library configparse. And this is done on __init__.pyfor viber/utils/.

Web Server for webhook
Now that we have the required keys and other settings loaded up and shared, let’s look at setting up the webserver for webhook.

It is important at this point that I also highlight for “security” reasons - only URLs with valid and official SSL certificate from a trusted CA is allowed by Viber for webhook. In case you are on a linux environment certbot might be your best friend to get a valid SSL certificate, however if you are using windows like I do you could make use of Ngrok as a reverse proxy tunnel for your project. We are going to run our python webserver on port 8080 therefore while you create the ngrok tunnel, point it to port 8080 of the local host and the tunnel address ngrok provide would be the webhook URL that is on the config.ini file.

Note: We are not going to discuss either on how to setup certbot no ngrok within this demonstration however you could refer this tutorial for ngrok and this one for certbot.

For our webserver we are going to use the library aiohttp. I hear you ask why not flask or Django - The key part of the aiohttp framework is that it is asynchronous, it can concurrently handle hundreds of requests per second without too much hassle. In comparison to frameworks such as flask, it’s incredibly performant. Use your mighty brain once again to conclude why in this case I chose aiohttp over other famous frameworks.

aiohttp server is built around aiohttp.web.Application instance. It is used for registering startup/cleanup signals, connecting routes etc.

You will hear me talk about handlers for aiohttp, a handler is a callable that accepts a single Request argument and returns a web Response derived instance.

A handler can also be a coroutine, in which case aiohttp.web will await the handler.

Handlers are connected to the web Application via routes. Routes can be registered using route tables and route decorators.

Our web application for the webhook is defined (line 6 of the below snippet) on webserver directory package (viber/utils/webserver/) as mentioned earlier.

The routes that we require for the web server are defined on routes.pyunder the same directory. (Handler – Line 9 of the below snippet, defining route table – Line 5 of the below snippet, registering route – Line 8 of the below snippet and adding the routes to the web app line 7 of the above snippet)

We will come back to explaining the handler mentioned here later within the project, for now let’s move forward with running the web server that we had defined. We do this on our main application module package (viber).

On the coroutine main() that our asyncio loop (line 35-37) runs, you could see that I have used an app runner (line 13-15) to start the web application. At a normal scenario - with aiohttp you would do run_app() to start a web application however run_app() provides a simple blocking API for running an application.

For starting the application asynchronously or serving on multiple HOST/PORT, app runners exists. As you could see from the above snippet within the same coroutine we are doing a lot more than just starting the web app, we are also starting schedulers to collect data for the helpers that application require and DB initializing for the first run (line 20-31).

After starting the app runner, we also wait for a good 2 seconds to give the application enough time for the Viber API to be available to set the webhook (line 17). Setting the webhook will be done by calling the set_webhook Viber API endpoint.

Once a set_webhookrequest is sent, Viber will send a callback to the webhook (in this case our web server) to check its availability and return a response to the user. These callback data (request) are what the root_route_post_handler(request)defined on our routes for the web server receives and handles.

In case the webhook is offline Viber will re-try to deliver the callback until HTTP status code 200 is received. There will be a retry attempt after 5 seconds, and then another after 1 minute and 5 seconds.

Any data that is posted to Viber API i.e. calling the Viber API endpoints from our program is handled by the function post(*args) defined under the module ViberApiResquestSender().

post(*args) function expects the following arguments:

uri - Instance attribute name for the respective Viber API end point defined under the class ViberMessageTypes.
payload - The data which you are posting to the API. Payloads used by the application are defined under the module ViberMessageTypes().

As already explained once we set our webhook, 1-on-1 conversation with our bot account will become available thus we will receive callback data from Viber when there is an update. To understand further more on how our application handles these callback data lets jump back to the root_route_post_handler(request) defined under our routes for the web server.

Validation of Data Received
Each callback will contain a signature on the JSON passed to the callback. The signature is HMAC with SHA256 that will use the authentication token as the key and the JSON as the value. The result will be passed as HTTP Header X-Viber-Content-Signature. We could use this to validate if the received callback data is from a valid source or otherwise bogus (line 12, 15 and validate_signature function(*args) from ViberApiResquestSender()module to calculate and compare the signatures as explained).

Handlers:
When the received data is confirmed from a valid source we call the function on_event(*args)from ViberHandlers()module. Basically, each callback data that is received from Viber contains a parameter called event - which tells us what kind of an update it is and the respective event handler tells the application what needs to be done when a certain update is received.

on_event(*args)function expects the callback data from Viber in JSON format(We collect these data on Line 10 of the routes.py) and this function checks for the following events and assigns the task to its respective event handler function:

Subscribed - Before the bot can send messages to a user, the user will need to subscribe to the bot account (Viber tends to call their bots, Public Accounts).
Unsubscribed - The user will have the option to unsubscribe from the Public Account. This will trigger an unsubscribed callback.
Message - This event is received when the bot account receives any form of an incoming message.

Each of the below explained handlers expects the same callback data from Viber in JSON format as received by on_event(*args)function. Ok, now let’s go through each handler one by one and investigate what they do:

on_subscription(*args)

When the received event is subscribed, we call this handler. The callback data received would be as below:

Viber recommends that we record the subscriber ID of each subscriber, as there’s no API for fetching all subscriber IDs for a bot (Line 38 of handlers.py we do this, it’s important we save these data – god knows what, you might even want to send a broadcast message to all your subscribers and these data comes in handy at such a situation).

Text Message (Welcome message)
Also, we do send a welcome message to the new user/subscriber. To do this we will first prepare the message object that needs to be posted to Viber.
In this case the welcome message is a text message the posted raw data to Viber API should be as the following example:

To generate a text message object with above formatted raw data we call text_message(*args) function from the module ViberMessageTypes()with the following arguments:

receiver: Unique Viber user id for receiver
text: text to include with the body of the message
tracking_data:  Allow the account to track messages and user’s replies. Sent tracking data value will be passed back with user’s reply
keyboard: if you are sending a keyboard with the message, the keyboard to include

General Parameters for Message Data
At this point, before we move any further lets also briefly look at available general parameters for the data that we post when we send a message (These parameters are common for all the message types i.e. text, media and etc)

Each API request must include an HTTP Header called X-Viber-Auth-Token containing the account’s authentication token however the message objects that are returned from the module ViberMessageTypes() does not contain the auth token. Therefore for each message object we create either it is a text message or otherwise - we add the token and do a bit of housekeeping and final preparation of the payload by calling prepare_payload(*args) function from ViberCommands()module. This function expects the following arguments:

message: message object
sender_name: senders name to display (you could leave this as None if you are defining sender)
sender_avatar: The sender’s avatar URL (you could leave this as None if you are defining sender)
sender: a dictionary with sender’s name and avatar (you could leave this as None if you are defining senders name and avatar with the above arguments)
receiver: Unique Viber user id for receiver
chat_id: you could leave this as None, it is used for internal usage.

Once the required payload is ready, we call the function post(*args)defined under the module ViberApiResquestSender() with arguments ‘send_message’ and the prepared payload to send the welcome message. This would be the same process that we will follow within the app to send any type of message.

on_unsubscribed(*args)

If the received event is unsubscribed, we call this handler. Received callback data would be as the following example:

We already know that the bot can only send messages to a user who is subscribed to it, therefore when once a user unsubscribes we remove that particular users details from our users collection from the DB (Line 43 of handlers.py).

on_message(*args)

If the received event is a message, we call this handler.

For now, we are only going to deal with the text messages that are received. A text message could be a command that was sent to the bot, or otherwise a response by a user to a conversation the bot has started (tracked using tracking data).

Validating Text (Bot Commands)
The main purpose in adding more than one bot command for this demonstration is to demo the different types of messages the bot can send, and this is where the helper modules come in play - to collect some real data with which we can demo the message types. Either way I do not plan to go in depth detailed explanation on helper modules since most of these modules are scrapers and it is for a different tutorial. Also, I hope and pray that our scraping magician and idiot Managing Director - Mr. PhoenixAtom writes a separate article on scraping.

As mentioned earlier for now we will only deal with text messages that are received. If the received text message does not contain tracking data we call the text_validator(*args)function from ViberCommands()module. This function expects the same callback data from Viber in JSON format as received by on_event(*args) function. Purpose of this function is to differentiate the commands to other text messages. We use the prefix “!” for commands, meaning we assume any text message starting with “!” as a command (Line 21 of the commands.py file). When the received text is a command we then call commands_checker(*args)from ViberCommands()module. This function expects the following arguments:

command: String value of the command.
receiver: Unique Viber user id for receiver

This function basically checks the received commands and perform respective relevant tasks according to the given/received command.

Rich Media Message / Carousel Content Message
The Rich Media message type allows sending messages with pre-defined layout, including height (rows number), width (columns number), text, images and buttons.

Commands (bills, resolutions, emergency_debates, approvals and others) formulates Rich Media Messages.

Below is an example of a Carousel Content Message, that allows a user to scroll through a list of items, each composed of an image, description and call to action button.

Each item on the list shown to the user is a button in the Rich Media message’s “Buttons” array. Sending one button is also permitted. The parameters for Rich Media message and its buttons are also used for Keyboards.

Also, you need to keep in mind that each button is limited to a maximum of 7 rows on Rich Media messages, and that forwarding is not supported for Rich Media messages.

Posted raw data to Viber API for Rich Media Messages should be as the following example:

Within our application to generate rich media messages we call the function rich_media(*args) from the module ViberMessageTypes(). This function expects the following arguments:

receiver: Unique Viber user id for receiver
data_list: list of data that needs to be included in the buttons for rich media messages.

For this demonstration to formulate these messages we scrape data from majilis.gov.mv and this is handled by the helper module MajilisCollection(). Since data scraping in real time is slow, we use schedulers to scrape data the mentioned data in given schedules and record them in our DB i.e. when ever a concerned command for the data is received we collect them from DB and send them for a faster response.

Picture message
For the purpose of this demonstration I am using unsplash.com (helper module: UnsplashPhotos()) to get random pictures i.e. to show how we can formulate and send a picture message (command: photo).

Posted raw data to Viber API for Picture Messages should be as the following example:

Within our application to generate picture messages we call the function picture_message(*args) from the module ViberMessageTypes(). This function expects the following arguments:

receiver: Unique Viber user id for receiver
text: lets call this the caption for the picture we are      sending
media: URL of the image (JPEG)
thumb: URL of a reduced size image (JPEG)
tracking_data: Allow the account to track messages and user’s replies. Sent tracking data value will be passed back with user’s reply
keyboard: if you are sending a keyboard with the message, the keyboard to include

Video Message
For the purpose of this demonstration I am using pixabay.com (helper module: PixbayVideos()) to get random videos i.e. to show how we can formulate and send a video message (command: video).

Posted raw data to Viber API for Video Messages should be as the following example:

Within our application to generate video messages we call the function video_message(*args)from the module ViberMessageTypes(). This function expects the following arguments:

receiver: Unique Viber user id for receiver
media: URL of the video (MP4, H264)
size:  Size of the video in bytes
thumbnail:  URL of a reduced size image (JPEG)
duration:  Video duration in seconds; will be displayed to the receiver
tracking_data: Allow the account to track messages and user’s replies. Sent tracking data value will be passed back with user’s reply
keyboard: if you are sending a keyboard with the message,      the keyboard to include

File Message
To demonstrate file message objects, we scrape data from gazette.gov.mv (this is done by the module GazetteCollection()). File messages are demonstrated with the command gazette. The trailing process for the command also makes use of keyboards and tracking data however we will come back to these two topics later within the documentation, for now we will only concentrate on the portion of file messages.

Posted raw data to Viber API for File Messages should be as the following example:

Within our application to generate file messages we call the function file_message(*args)from the module ViberMessageTypes(). This function expects the following arguments:

receiver: Unique Viber user id for receiver
url:  Size of the file in bytes
tracking_data: Allow the account to track messages and user’s replies. Sent tracking data value will be passed back with user’s reply
keyboard: if you are sending a keyboard with the message,      the keyboard to include

You might have noticed that we do not feed in size parameter for the function though it is required by the raw data. I did this since some of the web servers tend not to return proper headers with content-length which basically is the file size, i.e. the application will download the file to a temporary location and calculates its size on its own.

Location message
To demonstrate location messages, I have used FourSquare REST API – and its handled by the module FourSquare(). I had so much fun writing this part of the demonstration, as beautiful FourSquare’s REST API is it’s for anther tutorial. The trailing processes of command nearme for our application make use of location messages.

Posted raw data to Viber API for Location Messages should be as the following example:

Within our application to generate a location messages we call the function location_message(*args) from the module ViberMessageTypes(). This function expects the following arguments:

receiver: Unique Viber user id for receiver
lat: latitude
lon: longitude
tracking_data: Allow the account to track messages and user’s replies. Sent tracking data value will be passed back with user’s reply
keyboard: if you are sending a keyboard with the message, the keyboard to include

Contact message
The trailing processes of command nearme for our application also make use of contact message.

Posted raw data to Viber API for Contact Messages should be as the following example:

Within our application to generate a contact messages we call the function contact_message(*args)from the module ViberMessageTypes(). This function expects the following arguments:

receiver:  Unique Viber user id for receiver
name:  Name of the contact
contact: Phone number of the contact
tracking_data: Allow the account to track messages and user’s replies. Sent tracking data value will be passed back with user’s reply
keyboard: if you are sending a keyboard with the message, the keyboard to include

Sticker message
The trailing processes of command nearme for our application also make use of sticker_message.

Posted raw data to Viber API for Sticker Messages should be as the following example:

Within our application to generate a contact messages we call the function sticker_message(*args)from the module ViberMessageTypes(). This function expects the following arguments:

receiver:  Unique Viber user id for receiver
sticker_id:  Unique Viber sticker ID
tracking_data: Allow the account to track messages and user’s replies. Sent tracking data value will be passed back with user’s reply
keyboard: if you are sending a keyboard with the message, the keyboard to include

Keyboards
The Viber API allows sending a custom keyboard using the send_message API, to supply the user with a set of predefined replies or actions. The keyboard can be attached to any message type or sent on its own. Once received, the keyboard will appear to the user instead of the device’s native keyboard. The keyboards are fully customizable and can be created and designed specifically for the account’s needs. The client will always display the last keyboard that was sent to it.

As you have already noticed from the above explained message types and their respective message objects, keyboards can be attached to any message type and be sent and displayed together.

Following is an example of the posted raw data to Viber API to send a keyboard with a text message:

To demonstrate the same on our application I have used predefined keyboards on the module ViberKeyboards(). Commands gazette, nearme and admin make use of keyboards.

Tracking Data
Basically, tracking data allow the bot to track messages and user’s replies, in short to start conversation style messaging we put a tracker on the message sent so that the reply/response by the user can be tracked.

I have demonstrated this with the commands gazette, nearmeand adminhowever I am only going to explain you how the command nearmeworks.

Basically, the idea of this module is to recommend you with venues near you, when once you send your current location as an attachment (location attachment). As you send the command the bot will respond requesting you to send back the location, as it sends this text message it also adds the tracking data – “foursquare” i.e. the response/reply you send to the bot will have the same tracking data.

When once a message received with tracking data on_message(*args)handler assigns these data to the function attend(*args)from the module ViberTrackingDataAttendant(). This function expects the following arguments:

tracking_data: tracking data that was assigned to the received message.
data: callback data from Viber in JSON format as received by on_event(*args)function.

For the command nearme we first check if the received message type is a location message, if not we send a sticker message followed by a text message which tells that the user that the bot is expecting a location message not a text message, again we assign tracking data to this response as well so that the data that is received can be validated. If we do not assign the tracking data to this message the conversation track is broken unless you have a better logic in place to check for them data. As you can see this can run in a lengthy loop and to negate and break it we have also placed the command - !negatenearme. which would basically send a text message without any tracking data.

Final Thoughts

Well that pretty much concludes the project as we have already covered most of what Viber REST API has to offer. At least with which you could build a decent Viber bot. There are a few more API endpoints with which you could get user details and account details of the bot, which I would rather prefer covering on a different project.

Now let me end up this doc with a question and a statement, do I think this could be done any better? Definitely, it can be done better and I never get a project of mine totally finished – not because I am done reaching the end goals, but I learn something new everyday and find a better way of doing what I had done so far and I believe its your turn to make it better and for me to learn from you.