Introduction
Putting things off for a while instead of immediately doing them can be considered lazy. In fact, most of the time it probably is. However, there are times when it’s absolutely the right thing to do. Occasionally, one needs to delay a time-consuming job for a while; it needs to be queued for future execution so that something more important can be dealt with. For this to happen, you need a broker: someone who will accept messages (e.g. jobs, tasks) from various senders (i.e. a web application), queue them up, and distribute them to the relevant parties (i.e. workers) to make use of them – all asynchronously and on demand.
In this DigitalOcean article, we aim to introduce you to the RabbitMQ project: an open-source message-broker application stack which implements the Advanced Message Queuing Protocol (AMQP) to handle the entirety of the scenario we explained above.
Messaging, Message Brokers and Queues
Messaging is a way of exchanging certain data between processes, applications, and servers (virtual and physical). These messages exchanged, helping with certain engineering needs, can consist of anything from plain text messages to blobs of binary data serving to address different needs. For this to work, an interface managed by a third party program (a middleware) is needed… welcome Message Brokers.
Message Brokers are usually application stacks with dedicated pieces covering the each stage of the exchange setup. From accepting a message to queuing it and delivering it to the requesting party, brokers handle the duty which would normally be much more cumbersome with non-dedicated solutions or simple hacks such as using a database, cron jobs, etc. They simply work by dealing with queues which technically constitute infinite buffers, to put messages and pop-and-deliver them later on to be processed either automatically or by polling.
Why use them?
These message brooking solutions act like a middleman for various services (e.g. your web application). They can be used to greatly reduce loads and delivery times by web application servers since tasks, which would normally take quite bit of time to process, can be delegated for a third party whose sole job is to perform them (e.g. workers). They also come in handy when a more “guaranteed” persistence is needed to pass information along from one place to another.
When to use them?
All put together, the core functionality explained expands to cover a multitude of areas, including-but-not-limited-to:
Allowing web servers to respond to requests quickly instead of being forced to perform resource-heavy procedures on the spot
Distributing a message to multiple recipients for consumption (e.g. processing)
Letting offline parties (i.e. a disconnected user) fetch data at a later time instead of having it lost permanently
Introducing fully asynchronous functionality to the backend systems
Ordering and prioritising tasks
Balancing loads between workers
Greatly increase reliability and uptime of your application
and much more
RabbitMQ
RabbitMQ is one of the more popular message broker solutions in the market, offered with an open-source license (Mozilla Public License v1.1) as an implementation of Advanced Message Queuing Protocol. Developed using the Erlang language, it is actually relatively easy to use and get started. It was first published in early 2007 and has since seen an active development with its latest release being version 3.2.2 (December 2013).
How does it work?
RabbitMQ works by offering an interface, connecting message senders (Publishers) with receivers (Consumers) through an exchange (Broker) which distributes the data to relevant lists (Message Queues).
APPLICATION EXCHANGE TASK LIST WORKER
[DATA] -------> [DATA] ---> [D]+[D][D][D] ---> [DATA]
Publisher EXCHANGE Queue Consumer
How is it different than the others?
RabbitMQ, unlike some other solutions, is a fully-fledged application stack (i.e. a message broker). It gives you all the tools you need to work with, instead of acting like a framework for you to implement your own. Being extremely popular, it is really easy to get going using RabbitMQ and to find answers to your questions online.
Advanced Message Queuing Protocol (AMQP) in Brief
AMQP is a widely accepted open-source standard for distributing and transferring messages from a source to a destination. As a protocol and standard, it sets a common ground for various applications and message broker middlewares to interoperate without encountering issues caused by individually set design decisions.
Installing RabbitMQ
RabbitMQ packages are distributed both with CentOS / RHEL & Ubuntu / Debian based systems. However, they are – like with most applications – outdated. The recommended way to get RabbitMQ on your system is therefore to download the package online and install manually.
Note: We will be performing our installations and perform the actions listed here on a fresh and newly created VPS due to various reasons. If you are actively serving clients and might have modified your system, in order to not break anything working and to not to run into issues, you are highly advised to try the following instructions on a new system.
Installing on CentOS 6 / RHEL Based Systems
Before installing RabbitMQ, we need to get its main dependencies such as Erlang. However, first and foremost we should update our system and its default applications.
Run the following to update our droplet:
yum -y update
And let’s use the below commands to get Erlang on our system:
# Add and enable relevant application repositories:
# Note: We are also enabling third party remi package repositories.
wget http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
wget http://rpms.famillecollet.com/enterprise/remi-release-6.rpm
sudo rpm -Uvh remi-release-6*.rpm epel-release-6*.rpm
# Finally, download and install Erlang:
yum install -y erlang
Once we have Erlang, we can continue with installing RabbitMQ:
# Download the latest RabbitMQ package using wget:
wget http://www.rabbitmq.com/releases/rabbitmq-server/v3.2.2/rabbitmq-server-3.2.2-1.noarch.rpm
# Add the necessary keys for verification:
rpm --import http://www.rabbitmq.com/rabbitmq-signing-key-public.asc
# Install the .RPM package using YUM:
yum install rabbitmq-server-3.2.2-1.noarch.rpm
Installing on Ubuntu 13 / Debian 7 Based Systems
The process for downloading and installing RabbitMQ on Ubuntu and Debian will be similar to CentOS due to our desire of having a more recent version.
Let’s begin with updating our system’s default application toolset:
apt-get update
apt-get -y upgrade
Enable RabbitMQ application repository:
echo "deb http://www.rabbitmq.com/debian/ testing main" >> /etc/apt/sources.list
Add the verification key for the package:
curl http://www.rabbitmq.com/rabbitmq-signing-key-public.asc | sudo apt-key add -
Update the sources with our new addition from above:
apt-get update
And finally, download and install RabbitMQ:
sudo apt-get install rabbitmq-server
In order to manage the maximum amount of connections upon launch, open up and edit the following configuration file using nano:
sudo nano /etc/default/rabbitmq-server
Uncomment the limit line (i.e. remove #) before saving and exit by pressing CTRL+X followed with Y.
Managing RabbitMQ
As we have mentioned before, RabbitMQ is very simple to get started with. Using the instructions below for your system, you can quickly manage its process and have it running at the system start-up (i.e. boot).
Enabling the Management Console
RabbitMQ Management Console is one of the available plugins that lets you monitor the [RabbitMQ] server process through a web-based graphical user interface (GUI).
Using this console you can:
Manage exchanges, queues, bindings, users
Monitor queues, message rates, connections
Send and receive messages
Monitor Erlang processes, memory usage
And much more
To enable RabbitMQ Management Console, run the following:
sudo rabbitmq-plugins enable rabbitmq_management
Once you’ve enabled the console, it can be accessed using your favourite web browser by visiting: http://[your droplet's IP]:15672/.
The default username and password are both set “guest” for the log in.
Note: If you enable this console after running the service, you will need to restart it for the changes to come into effect. See the relevant management section below for your operating system to be able to do it.
Managing on CentOS / RHEL Based Systems
Upon installing the application, RabbitMQ is not set to start at system boot by default.
To have RabbitMQ start as a daemon by default, run the following:
chkconfig rabbitmq-server on
To start, stop, restart and check the application status, use the following:
# To start the service:
/sbin/service rabbitmq-server start
# To stop the service:
/sbin/service rabbitmq-server stop
# To restart the service:
/sbin/service rabbitmq-server restart
# To check the status:
/sbin/service rabbitmq-server status
Managing on Ubuntu / Debian Based Systems
To start, stop, restart and check the application status on Ubuntu and Debian, use the following:
# To start the service:
service rabbitmq-server start
# To stop the service:
service rabbitmq-server stop
# To restart the service:
service rabbitmq-server restart
# To check the status:
service rabbitmq-server status
And that’s it! You now have your own message queue working on your virtual server.
Configuring RabbitMQ
RabbitMQ by default runs with its standard configuration. In general, it does not require much tempering with for most needs as long as everything runs smoothly.
To learn about configuring it for custom needs, check out its documentation for Configuration.
