RabbitMQ and the AMQP protocol: I will listen to you too, just wait your turn

Hello,

I am writing this because I ran into a problem months ago and got past it with a very good and widely used solution. I wanted to share that solution briefly so that people who are as inexperienced in this area as I was can learn it too.

The problem

In a hobby project I built months ago, incoming requests were causing heavy memory use on the machine the project ran on. There was a lot of video and audio encoding involved. So once more than a certain number of requests arrived, something would break in the machine or in the project. In short, our problem was that the requests were all being processed at the same time.

The solution

So I searched for solutions online and learned that I could also handle it in code with approaches like FIFO (First In First Out) and LIFO (Last In First Out). Digging further I came across the idea of queuing protocols and the AMQP protocol (Advanced Message Queuing Protocol). After that I came across RabbitMQ, which I had heard about constantly without knowing exactly what it did.

AMQP is a protocol programmed at a low level, while RabbitMQ is a message queuing web application that can integrate with that protocol automatically. To understand the setup, I think you can draw the same relation between AMQP and RabbitMQ as between FTP and FileZilla. You could add P2P and torrent applications to the same set of examples.

I want to give you a surface level introduction to AMQP and RabbitMQ.

What AMQP is and how it came about

The protocol, which can also be written out as Advanced Message Queuing Protocol, is basically an application that lets you deliver messages you need to queue, or that you would benefit from queueing, to the services you specify, in order.

It is a technology developed by a joint working group of finance companies such as JP Morgan, Goldman Sachs, Barclays and Bank of America together with IT companies such as Microsoft, VMware, Red Hat and Cisco. JP Morgan actually started on it alone in 2003, and a working group was set up in 2005 to turn it into a protocol. It is another technology that shows the weight of the finance sector in how technology develops.

It is a protocol used in almost every product you buy from a website.

RabbitMQ, the big brother who says give it to me and I will queue it

We can call it a message queue system that can also use the AMQP protocol. To draw a picture of one way to use it:

1. We have a queue service, and it splits incoming requests into different queues and sends them on, and those requests build up in RabbitMQ.

2. RabbitMQ puts those requests in order and keeps track of them continuously. It stores them even when our systems are down, so we do not lose data.

3. We have a service listening to RabbitMQ, and it carries out the queued requests in the order set by the configuration.

It is used in a lot of places, from banking applications to the mail sending of websites and mobile apps. What makes RabbitMQ an advantage is that it gives you quite an easy development process. Someone who knows queue systems in theory will take a couple of hours to understand and get comfortable with RabbitMQ. It also helps you build scalable services together with Docker Swarm. It lets you set the number of consumers, the listeners that process the queued requests, according to the number of incoming requests. You can think of it as a kind of automated load balancer. I have been a professional developer for about three or four years, and I think Docker is the thing I have been happiest to learn. After that, learning these simple but indispensable queue systems widened my view quite a bit.

I want to send you the video I got the most out of. If you watch the ten minute video below you will definitely understand it well.

After that, the only thing left to do is read the RabbitMQ documentation. I hope this post is useful to you.

All the best.