Using Kafka as a Message Queue

I don’t usually just re-post stuff, but this article really hit the spot for me.

I like message queues – they are a wonderfully useful abstraction. Want to send a message from one component to another in a distributed system? Don’t let the components talk to each other directly – decoupling your components makes them more robust, and makes it far easier to swap in other service implementations at a later date.

But I am not here to discuss message queues in general. I’m here to discuss everyone’s (well, most people’s) favorite pub/sub system – Apache Kafka. Kafka is just there. Anyone doing big-data, streaming analytics, anything that requires reading in lots of data from different sources and pumping it into a processing system, is either using Kafka or evaluating it. It’s that successful.

But it’s not a message queue. It was not designed for it, and does not make any pretense to compete with the likes of RabbitMQ and ActiveMQ in this type of scenario.

But it’s there. If you already have a Kafka cluster all set-up and running nicely, maybe even running across data-centers (Kafka was built for this!), and you don’t need spectacular performance from your Message Queue, can you somehow use Kafka and spare yourself the setup and maintenance of another clustered system?

I was wondering how to go about this when I found this post on Software Mill’s feed. It’s worth a read:



A multi-network friendly Openstack VM image with netplugd

A while (long while. Sorry about that) ago, I posted about setting up a VM in an Openstack cloud so that it worked nicely with multiple networks. I also promised that I would explain how to automate this process.
So, in this post, I will explain how to create an Openstack VM image that will automatically set up networking for all of the Openstack networks that are connected to the VM.
The principles and techniques explained here should work similarly in other clouds that support multiple networks (like Amazon VPC)


  • Create a VM from your base image, with a single network. 
  • SSH into the instance and install netplugd.
  • Configure netplugd with an appropriate script, tweaked for your choice of Linux distribution.
  • Delete files that are not required, stop the instance and create a new image from the instance.
  • The new image will work correctly with all connected networks.

The Problem – VM Images with one configured NIC

As discussed in the previous post, setting up the networks in Neutron and starting a VM that is attached to them is not enough. The VM has to cooperate with the network setup, somehow. 
In many of the images provided by cloud vendors and Linux distributions, only one Network Interface Card – eth0 – is defined, so only the first network is available to the VM.
You can set up the other NICs using ssh (it network access is possible with only one NIC), or with a user-data script that runs during the VM start-up (I’m not a fan of this approach – more on that in a different post). But the best way (in my opinion, anyway) is by baking this behavior into your image.

The Solution – VM Images with NICs configured by netplugd

netplugd is “a daemon that notifies the status of one or more ethernet interfaces, calling an external script when something changes.” 
And that is exactly what we need. (Another option is ifupd). 

Let’s get to work.

Network Setup
I’ll be using the same network setup as in the previous post.
Base VM
Start by creating a VM that is attached to a single network, and attach a floating IP to the instance. The command line is similar to the ‘nova boot’ command from the previous post, you’ll just need to pass a single network ID instead of two. We’ll be using the same security group and key-pair as well.
And ssh into the instance:
ssh -i demo-keypair.pem ubuntu@<Floating-IP>
Now to setup netplugd:
sudo apt-get install netplug
sudo mv netplug /etc/netplug/netplug
sudo chmod +x /etc/netplug/netplug
You should read the netplug file. It is a straightforward script that detects network cards that are plugged in/out, and then edits the interfaces file accordingly.
Note: your choice of Linux distribution might keep the interfaces configuration in a different location or file format. Some tweaking of this file might be necessary. The script as provided works on Ubuntu 12.04. Feel free to send in a pull request if you want to add your configuration to the repo.
Now, let’s clean up the VM and close the session:
rm ~/.ssh/authorized_keys
Power off the instance
nova stop demo-vm
Wait until the VM fully shuts down. It should look something like this:
nova list –name demo-vm
Create a snapshot of the image:
nova image-create demo-vm “Ubuntu 12.04 with netplugd”
We can now delete the VM
nova delete demo-vm
Your image is now ready!
Let’s  take it out for a spin. We’ll use both networks this time:
nova boot –flavor standard.medium –image “Ubuntu 12.04 with netplugd” –security-groups demo-security-group –key-name demo-keypair –nic net-id=XXXXX-XXXX-XXXX-XXXX-XXXXXXXX –nic net-id=YYYYYY-YYYY-YYYY-YYYY-YYYYYYYYYYYY demo-vm
 Attach a floating IP to the instance and ssh into it.
 ssh -i demo-keypair.pem ubuntu@<Floating-IP> ifconfig
Note how both NICs are up and running? You now have a multi-network friendly Openstack Image. Enjoy.

One last thing, and this one always gets first-time users.
Just cause the NICs are up, doesn’t mean all is well in networking land. You are probably going to need to set up routing rules to determine which NIC should be used for each of your network packets. These rules depend on what you are trying to accomplish in this network configuration. But we’ll leave that for another post. For now, consult with your sysadmin.