Kafka, XMPP Chat, SSL and clustering married together

Posted on by Taras Filatov

QuickBlox has recently added the support for Kafka (pub-sub queue / messaging system).

We have made our Chat server write to Kafka so that it is easier for our enterprise users to apply analytics, trust & safety and external systems and tools to chat server generated data.

We have also enabled SSL encrypted connection via AWS load balancer so that you can connect Kafka consumers via a secure HTTPS connection.

This solution has been adjusted and tested to work in a scalable environment so that X Kafka brokers can work with Y Chat servers controlled by Z Kafka ZooKeepers. You may also connect multiple consumers if required. This is very useful for backup, analytics and real-time reaction / integration systems.

The configuration depends on architecture and scalability factors but here is an example diagram (clickable) showing the principal scheme.

Apache Kafka queue added to XMPP Chat server via SSL and clustered