mailgun/kafka-pixy repository overview

⁠Kafka-Pixy (gRPC/REST Proxy for Kafka)

Kafka-Pixy is a dual API (gRPC and REST) proxy for Kafka⁠ with automatic consumer group management. It is designed to hide the complexity of the Kafka client protocol and provide a stupid simple API that is trivial to implement in any language.

Kafka-Pixy is tested against Kafka versions 0.10.2.1, 1.1.1, and 2.1.0, but is likely to work with any version starting from 0.8.2.2. It uses the Kafka Offset Commit/Fetch API⁠ to keep track of consumer offsets. However Group Membership API⁠ is not yet implemented, therefore it needs to talk to Zookeeper directly to manage consumer group membership.

⁠Key Features:

• Automatic Consumer Group Management: Unlike in Kafka REST Proxy⁠ by Confluent⁠ clients do not need to explicitly create a consumer instance. When Kafka-Pixy gets a consume request for a group-topic pair for the first time, it automatically joins the group and subscribes to the topic. When requests stop coming for longer than the subscription timeout⁠ it cancels the subscription;
• At Least Once Guarantee: The main feature of Kafka-Pixy is that it guarantees at-least-once message delivery. The guarantee is achieved via combination of synchronous production and explicit acknowledgement of consumed messages;
• Dual API: Kafka-Pixy provides two types of API:
  • gRPC⁠ (Protocol Buffers⁠ over HTTP/2⁠) recommended to produce/consume messages;
  • REST (JSON over HTTP) intended for for testing and operations purposes, although you can use it to produce/consume messages too;
• Multi-Cluster Support: One Kafka-Pixy instance can proxy to several Kafka clusters. You just need to define them in the config file⁠ and then address clusters by name given in the config file in your API requests.
• Aggregation: Kafka works best when messages are read/written in batches, but from application standpoint it is easier to deal with individual message read/writes. Kafka-Pixy provides message based API to clients, but internally it aggregates requests and sends them to Kafka in batches.
• Locality: Kafka-Pixy is intended to run on the same host as the applications using it. Remember that it provides only message based API - no batching, therefore using it over network is suboptimal.