Home | All Posts

[01 Mar 2011] .NET Client for Kafka

It’s a bit early stage, but I forked the Kafka project on GitHub today to add a .NET client. Kafka is a distributed publish/subscribe messaging system and looks very promising. It is aimed at capturing activity stream data in the vein of Facebook’s Scribe and Cloudera’s Flume. I’ve never look to deeply into Scribe, but was very interested in Flume to the very close point of forking it to try to add some features. When I came across Kafka I made an immediate connection to its architecture (probably due to the simple publish/subscribe model) and thought of an area where I might make use of it.

The area where I might make use of it, involves a bit of .NET so I needed a .NET client that would communicate with Kafka for me. Since I couldn’t find anyone who was doing that, I figured I’d strike a few lines of code and get it done.

I executed the first commit to my fork which contained basic support for a .NET message producer. The basic approach to sending off one or more messages looks something like this:

string payload1 = "kafka 1.";
byte[] payloadData1 = Encoding.UTF8.GetBytes(payload1);
Message msg1 = new Message(payloadData1);

string payload2 = "kafka 2.";
byte[] payloadData2 = Encoding.UTF8.GetBytes(payload2);
Message msg2 = new Message(payloadData2);

Producer producer = new Producer("", 9092);
producer.Send("test", 0, new List<Message> { msg1, msg2 });

I need to get a basic consumer working next and then think about some more advanced features for the library.

Home | All Posts