noah

(makes ark fast)

A simple Clojure interface to the Kafka Streams API

This is not on Clojars yet.

blak3mill3r/noah {:git/url "https://github.com/blak3mill3r/noah" :sha "look it up"}

Why?

Kafka and Clojure seem like natural companions, both emphasizing immutability and the primacy of data.

What's inside?

• A Clojure interface for building topologies and running an app using the high-level Streams API
  • Clojure functions work on streams/tables
  • maps to provide options
  • surprisingly complete (it is generated reflectively)
• A few serialization choices including edn and nippy
• Test utilities exposing TopologyTestDriver
• A macro to ease the definition of a Transformer
  • Write access to StateStores
  • Punctuations
• Transducers
  • Fault-tolerant stateful transducers

Status

Working (with 2.0 cluster and 2.2 client libraries). Probably more interesting than useful at this point. This is a young project which is far from battle-tested and there are some inadequacies.

Kicking the tires

Clone this repo, then try

clj aot.clj
clj -A:test:runner

Usage

See the intro and the tests.

At this time no example app code is provided.

⚠️ If you want to make an app, you will want to AOT at least noah.serdes

---

Inadequacies

• There are some gaps in the wrapping, where you would still have to use ugly interop
  • Good news: all signatures of all public methods are exposed
  • Bad news: some types have no sugar; you should be able to get an idea from the docstrings
• No sugar for Processor, only Transformer
• Stateful transducers must be rewritten in order to instrument their state construction
• can't pass reader options for edn serialization, nor can you pass nippy options (well, you can, but it's DIY)
• probably a lot more, feel free to point them out

Reporting bugs

Please open an issue on Github if you spot something amiss.

Contributing

If you find it useful or interesting, I would love to hear about it. PRs are very welcome. It would be great to see a whole lot more transducers get added. I would also welcome feedback and discussion.

Acknowledgements

A tip of the hat is graciously offered to Bobby Calderwood, in whose experiments with KStream transducers I found much inspiration.

License

Copyright © 2018, 2019 Blake Miller

Distributed under the Eclipse Public License either version 1.0 or (at your option) any later version.