Recently I’ve taken on a challenge to set up a real-time event streaming platform that allows our developers to build complex event-based systems without having to worry about the infra, costs, growing complexity, and security. In this article I’ll talk about my learnings along the way. The goal is to touch on all the concepts and gotchas you need to be aware of and draw an example big picture. I won’t be going too deep in every single topic, nor will attempt to give you a full implementation tutorial as that could take me a lifetime. I will update the contents based on any feedback I may receive or if I learn something new to keep it educative. For most component-specific topics, I highly suggest reading the official documentation as anything I put here will soon get outdated.

# A bit of intro: The not so technical part

The Teams within our Area (department) at DPG Media offer solutions that allow serving videos and podcasts, playing games, on a large range of our brands. To make it more clear, any videos you see on ad.nl, any podcasts you can listen to on hln.be, and any puzzles you can solve on volkskrant.nl go through our services. As one of the largest media companies in both the Netherlands and Belgium, we serve content to millions of users per day.

As SecDevOps/Platform Engineers (Or whatever you want to call it), together with my 2 other colleagues, our role is to build a reliable, and secure Platform for our development teams that allows them to focus most of their energy on building features!

## Our current stack

Our Platform is built for most part on our multi-cluster Kubernetes instances, running on AWS EKS nodes that we manage ourselves using Karpenter. All of our applications and platform components are deployed using Kubernetes and we only use a small set of AWS resources where it would otherwise not make sense to use self-hosted solutions. In general we prefer self-hosting as much as possible where the company policies allow and not use AWS native or managed services.

## Why did we need a Streaming & Analytics Platform

Our developers are constantly building new features and experimenting with different ideas. Sooner or later every application needs to send some events. To give you some examples, for every game our users play, some streaks calculations need to be done, for our podcast solutions, we need to store the listening progress of our users so that they can continue where they left. For our videos, we need to be able to get visibility into what are the most replayed sections. Some of our smaller scale solutions were doing their own form of event streaming but the effort was scattered, and the solution was too expensive and not scalable.

To enable our devs to build such features quickly without getting bogged down into never-ending database growth hellish situations, we needed to offer a solution with the following characteristics:

1. It must be easy to use: If the learning curve for the developers is too steep, they will either need special training or will rely on some AI and make so many mistakes that will counter the whole point of this task. This should allow our developers to not only send events around but also do complex transformations, state storage, enriching and aggregating data as they flow without large latencies.
2. It must be resource and cost-optimized: Our scale is larger than what smaller companies may be used to, but it is by no means near a number in which you would feel a real challenge. Hence, spending too much money to be able to process such small amount of events would not be a good example of engineering. Modern computer hardwares are insanely capable and we should build better software solutions that do not waste all of that advancement effort, energy and money.
3. It must be stable and scalable: Event streaming at scale is HARD and if you don’t get it right, you can really shoot yourself in the foot, wasting months of effort and having to rebuild things. This solution will be used as a single foundation for many of our applications. This also means that if it does not perform well, it will turn into a single point of failure.

## Our first tenant and expected load

To build this and test it out, we needed a safe tenant to experiment with. Hence we started with a new telemetry service for our videos. A new addition to our largest component where the amount of data is the highest but the possible damages are the lowest.

When designing such solution, one of the most important questions is: how much load should it be able to handle?
For us, the calculation was slightly vague because prior to that we could not measure every user engagement that we would have liked to. We started with the load on our largest component (serving video content) which has been around a total of 6-10k requests, and if we drop the cached requests, it goes down to 2-3k rps. So we started by designing our event payload and counting how many events a user would be producing per page load and as they keep watching videos. With a few calculations we got to a rough estimation of 30k rps. Our average payload size is about 100 bytes. Let’s multiply that by 10 and say our events are 1kb each. 30,000 rps * 1000 bytes ~= 30MB/s throughput. 30MB/s is not a huge number, so we should be able to easily build something that can support this. In future we are certainly going to go way beyond that number, especially where we have to support existing json-formatted events, so we should be thinking about a much larger storage, but to keep things simple, let’s consider this for our POC.

# The choices: so many choices

A real-time event streaming platform that is able to support both transactional and analytical purposes consists of many components. Choosing each, carefully will save you many sleepless nights.

## The Event-Store: The heart of the system

This is arguably the biggest choice in this whole set up. Within our company, Kafka is the only allowed choice! It has a proven record and we have more people within the company who have hands-on experience with. So this made everything a lot simpler. Now all I had to do was to figure out what is the best way we can get our own instance of Kafka hosted somehow.

The bullet points below are the first comparison summary I wrote down in an internal document (Please don’t get offended if you work in one of these companies):

• Strimzi with vanilla Kafka: Great but needs disks and cross-AZ network traffic which makes it expensive.
• AWS MSK: Okayish at the highest price we can get.
• RedPanda: Good performance that we don’t need at not so much gain in storage and network costs.
• Confluent: Great but at a cost and needs licensing (bureaucracy).
• WarpStream: Great but not self-hostable. Needs licensing (bureaucracy). Diskless Kafka.
• AutoMQ: Amazing stuff at a reasonable trade-off. Diskless with the possibility of adding EBS for local WALs.
• Aiven: Too young and cloud-only. Diskless.

In this article I will not go too deep into the comparison of these solutions. There is enough comparison material available out there and many of these solutions offer a calculator for their pricing. Instead I will tell you what we ended up choosing and how we put it all together.

### Maintenance overhead of self-hosting vs a managed solution

Setting up a Kafka cluster is simple. Keeping it alive and up to date is not. With our current capacity of our Platform team, we needed to be careful with how much overhead we add to our daily lives. Gladly, Strimzi really minimises the amount of effort needed for maintenance. You would still need to learn how Strimzi and Kafka work in some depth but operating Kafka with Strimzi is orders of magnitude simpler than having to do all the clustering manually yourself.

Using a managed service, we of course no longer have to deal with any of the pain, but at a cost. [And honestly what is the fun in life if we use everything managed? What is so bad about taking up a challenge, show courage and fail responsibly?!]

### Cost

Once you start digging in features and costs of each of these solutions, you will soon realise that the cost is a huge factor. Some of these solutions are substantially cheaper than others at a very reasonable trade-off. Since our Platform runs on Kubernetes

Let’s consider our POC scenario:

Throughput: 30,000 events/sec * 1 KB = 30 MB/s
Ingress: 30 MB/s * 3600 * 24 * 30 ≈ 77 TB/month
Replication Volume: 77 TB * 2 (followers) = 154 TB/month cross-AZ Storage: 7 days retention ≈ 18 TB logical storage

Notes on the above image:

• Calculating the costs are not very straightforward so the values may not be exact but the proportion is valid.
• *1: For compute costs, an m7g.large EC2 instance is used
• *2: Both Strimzi and AWS MSK options support KIP-405: Kafka Tiered Storage so the storage cost could be lowered but I can’t easily calculate how much lower.
• *3: Redpanda also has a Kubernetes operator like Strimzi and can be self-hosted, however, Redpanda isn’t going to be happy with GP3 EBS disks and would require locally attached NVMe disks (For example c5d or c5ad EC2 instances)
• *4: Warpstream doesn’t have a self-hostable solution. On top of that, due to how it is architected, it adds a high latency to the events (depending on the pricing tier and the percentile, anywhere between 100 to 900ms). See more on their pricing page

Honorary mention: Confluent cloud being the de-facto solution out there for many large enterprises isn’t very clear about their pricing. This is an example they give in a sub section somewhere random in their pricing page:

The total throughput per month determines the $/GB applied to all Kafka throughput in that month. For example, if the total Kafka throughput for a given month is 300TB (307,200GB) for a Freight cluster, the price applied would be $0.025/GB for a total of $7,680.

But they say we can save %43 compared to baseline Apache Kafka (so in our case compared to Strimzi)

Bottom line of this section is that we do need a solution but our requirements are not at a scale that can justify a massive bill. We want the solution to be cheap at the start but able to scale as our needs grow. We do not want to do too much premature optimizations (just a little bit).

### Now what is KIP-405 (Zei er iemand kip?)

If you’ve been reading about Kafka recently, you may have heard about their Tiered Storage. This makes Kafka feel a lot more modern. Now you might ask yourself why would someone consider AutoMQ when Kafka already supports offloading old data to S3. The answer is that AutoMQ is architecturally still different than the KIP-405 design in that they do not use EBS volumes as a primary storage but only as a WAL storage. With KIP-405, the active (hot) data remains on the EBS volumes and older (cold) data is put in S3. That reduces costs compared to the “traditional” Kafka architecture but doesn’t completely get rid of the Cross-AZ communication and doesn’t make the pods stateless. AutoMQ is completely stateless. This page on AutoMQ docs explains this architectural differences nicely.

Source: Difference with Tiered Storage - AutoMQ Documentation

### What if we use Strimzi, but with AutoMQ?

Since there was no way I could convince my manager that we should pay $6k+ per month on this (Our entire production AWS bill is less than that btw), our options were limited.

One thing I learned along the way was that we can use Strimzi with AutoMQ images! This was a major finding for me because it meant that we could use the operator and pay the lowest possible cost. Our EBS disks would be 20GB max and we skip the majority of Cross-AZ Network communication.

AutoMQ’s GitHub page describes them as:

Diskless Kafka® on S3. 10x Cost-Effective. No Cross-AZ Traffic Cost. Autoscale in seconds. Single-digit ms latency. Multi-AZ Availability.

As you may have noticed, with AutoMQ’s architecture, fetching cold data requires calling S3 which could add a few milliseconds to the initial requests but for us, that slightly added latency is not an issue for the forseeable future. The majority of our applications will only be consuming fresh events and will not need replaying a large chunk of historical events, minimizing the GET calls to S3. If they do need the cold data, then they are also in all cases able to tolerate the slight delay.

So this was the choice that we made: Strimzi cluster using AutoMQ Kafka images.

## The Event Stream Processor: The Kidney of the system?

For this component you again have many options. The major competitors are Apache Flink, ksqlDB, Kafka Streams, and The usual suspects (Confluent Platform, AWS Managed services). These are all however a bit icky. If you ever tried to write a simple processing in Apache Flink, you will know that there’s no way you’re going to get past the beginner stage unless you dedicate years of your life to it and really study Flink in depth.

Remember our requirement:

It must be easy to use

And let’s say you did use Flink. You’d still need a DB to store your state in it if you are to query it.

While a solution like Flink is probably the most powerful and flexible thing you can get out there, we don’t need most of that. We need a simple way to chunk through a few hundred thousand events per second with high efficiency and allow near real-time (sub 5s) queries to it.

This is where I started becoming very interested in ClickHouseDB. ClickHouseDB is a columnar LSM-Tree (They say: “based on traditional log-structured merge (LSM) trees with novel techniques”) OLAP database. It has a direct integration with Kafka and offers more than enough real-time event processing features than we would need for the forseeable future. It scales nicely, supports replicated tables, easy partitioning, TTL setting per table, allows for event enrichment using Dictionaries, has incremental materialized views that simplify aggregations, support for Protobuf messages coming from a Kafka topic, and I could keep going. It is genuinely a piece of software to get excited about. And if all of that was not enough, recently they started offering clickhouse-operator for Kubernetes.

After giving it a test run and setting up a local instance to play with, I really enjoyed writing queries in it. ClickHouse’s Incremental Materialized views are very different than Postgres’s Materialized views, if you’re familiar with those. The naming might be confusing a little at the start and there’s a slight bit of mindset change when you start using ClickHouse MVs but when you get used to them, you suddenly realize what a powerful event transformation pipeline you have in your hands. The video on this page explains them very nicely.

ClickHouseDb matched all of the criteria I was looking for, it is simple, it is very powerful, and since we can self-host them in our Kubernetes clusters and have full control over the compute, we can host them at a very low price. So there’s that.

## Event schemas: Blood type? (This is getting out of hand)

Kafka is data-agnostic, so as an event-producer you could send an arbitrary byte array as the event message and Kafka couldn’t care less. The only thing that matters is that your producers and consumers understand the messages moving through the system. So early on in your event sourcing and streaming journey you have to decide on an event schema type.

### Choose either Avro or Protobuf

With Kafka, your top options are Avro, Protobuf, and if you are lazy, have too much money, and only have experience writing web apps, JSON. If you’re very hardcore, you could also define your own custom binary format if you are certain that your schema won’t change, but I wouldn’t go that way. Your schema will change 2 hours after your first release because “you forgot to count for that thing there”. Initially I really wanted to try Avro because the Kafka ecosystem has a good support for it and it removes the headache of code generation. However, we quickly realised that there are a few issues that I wasn’t very comfortable with:

• Client library support for it for our Swift client apps weren’t great.
• I wasn’t really sure about having to attach the schema to messages and how much overhead that would add.
• Lack of field tags in Avro and it’s positional layout means that you could accidentally get schema mismatches and introduce silent failures. Since our schema changes quickly, this wasn’t a risk I was ready to take and I was quite sure that this would be a mistake we would make multiple times. Whether this is seen as an advantage or disadvantage really depends on the scale, rate of changes, and the organization topology you work with. Here’s a good in depth comparison (Also included in the DDIA Book).

So then I started leaning towards and old friend. I had prior experience with Protobuf when using GRPC based transport and was generally positive about it in the past, except the tooling and code generation headaches. Luckily these days Buf exists. Buf simplifies the code generation and let’s you choose between their free remote plugin or locally installed protobuf plugins to generate code. You don’t necessarily need Buf, but it does removes some pain, especially regarding managing protobuf generator versions and keeping them up to date.

The reason the title of this section is opinionated is that when you’re getting so much data in, you should be ready to get out of your comfort zone of “JSON is human readable”. Projects like Protobuf and Avro rely on clever techniques to reduce the total data size that needs to be transmitted, computed and stored. See Protobuf’s varint documentation for example which is aiming to minimise the space needed to send an int over the wire. JSON, however, is meant to be easy to be read and modified by humans. So they’re built for totally different purposes. If you get 5 requests per seconds on an endpoint, you probably shouldn’t over-optimize and use binary formats. But at a higher scale, optimizing for storage and network costs is cruicial. By choosing a more compressed data format, you are reducing costs drastically. If you have your event payload at the hand, you can use a tool like protobufpal or to write a simple script to calculate the difference between a JSON message vs Protobuf. If your event contains many fields and some have long names, you are pretty much guaranteed to get 50%+ reduction rate. For us that is really the case. Our average JSON message payload (without tabs and spaces) was about ~290 bytes whereas the Protobuf message was only about 100 bytes. That is almost 3x savings in both Storage and network cost. Also in terms of serializing and deserializing performance (compute resources needed), Protobuf and Avro are superior, very largely due to them not having to scan through a schemaless string. You could argue that we could win some by using very small fields names like pl instead of platform or t instead of timestamp_ms, but then you’re defeating a big purpose of JSON: readability.

#### Schema evolution

I don’t think I need to go into depth with this one. There are countless articles including the one I referenced earlier that explain this way better I could.

The goal is, you must be able to introduce changes and be able to revert those changes with minimal damage. Protobuf, Avro, and JSON are all nicely evolvable. There are details regarding the differences between Avro and Protobuf which makes the choice very use case dependant.

#### ClickHouse’s Protobuf support

ClickHouse has support for all the 3 aforementioned schema types.

My experience with it’s Protobuf parser together with Kafka Table Engine was mostly positive but I sometimes had to look a bit further than just the documentaion. I’ll explain how you can set it up in a further section.

### Schema registry and ClickHouse

ClickHouse doesn’t work nicely with Confluent Schema Registry out of the box. The reason being that schema registry prepends 5 bytes (1 Magic byte + 4 bytes schema ID) to the message on the producer side and expects those 5 bytes on the consumer side to be able to parse the message. If you are planning to use schema registry, you would either need to put KafkaConnect or similar in-between your broker and ClickHouse or use ClickPipes. You could also skip schema-registry and move the parsing logic to your producer and consumer side. This is a simpler approach if you only have a single schema per topic. You could also selectively use schema registry where needed and not use it in cases like this. There’s no one forcing you either way.

Note: After I wrote the above paragraph, I’ve checked ClickHouse documentation again and noticed that the Kafka Table Engine now has a kafka_schema_registry_skip_bytes setting but I haven’t tested it out yet.

### Code generation for .proto files

We have all of our Protobuf schemas in a single repo. That allows us to have a unified CI pipeline that generates libraries for all of our client applications, ie: Java/Kotlin, Swift, TypeScript, Elixir, Golang. When we modify a certain schema file, we trigger the code generation and package the generated code to our package registries. The packages are then added as a dependency to our applications. Applications use the definitions in these packages together with a protobuf library to turn their model instances into protobuf formatted binary and use an HTTP library to send that to our gateway.

## The gateway: The mouth?

Unlike your mouth, this component is optional and whether you need it depends completely on your specific setup. As long as you have your data within your private network and Kubernetes cluster, you could talk to the Kafka broker directly with some level of authentication and there would be no need for a middleman. However, when your event producing clients reside outside your K8s cluster (in our case, our mobile apps and websites), you need a secure gateway to let clients send data to. You may also not be able to introduce a completely new protocol in the clients. We operate in a large organization that consists of many application and web development teams that have complex rules on the libraries and dependencies they use and one team cannot simply add a new library to the stack without a large amount of coordination. For this reason, we really needed to rely on the good old HTTP protocol for sending telemetry data. This wasn’t a major issue. We can write whatever binary data comes out of our protobuf library into a POST payload and have a server forward it to our Kafka cluster.

For this purpose you could use an existing solution such as Confluent’s Kafka Proxy or Strimzi’s Kafka Bridge. I however have strong opinions on some matters and like to make my life harder by creating things in certain ways that serve one purpose and do it in the way I can. So I created a simple service in Golang with the standard net/http package and made sure that there will be no runtime memory allocations (at least by my code. I can’t control what net/http does.) on the hot path by using a sync.Pool for the byte buffers. It does exactly what we need and does not waste any compute than strictly necessary. The core code is less than 200 loc including whitespaces and contains another 200 lines of metrics and observabilty generating code. In practice it handles 20k rps with 2 pods each using 40-50Mi RAM usage and about 300m CPU each. While I also love Rust, it would’ve been a bit overkill for this.

### Authentication limitations for anonymised data

There has some big limitations when collecting data anonymously from web and mobile clients. While there are some native attestation APIs for mobile clients and you could do some level of CAPTCHA or trust tokens on web, there’s no simple bulletproof way of telling a bad actor from a normal user. One way to get around this is to create dump ingestors that only validate a certain number of things such as payload size and the schema and just pass it to Kafka. You would then need to do a round of validation afterwards which is less than optimal, but it makes your system more resistant towards DDoS attacks.
On top of that you can also separate your ingress for the high-throughput anonymous telemetry data from the very important events and have less strict rules for the anonymous ones and heavy rate-limitting and authentication for your vital events. This guarantees that you will never miss important events and that you have a strong authentication for those while your less important data can be ingested with less pain.

# Putting it all together

Great! Now we have a full picture of what we want to build. Let’s continue building our POC of a Telemetry Dashboard for our Video content.

Note that I’ve stripped every bit of details from this, for example: The VPC and Networks involved, S3 Buckets and EBS Volumes (via Kubernetes PVCs and SC) attached to Kafka and Clickhouse, Authentication, Network policies etc. This is just to give you a general idea.

From this point your task would be to:

1. Set up the Strimzi operator.
2. Set up a chart that let’s you create a new instance of a Kafka cluster and all it needs to be usable by other applications.
3. Set up the gateway.
4. Set up code generation.
5. Set up the Clickhouse operator.
6. Set up a chart that let’s you create a new instance of ClickhouseDB and all it needs to be usable by other applications.
7. Test the flow.
8. Improve!

For the most part, you would be better off reading the official installation and usage documentation. In the next sections, I’ll be explaining the harder bits that you won’t immediately notice.

## Strimzi and AutoMQ: The crux

### Don’t vibe this part. Learn the theory and test your learnings in practice

This is the part that took me the longest to fully understand, implement, and test. I’ll be honest with you; There’s no way to fast-forward this step and you must absolutely understand a bunch of concepts before you can consider this a job well done. You may be able to speed up the learning process by using LLMs to explain things to you but I absolutely discourage you against letting an AI take the control of this implementation while you “Yes” your way to a grand failure. Kafka is a complex distributed system and should be treated as such. Strimzi helps to simplify the set up, but it does not remove the need to understand the underlying system.

### Strimzi operator and AutoMQ version compatibility

Before you start with this setup, you must be aware that you’re now dealing 3 software versions:

• Strimzi Operator version
• AutoMQ version
• (And each AutoMQ Version is compatible with a specific) Kafka version

So it is essential that you first find the exact version of each component that matches the rest.
I would start here. On that page you can see which Strimzi Operator version is compatible with which Kafka version. Then go to AutoMQ’s releases page. There you can find a file in format automq-{AUTOMQ_VERSION}_kafka-{KAFKA_VERSION}.tgz. At the time of writing this article, the latest AutoMQ-supported Kafka version is 3.9.1, so the latest Strimzi version I could use is 0.47.0 (which is compatible with Kafka versions 3.9.0, 3.9.1, 4.0.0).

This is something you need to take into account when upgrading any of the components in the chain. You will risk possible data corruption/loss if you deploy mismatching versions, so I highly encourage you setting up an acceptance environment which you can evaluate any changes before pushing those changes to prod.

As an example, here’s how you could pass versions to the strimzi-kafka-operator Helm chart at the time of writing this article:

strimzi-kafka-operator:
  extraEnvs:
    - name: STRIMZI_KAFKA_IMAGES
      # We use these images via our ECR Pull-through cache but you could also just reference them directly from Docker Hub and Quay like below
      value: |
        3.9.0=automqinc/automq:1.6.4-strimzi
        3.9.1=automqinc/automq:1.7.2-strimzi
        4.0.0=quay.io/strimzi/kafka:0.47.0-kafka-4.0.0

I highly recommend reading the following docs:

• This page explains most of the custom changes you need to apply when installing Strimzi together with AutoMQ.
• Deploying and Managing Strimzi Kafka clusters and honestly every page in this documentation will unravel the mystery more and more.
• Check out the Chart’s documentation on the exact tag that you use.

#### Update steps

If you ever want to upgrade the components, do it in the following order:

1. Upgrade strimzi-kafka-operator chart to a version that matches at least the latest automq’s supported Kafka version: https://strimzi.io/downloads/
2. Modify the STRIMZI_KAFKA_IMAGES env in the Helm chart to match the Kafka versions with the right AutoMQ version.
3. Upgrade Kafka cluster CRDs to match the latest supported Kafka version STRIMZI_KAFKA_IMAGES

### Creating a Kafka cluster

Now that you have the operator set up, it’s time to create a cluster. From this point onwards, you must master navigating your way through this documentation page. This page has more info than I could ever explain and it is absolutely necessary to check out all of the fields in every CRD you’re defining. Your set up needs would be different than ours, so it’s important that you’re aware of any options and configuration fields.

The following is the way I’ve set it up so far is as follows. You may choose a completely different path.

#### Kafka CR

A single Kafka instance per Kubernetes cluster. This will be the broker that handles everything we need.

• NodePools: We have NodePools: enabled. This is a great feature that simplifies configuring nodes separately while sharing some of the config. It also simplfies the Kafka cluster instance.
• Kraft: Absolutely enabled!
• Authorization: We rely on a ServiceMesh for our cross-pod traffic and mTLS, so for this set up we use simple authorization with SASL Scram SHA-512 authentication. I’ll explain this further in the KafkaUser section. When exposing multiple listeners, you have to understand that once the cluster is using simple authorization, you can’t use use internal port 9092 and expect to be able to connect to the brokers without authenticating yourself and being authorized to execute certain actions.
• ServiceAccount and IAM Role: You need to create an AWS IAM Role and make sure that the role’s trust policy allows connections from the service account. [Read more]. The role must have a policy that allows the following actions from the AutoMQ pod on the Buckets that AutoMQ uses (The ones you hopefully created by now when following this guide):
  • s3:PutObject
  • s3:GetObject
  • s3:AbortMultipartUpload
  • s3:DeleteObject
  • s3:ListBucket

#### KafkaNodePool CRs

For production, you should definitely separate your broker node pools from your controller node pools. We use 20GB of gp3 EBS volumes per broker node. When selecting resources for your nodes I would recomment starting at around cpu: 1000m, memory: 4Gi (both request and limit) for broker pods and cpu: 500m, memory: 2Gi for controller pods with a KAFKA_HEAP_OPTS value that leaves some breathing space available for those pods. Later on you can monitor and modify your values as needed but start small as possible. Kafka is able to process millions of messages per second on reasonably small instances if configured properly.

#### KafkaUser and KafkaAccess Operator

Once you enable authoriation you will need to start using KafkaUser resources to define how your user will be able to authenticate themselves and what are they authorized to do in the cluster. This is a great feature and works nicely. It creates a secret for the user and you can just inject that secret to the application pods that need to use it. However, it is common practice to put your Kafka Broker pods in a dedicated namespace and have your applications defined in a separate namespace. That’s where you start needing the Kafka Access Operator. It does a simple job, you define a KafkaAccess resource for a given KafkaUser and give it a namespace and it will take the secret from your broker namespace and put it in the target namespace.

AutoMQ’s Autobalancer: When you have Authorization set up for the cluster, you must make sure that AutoMQ’s autobalancer either has a user assigned to it on both your broker and controller nodes (recommended), or is a superuser. To tell autobalancer to use a specific user, set the following in your kafka config:

autobalancer.client.listener.name: SASLSCRAM-9094
autobalancer.client.auth.security.protocol: SASL_PLAINTEXT
autobalancer.client.auth.sasl.mechanism: SCRAM-SHA-512
autobalancer.client.auth.sasl.jaas.config: "${strimzienv:AUTOBALANCER_JAAS_CONFIG}"

And then pass the AUTOBALANCER_JAAS_CONFIG in your KafkaNodePool:

spec:
  template:
    kafkaContainer:
      env:
        - name: AUTOBALANCER_JAAS_CONFIG
          valueFrom:
            secretKeyRef:
              name: {{ .Release.Name }}-user-autobalancer
              key: sasl.jaas.config
              optional: true

Note: strimzienv must be named like this. It is an alias defined for org.apache.kafka.common.config.provider.EnvVarConfigProvider

### KafkaTopic CRs

Now we have our brokers and controllers clustered together and we have a user that can do great things! Time to create a topic. Creating a topic is done using a KafkaTopic resource. As usual, I recommend reading the official documentation for the details. There aren’t many gotchas if you know how Topics work in Kafka.

#### How many partitions do you need?

This is another topic you absolutely don’t want to take lightly. Unfortunately explaining this would take another book and is out of the scope of this article. The video on this page explains the basics nicely but when you get your hands on it and start consuming events it becomes quite important that you set your partition size correctly.

If you’re already overwhelmed with all of the info so far and just want to get something up and running for testing I recommend the following approach:

• If you don’t care about the ordering of events and your consumers should be able to just process without coordination on the order, it is safer to over-provision than to under-provision. The number of your consumers must be lower than or equal to your partitions. In other terms, a consumer may be assigned to multiple partitions but a single partition may only be consumed by one consumer at a time. For example:
  • If you have 10 partitions and 100 consumers, only 10 consumers are active and the other 90 are sitting idle.
  • If you have 100 partitions and 10 consumers, all 10 consumers are active and they can consume from all of the 100 partitions in parallel.
• If you care about the ordering and your usecase is simple and one consumer could do, just use a single partition.

The moment you decide this is something you want to put in production, it is absolutely necessary that you understand how you should partition your topics based on factors such as, scale of stored data, if parallelism is needed or even possible, or if global order of events matter.

#### How do I partition

With similar importance to the previous section, once you decide to have more than a single partition, you should partition your events in such a way that your messages are spread equally on all partitions. That means choosing the right “key” for your data so that no partition gets assigned a disproportionately large number of messages. Such partition would be called a “Hot Partition”, but not “Hot” as in “Sexy”.

If you’re designing the platform but not the applications like me, you need to explain this to your developers. I’ve decided to design the Gateway component in such a way that will allow 3 types of event production:

1. Producer let’s the Gateway component decide on a partition key based on the message contents. This is low effort way and works for majority of small scale cases.
2. Producer passes a partition key and let’s the Gateway (well, the producer Kafka client inside the Gateway) do the hashing and partition number assignment. This is a better choice for when the producer knows a good partition key candidate such as the user_id or product_id.
3. Producer does the partitioning on their side and passes an explicit partition number. This one can be trickier than the other solutions as when you increase your partition size, you would need to update the producers to produce to the new partitions as well.

#### How do I name my topics?

This article should help you. But the important part is that you come up with a sturdy naming pattern at the beginning and not delay the decision to later. It will be much harder later to change things once they’re stablished.

## ClickHouseDB Set up

We can now start ingesting the data that is produced to a Kafka topic.
Regarding the clickhouse-operator setup, the only note I could add is that you might be at first confused to see that there are 2 active clickhouse-operator projects out there:

• The new official one from ClickHouse: -> This is the one we’re using
• The one from Altinity

Other than that a standard set up would do

### KeeperCluster and ClickHouseCluster CRs

Once the operator is in place you can start creating your HA ClickHouse cluster instances. Their official documentation along with their examples folder are great places to start.

The following sections explain the configurations that I found a bit interesting.

#### Start with very small database instances and let them break

The default configuration of the database instances has some quirks which you may miss at the start. If you’re new to Clickhouse and self-hosting a distributed database, I suggest you start with very small instances and put them under heavy load until they break. Only then try to figure out how to tweak it for your specific goals and learn along the way. If you start directly under full load in production without optimizing first, you will get absolutely bashed.

#### Some System Tables grow indefinitely if you don’t configure their Partitioning and TTL strategies

Examples are the part_log, query_log, text_log, metric_log, and the asynchronous_metric_log tables. Gladly you can configure them to clean up after a certain amount of time by setting a partition_by and ttl just like you would with any other ClickHouse Tables. To give you an example, you could do:

apiVersion: clickhouse.com/v1alpha1
kind: ClickHouseCluster
metadata:
  # ...
spec:
  settings:
    extraConfig:
      metric_log:
        # This interesting syntax took me a while to figure out, so I'm very proud to present it here today!
        # See https://clickhouse.com/docs/concepts/features/configuration/server-config/configuration-files#example-1 for more examples
        "@remove": 1
      query_log:
        database: system
        partition_by: toYYYYMM(event_date)
        table: query_log
        ttl: event_date + INTERVAL 14 DAY DELETE
      # Repeat the same for the rest depending on your needs

#### Named Collections are great

Named collections provide a way to store collections of key-value pairs to be used to configure integrations with external sources. You can use named collections with dictionaries, tables, table functions, and object storage. Source: Named collections

What this means for you is that instead of hardcoding your Kafka username and password in your Kafka Table Engine (which I’ll show you later), you can inject them as env var in your ClickHouseDB pods as secrets like so:

apiVersion: clickhouse.com/v1alpha1
kind: ClickHouseCluster
metadata:
  # ...
spec:
  containerTemplate:
    env:
      - name: CLICKHOUSE_MY_APP_USER_PASSWORD
        valueFrom:
          secretKeyRef:
            key: password
            name: my-app-user-password 
      - name: KAFKA_SASL_MECHANISM
        valueFrom:
          secretKeyRef:
            key: saslMechanism
            name: kafka-user-my-events-db # Assuming you've already created this and the other secrets the way you're used to create your secrets
      - name: KAFKA_SASL_USERNAME
        valueFrom:
          secretKeyRef:
            key: username
            name: kafka-user-my-events-db
      - name: KAFKA_SASL_PASSWORD
        valueFrom:
          secretKeyRef:
            key: password
            name: kafka-user-my-events-db
  settings:
    extraConfig:
      named_collections:
        kafka_connection:
          kafka_broker_list: <placeholder-your-kafka-endpoint>:<your-kafka-port>
          kafka_sasl_mechanism:
            - '@from_env': KAFKA_SASL_MECHANISM
          kafka_sasl_password:
            - '@from_env': KAFKA_SASL_PASSWORD
          kafka_sasl_username:
            - '@from_env': KAFKA_SASL_USERNAME
          kafka_security_protocol: sasl_plaintext

Then where you create your Kafka Table engine, you could use the configuration just by referencing the named collection like this:

CREATE TABLE IF NOT EXISTS my_kafka_events_queue (
    -- your fields here
) ENGINE = Kafka(kafka_connection) SETTINGS
  -- The rest of your settings go here, for example
  kafka_group_name = 'clickhouse_analytics_group',

#### Creating users using extraUsersConfig setting

You can create new users in your ClickHouseCluster like so:

apiVersion: clickhouse.com/v1alpha1
kind: ClickHouseCluster
metadata:
  # ...
spec:
  containerTemplate:
    env:
      - name: CLICKHOUSE_MY_APP_USER_PASSWORD
        valueFrom:
          secretKeyRef:
            key: password
            name: my-app-user-password # Assuming you've already created this secret somehow
  settings:
    extraUsersConfig:
      users:
        my-user:
          grants:
            - query: GRANT ALL ON *.* # Reduce this to what your user needs
          password:
            - '@from_env': CLICKHOUSE_MY_APP_USER_PASSWORD
          profile: default # You can customize this: https://clickhouse.com/docs/concepts/features/configuration/settings/settings-profiles
          quota: default # See: https://clickhouse.com/docs/concepts/features/configuration/settings/settings-users#user-namequota

#### A dangerous backdoor by default: The default user is created without a password

If you don’t set the spec.settings.defaultUserPassword, a user with username default will be created with no password set. This will be a clear backdoor, of course. So make sure to set it on your ClickHouseCluster CRs.

### Creating ClickHouseDB Tables

At this stage, the first thing you need to do is to quickly glance through this and related pages in the official ClickHouseDB documentation.
Once you have sort of an idea how this will be set up, go to that link I mentioned earlier on incremental materialized views, watch the video, then navigate to the page that explains the concepts in more depth. Once you’re done there, go back to the Kafka Table Engine page and try to set up a table.

Let’s begin together with a simple example. Given your event’s protobuf schema looks like this:

syntax = "proto3";

package my_analytics.v1;

option java_multiple_files = true;
option java_package = "com.example.team.telemetry.schemas.myanalytics.v1";

enum Platform {
  PLATFORM_UNSPECIFIED = 0;
  PLATFORM_WEB_FRONTEND = 1;
  PLATFORM_ANDROID = 2;
  PLATFORM_IOS = 3;
}

message MyAnalyticsEvent {
  string device_id = 1;
  string video_id = 2;
  Platform platform = 3;
}

Note: For the sake of simplicity, we will assume that all of these fields will be set and won’t be null (even though they could be seen as null with Protobuf v3 spec)

Your first table’s will look something like this:

CREATE TABLE IF NOT EXISTS raw_events_queue (
    device_id String,
    video_id String,
    platform Enum8(
        'PLATFORM_UNSPECIFIED' = 0,
        'PLATFORM_ANDROID' = 1,
        'PLATFORM_IOS' = 2,
        'PLATFORM_WEB' = 3
    ),
    -- Maybe some more fields
) ENGINE = Kafka(kafka_connection) SETTINGS
    format_schema_source = 'string',
    format_schema = 'syntax = "proto3"; package my_analytics.v1; option java_multiple_files = true; option java_package = "com.example.team.telemetry.schemas.myanalytics.v1"; enum Platform { PLATFORM_UNSPECIFIED = 0; PLATFORM_WEB_FRONTEND = 1; PLATFORM_ANDROID = 2; PLATFORM_IOS = 3; } message MyAnalyticsEvent { optional string device_id = 1; optional string video_id = 2; optional Platform platform = 3; }',
    format_schema_message_name = 'MyAnalyticsEvent',
    kafka_topic_list = '<comma-separated-list-of-your-kafka-topics>',
    kafka_format = 'ProtobufSingle',
    kafka_group_name = 'my_analytics_clickhouse',
    -- Some other settings that I'll explain in a bit
    ;

This raw_events_queue table will act as a stream of events. It will not actually store any of the incoming messages. To actually use the events from this stream we need to somehow write them to a table. Now is the time that all of that Materialized View knowledge comes in handy.

Fun fact: This raw_events_queue will not start consuming data from the topic yet. It will only do so once the first Materialized View starts reading from it.

Now let’s design a simple aggregation just for the sake of learning the flow. It will not have any real use at this point.

-- 1. Create the Target Table
CREATE TABLE IF NOT EXISTS video_play_stats (
    video_id String,
    platform Enum8(
        'PLATFORM_UNSPECIFIED' = 0,
        'PLATFORM_ANDROID' = 1,
        'PLATFORM_IOS' = 2,
        'PLATFORM_WEB' = 3
    ),
    total_plays UInt64
) ENGINE = SummingMergeTree(total_plays)
ORDER BY (video_id, platform);

-- 2. Create the Materialized View
CREATE MATERIALIZED VIEW IF NOT EXISTS video_play_stats_mv 
TO video_play_stats AS
SELECT
    video_id,
    platform,
    toUInt64(1) AS total_plays
FROM raw_events_queue;

The flow of data will be as follows:

1. When the video_play_stats_mv Materialized View is added, it triggers the raw_events_queue to start consuming events from the topics you specified with kafka_topic_list.
2. As the events arrive, the raw_events_queue table parses them given the Protobuf defition and video_play_stats_mv reads the fields exposed by raw_events_queue.
3. The video_play_stats_mv will then write those to video_play_stats (note the TO video_play_stats syntax)
4. The video_play_stats table uses a SummingMergeTree which as the name suggests will sum the records based on grouping rules. In this example, the ORDER BY (video_id, platform) is what does the grouping. Clickhouse then sums total_plays as explicitly defined in the SummingMergeTree(total_plays) syntax.

I think this example should be a good one to help you understand the basic flow (but going beyond this on the specifics of ClickHouseDB Table Engines is again beyond the scope of this article).

#### Parallel consumers

By setting kafka_num_consumers and kafka_thread_per_consumer fields you can specify how many consumers will be consuming data in parallel from your Kafka topic. (See the partitioning section above for more info)

Example usage:

CREATE TABLE IF NOT EXISTS raw_events_queue (
) ENGINE = Kafka(kafka_connection) SETTINGS
    -- How many consumers will be consuming in parallel (per replica. So if you have 2 instances active, you will actually see 16 active consumers)
    kafka_num_consumers = 4,
    -- You could set this to 0 in your acceptance environment to reduce overhead and costs if your CPU count is less than your consumer count
    kafka_thread_per_consumer = 1,
    -- Other settings

#### Error handling with kafka_handle_error_mode

Clickhouse will by default crash on any Deserialization errors and you absolutely would want to avoid that. Gladly, that’s not that complicated (Although some versions such as v26.5 had a bug that would cause the DB to SegFault. This was fixed when I upgraded to v26.7 (not sure which specific patch version contained the fix.))

The kafka_handle_error_mode setting gives you two ways to handle errors:

• stream: adds _error and _raw_message virtual columns to your table. All the materialized views consuming from this table downstream MUST only read the records WHERE len(_error) = 0. If you just set this setting to stream and forget to add the WHERE clause, you will still crash the db.
• dead_letter_queue: In this case the error data is stored in system.dead_letter_queue. I wouldn’t recommend this approach because it’s a bit rigid.

My prefered solution to have a DLQ or a way to see what went wrong is to use kafka_handle_error_mode: stream and create a table and a materialized view that only reads the errored records:

CREATE TABLE my_kafka_events_queue_errors (
    error_time DateTime DEFAULT now(),
    raw_message String,
    error_reason String
) ENGINE = MergeTree
ORDER BY error_time
TTL error_time + INTERVAL 7 DAY;

CREATE OR REPLACE MATERIALIZED VIEW my_kafka_events_queue_errors_mv
TO my_kafka_events_queue_errors
AS
SELECT
    now() AS error_time,
    _raw_message AS raw_message,
    _error AS error_reason
FROM my_kafka_events_queue
WHERE length(_error) > 0;

Kafka Table engine also gives you a kafka_skip_broken_messages which is quite a mystery for me so far. I can’t fully visualize how this works together with the kafka_handle_error_mode option. @clickhouse if you read this could you improve the doc on this field, please? Thanks in advance!

#### Using nested proto structures with oneof fields

As your event grows, you need to decide if you want to have a flat message structure or a nested one. Both options are valid depending on your context. In protobuf, if you choose to nest fields, you would use the oneof keyword. Let’s expand our example slighly:

-- redacted for simplicity
message MyAnalyticsEvent {
  -- redacted for simplicity
  optional Platform platform = 3;

  oneof event_payload {
    VideoStarted video_started = 4;
    VideoEnded video_ended = 5;
  }
}

message VideoStarted {
  uint64 time_to_first_frame_ms = 1;
}

/* This is a valid protobuf message which needs a workaround to use in ClickHouse which I will explain if you read a bit further */
message VideoEnded { }

Gladly Clickhouse supports this too by setting input_format_protobuf_oneof_presence = 1 on the table (where the rest of the settings are) but it does require some extra work and there are gotchas. Please refer to this page for more info.

For this example we would need to modify our table like so:

CREATE TABLE IF NOT EXISTS raw_events_queue (
    -- Other fields are irrelevant for this example
    event_payload Enum8(
        'none' = 0,
        'video_started' = 4,
        'video_ended' = 5,
    ),
    video_started_time_to_first_frame_ms Nullable(UInt64),
    video_ended_dummy Nullable(Bool)
) ENGINE = Kafka(kafka_connection) SETTINGS
    -- We've changed below setting
    format_schema = 'syntax = "proto3"; package my_analytics.v1; option java_multiple_files = true; option java_package = "com.example.team.telemetry.schemas.myanalytics.v1"; enum Platform { PLATFORM_UNSPECIFIED = 0; PLATFORM_WEB_FRONTEND = 1; PLATFORM_ANDROID = 2; PLATFORM_IOS = 3; } message MyAnalyticsEvent { optional string device_id = 1; optional string video_id = 2; optional Platform platform = 3; oneof event_payload { VideoStarted video_started = 4; VideoEnded video_ended = 5; } } message VideoStarted { uint64 time_to_first_frame_ms = 1; } message VideoEnded { bool dummy = 1; }',
    -- We've added below setting
    input_format_protobuf_oneof_presence = 1,
    -- Other settings are irrelevant for this example
    ;

I would like to take your attention specifically to the end of the value of the format_schema setting: message VideoEnded { bool dummy = 1; }. This is a way to get around a ClickHouse limitation. ClickHouse’s input_format_protobuf_oneof_presence cannot detect empty protobuf messages. This affects VideoEnded and any other events that don’t have any nested fields. We have a dummy field in such events to work around this. This field does not need to be set by our event producers. You can also not even add this dummy field in your main schema to keep it clean and only add it to the schema that ClickHouse uses. Your clients then won’t need to see an ugly dummy field in your generated sdks.

This is quite an annoying issue because nesting with protobuf is quite nice for the readability of the generated code and reduces possible human “forgetting” errors. Hopefully ClickHouse addresses this at somepoint (Thanks in advance!)

#### Other tweaks

From this point onwards, you can tweak kafka_max_block_size, kafka_flush_interval_ms, kafka_poll_timeout_ms values to something that matches your rate of event production and the latency your db record consumers expect.

## NetworkPolicies

Slightly unrelated to the core of this article but I suggest you use Kubernetes NetworkPolicies on top of mTLS and the Authentication methods that Clickhouse and Kafka provide.
You will have certain active routes for example from your backend application to Clickhouse and from Clickhouse to Kafka brokers. This not only adds to your security, but also makes sure that applications get an explicit permission to talk to certain pods. This will help you understand your producer and consumer relations better and make migrations a lot less of a guess-work.

# Testing it

That was it! If you’ve followed this and made stuff, you now have a fully fledged Event Streaming & Analytics platform!

At this point you just need to start producing events and see how your system performs.

## Dashboards and observability

Each of components we used exposes lots of metrics which you can listen to and create as many alerts and dashboards as you want. If you also make your custom gateway, make sure to expose some metrics you can visualize. Otherwise you’ll be walking around blindly.

For Kafka, I’ve set up an instance of Kafbat-UI (formerly Kafka-UI) and it does a great job so far. I’m happy.
For ClickHouseDB I use CH-UI and I really like the built-in dashboard it offers.

Strimzi has a bunch of dashboards you could use, so does AutoMQ. I suggest grabbing an existing one and tweaking them to your needs.

# How it is going: Spoilers

## Strimzi

I had my doubts regarding Strimzi before I started using it because setting up a Kafka cluster with Strimzi also has a lot of edge cases and complexities, but it was definitely worth the learning journey and once it is set up, it “just works”.

## AutoMQ

AutoMQ is really great! For our usage it delivers great and the cost matches exactly what we expected.

The downsides are:

• Documentation is in some cases scattered or hard to find (but it’s there somewhere)
• As the time of writing [July 2026], AutoMQ still doesn’t support Kafka v4+ while that version is already out for quite some time already.

## Cost-optimization vs Total Cost of Ownership

In the Choices section above I’ve explained that one big reason I’ve chosen this solution was because it was low on Cloud costs, however we always have to consider the total cost of building and maintaining such solution. In practice we’re not spending that much time on the platform after the initial implementation. Sure, it took me a couple of weeks (8 maybe) to do the initial platform set up and the Strimzi and ClickHouse cluster needed some occasional fixes and tuning at the start when the platform was young but they are truely low cost in the sense that they don’t need to be actively modified. The company now pays less for maintaining this solution and has aquired more knowledge along the way. I truely believe this is how we should deal with things. Take on big challenges and learn. Be ready to fail. Sure buying is easy and you don’t have to wake up at night, but then you could also invest in your peoepl so they become better at building solution that doesn’t need them to wake up at night. But that’s for another article!

The part regarding educating the developers on how to use these tools could be seen as a slightly different story. We’re unfortunately in a bullshit generating AI era and cannot force people to rely on their own learnings and capacity. We can encourage, but not gatekeep. It probably takes more energy and effort to deal and try to educate the slop generating developers than to set up this whole platform. This is unfortunately the new reality and is not limited to this project but is an ongoing issue on all fronts. Even if that was not an issue, any other solution we would’ve chosen, we needed to put a similar amount of effort, teaching our developers. Arguably much more effort if we would’ve chosen Apache Flink instead of ClickHouseDB for example.

In terms of cloud cost, in practice it costs us around between $750 to run the whole stack, including Strimzi nodes, S3 buckets, Network traffic, Clickhouse nodes, EBS volumes etc. This will not increase much because the Strimzi cluster will be able to handle much more than our current needs without any upscaling. New applications will need dedicated Clickhouse instances. I count that as the cost of the application and not that of the platform.

# What can be improved

## Knowing cost of operation per tenant/topic

Since we have got everything on the same Kafka cluster, we could just look at the size of each topic’s messages and deduce something from it but it would be nicer to have a better mechanism to extract the total cost of operation per tenant. Something to consider soon.

## I would like a way to get more visibility into Events for debugging purposes

As events start flowing in at a rate of thousands per second, it becomes harder to find and debug a specific subset of events based on certain characteristics. You could tap into a topic with a new consumer with a simple script and do that but it would be nicer to have a unified simple way to achieve this at scale.

# Feedback

Thanks if you read so far! I hope this was interesting and I would love to hear some feedback.
Please reach out if you have any tips for me to improve this article. Cheers!