Kafka Streams Topology: Sandbox vs Cluster

No, a Kafka Streams topology does not start in a sandbox; it runs on your Kafka cluster’s brokers, but the topology itself is defined and built in your application code, which can run anywhere. The confusion arises because the development and testing of a topology often happens in a local sandbox, while the actual execution of the topology—the stream processing tasks—occurs on the cluster.

Background: What Is a Kafka Streams Topology?

Kafka Streams is a client library for building stream processing applications. A topology is a directed acyclic graph (DAG) of processors, where each node represents a processing step (like filtering, mapping, aggregating) and each edge represents a data flow between processors. The topology is defined in your application code using the Kafka Streams DSL (Domain-Specific Language) or the lower-level Processor API.

When you write a Kafka Streams application, you create a Topology object, then build a KafkaStreams instance from it. The KafkaStreams instance connects to your Kafka cluster and starts executing the topology. The topology itself is not stored on the cluster; it is a runtime construct that lives in your application’s memory. The cluster stores the data (in topics) and coordinates the execution via consumer groups and internal topics.

How It Works: Where the Topology Actually Runs

The Kafka Streams library partitions the topology into tasks, each responsible for a subset of the data partitions. These tasks are distributed across the application instances you launch. Each instance runs its own copy of the topology code, but they coordinate via the Kafka cluster to share work. The cluster’s brokers are involved in storing the input and output topics, as well as internal topics used for state stores and repartitioning.

Here’s a step-by-step of what happens:

• You write and compile your Kafka Streams application with the topology definition.
• You launch one or more instances of the application (e.g., on a server, in a container, or on a local machine).
• Each instance connects to the Kafka cluster (specified by bootstrap.servers) and joins a consumer group.
• The library assigns partitions to each instance, and the topology starts processing records from those partitions.
• As records flow, the library may create internal topics (prefixed with streams-) on the cluster to handle repartitioning or changelog topics for state stores.

So the topology runs on the machines where your application instances run, not on the brokers themselves. The brokers only store data and serve it to the consumers and producers.

Sandbox vs Cluster: Development and Testing

In practice, you almost always develop and test your topology in a sandbox—a local environment with a single-node Kafka cluster (often using kafka-stack-docker-compose or the embedded Kafka from Spring Boot). This sandbox mimics the cluster for development purposes. But when you deploy the application to production, the topology runs on your real cluster’s data, coordinated by the brokers.

The common misconception is that the topology is “uploaded” to the cluster like a JAR to a serverless platform. That is not how Kafka Streams works. Instead, you deploy your application as a standalone JVM process (or a set of processes) that connects to the cluster. The cluster does not execute your code; your code executes and uses the cluster for data storage and coordination.

How to Choose: Where to Run Your Kafka Streams App

You can run your Kafka Streams application on any machine that can reach your Kafka cluster. Common choices include:

• Dedicated application servers (e.g., on-premises VMs or bare metal).
• Container orchestration platforms like Kubernetes, where you deploy a Deployment or StatefulSet.
• Serverless platforms (e.g., AWS Lambda with custom runtime) but note that Kafka Streams is stateful and long-running, so serverless may not be ideal unless you use external state stores.
• Your local machine for testing, connecting to a remote cluster.

The key is that the application instances must be able to connect to the cluster’s bootstrap brokers and have enough resources (CPU, memory, disk) for processing and state stores. The cluster itself does not need to know about your topology; it just sees consumers and producers.

Common Problems and Misconceptions

One frequent issue is that developers think they must deploy their topology to the Kafka brokers. This leads to confusion when they cannot find a way to “submit” a topology. Another problem is that when testing locally, they use a sandbox Kafka cluster (like kafka-topics scripts) but then try to connect to a production cluster without adjusting security settings (SSL, SASL).

Another misconception is that the topology is serialized and sent to the cluster. In reality, the topology is just a Java object graph in your application. The cluster only sees the records you produce and consume. This means you can change your topology without restarting the cluster, but you must restart your application instances to apply the new topology.

Comparing Kafka Streams to Other Stream Processing Tools

Kafka Streams differs from tools like Apache Flink or Apache Spark Streaming, which often run as a cluster of their own. With Flink, you submit a job to a Flink cluster, and that cluster executes the topology. With Kafka Streams, you are the cluster—you run your own application instances and they coordinate via Kafka’s consumer groups.

This has pros and cons. Kafka Streams is simpler to deploy because you don’t need a separate processing cluster. But it means you must manage the lifecycle of your application instances yourself. Also, Kafka Streams is tightly coupled to Kafka, while Flink can read from many sources.

Costs and Resource Considerations

Running Kafka Streams applications incurs costs beyond the Kafka cluster itself. You need compute resources for your application instances. The more partitions and state stores, the more memory and disk you need. Also, each instance consumes network bandwidth to read and write to the cluster.

There is no additional licensing cost for Kafka Streams—it’s part of Apache Kafka. But you may incur costs for the infrastructure (cloud instances, storage) and for the increased load on your Kafka cluster due to internal topics and additional consumers.

Safety and Best Practices

When deploying Kafka Streams, follow these safety practices:

• Always test in a sandbox that mirrors your production cluster’s configuration (including security).
• Use exactly-once semantics if you need to avoid duplicates; this requires configuring processing.guarantee=exactly_once_v2.
• Monitor your application’s lag, throughput, and state store sizes.
• Set appropriate replication factor for internal topics (defaults to the cluster’s default, but you can override).
• Gracefully shut down your instances to allow rebalancing.

Frequently Asked Questions

Do I need to install Kafka Streams on my cluster?

No, Kafka Streams is a client library. You add it as a dependency to your Java application. The brokers do not need any additional software. Your application connects to the brokers as a normal consumer/producer.

Can I run Kafka Streams locally without a cluster?

You can use the embedded Kafka broker from the kafka-streams-test-utils artifact for unit tests. For integration tests, you can run a single-node Kafka cluster in Docker. But for production, you need a real cluster (even if it’s a single broker).

What happens if my application crashes?

If an instance crashes, the consumer group rebalances, and other instances take over the partitions. State stores are restored from changelog topics. If all instances crash, processing stops until you restart them.

Can I change my topology without stopping the application?

No, you must stop the application, deploy the new code, and restart. The topology is compiled into the code. You cannot dynamically add processors at runtime.

Is Kafka Streams the same as KSQL?

No. KSQL (now ksqlDB) is a separate streaming SQL engine that runs on a cluster of its own. Kafka Streams is a library you embed in your application. ksqlDB uses Kafka Streams internally but provides a SQL interface.

Related

• Robot Submarines and Sonar: Public Evidence
• Complacent Photographer Signs & How to Overcome
• How Astronomers Use Webb & Hubble for Star Formation