> ## Documentation Index
> Fetch the complete documentation index at: https://sequinstream.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# How to stream Postgres to Kafka

> Build real-time data pipelines by streaming Postgres changes to Kafka. Learn how to go from dev to prod with a Sequin Kafka sink.

This guide shows you how to set up Postgres change data capture (CDC) and stream changes to Kafka using Sequin.

With Postgres data streaming to Kafka, you can trigger workflows, keep services in sync, [build audit logs](/how-to/create-audit-logs), [maintain caches](/how-to/maintain-caches), and more.

By the end of this how-to, you'll have database changes flowing to a Kafka topic.

<Tip>
  This is the how-to guide for streaming Postgres to Kafka. See the [quickstart](/quickstart/kafka) for a step-by-step walkthrough or the [reference](/reference/sinks/kafka) for details on all configuration options.
</Tip>

## Prerequisites

If you're self-hosting Sequin, you'll need:

1. [Sequin installed](/running-sequin)

2. [A database connected](/connect-postgres)

3. A Kafka cluster ready to go

4. A Kafka cluster ready to go

## Basic setup

### Prepare your Kafka cluster

You'll need a Kafka cluster and topic ready for Sequin to stream changes to. You can use either a local cluster for development or a cloud-hosted Kafka service in production.

#### Local development with Docker

For local development with Sequin, you can quickly spin up a Kafka cluster using Docker Compose:

<Accordion title="docker-compose.yaml">
  ```yaml theme={null}
  version: '3'
  services:
    zookeeper:
      image: bitnami/zookeeper:latest
      ports:
        - "2181:2181"
      environment:
        - ALLOW_ANONYMOUS_LOGIN=yes
      networks:
        - kafka-net

    kafka:
      image: bitnami/kafka:latest
      ports:
        - "9092:9092"
        - "9093:9093"
      environment:
        - KAFKA_BROKER_ID=1
        - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
        - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
        - KAFKA_CFG_LISTENERS=INTERNAL://:9093,EXTERNAL://:9092
        - KAFKA_CFG_ADVERTISED_LISTENERS=INTERNAL://kafka:9093,EXTERNAL://localhost:9092
        - KAFKA_CFG_INTER_BROKER_LISTENER_NAME=INTERNAL
        - ALLOW_PLAINTEXT_LISTENER=yes
      depends_on:
        - zookeeper
      networks:
        - kafka-net

  networks:
    kafka-net:
      driver: bridge
  ```
</Accordion>

Start the cluster:

```bash theme={null}
docker-compose up -d
```

Create a topic for Sequin to publish to:

```bash theme={null}
docker exec -it <kafka-container-id> kafka-topics.sh \
  --create \
  --topic my-topic \
  --bootstrap-server kafka:9093 \
  --partitions 1 \
  --replication-factor 1
```

#### Production Kafka

For production environments, Sequin works with major managed Kafka providers as well as self-hosted Kafka clusters. To prepare your Kafka cluster for Sequin, you'll need to:

1. Create a dedicated topic for Sequin to publish to
2. Create credentials (username/password) for Sequin to use
3. Configure appropriate ACLs to allow Sequin to publish to your topic

<Info>
  Sequin currently supports SASL PLAIN, SCRAM-SHA-256, and SCRAM-SHA-512 authentication mechanisms with TLS encryption. Custom client certificates for verifying TLS connections are coming soon, [let us know if you need them](https://github.com/sequinstream/sequin/issues/592).
</Info>

Once your cluster is ready, make note of:

* Bootstrap server addresses
* Topic name
* Authentication credentials
* Whether TLS is required

You'll need these details when configuring your Kafka sink in the next section.

## Create Kafka sink

Navigate to the "Sinks" tab, click "Create Sink", and select "Kafka Sink".

### Configure the source

<Steps>
  <Step title="Select source tables">
    Under "Source", select the schemas and tables you want to stream data from.
  </Step>

  <Step title="Add filters (optional)">
    Add [filters](/reference/filters) to the sink to control which database changes are sent to your Kafka topic.
  </Step>

  <Step title="Specify backfill">
    You can optionally indicate if you want to [backfill](reference/backfills) of all or a portion of the table's existing data into Kafka. Backfills are useful if you want to use Kafka to process historical data. For example, if you're materializing a cache, you might want to warm it with existing rows.

    You can backfill at any time. If you don't want to backfill, toggle "Backfill" off.
  </Step>

  <Step title="Specify message grouping">
    Under "Message grouping", you'll most likely want to leave the default option selected to ensure events for the same row are sent to [Kafka in order](reference/sinks/kafka#message-ordering-and-partitioning).
  </Step>
</Steps>

### Configure Kafka

<Steps>
  <Step title="Enter Kafka connection details">
    Fill in your Kafka connection details:

    * **Hosts** (required): A comma-separated list of `host:port` pairs (e.g., `kafka1.example.com:9092,kafka2.example.com:9092`)
      * This can be either the full list of Kafka brokers or just the bootstrap server
    * **Topic** (required): The Kafka topic to publish messages to (max 255 characters)
    * **Username**: Username for SASL PLAIN authentication (optional)
    * **Password**: Password for SASL PLAIN authentication (optional)
    * **TLS**: Enable TLS/SSL encryption for the connection
  </Step>

  <Step title="Create the sink">
    Give your sink a name, then click "Create Kafka Sink".
  </Step>
</Steps>

## Verify & debug

To verify that your Kafka sink is working:

1. Make some changes in your source table
2. Verify that the count of messages for your sink increases in the Sequin web console
3. Using your Kafka consumer tools, check your topic:
   ```bash theme={null}
   kafka-console-consumer --bootstrap-server localhost:9092 --topic your-topic --from-beginning
   ```
   You should see the messages from Sequin appear in the topic.

If messages don't seem to be flowing:

1. Click the "Messages" tab to view the state of messages for your sink
2. Click any failed message
3. Check the delivery logs for error details, including any Kafka connection errors

## Next steps

* **Setup a processor**

  Now that your Postgres data is flowing into Kafka, you can setup a consumer group to read from the topic and process the data.

  Refer to the [Kafka sink reference](/reference/sinks/kafka) for the shape of messages that Sequin will publish to Kafka.

* **Deploy your implementation**

  When you're ready to deploy your implementation, see "[How to deploy to production](/how-to/deploy-to-production)".

* **Advanced configuration**

  For more about how Kafka sinks work, see the [Kafka sink reference](/reference/sinks/kafka).