Skip to main content
In this quickstart, you’ll create a real-time data pipeline that streams changes from a Postgres database to a Google Cloud Pub/Sub topic. You’ll:
  • Boot Sequin
  • Connect to a sample playground database
  • Configure a Pub/Sub topic to receive database changes
  • See your changes flow in real-time
By the end, you’ll have hands-on experience setting up Postgres change data capture (CDC) with Sequin and GCP Pub/Sub.
This is the quickstart for streaming Postgres to GCP Pub/Sub. See the how-to guide for an explanation of how to use the GCP Pub/Sub sink or the reference for details on all configuration options.

​
Prerequisites

  • A Google Cloud project with Pub/Sub enabled
  • A service account with Pub/Sub permissions
For this quickstart, you can use an existing service account with broad Pub/Sub permissions. For production use cases, we recommend creating a dedicated service account for Sequin with these specific roles:
  • roles/pubsub.publisher
  • roles/pubsub.viewer
For more information on creating a service account, see the how-to guide for Pub/Sub.

Create a Pub/Sub Topic

If you don’t already have a Pub/Sub topic:
  1. Navigate to the Pub/Sub Topics page in your Google Cloud Console
  2. Click “CREATE TOPIC”
  3. Enter a topic ID for this tutorial (e.g., “products”)
  4. Click “CREATE”
Make note of your:
  • Google Cloud project ID (e.g., “my-project-123”)
  • Pub/Sub topic ID (e.g., “products”)

Run Sequin

The easiest way to get started with Sequin is with our Docker Compose file. This file starts a Postgres database, Redis instance, and Sequin server.
1

Create directory and start services

  1. Download sequin-docker-compose.zip.
  2. Unzip the file.
  3. Navigate to the unzipped directory and start the services:
cd sequin-docker-compose && docker compose up -d

curl -L https://github.com/sequinstream/sequin/releases/latest/download/sequin-docker-compose.zip -o sequin-docker-compose.zip \
  && unzip sequin-docker-compose.zip && rm sequin-docker-compose.zip
cd sequin-docker-compose && docker compose up -d

git clone [email protected]:sequinstream/sequin.git
cd sequin/docker
docker compose up -d
2

Verify services are running

Check that Sequin is running using docker ps:
docker ps
You should see output like the following:
CONTAINER ID   IMAGE                           COMMAND                  CREATED          STATUS                    PORTS                              NAMES
bd5c458cabde   sequin/sequin:latest            "/scripts/start_comm…"   11 seconds ago   Up 9 seconds              4000/tcp, 0.0.0.0:7376->7376/tcp   sequin-sequin-1
3bacd89765e7   grafana/grafana                 "/run.sh"                11 seconds ago   Up 11 seconds             0.0.0.0:3000->3000/tcp             sequin-sequin_grafana-1
3ad41319a66c   postgres:16                     "docker-entrypoint.s…"   11 seconds ago   Up 11 seconds (healthy)   0.0.0.0:7377->5432/tcp             sequin-sequin_postgres-1
6139a5fc4e80   redis:7                         "docker-entrypoint.s…"   11 seconds ago   Up 11 seconds             0.0.0.0:7378->6379/tcp             sequin-sequin_redis-1
7e07a5b052de   prom/prometheus                 "/bin/prometheus --c…"   11 seconds ago   Up 11 seconds             0.0.0.0:9090->9090/tcp             sequin-sequin_prometheus-1
Sequin, Postgres, Redis, Prometheus, and Grafana should be up and running (status: Up).

Login

The Docker Compose file automatically configures Sequin with an admin user and a playground database.Let’s log in to the Sequin web console:
1

Open the web console

After starting the Docker Compose services, open the Sequin web console at http://localhost:7376:
Sequin login page, allowing login with default credentials
2

Login with default credentials

Use the following default credentials to login:
  • Email:
[email protected]
  • Password:
sequinpassword!

View the playground database

To get you started quickly, Sequin’s Docker Compose file creates a logical database called sequin_playground with a sample dataset in the public.products table.Let’s take a look:
1

Navigate to Databases

In the Sequin web console, click Databases in the sidebar.
2

Select playground database

Click on the pre-configured sequin-playground database:
Playground database
The database “Health” should be green.
3

View contents of the products table

Let’s get a sense of what’s in the products table. Run the following command:
docker exec -i sequin-sequin_postgres-1 \
  psql -U postgres -d sequin_playground -c \
  "select id, name, price from products;"
This command connects to the running Postgres container and runs a psql command.
You should see a list of the rows in the products table:
  id |         name          | price 
----+-----------------------+-------
  1 | Avocados (3 pack)     |  5.99
  2 | Flank Steak (1 lb)    |  8.99
  3 | Salmon Fillet (12 oz) | 14.99
  4 | Baby Spinach (16 oz)  |  4.99
  5 | Sourdough Bread       |  6.99
  6 | Blueberries (6 oz)    |  3.99
(6 rows)
We’ll make modifications to this table in a bit.

Create a Pub/Sub Sink

With the playground database connected, you can create a sink. This sink will send changes to the products table to your Pub/Sub topic:
1

Navigate to Sinks

Click “Sinks” in the sidebar navigation, then click “Create Sink”.
2

Select sink type

Select “Google Cloud Pub/Sub” as the sink type and click “Continue”.
3

Note "Source" configuration

In the “Source” card, note that the sequin-playground database is selected and all schemas and tables are included. Leave these defaults:
Source card showing the sequin_playground database and products table
4

Setup a backfill

In the Inital backfill card, select the public.products table to initate a backfill when the sink is created.
Initial backfill card with public.products table selected
5

Configure "Pub/Sub Configuration"

In the “Pub/Sub Configuration” card, enter your Pub/Sub details:
  • Project ID: Your Google Cloud project ID
  • Topic ID: The Pub/Sub topic to stream to (e.g., “products”)
  • Service Account Credentials: Paste your service account JSON key file contents
Pub/Sub configuration card
6

Test the connection

At the bottom of the form, click the “Test Connection” button. If you provided proper credentials, it should succeed.
Sequin can connect to your Pub/Sub topic.
7

Create the sink

You can leave the rest of the defaults. As configured, the Pub/Sub topic will first receive a backfill of all rows currently in the products table. Then, it will receive all changes to the products table in real-time.Click “Create Sink” to finish setting up your Pub/Sub sink.

See changes flow to your Pub/Sub topic

On the new sink’s overview page, you should see the “Health” status turn green, indicating data is flowing to your Pub/Sub topic.Let’s confirm messages are flowing:
1

Messages tab

Click the “Messages” tab. You’ll see a list of the recently delivered messages:
List of recently delivered messages
Sequin indicates it backfilled the products table to your Pub/Sub topic.
2

View in Pub/Sub Console

In the Google Cloud Console:
  1. Navigate to your Pub/Sub topic
  2. Click on the “MESSAGES” tab
  3. Under “Step 2”, select a Cloud Pub/Sub subscription
  4. Click “PULL” to see messages in the topic
You should see a list of messages appear in the table below. The messages are read events from the initial backfill of the products table.
Messages are flowing from Sequin to your Pub/Sub topic.
3

Make some changes

Let’s make some changes to the products table and see them flow to your topic.In your terminal, run the following command to insert a new row into the products table:
docker exec -i sequin-sequin_postgres-1 \
  psql -U postgres -d sequin_playground -c \
  "insert into products (name, price) values ('Organic Honey (16 oz)', 12.99);"
In the Pub/Sub Console, click “PULL” again to see the new message appear.Feel free to try other changes:
docker exec -i sequin-sequin_postgres-1 \
  psql -U postgres -d sequin_playground -c \
  "update products set price = 7.99 where name = 'Avocados (3 pack)';"

docker exec -i sequin-sequin_postgres-1 \
  psql -U postgres -d sequin_playground -c \
  "update products set name = 'Organic Avocados (3 pack)' where name = 'Avocados (3 pack)';"

docker exec -i sequin-sequin_postgres-1 \
  psql -U postgres -d sequin_playground -c \
  "delete from products where name = 'Blueberries (6 oz)';"
Each change will appear in your Pub/Sub topic within a few seconds.
Great work!
You’ve successfully:
  • Set up a complete Postgres change data capture pipeline
  • Loaded existing data through a backfill
  • Made changes to the products table
  • Verified changes are flowing to your Pub/Sub topic

​
Ready to stream

Now you’re ready to connect your own database to Sequin and start streaming changes:

Guide: Connect Postgres

Connect your Postgres database to Sequin.

Guide: Setting up a Pub/Sub sink

Setup a Pub/Sub sink to send changes to your topic.