Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -40,5 +40,7 @@ foreign/node/.npm
*.out
go.work
core/bench/dashboard/frontend/dist
core/integration/tests/connectors/postgres/cdc_restart_connectors/*
!core/integration/tests/connectors/postgres/cdc_restart_connectors/config.toml
LICENSE-binary
**/LICENSE-binary
31 changes: 23 additions & 8 deletions core/connectors/sources/postgres_source/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,9 +41,7 @@ primary_key_column = "id"
# Custom query (optional)
custom_query = "SELECT * FROM $table WHERE id > $offset ORDER BY id LIMIT $limit"

# CDC options (optional)
enable_wal_cdc = false
publication_name = "iggy_publication"
# CDC options (only used when mode = "cdc")
replication_slot = "iggy_slot"
Comment thread
mmodzelewski marked this conversation as resolved.
capture_operations = ["INSERT", "UPDATE", "DELETE"]
cdc_backend = "builtin"
Expand All @@ -69,9 +67,7 @@ cdc_backend = "builtin"
| `processed_column` | string | none | Boolean column to mark as processed |
| `primary_key_column` | string | tracking_column | PK for delete/mark operations |
| `custom_query` | string | none | Custom SQL with parameter substitution |
| `enable_wal_cdc` | bool | `false` | Enable WAL-based CDC |
| `publication_name` | string | `iggy_publication` | Logical replication publication |
| `replication_slot` | string | `iggy_slot` | Replication slot name |
| `replication_slot` | string | `iggy_slot` | Replication slot name (only used when `mode = "cdc"`) |
| `capture_operations` | array | `["INSERT","UPDATE","DELETE"]` | CDC operations to capture |
| `cdc_backend` | string | `builtin` | `builtin` or `pg_replicate` |
| `verbose_logging` | bool | `false` | Log at info level instead of debug |
Expand Down Expand Up @@ -267,13 +263,33 @@ CDC requires PostgreSQL logical replication setup:
```toml
[plugin_config]
mode = "cdc"
enable_wal_cdc = true
tables = ["users", "orders"]
capture_operations = ["INSERT", "UPDATE", "DELETE"]
```

The `pg_replicate` backend requires the `cdc_pg_replicate` feature flag at build time.

### Slot Naming

Each CDC connector must use a unique `replication_slot`. Setup accepts any
pre-existing `test_decoding` slot, so two connectors pointed at the same
database with the default `replication_slot = "iggy_slot"` will silently
share one slot. `pg_logical_slot_get_changes` consumes changes on read, so
each connector only sees a subset of the other's changes instead of erroring.
Set an explicit, distinct `replication_slot` per connector instance.

### Decommissioning

A replication slot retains WAL for as long as it exists, regardless of
whether a connector is still consuming it (`max_slot_wal_keep_size` defaults
to `-1`, i.e. unbounded). Dropping a connector without dropping its slot
leaves an orphaned slot that accumulates WAL indefinitely and can fill the
disk. When decommissioning a CDC connector, drop its slot:

```sql
SELECT pg_drop_replication_slot('iggy_slot');
```

## Example Configs

### Basic Polling (JSON Mode)
Expand Down Expand Up @@ -344,7 +360,6 @@ batch_length = 100
[plugin_config]
connection_string = "postgresql://user:pass@localhost:5432/mydb"
mode = "cdc"
enable_wal_cdc = true
tables = ["users", "orders"]
capture_operations = ["INSERT", "UPDATE"]
```
Expand Down
89 changes: 89 additions & 0 deletions core/connectors/sources/postgres_source/src/cdc_fixtures.rs

Large diffs are not rendered by default.

Loading
Loading