Skip to content

Add support for Data Modifying Common Table Expressions (CTEs) in PostgreSQL backend - #821

Merged
LaurentRDC merged 10 commits into
haskell-beam:masterfrom
kushagarr:cte_support_postgres
Jul 16, 2026
Merged

Add support for Data Modifying Common Table Expressions (CTEs) in PostgreSQL backend#821
LaurentRDC merged 10 commits into
haskell-beam:masterfrom
kushagarr:cte_support_postgres

Conversation

@kushagarr

@kushagarr kushagarr commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

Summary

This PR adds PostgreSQL-specific support for data-modifying and explicitly
materialized common table expressions in beam-postgres, without changing
the portable CTE API or SQL99 abstractions in beam-core.

The implementation introduces a placement-indexed PgWith builder which
captures PostgreSQL’s rule that data-modifying CTEs must be attached to a
top-level statement. It supports WITH statements terminating in SELECT,
INSERT, UPDATE, or DELETE, while preserving compatibility with existing
portable With Postgres helpers.

A complete WITH ... <terminal statement> is represented as one Beam
statement and sent to PostgreSQL in one database round trip.

PostgreSQL-specific builder

PgWith is indexed by PgCtePlacement:

  • PgCteNestedAllowed represents SELECT-only CTE blocks which may be embedded
    in a subquery.
  • PgCteTopLevelOnly represents blocks containing data-modifying CTEs.

pgSelectWithNested accepts only nested-safe blocks.
pgSelectWithTopLevel, pgInsertWith, pgUpdateWith, and pgDeleteWith
produce top-level statements and therefore accept either placement.

The placement parameter has a nominal role, preventing Data.Coerce from
relabelling a top-level-only block as nested-safe.

Existing helpers built with With Postgres can be composed using pgLiftWith.
Lifted and PostgreSQL-specific CTEs use the same name supply, so their generated
CTE names cannot collide.

Recursive construction is restricted to SELECT-only blocks. pgToTopLevel
allows a completed recursive SELECT block to feed a later data-modifying CTE
without permitting a modifying CTE to refer recursively to itself.

Supported CTEs

SELECT CTEs can be introduced with:

  • pgSelecting
  • pgSelectingWith PgCteDefault
  • pgSelectingWith PgCteMaterialized
  • pgSelectingWith PgCteNotMaterialized

Explicit MATERIALIZED and NOT MATERIALIZED modifiers require PostgreSQL 12
or later. PgCteDefault emits no modifier and retains PostgreSQL’s normal
planner behaviour and compatibility with older server versions.

Reusable data-modifying CTEs are provided by:

  • cteInsertReturning
  • cteUpdateReturning
  • cteDeleteReturning

Their RETURNING output is exposed through ReusableQ and can be consumed by
later CTEs or the terminal statement using reuse.

Side-effect-only variants are also provided:

  • cteInsert
  • cteUpdate
  • cteDelete

These omit RETURNING, return (), and still execute when the surrounding
top-level statement runs.

Zero-column projections

Reusable CTE projections containing no fields are supported.

For a zero-column SELECT CTE, Beam omits PostgreSQL’s optional CTE column-alias
list. This preserves the query’s row cardinality without exposing any result
fields.

PostgreSQL requires a data-modifying RETURNING clause to contain at least one
expression. For a zero-field cteInsertReturning, cteUpdateReturning, or
cteDeleteReturning projection, Beam therefore emits a private
NULL::boolean sentinel:

WITH "cte0"("res0") AS
       (DELETE FROM "source"
        WHERE ...
        RETURNING NULL::boolean)
SELECT FROM "cte0" AS "t0"

The sentinel is not exposed to Beam’s result decoder. It preserves one
zero-field result row for every affected row, allowing the reusable result to
participate in COUNT(*), EXISTS, joins, and repeated reuse.

When no later CTE or terminal statement needs the affected rows, the
side-effect-only builders should be used instead; they omit RETURNING
entirely.

Terminal statements and no-op handling

A PgWith block may terminate with:

  • pgSelectWithTopLevel
  • pgInsertWith
  • pgUpdateWith
  • pgDeleteWith

The DML consumers retain Beam’s existing SqlInsert, SqlUpdate, and
SqlDelete wrappers, so the existing PostgreSQL returning API remains
available for the terminal statement.

Empty inserts and identity updates remain Beam no-ops. Since PostgreSQL cannot
execute a bare WITH block, accumulated CTE definitions are discarded when
there is no terminal statement.

This PR also fixes pgSelectWith with an empty CTE block: it now renders the
underlying SELECT directly instead of producing an invalid empty WITH
clause.

Documentation

The Haddock documentation and PostgreSQL user guide include paired Beam and
generated-SQL examples covering:

  • lifted portable helpers
  • explicit materialization
  • reusable and side-effect-only modifying CTEs
  • zero-column projections
  • SELECT, INSERT, UPDATE, and DELETE terminal statements

The Chinook documentation fixture is pinned to the revision matching its
verified checksum, making clean documentation builds reproducible.

Testing

The test suite covers:

  • mixed SELECT, INSERT, UPDATE, and DELETE CTE definitions
  • SELECT and DML terminal consumers
  • MATERIALIZED and NOT MATERIALIZED
  • reusable and side-effect-only modifying CTEs
  • recursive SELECT CTEs feeding later modifications
  • parameter ordering across CTE bodies and terminal statements
  • shared naming across lifted and native builders
  • empty inserts and identity updates
  • zero-column SELECT and data-modifying projections
  • zero-column cardinality through COUNT(*), EXISTS, joins, and repeated reuse
  • negative type tests for invalid nesting, recursive modification, coercion,
    and reuse of side-effect-only CTE results
  • Hedgehog properties comparing PostgreSQL execution with pure Haskell models

- Introduced `pgWithSyntax` to prefix PostgreSQL statements with CTEs.
- Added new types for data-modifying CTEs: `PgDataModifyingCommonTableExpressionSyntax` and `PgCommonTableExpressionSyntax`.
- Implemented type classes for handling CTEs in SQL queries, ensuring correct placement of data-modifying CTEs.
- Created unit and integration tests for rendering and type safety of CTEs, including negative tests for invalid placements.
- Updated documentation to reflect new CTE capabilities and usage examples.
- Modified test setup to use PostgreSQL version 18.4 for better compatibility.
@kushagarr kushagarr changed the title WIP : Add support for Common Table Expressions (CTEs) in PostgreSQL backend WIP : Add support for Data Modifying Common Table Expressions (CTEs) in PostgreSQL backend Jul 11, 2026
@kushagarr kushagarr changed the title WIP : Add support for Data Modifying Common Table Expressions (CTEs) in PostgreSQL backend Add support for Data Modifying Common Table Expressions (CTEs) in PostgreSQL backend Jul 11, 2026
@kushagarr
kushagarr marked this pull request as ready for review July 11, 2026 08:42
Comment thread beam-core/Database/Beam/Backend/SQL/SQL99.hs Outdated
- Updated CTE handling in tests to reflect PostgreSQL's behavior with zero-column CTEs and side-effect-only CTEs.
- Added new tests for materialization execution and rendering, ensuring compliance with PostgreSQL 12+ features.
- Refined error handling for invalid CTE placements, particularly for side-effect-only operations.
- Improved documentation on PostgreSQL-specific CTE usage, including materialization options and nested CTEs.
- Adjusted dependencies in the SQLite library for compatibility.
- Ensured reproducibility of CI runs by pinning PostgreSQL server version.
@kushagarr
kushagarr requested a review from LaurentRDC July 15, 2026 04:34
Comment thread beam-postgres/test/Database/Beam/Postgres/Test/CTE.hs
Comment thread docs/user-guide/backends/beam-postgres.md
Comment thread beam-postgres/Database/Beam/Postgres/Syntax.hs Outdated
Comment thread beam-postgres/Database/Beam/Postgres/Full.hs Outdated
@kushagarr
kushagarr requested a review from LaurentRDC July 15, 2026 19:01
Comment thread beam-postgres/beam-docs.sh Outdated
Comment thread beam-postgres/beam-postgres.cabal
Comment thread beam-postgres/Database/Beam/Postgres/Full.hs
@LaurentRDC

Copy link
Copy Markdown
Member

We're getting really close!

@kushagarr
kushagarr requested a review from LaurentRDC July 16, 2026 13:20
@kushagarr

Copy link
Copy Markdown
Contributor Author

It closes #290

@LaurentRDC LaurentRDC linked an issue Jul 16, 2026 that may be closed by this pull request

@LaurentRDC LaurentRDC left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome, thanks @kushagarr!

@LaurentRDC
LaurentRDC merged commit 2852cc9 into haskell-beam:master Jul 16, 2026
13 checks passed
@kushagarr
kushagarr deleted the cte_support_postgres branch July 17, 2026 08:32
@LaurentRDC

Copy link
Copy Markdown
Member

@kushagarr released in beam-postgres 0.6.3.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support Data-modifying CTE's

2 participants