Skip to content

http-security.md

Latest commit

 

History

History
351 lines (259 loc) · 14.5 KB

http-security.md

File metadata and controls

351 lines (259 loc) · 14.5 KB
Raw

HTTP Security in pg_durable

This document describes the security model for df.http() — the durable HTTP activity that lets workflows make outbound HTTP(S) requests from within the PostgreSQL background worker.

Table of Contents

  1. Feature Flags
  2. Three-Layer Security Model
  3. Layer 0: PostgreSQL Privilege Check
  4. Layer 1: IP Blocklist (SSRF protection)
  5. Layer 2: Endpoint Allow-List
  6. Additional Hardening
  7. Audit Logging
  8. Error Messages
  9. Out of Scope

1. Feature Flags

Outbound HTTP access is controlled entirely by Cargo features at build time. The database cannot override these choices — they cannot be changed with GUCs or SQL.

Feature What is allowed Use case
(none) Nothing — df.http() errors immediately at DSL time and at execution time Deployments that don't need HTTP
http-allow-azure-domains Subdomains of the Azure allow-list plus api.github.com; bare IPs blocked; redirects blocked Production
http-allow-test-domains Everything in http-allow-azure-domains plus httpbingo.org E2E testing; implies http-allow-azure-domains
http-allow-all All URLs; SSRF IP blocklist and allow-list are both disabled Local development only

The scripts and CI use http-allow-test-domains so that the HTTP E2E tests pass — this includes the source-built Dockerfile used for local dev and CI. The released Debian packages are built with http-allow-azure-domains, so the published Docker image (Dockerfile.release, which installs that package) inherits the http-allow-azure-domains policy.

When no feature is set

df.http() fails at the point df.http() is called in SQL with:

df.http() is disabled. Rebuild with the 'http-allow-azure-domains' Cargo feature to enable outbound HTTP requests.

Because df.nodes rows can be inserted by hand (bypassing the DSL), the same block is enforced again at execution time inside execute_http.rs via validate_url_allowlist.

2. Three-Layer Security Model

┌──────────────────────────────────────────────────────────┐
│  Layer 0: PostgreSQL Privilege Check                     │
│                                                          │
│  • submitted_by role must have EXECUTE on df.http()      │
│  • Checked at execution time against the live catalog    │
│  • Blocks bypass via raw df.start() JSON injection       │
│  • Runs before any network activity                      │
│                                                          │
├──────────────────────────────────────────────────────────┤
│  Layer 1: IP Blocklist (SSRF protection)                 │
│                                                          │
│  • Private/reserved IP ranges blocked after DNS          │
│  • IPv4-mapped IPv6 (::ffff:A.B.C.D) unwrapped + checked │
│  • IP literals in URLs blocked before DNS                │
│  • DNS rebinding prevented via inline resolver check     │
│  • Disabled only under http-allow-all                    │
│                                                          │
├──────────────────────────────────────────────────────────┤
│  Layer 2: Endpoint Allow-List                            │
│                                                          │
│  • Bare IPv4/IPv6 addresses always rejected              │
│  • Hostname must match an approved suffix or exact name  │
│  • Disabled (allow everything) under http-allow-all      │
│  • Empty (block everything) when no http feature set     │
│                                                          │
└──────────────────────────────────────────────────────────┘

All three layers run inside execute_http.rs before the request is sent. There is no GUC, no table override, no superuser bypass for Layers 1 and 2.

3. Layer 0: PostgreSQL Privilege Check

3.1 Purpose

A user granted df.grant_usage() can call df.start() directly with a hand-crafted Durofut JSON string, inserting an HTTP node without ever calling df.http(). The DSL-time guard inside df.http() does not run in that path.

To close this gap, execute_http checks at execution time whether the submitted_by role recorded in the node still holds EXECUTE privilege on df.http(). If the role's grant has been revoked since the node was created, the node fails immediately.

3.2 Mechanism

execute_http runs the following check before any network activity:

SELECT has_function_privilege($submitted_by::regrole,
    'df.http(text,text,text,jsonb,integer)'::regprocedure,
    'EXECUTE')

has_function_privilege honours PostgreSQL's standard privilege model: superusers always return true; regular roles return true only when an explicit GRANT EXECUTE ON FUNCTION df.http(text, text, text, jsonb, integer) TO <role> (or a role that inherits one) is in effect.

3.3 Managing access

HTTP access is opt-in and separate from general df access.

Granting access

Use df.grant_usage() with include_http => true:

SELECT df.grant_usage('my_role', include_http => true);

Or grant directly:

GRANT EXECUTE ON FUNCTION df.http(text, text, text, jsonb, integer) TO my_role;

df.grant_usage('my_role') (without include_http) grants all standard df privileges but not df.http(). HTTP access must be explicitly opted in to.

Revoking access

To remove HTTP access without removing all df access:

REVOKE EXECUTE ON FUNCTION df.http(text, text, text, jsonb, integer) FROM my_role;

After this, any existing or future HTTP nodes submitted by my_role will fail at execution time with a "permission denied" error. All other df functions remain accessible.

df.revoke_usage('my_role') removes all df access, including df.http().

PUBLIC grant and upgrades

Fresh installs (v0.2.0+) have EXECUTE on df.http() revoked from PUBLIC at CREATE EXTENSION time. Installs that upgraded from v0.1.1 retain the PUBLIC grant that v0.1.1 issued — the upgrade script does not revoke it.

If an upgraded install should enforce opt-in HTTP permissions, the admin must run manually:

REVOKE EXECUTE ON FUNCTION df.http(text, text, text, jsonb, integer) FROM PUBLIC;

When df.grant_usage(role, include_http => false) is called and the role still has effective HTTP access via the PUBLIC grant (or another inherited grant), a WARNING is emitted to signal that the revocation had no net effect.

3.4 Admin function protection

df.grant_usage() and df.revoke_usage() are admin-only functions. EXECUTE is revoked from PUBLIC at CREATE EXTENSION time, so only superusers can call them.

Caution: df.grant_usage() internally runs GRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA df, which temporarily includes df.grant_usage() and df.revoke_usage() themselves before the function immediately revokes them from the target role. If an admin replicates the blanket GRANT manually without the matching REVOKEs, the target role will gain access to these admin helpers. Always use df.grant_usage() rather than hand-crafting the equivalent GRANT statements.

3.5 Feature-flag interaction

The privilege check runs regardless of which HTTP Cargo feature is enabled. When no HTTP feature is compiled in, the request is still blocked later by the DSL-time guard and by execution-time URL validation, but the privilege check remains compiled in and still runs before any network activity.

4. Layer 1: IP Blocklist (SSRF protection)

4.1 Blocked IPv4 ranges

CIDR Description
0.0.0.0/8 "This" network
10.0.0.0/8 RFC 1918 private
127.0.0.0/8 Loopback
169.254.0.0/16 Link-local — includes cloud metadata at 169.254.169.254
172.16.0.0/12 RFC 1918 private
192.168.0.0/16 RFC 1918 private

4.2 Blocked IPv6 ranges

Range Description
::/128 Unspecified
::1/128 Loopback
fe80::/10 Link-local
fc00::/7 Unique local (ULA)

4.3 IPv4-mapped IPv6 handling

Addresses of the form ::ffff:A.B.C.D are unwrapped to their embedded IPv4 before the blocklist check, preventing bypasses like ::ffff:169.254.169.254.

4.4 DNS rebinding prevention

The SSRF-safe DNS resolver (SsrfSafeResolver) wraps the system resolver and filters blocked IPs inline — the same IP that passes the check is the one used for the TCP connection. There is no window for a rebinding attack.

Only the single IP address that reqwest actually connects to is checked. If DNS returns multiple A/AAAA records, the others are not checked because they are never used. This is intentional, not a gap — checking unused addresses would create false positives without any security benefit.

4.5 IP literals in URL

Bare IP literals in URLs (e.g. http://169.254.169.254/...) bypass DNS entirely — reqwest connects directly without calling the resolver. validate_url_allowlist blocks all bare IPs unconditionally, so these never reach the resolver.

5. Layer 2: Endpoint Allow-List

5.1 Azure domains (always present with http-allow-azure-domains)

Only subdomains of the following suffixes are permitted. Apex domains (e.g. blob.core.windows.net without a subdomain label) are rejected.

Suffix Service
.blob.core.windows.net Azure Blob Storage
.blob.storage.azure.net Azure Blob Storage (secondary)
.queue.core.windows.net Azure Queue Storage
.table.core.windows.net Azure Table Storage
.file.core.windows.net Azure Files
.azurewebsites.net Azure App Service
.azure-api.net Azure API Management
.documents.azure.com Azure Cosmos DB
.servicebus.windows.net Azure Service Bus
.openai.azure.com Azure OpenAI
.cognitiveservices.azure.com Azure Cognitive Services
.vault.azure.net Azure Key Vault
.redis.cache.windows.net Azure Cache for Redis
.database.windows.net Azure SQL Database
.kusto.windows.net Azure Data Explorer
.azurefd.net Azure Front Door
.azureedge.net Azure CDN
.azure-devices.net Azure IoT Hub
.trafficmanager.net Azure Traffic Manager
.cloudapp.azure.com Azure Cloud App

5.2 Exact-match domains (always present with http-allow-azure-domains)

Matched exactly — subdomains and lookalikes are rejected.

Domain Purpose
api.github.com GitHub API

5.3 Test domains (additional with http-allow-test-domains)

Domain Purpose
httpbingo.org HTTP echo service (used in HTTP E2E tests)

5.4 Bare IP rejection

All bare IPv4 and IPv6 addresses are rejected by validate_url_allowlist regardless of feature flag — even under http-allow-azure-domains. Because the allowlist blocks all bare IPs, there is no separate IP-literal check; the allowlist is the definitive gate for IP-literal URLs.

6. Additional Hardening

6.1 Scheme restriction

Only http:// and https:// are accepted. All other schemes (file://, ftp://, gopher://, etc.) are rejected before any DNS resolution or connection attempt.

6.2 Redirect blocking

reqwest is built with Policy::none() (no redirect following). This prevents redirect-based bypasses where an attacker hosts a public server that returns a 302 Location: http://169.254.169.254/... — since the redirect target is an IP literal, the DNS resolver would never be called.

7. Audit Logging

Every HTTP attempt (allowed or blocked) is logged via ctx.trace_info with:

  • submitted_by — the role that called df.start() at the time the node was created (captured as current_user in the DSL and stored in FunctionNode)
  • url — the requested URL
  • Block reason tag — (scheme), (allowlist), or (ip) in the log prefix

Resolved IP addresses are not included in error messages or logs to avoid leaking internal network topology to potentially malicious users.

8. Error Messages

Scenario Message
No EXECUTE privilege on df.http() Blocked: role '{role}' does not have EXECUTE privilege on df.http(). Grant EXECUTE ON FUNCTION df.http(text,text,text,jsonb,integer) TO {role} to allow HTTP requests.
HTTP disabled (no feature) Blocked: outbound HTTP requests are disabled. Rebuild with the 'http-allow-azure-domains' Cargo feature to enable them.
Unsupported scheme Blocked: unsupported URL scheme '{scheme}'. Only http and https are allowed.
Bare IP address Blocked: requests to bare IP addresses are not permitted. Use an approved Azure service hostname instead.
Non-allowed domain Blocked: '{host}' is not in the allowed endpoint list. Only requests to approved Azure service domains are permitted.
Blocked IP (literal or DNS) Blocked: the resolved IP address for '{host}' is in a restricted range. df.http() cannot access private or internal network addresses.
DSL-time (no feature) df.http() is disabled. Rebuild with the 'http-allow-azure-domains' Cargo feature to enable outbound HTTP requests.

9. Out of Scope

These items are deferred to a future customer-level access control spec:

Item Notes
Per-role URL/domain allowlists configurable by admins GUC or table-driven
Rate limiting DoS mitigation, not SSRF
Response size limits Resource management
Port restrictions Low value at this layer
Egress filtering to attacker-controlled domains Separate threat (T9)
Azure Private Endpoint Private Endpoints assign private RFC 1918 addresses to Azure services, which the IP blocklist currently blocks. Supporting Private Endpoints requires a targeted exemption mechanism that does not open all private ranges. Design is deferred to a future spec.