Audience: operators.
End to end, with a proof point at every step — what to look at, and what it should say. The PostgreSQL Targets page is the reference for what each collector does; this is the ordered procedure for getting one collecting and knowing that it is.
Status. This started as the procedure derived from the code, written before the service had ever been pointed at a PostgreSQL target. That sentence is now false in the direction you want: the service has since run PostgreSQL targets end to end — connect probe → collector dispatch → COPY into a store → read back out — on live fleets, self-hosted and Aurora/RDS both, and several steps below carry corrections only a real run could have found: the TLS trap in step 2 came off the first real run of this runbook, the statement-stats growth cap came off a 50-cluster fleet's first 36 hours (
PgStatementStatsCollector's own remarks tell that story), and thepgstattuplefailure mode step 7 describes was observed on thepostgresmaintenance database of the first production target.Log text below is still quoted from the source that emits it, and every count is re-derived from
CollectorCatalogand the store's migration ladder rather than remembered. The instruction stands: if what you see disagrees with this file, trust the target, and correct this file from what you actually see.
| Check | ||
|---|---|---|
| A target | Aurora PostgreSQL, or self-managed PostgreSQL 10 or later. An older version stores NULL for a column it lacks, and pg_io_stats needs 16 (see Oldest PostgreSQL version per collector) |
SELECT version(); |
| A login on it | pg_monitor, password auth |
step 1 |
| A store | Managed (bundled) or your own PostgreSQL | Operations, The Store |
| The service | A current release, installed as Install as a Windows Service describes | PerformanceMonitor.Darling.Service.exe --version |
| Network | The service host must reach the target's port 5432 | nc -z <host> 5432 |
On Aurora, "the target" is one instance, not the cluster. A writer and a reader are two entries with two identities, because they have two sets of statistics — see step 3 for what changes between them.
Read-only. Nothing is created and no setting changes, unlike a SQL Server target with its Extended Events sessions.
CREATE ROLE darling_monitor WITH LOGIN PASSWORD '<password>';
GRANT pg_monitor TO darling_monitor;On Aurora/RDS run the GRANT as an rds_superuser. No superuser is needed for the monitoring login.
Proof: connect as the login and confirm it can see other backends, which is the whole point of
pg_monitor — without it the statistics views quietly narrow to the connecting user's own sessions and
fleet monitoring becomes self-monitoring with no error anywhere.
SELECT count(*) AS visible_backends,
count(*) FILTER (WHERE query = '<insufficient privilege>') AS hidden
FROM pg_stat_activity;hidden must be 0. If it equals visible_backends - 1, the grant did not take.
Optional, for the five extension-backed collectors. The biggest one first: pg_statement_stats,
the per-query-shape collector, reads pg_stat_statements — on Aurora it reads
aurora_stat_statements(), the same view plus Aurora-only columns, and on everything else the vanilla
view with those columns honestly NULL (#2625). Check for it:
SELECT count(*) FROM pg_extension WHERE extname = 'pg_stat_statements';If that returns 0 you need pg_stat_statements in shared_preload_libraries — a parameter-group change
plus a reboot on Aurora/RDS — and then CREATE EXTENSION pg_stat_statements; in the database Darling
connects to. One installation covers the whole instance; the extension is keyed by dbid and tracks all
databases. The other four, from the collectors' own declared dependencies: pg_buffer_usage
(pg_buffercache — a plain CREATE EXTENSION, no restart), pg_kernel_stats (pg_stat_kcache, which
sits on top of pg_stat_statements), pg_predicate_stats (pg_qualstats — preloaded, and additionally
CREATE EXTENSION per database), and pg_wait_sampling (the pg_wait_sampling module — preload-only,
so check it with SHOW shared_preload_libraries;, not pg_extension). Skipping any of them is fine:
the collector records a non-fatal skip naming exactly which install is missing (step 10), and the other
collectors are unaffected.
pg_wait_sampling is the one whose absence is no longer a skip (#3604). Without the extension the same
collector takes its service-sampler arm: it polls pg_stat_activity once a second for a 30-second
window every five minutes and accumulates what it saw, so a stock target has a wait profile from its first
cycle rather than an empty chart. That arm needs nothing beyond the pg_monitor grant in step 1 —
pg_stat_activity's wait columns for other users' backends come from pg_read_all_stats, which
pg_monitor carries, and pg_stat_clear_snapshot() and pg_sleep() are callable by any role. It is a
floor, not parity: a wait shorter than a second is seen with probability roughly its length over a second,
and nothing between windows is seen. Every read of that table (get_pg_wait_sampling, the Viewer's panel)
says which arm fed it as instrument — extension_sampled or service_sampled — and the three-tier
selection (Aurora native › extension › service sampler) is made once when the service connects, so after
installing the extension let the service reconnect to move up a tier.
pg_deadlocks, pg_plan_capture and pg_log_events read the server log with pg_read_file(), and that is the one read
in this step pg_monitor does not cover. It takes BOTH halves — the role does not carry the function's
EXECUTE, because pg_read_file's ACL is postgres=X/postgres (measured on #2566; granting the role
alone changes nothing) — and the EXECUTE half lives in each database's own catalog, so run this in the
database Darling connects to, not in postgres for good measure:
GRANT pg_read_server_files TO darling_monitor;
GRANT EXECUTE ON FUNCTION pg_read_file(text), pg_read_file(text, bigint, bigint), pg_read_file(text, bigint, bigint, boolean) TO darling_monitor;
GRANT EXECUTE ON FUNCTION pg_catalog.pg_read_binary_file(text, bigint, bigint) TO darling_monitor;The third grant is not optional in practice (#4046):
pg_read_file() returns text, which PostgreSQL validates against the database encoding before this
process ever sees a row, so one byte that is not valid UTF-8 anywhere in the 4 MB tail — which a failed
login can plant with nothing but a bad role or database name — fails the WHOLE read for as long as that
byte sits in the window, blinding all three collectors at once. pg_read_binary_file() returns bytea,
which carries no such check, and the three collectors switch to it on their own once it is granted, with no
restart and no config change: on the next cycle after a read that failed on such a byte, otherwise within an
hour. That applies to a database whose encoding is UTF8 or SQL_ASCII. PostgreSQL checks SQL_ASCII text as
UTF-8 on its way to the collector, so a SQL_ASCII database meets the same failure and gets the same fix. In any
other encoding, the collectors stay on pg_read_file(), because the binary route decodes the log as UTF-8.
LATIN1 accepts every byte except NUL, so a planted byte never fails its read. The EUC encodings and WIN1252 can
still fail it, and the grant does not help there. The fault message says so, and
#4062 tracks a fix.
Issued in a different database on the same cluster, the grants change nothing and the failure looks
identical — measured on a live PG18 target, where the in-database grant flipped pg_deadlocks from
PERMISSIONS to SUCCESS on the next cycle, no restart needed. Self-hosted only: on Aurora/RDS there
is no filesystem, pg_read_server_files is not grantable, and all three collectors take the RDS log API
route instead — that is the IAM subsection below.
Proof: as the monitoring login, in the same database, mirror the collectors' own read:
SELECT pg_catalog.pg_read_file('log/' || name, 0, 64) FROM pg_catalog.pg_ls_logdir() ORDER BY modification DESC, name DESC LIMIT 1;A log line comes back. permission denied for function pg_read_file means the EXECUTE half is missing
in this database; until both halves are in place, all three collectors fail every cycle as PERMISSIONS,
with an error_message naming this exact pair and the database to run it in.
pg_server_config reads pending_restart off pg_settings, which is per-connection: on Windows, where a
new backend rebuilds its settings from the postmaster's saved values on every connect, that column can read
false for a setting that changed and reloaded but is still waiting for a restart — and this collector opens
a fresh connection every cycle. On Unix this does not happen: the postmaster itself applies the reload, and
every backend it forks inherits the flag, so pg_settings.pending_restart is already correct there. Granting
the two objects below on a non-Windows target fixes nothing, and the collector does not ask for them there —
skip this section on Linux, RDS, Aurora, Azure and Cloud SQL.
On a Windows target, pg_file_settings closes the gap: it is read from the file, not backend-local state. It
needs two grants, both superuser-only by default:
GRANT SELECT ON pg_file_settings TO darling_monitor;
GRANT EXECUTE ON FUNCTION pg_show_all_file_settings() TO darling_monitor;Both are superuser-only for a reason: the view returns every uncommented line of every configuration file,
including lines that are not the running value — a superseded or misspelled primary_conninfo with its
password still sitting above the current one, for example — and each file's path. Grant them only if that
exposure is acceptable for this role. The collector reads nothing from the view but a setting's name and its
error text; get_pg_server_config and get_pg_logging_audit say so in the caveat they attach when the
grants are missing on a Windows target.
This is a different axis from the grant above — it authorizes the monitoring host's AWS identity,
not the PostgreSQL login. On Aurora/RDS there is no local log directory a SQL session can read with
pg_read_file(); plan capture, deadlock detection and the log-event pipeline instead pull the log tail
through the RDS control plane. The CPU and host-memory collector also calls AWS, to read Performance Insights.
Attach this to the instance role/profile the Darling service actually runs as, or, for a server that has its own
role, to that role (see the next subsection):
{
"Effect": "Allow",
"Action": [
"rds:DescribeDBClusters",
"rds:DescribeDBInstances",
"rds:DescribeDBLogFiles",
"rds:DownloadDBLogFilePortion"
],
"Resource": [
"arn:aws:rds:<region>:<account-id>:cluster:*",
"arn:aws:rds:<region>:<account-id>:db:*"
]
}Add a second statement for Performance Insights:
{
"Effect": "Allow",
"Action": "pi:GetResourceMetrics",
"Resource": "arn:aws:pi:<region>:<account-id>:metrics/rds/*"
}DescribeDBClusters resolves an Aurora cluster to its current writer instance; the log actions list and
read the log file itself. DescribeDBInstances looks up the instance's DbiResourceId, the identifier that
Performance Insights uses. pi:GetResourceMetrics reads the CPU and host-memory series from Performance
Insights. Darling also uses the two Describe actions to check that the server's host is the endpoint AWS
reports for that instance or cluster, before it reads any logs or Performance Insights data for it; a host
that does not match, or a server that does not accept a fresh login, is recorded as a PERMISSIONS outcome and nothing is read. Self-managed PostgreSQL doesn't need this at all — it reads pg_read_file() directly,
which is what the log-reader grants above are for. Skipping this on Aurora/RDS is not silent: both
collectors log the missing action by name (see step 10).
The service asks AWS for the cluster in the account that its credentials belong to. If the cluster is in
another account, the host's own identity cannot see it, and the calls fail with is not authorized to perform.
Each server can use its own AWS role instead, set on that server alone:
{
"name": "aurora-orders-writer",
"engine": "postgres",
"host": "orders.cluster-abc123.us-east-1.rds.amazonaws.com",
"awsRoleArn": "arn:aws:iam::123456789012:role/darling-monitor",
"awsExternalId": "REPLACE with your external ID"
}awsRoleArn is an IAM role ARN. awsExternalId is optional, needs the role, and is 2 to 1224 characters
(letters, digits and _ + = , . @ : / -). Use a random value that cannot be guessed, such as a UUID. The sample value above is a placeholder with spaces in it, which the service refuses as written, so it has to be replaced with your own. The store keeps
the external ID as plain text, unlike the passwords, which it seals. The web, MCP and the desktop viewer never show it
back; they show only whether one is set. AWS does not treat the external ID as a secret either: it is not a
credential, it can be seen by anyone with permission to view the role, and its job is to keep a role from being
assumed on someone else's behalf (see
External IDs for third party access
in the AWS IAM User Guide). Both apply to a PostgreSQL target only. Darling assumes the role
through AWS STS and uses it for every AWS call it makes for that server. Those calls are the Describe calls
that check the host, instance CPU and host memory (pg_cpu_utilization, through RDS and Performance Insights),
and the log reads behind pg_plan_capture, pg_deadlocks and pg_log_events. A server with no awsRoleArn uses the
process's own AWS credentials, exactly as before. Two servers can name two different roles, and a server's
role is never used for another server.
To set it up:
-
In
123456789012(the cluster's account), create the role and attach the policies from the IAM section above. Leave its maximum session duration at one hour or more. Darling asks for one-hour sessions and renews them itself, about 15 minutes before they expire. -
Give the role a trust policy that names the AWS identity the Darling service runs as. If you use an external ID, require it with an
sts:ExternalIdcondition that matchesawsExternalId:{ "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::<host-account-id>:role/<darling-host-role>" }, "Action": "sts:AssumeRole", "Condition": { "StringEquals": { "sts:ExternalId": "REPLACE with your external ID" } } } -
In the host's account, allow the host's identity to call
sts:AssumeRoleonarn:aws:iam::123456789012:role/darling-monitor. -
Add
awsRoleArn(andawsExternalId) to the server, by one of the paths below.
The session name is darling- and the first 12 hex digits of the install id, so the assumed sessions can be
told apart in CloudTrail. Darling calls STS in the region of the target it is reading.
A role is a server setting like host. You can put it in the
darling.json entry (the file seeds the registry once, see step 2), through add_servers and edit_server
(aws_role_arn, aws_external_id), on the web Servers page, in the desktop viewer's Add and Edit server
dialog, and with --add-server. The service runs a role only if it is allowed:
-
allowedAwsRolesindarling.jsonis a list of role ARNs (matched exactly) and 12-digit account ids (every role in that account). A bare account id is an account in theawspartition. For an account in the China or GovCloud partition, write the partition first:aws-cn:123456789012oraws-us-gov:123456789012:"allowedAwsRoles": [ "arn:aws:iam::123456789012:role/darling-monitor", "123456789012" ]
-
The effective list is that list plus the
awsRoleArnof every entry indarling.json's ownservers[]. A role you write in the file is allowed by being there.allowedAwsRolesis what lets the web, MCP, the desktop viewer and--add-serverset a role. -
With no list, the web and MCP cannot set a role. A role set through them has to be on the list when you save, and the answer says so:
The web and MCP cannot set an AWS role until darling.json lists the roles they may use. Add allowedAwsRoles to darling.json, then restart the service.(no list) orThat AWS role is not on the list of roles the web and MCP may set. Add it to allowedAwsRoles in darling.json and restart the service.(a list without that role).--add-serverruns on the host and is not held to the list when it saves, but the service still runs only a listed role. -
The list is read when the service starts. Restart the service after you change
allowedAwsRoles. -
A role the list does not allow is never assumed, even if a stored row already names it. Remove the role (or list it) to clear the message in step 10.
On the web and in edit_server, leaving the role blank (or omitting it) keeps the stored role.
Sending null removes it, and the external ID goes with it. Removing a role is always allowed. Changing the role on a
server that has an external ID stored needs the external ID in the same request (a new one, or null):
Changing the AWS role needs the external ID with it: send the external ID again, or clear it. The web and
MCP never return the external ID. They show only whether one is set (aws_external_id_set). Omitting the external ID keeps
the stored one. The desktop viewer's dialog is filled with the stored role, so
emptying that box removes the role. Its external ID box does not show the stored ID: it reads "set, leave blank to
keep" when one is stored, typing in it replaces the ID, and the "Clear external ID" box removes it. A role-only change takes
effect at the next sweep, with no restart: the server's connection is rebuilt with the new role.
The service logs and records the role ARN and ExternalId = set or ExternalId = none. It never writes the
external ID itself. A diagnostics bundle replaces the role ARN and account id with aliases and removes the
external ID.
Which path you use depends on whether this store has ever been seeded, and getting this wrong is the
most likely way to conclude the feature is broken when it is not. darling.json seeds
config.config_monitored_servers once, when that table is empty. After that the registry is
authoritative and a darling.json edit adds nothing.
Fresh store — edit darling.json:
{
"name": "aurora-orders-writer",
"engine": "postgres",
"host": "orders.cluster-abc123.us-east-1.rds.amazonaws.com",
"auth": "sql",
"username": "darling_monitor",
"encryptedPassword": "<output of --encrypt-password>",
"trustServerCertificate": true
}Existing install — use add_servers through MCP, which is the designed path and takes effect within one
collection sweep, no restart:
[{"host": "orders.cluster-abc123.us-east-1.rds.amazonaws.com",
"engine": "postgres", "auth": "SQL",
"username": "darling_monitor", "password": "...",
"trust_server_certificate": true}]It probes before saving, so a bad credential or an unreachable host comes back as connection_failed
without leaving a broken row behind — and an added PostgreSQL target's detail reports the same
writer/reader, Aurora, and collectors-apply facts step 3 describes.
Check which situation you are in before assuming:
SELECT count(*) AS registered FROM config.config_monitored_servers;Three things go wrong here and each has a specific symptom:
authmust be"sql". There is no integrated-auth path for PostgreSQL. An entry asking for it fails config validation with a message saying so, rather than failing later at connect.trustServerCertificate: trueis usually needed on Aurora. The default is full certificate verification (SslMode=VerifyFull) and Aurora presents an RDS CA a stock trust store does not know. Without it, connect fails on certificate validation — which reads like a network problem and is not.- A target with TLS switched off entirely needs
"encryptMode": "optional".trustServerCertificaterelaxes verification, not the requirement — so a stock self-hosted PostgreSQL runningssl = off, which is the normal shape for a local or lab instance, is unreachable untilencryptModeisoptional(SslMode=Prefer). Found the hard way on the first real run of this runbook: fail-closed TLS is correct, and the failure gives no hint that the fix is a different setting than the one you already tried. nameis the storage identity. It derivesserver_id, so renaming an existing entry orphans its history under the old id rather than moving it.enginetypos behave differently by path, on purpose. Indarling.jsonan unrecognized engine resolves to SQL Server rather than throwing, so one bad line cannot stop the service from monitoring everything else.add_serversrefuses it — onboarding is one deliberate act, and"postgress"silently becoming a SQL Server target produces a connection failure against 5432 with nothing naming the cause.
The password slot takes env:NAME and file:/path references as well as a DPAPI blob; on non-Windows
hosts use those, since DPAPI is Windows-only.
PerformanceMonitor.Darling.Service.exe --test-connection
Proof: a [PASS] line that reports PostgreSQL facts, ending in how many collectors will actually run.
[PASS] aurora-orders-writer: PostgreSQL 17 (server_version_num 170007), writer, Aurora — all 29 PostgreSQL collectors apply
Read the count. It is computed by asking the same gate the collector runner asks, so it is the real
answer, and it is the difference between "this is configured" and "this will collect". A gated-off
collector is named in the line itself — 25 of 29 PostgreSQL collectors apply (skipped: ...) — and the
reasons come from the collectors' own gates in CollectorCatalog:
| Target | Applies | Skipped, and why |
|---|---|---|
| Aurora writer, PG 16+ | 29 of 29 | — |
| Aurora reader | 25 of 29 | pg_autovacuum_stats, pg_index_usage_stats, pg_table_bloat_stats, pg_index_bloat — all four are writer-only: a standby's per-table statistics are either zeros or its own, and both readings are wrong for the cluster |
| Self-managed 16+ writer | 27 of 29 | pg_wait_stats, pg_cpu_utilization — one reads aurora_stat_system_waits(), the other AWS Performance Insights; both gate on Aurora detection |
| Self-managed 15 reader | 22 of 29 | all of the above, plus pg_io_stats (needs pg_stat_io, PostgreSQL 16+) |
A PostgreSQL 13 target skips no collector for its version, but pg_write_stats stores NULL in its WAL columns, because the pg_stat_wal source needs PostgreSQL 14.
If the count is lower than the table says it should be, the probe disagrees with you about the target —
check writer/reader and Aurora/not Aurora in the same line before touching anything else.
A [FAIL] carries the driver's own error. The three common ones:
| Error | Cause |
|---|---|
28P01: password authentication failed |
wrong password, or the secret reference did not resolve |
| certificate / SSL validation failure | trustServerCertificate not set on Aurora (step 2) |
Timeout ... connection attempt |
security group, VPN route, or wrong endpoint — not credentials |
The verb exits 0 only when the config is valid and every server is reachable, so it works as a deployment gate. One caveat inherited from the SQL Server path: it connects as you, the console user. For a PostgreSQL target that is the same credential the service will use (it comes from the config, not the ambient identity), so the caveat that matters for integrated auth does not apply here.
Adding the first PostgreSQL target does not need a store change of its own, but starting this build
against an existing store migrates it — applied automatically on start, forward-only, no
down-migration. The first PostgreSQL collector tables arrived as rungs V63–V69 and the registry's
engine/port columns as V70, and the ladder has kept climbing with the collectors since (V71 blocking
edges, V83–V95, plan capture at v99, the PostgreSQL alert count knobs at V122, ...): a current build
carries any older store to the newest version. StorageVersion.SchemaVersion is the source of truth (214 at
3.11.0), and step 5's proof line quotes whatever it says at your build.
Before starting, if your store is unmanaged and has TimescaleDB, re-derive the background-worker settings. Nearly every collector table becomes a hypertable. The required numbers move whenever collectors are added, and undersizing does not error — it silently stops compression and retention from running. See Background workers; this build's first start migrates the store and can add hypertables. After that start, count your hypertables and continuous aggregates with the two queries there and work its formula. For example, a store with 72 hypertables and 29 continuous aggregates gets 128 and 139 on an 8 GB host. It gets 74 and 85 at 4 GB or less, and 177 and 188 at 12 GB or more. Both settings need a server restart. Managed mode does this itself.
& "C:\Program Files\PerformanceMonitorDarling\PerformanceMonitor.Darling.Service.exe"
If you built the service from source, run the exe that the build wrote instead:
Darling\PerformanceMonitor.Darling.Service\bin\Release\net10.0\PerformanceMonitor.Darling.Service.exe.
There is no --console flag — the bare executable is console mode, and the Windows-service lifetime is a
no-op when there is no service host. Worth stating because looking for the flag and not finding it reads like
a missing feature.
Proof — two lines, in order. The store migrated (the applied count is however many rungs this store was behind — on a fresh store it is all of them):
Postgres store ready (schema v<N>, <M> migration(s) applied)
The target was probed as PostgreSQL:
Connected to PostgreSQL target 'aurora-orders-writer': major 17 (server_version_num 170007), writer, Aurora: True, pg_wait_sampling: n/a (engine_cumulative tier), pg_stat_statements: extension 1.10 — PostgreSQL 17.7 ...
If this line says writer for something you believe is a reader, or Aurora: False for an Aurora
instance, stop — every gate downstream keys off these and the collection you get will not be what you
expect.
The per-collector row-count lines exist too, but they are Debug-level since #3102, so a default
console shows the two lines above and not these. Raise the service's own namespace
(Logging__LogLevel__PerformanceMonitor.Darling.Service=Debug, documented in the README) to see them —
or skip straight to step 6, which proves the same thing from collection_log at any log level:
[aurora-orders-writer] pg_wait_stats => 47 rows (sql:31ms, pg:9ms)
[aurora-orders-writer] pg_xmin_horizon => 4 rows (sql:12ms, pg:4ms)
The per-database collectors name their database, one line per database per cycle:
[aurora-orders-writer] pg_autovacuum_stats [orders] => 19 rows (sql:88ms, pg:6ms)
Against the store, not the target:
SELECT collector_name,
count(*) AS runs,
max(collection_time) AS latest,
sum(rows_collected) AS rows_total,
count(*) FILTER (WHERE status <> 'SUCCESS') AS non_success,
max(error_message) AS last_error
FROM collect.collection_log
WHERE server_id = (SELECT server_id FROM collect.servers WHERE server_name = 'aurora-orders-writer')
AND collector_name LIKE 'pg\_%'
GROUP BY collector_name
ORDER BY collector_name;Within a couple of minutes, every per-minute collector the pre-flight count promised should appear
with non_success = 0. The hourly and daily ones arrive on their own cadences — step 7's table says
when each is due, so an absence there is a schedule, not a failure, until its first interval has passed.
status is one of exactly six values — SUCCESS, PERMISSIONS, EXTENSION_MISSING, ERROR,
SESSION_MISSING, YIELDED — and step 10 covers what a non-SUCCESS one means.
A collector that is absent entirely was gated off, not failing: cross-check it against the skipped list from step 3.
Two different waits, and conflating them is the most likely way to mistake a working install for a broken one. A row exists after the first cycle. A reader — the MCP tool — needs two samples before it can difference a cumulative counter, so the first read after startup legitimately shows zero activity.
All 29, from CollectorScheduleDefaults — the shared table both SKUs schedule by:
| Collector | Cadence | First row | First meaningful read |
|---|---|---|---|
pg_wait_stats |
1 min | 1 min | 2 min (counters need two samples) |
pg_statement_stats |
1 min | 1 min | 2 min |
pg_xmin_horizon |
1 min | 1 min | 1 min (levels, not counters) |
pg_replication_slots |
1 min | 1 min | 2 min (growth needs two) |
pg_replication_stats |
1 min | 1 min | 1 min (a sample, not a counter) |
pg_io_stats |
1 min | 1 min | 2 min |
pg_write_stats |
1 min | 1 min | 2 min |
pg_database_stats |
1 min | 1 min | 2 min |
pg_blocking |
1 min | 1 min | 1 min (a sample, not a counter) |
pg_session_states |
1 min | 1 min | 1 min (a sample, not a counter) |
pg_lock_stats |
1 min | 1 min | 1 min (a sample, not a counter) |
pg_wraparound_stats |
5 min | 5 min | 5 min (levels) |
pg_deadlocks |
5 min | 5 min | the first deadlock reported — an event log, not a counter |
pg_log_events |
5 min | 5 min | the first classified line — an event log; each family carries rows only while its log_* setting is on (#3601). A temp_file event carries the spill's exact bytes once log_temp_files is on (#3602); an autovacuum event carries the run's duration, pages, tuples, buffers and WAL once log_autovacuum_min_duration is set, and get_pg_autovacuum_health shows them per table as recent_runs (#3603) |
pg_cpu_utilization |
5 min | 5 min | 5 min (Performance Insights backfills the 1-minute points) |
pg_autovacuum_stats |
60 min | 60 min | 2 h (growing/flat needs two) |
pg_table_bloat_stats |
60 min | 60 min | 2 h (growing/flat needs two) |
pg_server_config |
60 min | 60 min | 2 h for the changes read (a change needs two snapshots) |
pg_plan_capture_readiness |
60 min | 60 min | 60 min (facets are levels) |
pg_plan_capture |
60 min | 60 min | the first plan auto_explain logs past its threshold |
pg_wait_sampling |
5 min | 5 min | 10 min (counters; on the service-sampler arm the second cycle also fills the first window) |
pg_kernel_stats |
60 min | 60 min | 2 h (counters) |
pg_predicate_stats |
60 min | 60 min | 2 h (counters) |
pg_buffer_usage |
60 min | 60 min | 60 min (residency is a level) |
pg_database_size_stats |
60 min | 60 min | 2 h (growth needs two samples) |
pg_index_usage_stats |
24 h | 24 h | 48 h (a scan count is a difference) |
pg_index_bloat |
24 h | 24 h | 24 h, complete (see below) |
pg_column_stats |
24 h | 24 h | 24 h (levels; "it moved" needs two) |
pg_extension_availability |
24 h | 24 h | 24 h (levels) |
The three extension-backed hourly ones (pg_kernel_stats, pg_predicate_stats, pg_plan_capture) are
hourly for the fleet's sake rather than the data's: on a managed target they can only ever record a
non-fatal skip, and a five-minute cadence would be hundreds of skip rows per target per day saying the
same thing. pg_wait_sampling left that group in #3604: it is gated off Aurora (which cannot load the
module) and takes the service-sampler arm everywhere else, so it records that skip nowhere, and its
five-minute cadence is the sampler arm's duty cycle (30 s of every 300). Their counters lose nothing at the longer interval — with one exception the
schedule's own remarks spell out: pg_plan_capture's self-hosted route is a fixed 4 MB log tail with
no resume marker, so a server logging faster than that window covers silently loses plans between reads,
and a self-hosted operator lowers the cadence per server rather than relying on the default.
The per-database collectors are the ones that surprise people — seven of them at the time of writing,
which is every collector whose RunsPerDatabase returns true, so count them from CollectorCatalog
rather than from this sentence. pg_autovacuum_stats and pg_table_bloat_stats take an hour before the
first row and two before "growing or flat" can be answered; pg_index_usage_stats takes a day, and
two before a windowed scan count exists at all.
pg_index_bloat used to be the one whose first-answer column needed reading carefully, and since
#3234 it is not. It estimates from catalog statistics and reads no index page, so one cycle covers
every index rather than a rotating slice: a day gets you the whole census, not a fraction of it. What
it needs instead is the pg_read_all_data grant below — pg_stats supplies the column widths the
model rests on, and without them every row reports the widths-not-visible reason rather than a number.
It no longer needs the pgstattuple extension at all; that is now the ON-REQUEST route to an exact
figure, and every row carries the pgstatindex call for its own index.
Read the reason on a row that has no estimate, because two of them are permanent. A never-analyzed
parent needs an ANALYZE; invisible column widths need the grant. But a partial index cannot be
modelled — the only row count available is the parent table's, and the index holds just the rows
matching its predicate — and a deduplicated one cannot either, because PostgreSQL 13+ stores
duplicate keys once in a posting list, so real storage is denser than per-tuple arithmetic allows and a
correct model still over-predicts. Those two are what the exact command is for, at any grant and any
statistics freshness.
It remains the complete btree census with no size floor, while pg_index_usage_stats floors at
64 kB, which is why the two report row counts differing by roughly 65% on the same target: the gap is
entirely indexes too small for usage statistics to be worth recording.
Those cadences are deliberate — a PostgreSQL connection is bound to one database for life, so per-database collection costs one connection per database per cycle, and index usage is a structural question that an hourly sample would re-record 24 times a day for nothing.
pg_table_bloat_stats shares pg_autovacuum_stats' hour on purpose rather than by copying: it measures
the DAMAGE whose CAUSE that one measures, and "vacuum fell behind at 14:00 and bloat grew" is only a
sentence the data can support if both are sampled on the same grain.
Through MCP — a read per collector, plus the trend, detail and config-diff readers that sit on top of
them; more than 30 get_pg_* tools in all, registered by the same service:
| Tool | Answers |
|---|---|
get_pg_wait_stats |
what the instance waits on (Aurora only — get_pg_wait_sampling covers everything else) |
get_pg_wait_sampling |
the sampled wait profile from the pg_wait_sampling extension — the stock-PostgreSQL counterpart of the above, with a waiting-vs-working split |
get_pg_top_queries |
top query shapes by total time, wherever pg_stat_statements is installed; Aurora adds the storage-vs-cache I/O split and per-statement peak memory |
get_pg_plans |
captured execution plans from auto_explain, grouped by plan shape, literals redacted before storage |
get_pg_plan_capture_readiness |
whether the target can capture plans at all, facet by facet in causal order, with the remedy for each unmet step |
get_pg_logging_audit |
whether the target's logging settings (log_lock_waits, log_temp_files, log_autovacuum_min_duration, log_checkpoints, log_connections/log_disconnections, log_min_duration_statement) are producing the lines they could — per setting: verdict, what it unlocks, the recommended value with its cost, and the remedy in your hosting flavour's syntax; judged from the stored pg_server_config snapshot, so it answers once pg_server_config has its first snapshot (step 7: 60 min) |
get_pg_wraparound_risk |
XID and MultiXact freeze headroom — how close to a write outage |
get_pg_xmin_horizon |
why vacuum is reclaiming nothing, attributed to the specific holder |
get_pg_replication_slots |
slot health, and whether retained WAL is still growing |
get_pg_replication_stats |
the CONNECTED replicas: send/replay lag, with the window's worst beside the latest |
get_pg_autovacuum_health |
tables ranked by how far past their own trigger threshold, each with recent_runs — what its automatic vacuums and analyzes cost, from the log_autovacuum_min_duration reports in pg_log_events (#3603) |
get_pg_io_stats |
I/O by (backend type, object, context) — who, what, and why (PostgreSQL 16+) |
get_pg_io_trend |
one (backend type, context) pair's rates and hit ratio over time |
get_pg_cpu_utilization |
instance CPU from AWS Performance Insights (Aurora/RDS only) |
get_pg_kernel_stats |
OS-level CPU and device I/O per query shape (pg_stat_kcache) — the burning-vs-waiting split beside get_pg_top_queries |
get_pg_predicate_stats |
which columns queries filter on and how selectively (pg_qualstats) — the evidence behind an index idea |
get_pg_blocking |
blocking chains that were SAMPLED, with the root attributed |
get_pg_lock_stats |
contended lock modes and relations over time, sampled from pg_locks |
get_pg_deadlocks |
deadlocks parsed from the server log, one row per distinct deadlock |
get_pg_deadlock_detail |
one deadlock in full: the complete wait graph and every participant's SQL, normalized |
get_pg_log_events |
the server log, classified: errors (WARNING and worse), connections, lock waits, spills with their exact bytes per file (#3602), autovacuum / autoanalyze runs with their duration, pages, tuples, buffers and WAL (#3603), plus checkpoints recognised for later structure — messages as PostgreSQL wrote them with any SQL in them normalized, filtered by family and min_severity, the page saying what bounded it. The "check the error log" read (#3601) |
get_pg_database_stats |
temp-file spills, cache hit ratio, deadlocks, commit/rollback split, and the window's peak connected backends per database (peak_numbackends, a level — null on a pre-V133 history, not 0) |
get_pg_database_trend |
one database's spills, hit ratio, deadlocks and rollback share over time, in buckets sized to the window, each with its worst interval |
get_pg_index_usage |
which indexes nothing scans — and whether each one can actually be dropped |
get_pg_index_bloat |
how much of each index is reclaimable, estimated from statistics with its accuracy stated — read the reason on a row with no estimate, and exact_measurement_command when you need certainty on one index |
get_pg_table_bloat |
how much space the vacuum lag above has cost, as an estimate with its own error stated |
get_pg_column_stats |
the planner's own per-column inputs — read coverage before acting on the ranking |
get_pg_buffer_usage |
what is resident in shared buffers, per relation (pg_buffercache) |
get_pg_extensions |
which monitoring-relevant extensions are installed, outdated, merely available, or absent — per database |
get_pg_write_stats |
checkpoints (timed vs REQUESTED), background writer, WAL volume across the window |
get_pg_server_config |
the settings snapshot, non-defaults first, pending_restart reported loudly |
get_pg_server_config_changes |
what changed and when, old and new value side by side |
get_pg_session_states |
who is holding a transaction open — and whether they actually pin the xmin horizon |
get_pg_wait_trend |
one wait event as a time series |
get_pg_query_duration_trend |
one statement's per-execution cost over time — the regression read |
Proof, and the trap: on a healthy target most of these are supposed to be boring. Do not read
"nothing alarming" as "not collecting" — check collection_log (step 6) for that. Distinguish:
- Rows, all classified
ok— collecting, and the target is healthy. Success. - No rows, collector ran with
rows_collected = 0— the target genuinely has none of that thing. No replication slots is the common one, and it is good news. - No rows, collector never ran — gated off (step 3), not yet due (step 7), or failing (step 10).
Five results that look like bugs and are not:
get_pg_io_statson Aurora reports write counters not tracked. Correct: Aurora backends do not write data files, the storage layer does, so those columns are NULL — which is why the tool reports trackedness instead of letting a NULL read as a zero.get_pg_replication_slotsempty on a reader is per-instance, not a cluster all-clear. Slots live on the writer. Same for autovacuum state, index usage and both bloat surfaces — all four are writer-only collectors.get_pg_table_bloatorget_pg_index_bloatreporting most of its rows with a suppressed estimate is almost always a permissions gap rather than a missing ANALYZE, and it is a step in this runbook thatGRANT pg_monitoralone does not satisfy. See the note below. Index bloat joined this list in #3234, when it stopped walking pages and started modelling frompg_stats.get_pg_session_statesreporting a session idle in transaction for an hour withpeak_horizon_ageof-1is not a contradiction and not a rounding artefact. It means the session pins nothing: a READ COMMITTED transaction releases its snapshot at the end of each statement, and one whose write matched no rows never got a transaction id to hold. Both were measured on a live PostgreSQL 16.15 instance. Terminating such a session reclaims not one dead row, which is exactly why the tool says so instead of letting the duration imply otherwise.get_pg_database_statsreportingstats_reset_countabove zero is the tool working, not a fault. The counters it reads are cumulative since the lastpg_stat_reset(), so a reset zeroes them; the window totals become LOWER BOUNDS and the tool says so rather than letting the reset surface as a negative rate or a spike. A crash restart shows the same way.
pg_monitor is enough for every collector here except the two self-hosted log readers — their grants are
step 1's log-reader subsection, and they fail loudly — and the THREE that read pg_stats: table bloat,
index bloat since #3234, and per-column statistics. The pg_stats way of failing is the one worth
knowing about, because it does not look like a failure. pg_stats is filtered by has_column_privilege(..., 'select'), and pg_monitor confers
no SELECT on user tables — so the monitoring role sees zero
rows in pg_stats and the estimator, fed nothing, returns confident large numbers. Measured against a
pg_monitor-only role on a live PostgreSQL 16 target: 88.59% reported for a table whose true bloat is
0.50%, 95.03% for one that is really 74.82%, 22.57% for one that is really 0.46%.
Darling does not publish those numbers — estimate_unavailable is set on every such row and the read
suppresses the figure rather than captioning it — but the result is a bloat surface that reports nothing
useful.
The pg_column_stats collector has the same dependency and fails more quietly still: it reads pg_stats
directly, so without this grant it returns zero rows and logs SUCCESS. Measured on a live Aurora
target carrying 361 tables over the collector's size floor, 107 of them analyzed: the collector ran, succeeded,
and collected nothing. An empty per-column statistics panel on a busy database means this grant is missing,
not that the planner has no statistics.
You no longer have to know that from here. Until #3154 that sentence was the only place the distinction
was written down, and nobody reads a runbook while looking at an empty grid: the panel and
get_pg_column_stats both listed the size floor and the privilege filter and selected neither. They now name
which one applies, and report it on a POPULATED result too — a login that can read four tables of twenty
produces a ranking that looks complete. get_pg_column_stats returns the arm as its own coverage field, and
every value it can take means something different to you:
coverage |
what it means | your move |
|---|---|---|
StatisticsNotVisible |
tables clear the floor, none of their statistics are readable by the monitoring login | this grant |
PartialVisibility |
some are readable, the rest are not — the ranking you are looking at is a subset | this grant; treat the result as partial until then |
BelowSizeFloor |
nothing on the server is large enough for this collector to read | nothing; there is nothing to collect |
FullyMeasured |
every table above the floor is represented | nothing |
CollectionFault |
tables clear the floor AND are readable, and nothing was stored anyway | ours — neither cause explains it, so raise it |
EvidenceStale |
rows exist, yet no table clears the floor in the evidence window | nothing on the grant; the rows describe tables that may have since shrunk, been truncated or been dropped |
Undetermined |
the two counts could not be established | nothing; see below |
The two counts behind every verdict come from pg_table_bloat_stats, which collects on writers only and
hourly, so on a read replica — where pg_column_stats does run — the answer is honestly Undetermined
rather than a guess. They are measured over a fixed 24-hour lookback while the row count is measured over
whatever window you asked for, which is why EvidenceStale exists and why the census states the span beside
each figure: do not draw a ratio between two numbers spanning different intervals.
The fix is one grant:
-- PostgreSQL 14 and newer
GRANT pg_read_all_data TO darling_monitor;
-- PostgreSQL 13: the role does not exist, so grant it per schema
GRANT USAGE ON SCHEMA public TO darling_monitor;
GRANT SELECT ON ALL TABLES IN SCHEMA public TO darling_monitor;
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT ON TABLES TO darling_monitor;Verified on the live rig: with pg_read_all_data added, a pg_monitor role's estimates became
byte-identical to a superuser's. This is a genuine widening of what the monitoring role can read, so it is
a decision to take deliberately rather than a step to run — every other collector works without it, and a
fleet that does not want it simply gets measured sizes and dead-tuple counts from this surface instead of
an estimate, and an empty per-column statistics panel.
pg_read_all_data is not the only route, and it is worth knowing the alternative exists before deciding,
because the two fail in different directions.
Datadog takes the other route. Its PostgreSQL setup has you create a datadog schema in each
monitored database holding a SECURITY DEFINER function, and grant the monitoring role EXECUTE on that
rather than SELECT on the data. A SECURITY DEFINER function runs with its OWNER's privileges, so a
low-privilege login can obtain one specific privileged answer without being able to read anything else.
Applied here, that would be a function returning pg_stats rows for a table — the monitoring role gets
column statistics and still cannot SELECT a single row of customer data.
GRANT pg_read_all_data |
helper function | |
|---|---|---|
| what it widens | SELECT on all data, cluster-wide | EXECUTE on one function |
| objects created in the customer's database | none | one schema + one function, per database |
| install | one statement, once | DDL in every database, repeated for every database added later |
| upgrade | nothing to upgrade | the function is versioned code that ships with the product |
| removal | REVOKE |
DROP, and something has to remember it is there |
| PostgreSQL 13 | role does not exist — explicit GRANT SELECT per schema |
works |
| who must run it | someone who can grant a role | someone who can create objects and own the definer |
The honest summary of the trade. The grant is one statement and no footprint, and it hands over more than the collector needs. The helper hands over exactly what the collector needs and puts product-owned code inside the customer's database forever — which is a support obligation, not just an install step: it has to be versioned, upgraded in place, and removable, and a database created after onboarding silently has no helper until something notices.
Which is why a monitoring vendor might reasonably choose either. A product that cannot ask for
pg_read_all_data — because its customers will not grant it, or because it must work on PostgreSQL 13
where the role does not exist — has the helper as its only route to the same data.
Darling ships neither today. The grant is documented above and the helper is not implemented. If a
fleet will not widen the role, the current behaviour is the honest one: pg_column_stats collects nothing,
the bloat estimate reports estimate_unavailable, and every other collector is unaffected.
Eight alert families fire on a PostgreSQL target, in two groups that are tuned differently.
The three Tier 0 outage predictors — PostgreSQL Wraparound Risk, PostgreSQL Vacuum Horizon Blocked and PostgreSQL Replication Slot Retention, reading pg_wraparound_stats, pg_xmin_horizon
and pg_replication_slots:
- Evaluated on the 30-second alert sweep, after the shared SQL Server sweep, gated on the probed engine.
- Read only data collected in the last 2 hours. A stale target alerts on nothing — which is what the separate collection-stopped self-alert is for.
- Delivered through the same deliverer, history and mute rules as every SQL Server alert.
- Thresholds derive from the target's own settings — wraparound grades against that cluster's
autovacuum_freeze_max_age, not a constant — and these three have no knob, by decision rather than by omission: each bar is a ratio of a setting the row itself carries, so there is no number a user would set differently, andpostgres-alerting-design-note.mdis the record of that decision. Do not widen it into "PostgreSQL alert thresholds are not configurable"; the next group's are.
The five that ride alongside carry the SAME alert names a SQL Server target raises — deliberately, so
a mute rule, a history filter or a dashboard built against Deadlocks Detected does not have to know
which engine a server runs. They are switched and tuned through the shared alerts settings, and the
same seed-once rule as step 2 applies: the alerts section of darling.json seeds
config.config_alert_settings when that table is empty, and after that the store is authoritative —
tune through get_alert_settings / update_alert_settings over MCP or the Viewer's alert settings, and
the running service picks the change up on its next sweep, no restart:
| Alert | Reads | Enabled by | Threshold |
|---|---|---|---|
Deadlocks Detected |
pg_deadlocks |
deadlockEnabled — shared with SQL Server |
pgDeadlockCountThreshold (deadlocks.pg_count_threshold on the MCP surface): distinct deadlocks inside the rolling one-hour window; default 1, floor 1 (#3444, V122) A fire pages when the server had 5 or more distinct reports in the last hour or when a statement pair is new (not seen in the last 30 days). Otherwise the fire is held back from paging and recorded as one digest row for the daily digest. The pair is the sorted statements of the report's graph, counted in the 30-day signature store under a pg: key. A report with no statements falls back to its lock shape. The alert text lists the statement pairs with their report counts. |
Blocking Detected |
pg_blocking |
blockingEnabled — shared |
pgBlockingCountThreshold (blocking.pg_count_threshold): distinct ROOT blockers inside the rolling one-hour window; default 3 since #7083 (V235 moves a store still at 1), floor 1 (#3444, V122). Pages when that many roots exist and the worst victim wait is 15 seconds or more, or when one root's worst victim waited blockingWaitSecondsThreshold (60 s by default; 0 turns that route off). A count at the bar with every wait under 15 seconds writes one held digest row for that burst, and another only when a newer capture arrives after the cooldown |
Long-Running Query |
pg_session_states |
longRunningQueryEnabled — shared |
longRunningQueryThresholdMinutes — the shared value, judged against the most recent capture |
Poison Wait |
pg_wait_stats (Aurora only — the IPC pair BtreePage / BufferIo) |
poisonWaitEnabled — shared |
the shared accumulation bars: one backend continuously stuck across the ten-minute window warns, ten is critical; no per-engine knob, by the same reasoning as the trio. On PostgreSQL a warning also needs the previous ten-minute window at or over the same bar, 20 minutes in all (#7082). Critical does not A warning that has not held for two windows writes one digest row for the daily digest and pages nothing. |
High CPU |
pg_cpu_utilization (Aurora/RDS only) |
cpuEnabled — shared |
cpuThresholdPercent — the shared value, sustained across the same consecutive-sample gate SQL Server uses. A sample counts only when both capacity use (percent of the ACU ceiling) and instance CPU are at or over it (#7082) Capacity at the ceiling with instance CPU under the bar pages nothing and writes at most one digest row per cooldown per server (reason: capacity at ceiling, instance CPU N%). A sample with no capacity figure counts on instance CPU alone only when the target is known to be provisioned: its oldest stored sample in the last 7 days is at least an hour old and none of them carries a capacity figure, which is what a provisioned (non-serverless) instance gives. The alert text then says instance CPU only. On a serverless instance a sample with no capacity figure is left out of the gate (neither a breach nor a clear). |
The two PostgreSQL count knobs are deliberately not their SQL Server neighbours
(deadlockCountThreshold, blockingCountThreshold), and a store that upgrades without touching them
fires exactly where it did. The SQL Server figures are calibrated against surfaces a PostgreSQL target
does not have — a deadlock health band, engine-recorded blocked-process reports — while the PostgreSQL
blocking count is distinct roots in a periodic SAMPLE of pg_stat_activity (step 7: pg_blocking is a
sample, not an event log). Moving one does not move the other; the enabled switch in each pair governs
both engines; both PostgreSQL keys are inert on a store with no PostgreSQL targets.
Where they land. Every alert above goes to every channel configured in the Viewer's Settings →
Notifications by default. Since V131 (#3598) that is the parent channel set, and a sparse routes table
layered over it (Settings → Notifications → Manage Notification Routes…, read back by
get_notification_routes) can send an alert family somewhere else: all eight PostgreSQL alerts are
performance — the pages family — so a route on performance moves them together with their SQL Server
namesakes, an exact-metric route on PostgreSQL Wraparound Risk moves that one alone, and an empty
channel on a route inherits the parent's. The pre-routes workaround — a generic-webhook template branching
on {{metric}} in an external router — still works. The Darling README's
notification routes
section has the taxonomy and the resolution order.
Proof on a healthy target is silence, which is unfalsifiable, so verify the path rather than the
outcome: confirm the collectors backing it have fresh rows (step 6 — pg_wraparound_stats,
pg_xmin_horizon, pg_replication_slots, and pg_deadlocks / pg_blocking / pg_session_states for
the second group), that get_alert_settings reports the two pg_count_threshold values you expect, and
that a SQL Server alert has delivered through the same deliverer at some point. Nothing here fires on a
healthy cluster, and that is the design: a predictor that cries wolf gets muted, and a muted outage
predictor is worse than none.
A collector that fails is classified rather than logged as ERROR forever. The store's non-fatal bucket
is named PERMISSIONS, and things landing in it are not always missing grants — the error_message
says which kind it is. A missing extension a collector declares gets its own EXTENSION_MISSING status
(#3240) rather than that bucket, so an uninstalled optional module never reads as a grant problem.
status |
error_message says |
Actually means | Fix |
|---|---|---|---|
PERMISSIONS |
a missing grant | it is one | GRANT pg_monitor (step 1) |
PERMISSIONS |
permission denied for function pg_read_file, names the grant pair |
the self-hosted log readers (pg_deadlocks, pg_plan_capture) lack the log-reader grants |
step 1's self-hosted subsection — run it in the database Darling connects to; issued elsewhere on the cluster it changes nothing |
EXTENSION_MISSING |
names the extension, "NOT a missing grant" | that extension never created where the collector connects | CREATE EXTENSION per the message (step 1's optional half), or leave it uninstalled and accept the gap |
PERMISSIONS |
"NOT a missing grant", not implemented | reading something this engine lacks | nothing — expected off Aurora |
PERMISSIONS |
"NOT a missing grant", feature disabled | switched off in the parameter group | enable it, or accept the gap |
PERMISSIONS |
is not authorized to perform: rds:Describe.../rds:Download..., names an IAM role ARN |
the monitoring host's IAM role (or the server's own AWS role, once assumed) lacks the AWS-level grant plan capture/deadlocks need on Aurora/RDS | attach the IAM policy in step 1's IAM subsection to the role the message names. A DB-side grant cannot fix this, because it is a different identity entirely |
PERMISSIONS |
The AWS role <arn> on this server is not in allowedAwsRoles in darling.json, so it was not used. Nothing was read this cycle. Add the role or its account id to the list and restart the service, or clear the role on this server. |
the server names a role the list does not allow, so Darling never assumed it | list the role ARN or its account id in allowedAwsRoles and restart the service, or remove the role from the server |
PERMISSIONS |
Role <arn> is in AWS partition aws-cn, but this target's region <region> is in partition aws. A role can be assumed only inside its own partition. Nothing was read this cycle. Set a role from the target's partition on this server, or clear the role, or correct the server's host name if it names the wrong region. |
the role and the target's region are in different AWS partitions | set a role from the region's partition on the server, clear the role, or correct the host name if it names the wrong region |
PERMISSIONS |
AWS refused to let the monitoring host assume role <arn> (with the external ID set on this server). Nothing was read this cycle. One of these is wrong: the role does not exist, its trust policy does not trust the host's AWS identity, the external ID does not match the trust policy's sts:ExternalId condition, or the host's identity has no sts:AssumeRole permission on the role. AWS gives the same answer for all four., or the same with (without an external ID) |
STS denied the assume-role call | check the four named things in step 1's cross-account subsection. The message never shows the external ID, only whether one is set |
PERMISSIONS |
AWS STS is not active in region <region> for the monitoring host's AWS account, so role <arn> could not be assumed. Nothing was read this cycle. Enable that region for the AWS account that owns the role and for the monitoring host's account (AWS account settings, Regions), or set a role in a region that is enabled, or correct the server's host name if it names the wrong region. |
STS is switched off for that region in the host's account or the role's account | enable the region for both accounts (AWS account settings, Regions), set a role in an enabled region, or correct the host name |
ERROR |
AWS rejected the monitoring host's own credentials while it assumed role <arn>, then (<AWS error code>) when AWS gave one, then . The credentials the service runs with are expired or wrong, or the host's clock is off. Nothing was read this cycle. Fix the host's credentials or clock; a new attempt is made within a minute. |
the host's own credentials, not the role, are the problem | fix the host's credentials or clock |
ERROR |
The monitoring host has no AWS credentials to assume role <arn> with. Nothing was read this cycle. Give the service an AWS identity to sign in with (an instance profile, a container task role, environment variables or a shared credentials profile it can read), then restart the service., then (SDK exception <type>, AWS error code <code>) when the SDK gave them |
the host has no AWS identity to assume the role from | give the service an AWS identity (step 1's IAM subsection). The message holds the SDK exception type and AWS error code only; the SDK's own text is in the service log |
ERROR |
Assuming role <arn>: AWS STS could not be reached or answered with an error. Nothing was read this cycle. A new attempt is made within a minute., then (SDK exception <type>, AWS error code <code>) when the SDK gave them; or Assuming role <arn>: AWS STS did not answer within <n> seconds. |
STS, or the host's credential lookup, did not answer in time, or STS answered with a fault | a new attempt is made within a minute. The SDK's own text is in the service log |
ERROR |
a statement timeout | the query was too slow once | usually transient; deliberately does not drop the connection, so a slow query cannot cause a reconnect storm |
ERROR |
anything else | unclassified | read error_message; this is the bucket that wants a bug report |
YIELDED |
lock contention | the collector stepped aside | none today — no PostgreSQL collector opts into the lock-timeout yield |
Connection-level failures (the 08 class, 57P0x) force a reconnect and a re-probe, so a failover
re-probes the target — which is how a promoted reader stops being gated as a standby.
Symptoms with no error anywhere, the ones that cost the most time:
| Symptom | Cause |
|---|---|
The connect line says SQL Server, or the error mentions SqlException / a TDS handshake against 5432 |
the target lost its engine on the way through the registry. Requires store schema v70+: SELECT name, engine, port FROM config.config_monitored_servers; — if engine is not a column, this build predates the fix and no darling.json edit will help |
| Autovacuum health reports everything fine on a cluster you know is behind | you are reading a reader. It reports all zeros, not an error. Measured: writer 13,654,458 dead tuples, reader 0, same cluster and tables |
| Added a target to darling.json and nothing happened | the store was already seeded; use add_servers (step 2) |
pg_wait_stats empty, everything else fine |
not Aurora. Core PostgreSQL has no cumulative wait counters at all, and the gap message says so — pg_wait_sampling answers the same question from its extension, or from the service sampler when the extension is absent |
get_pg_wait_sampling says instrument: service_sampled |
expected on a stock target without pg_wait_sampling — the floor tier (#3604). Every share is of one-second polls in a 30-second window per five-minute cycle; install the extension and let the service reconnect to reach extension_sampled |
get_pg_top_queries empty on self-hosted |
pg_stat_statements not installed (step 1's optional half) — since #2625 the collector reads the vanilla view anywhere the extension exists |
Only some databases in pg_autovacuum_stats |
by design: datallowconn and non-template only, minus rdsadmin on a managed instance (it rejects every customer principal) and your excludedDatabases |
| Store stopped compressing after adding targets | background workers (step 4). Silent — check the postmaster log for "out of background workers" |
remove_server through MCP, or the Viewer's Manage Servers dialog — not by deleting the darling.json
entry, which the registry ignores (step 2). Collection stops within a sweep; the collected history stays
under its server_id and ages out on the normal retention horizons. Nothing was ever created on the
monitored instance, so there is nothing to clean up there — dropping darling_monitor is optional and
unrelated to Darling's state. If you enabled auto_explain for plan capture, that parameter is yours to
keep or revert; Darling only ever read the log it produced.
This list used to be longer, and everything struck from it is covered in the steps above — each strike names the step that now carries it, because a runbook that says "not built" about a shipped feature sends a new operator away from the thing that would have answered their first week's questions (#3608):
- Plan capture shipped:
pg_plan_capture— #2566 self-hosted via the server log, #2538/#2692 on Aurora/RDS via the log API — withpg_plan_capture_readinessnaming any missing precondition. The grants are step 1's log-reader and IAM subsections, the cadence trap is in step 7, andget_pg_plans/get_pg_plan_capture_readinessare in step 8's table. Its siblingget_pg_logging_audit(#3607) applies the same facet-and-remedy shape to the rest of the logging surface — lock waits, temp files, autovacuum runs, checkpoints, connection churn, slow statements — so the two together are the onboarding answer to "is this target telling us everything it could"; read both before concluding anything from an empty target-side log. - Blocking chains shipped:
pg_blocking(step 7 — a sample, not an event log),get_pg_blocking(step 8) and theBlocking Detectedalert with its own count knob (step 9). The blocking design note records what building it changed. - Configurable alert thresholds shipped for the alerts that have a number worth moving:
pgDeadlockCountThresholdandpgBlockingCountThreshold(#3444, V122), onget_alert_settings/update_alert_settings, the Viewer anddarling.json. Step 9 has the table. The three Tier 0 predictors still carry no knob, and that is a decision the alerting design note defends rather than a gap — their bars are ratios of the target's own settings. - The Viewer's PostgreSQL surfaces shipped (#2530 — a PostgreSQL target gets seven inner tabs in place of the nineteen SQL Server ones).
What genuinely remains, re-checked against dev at the time of this revision:
- Scheduled analysis findings. The analysis pass now runs for a PostgreSQL target: the service
routes by the registry's
engine_kind, measures the 24-hour data-span gate onpg_database_statsand scores the PostgreSQL-target vocabulary (PG_*/CONFIG_PG_*/ANOMALY_PG_*). The #3542 v1 detector families have landed, so a pass over a target with a day of history can return findings;analysis_stateclears on the first real pass (no more "does not apply" tombstone). A target whose registry row has no engine stamp yet says so in its insufficient-data message and is measured correctly from its next connect. - Knobs on the three Tier 0 predictors. Not a gap in the sense the rest of this list is — see above
and step 9 — but listed so nobody goes looking for a
pgWraparound...setting that does not exist. - The
pg_statshelper-function route. Step 8 documentspg_read_all_dataand theSECURITY DEFINERalternative; only the grant is implemented. A fleet that will not widen the role gets measured sizes and suppressed estimates, exactly as step 8 describes. - What a per-server AWS role covers, and what it does not.
awsRoleArnchanges which AWS identity makes the RDS and Performance Insights calls for that server (step 1's cross-account subsection). It does not change the database login, which stays the SQL login from step 2, and it does not apply to SQL Server targets. - Charts on the PostgreSQL tabs. Both UIs render tables over the window — no correlated timeline, and no drill-down from a blocking root to the sessions behind it.