Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions CHANGES.md
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,10 @@ To be released.
and other containers remain rejected. This keeps portable actors that
publish container terms such as Mastodon's `attributionDomains`
verifiable. [[#1234], [#1353]]
- Fixed `FedifySpanExporter.forceFlush()` and `shutdown()` returning before
pending trace writes finish. Added a `waitUntil` exporter option to keep
serverless invocations alive for each export attempt; use it with
`SimpleSpanProcessor` on Cloudflare Workers. [[#1382], [#1415]]
- Fixed log entries emitted during an individual outbound activity delivery
or inbound activity processing carrying the enclosing HTTP request's or
queue worker's `traceId`/`spanId` instead of the delivery or processing
Expand All @@ -110,7 +114,9 @@ To be released.
[#1328]: https://github.com/fedify-dev/fedify/pull/1328
[#1353]: https://github.com/fedify-dev/fedify/pull/1353
[#1359]: https://github.com/fedify-dev/fedify/pull/1359
[#1382]: https://github.com/fedify-dev/fedify/issues/1382
[#1396]: https://github.com/fedify-dev/fedify/pull/1396
[#1415]: https://github.com/fedify-dev/fedify/pull/1415

### @fedify/backfill

Expand Down
9 changes: 9 additions & 0 deletions changes.d/fedify/span-exporter-flush.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
---
links:
'#1382': https://github.com/fedify-dev/fedify/issues/1382
'#1415': https://github.com/fedify-dev/fedify/pull/1415
---
- Fixed `FedifySpanExporter.forceFlush()` and `shutdown()` returning before
pending trace writes finish. Added a `waitUntil` exporter option to keep
serverless invocations alive for each export attempt; use it with
`SimpleSpanProcessor` on Cloudflare Workers. [[#1382], [#1415]]
14 changes: 14 additions & 0 deletions claude-plugin/skills/fedify/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -477,6 +477,20 @@ its internals. For trace persistence, `@fedify/fedify/otel` exports
`FedifySpanExporter`, which writes traces to a `KvStore` so the
*@fedify/debugger* dashboard can render them.

On Cloudflare Workers, use `SimpleSpanProcessor` and configure the exporter's
`waitUntil` option (since 2.5.0) with a request-aware callback, or create the
exporter per request with `ctx.waitUntil.bind(ctx)`. A shared exporter must
not retain one request's execution context. In a `finally` block around
`await federation.fetch()`, pass `tracerProvider.forceFlush()` to
`ctx.waitUntil()` so the flush is scheduled on both success and failure.
With `SimpleSpanProcessor`, this also covers pending asynchronous resource
attributes. In `@opentelemetry/sdk-trace-base` 2.7.1, a batch already waiting
for resource attributes can be missed by a later provider flush, so do not rely
on that flush with `BatchSpanProcessor`. Batch timers alone do not keep an
invocation alive. The exporter's `forceFlush()` and `shutdown()` wait for
export attempts already started; storage failures are logged and reported
through the export callback.

> [!CAUTION]
> Initialize the OpenTelemetry SDK *before* importing Fedify. Later
> registration leaves earlier spans untraced.
Expand Down
96 changes: 96 additions & 0 deletions docs/manual/opentelemetry.md
Original file line number Diff line number Diff line change
Expand Up @@ -1281,6 +1281,95 @@ const federation = createFederation<void>({

:::

### Keeping serverless exports alive

*The `waitUntil` option is available since Fedify 2.5.0.*

`FedifySpanExporter` writes asynchronously. On Cloudflare Workers, pass a
`waitUntil` callback so an invocation can stay alive while its trace records
are written. Use `SimpleSpanProcessor` to start exporting when each span ends.
The exporter calls the callback synchronously from `export()` with a promise
that fulfills after that export attempt and its result callback finish.
Storage failures are logged and reported to the processor; the lifetime promise
does not reject. Exceptions thrown by the callback are logged and ignored.

For a request-scoped exporter, pass the current execution context's bound
`ctx.waitUntil()` method:

~~~~ typescript twoslash
import type { KvStore } from "@fedify/fedify";
declare const kv: KvStore;
declare const ctx: { waitUntil(promise: Promise<void>): void };
// ---cut-before---
import { FedifySpanExporter } from "@fedify/fedify/otel";
import {
BasicTracerProvider,
SimpleSpanProcessor,
} from "@opentelemetry/sdk-trace-base";

// Create these inside the request handler and pass tracerProvider to Fedify.
const exporter = new FedifySpanExporter(kv, {
waitUntil: ctx.waitUntil.bind(ctx),
});
const tracerProvider = new BasicTracerProvider({
spanProcessors: [new SimpleSpanProcessor(exporter)],
});
~~~~

A shared exporter needs a callback that finds the current invocation, such as
`waitUntil()` imported from `cloudflare:workers`. Do not keep one request's
bound `ctx.waitUntil()` in a shared exporter. The callback must not log
through a LogTape sink that uses the same callback, since that can recurse.

The callback only protects work once the processor calls the exporter. With
pending asynchronous resource attributes, even `SimpleSpanProcessor` can defer
that call. `BatchSpanProcessor` buffers spans, and its timer cannot be relied
on to run after the response. To cover processor-side pending work, pass
`tracerProvider.forceFlush()` to `ctx.waitUntil()` in a `finally` block around
`federation.fetch()`. This schedules the flush even if request handling fails:

~~~~ typescript twoslash
import type { Federation } from "@fedify/fedify";
import type { BasicTracerProvider } from "@opentelemetry/sdk-trace-base";
declare const federation: Federation<void>;
declare const tracerProvider: BasicTracerProvider;
declare const request: Request;
declare const ctx: { waitUntil(promise: Promise<void>): void };
// ---cut-before---
async function handleRequest(): Promise<Response> {
try {
return await federation.fetch(request, { contextData: undefined });
} finally {
ctx.waitUntil(tracerProvider.forceFlush());
}
}
~~~~

Use `SimpleSpanProcessor` together with the `finally` flush above. In
`@opentelemetry/sdk-trace-base` 2.7.1,
`BatchSpanProcessor` can remove a batch from its buffer before waiting for
asynchronous resource attributes. A later provider flush does not wait for
that in-progress batch, and the exporter's `waitUntil` callback has not yet
been called. An explicit flush therefore cannot reliably protect those
exports from invocation termination.

`exporter.forceFlush()` only waits for exports already handed to the
exporter, plus any that start while it waits. It completes when the exporter
becomes idle, so continuous exports can keep it pending.

On Node.js and other long-lived runtimes, stop accepting requests, await
`tracerProvider.forceFlush()`, then await `tracerProvider.shutdown()` during
graceful shutdown. The exporter's `shutdown()` also waits for pending export
attempts. The exporter's `forceFlush()` and `shutdown()` do not reject for a
failed storage attempt.

Lifetime extension is subject to the platform's limits. For HTTP-triggered
Workers,
[`waitUntil()` allows up to 30 seconds after the invocation ends][workers-waituntil],
shared across the request's calls. It does not guarantee durable delivery.

[workers-waituntil]: https://developers.cloudflare.com/workers/runtime-apis/context/#waituntil

### Querying stored traces

The `FedifySpanExporter` provides methods to query stored trace data:
Expand Down Expand Up @@ -1335,6 +1424,13 @@ Each `TraceActivityRecord` contains:

The `FedifySpanExporter` constructor accepts the following options:

`waitUntil`
: A callback invoked synchronously for each export with a never-rejecting
completion promise. Use it to extend the current serverless invocation's
lifetime; see
[Keeping serverless exports alive](#keeping-serverless-exports-alive).
Available since Fedify 2.5.0.

`ttl`
: The time-to-live for stored trace data. If not specified, data will be
stored indefinitely (or until manually deleted). This is useful for
Expand Down
Loading
Loading