Skip to content

[api][runtime][python] Add minimal Event lineage#923

Open
rosemarYuan wants to merge 3 commits into
apache:mainfrom
rosemarYuan:feature/eventlineage
Open

[api][runtime][python] Add minimal Event lineage#923
rosemarYuan wants to merge 3 commits into
apache:mainfrom
rosemarYuan:feature/eventlineage

Conversation

@rosemarYuan

@rosemarYuan rosemarYuan commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Linked issue: Closes #841

Related discussions:

Purpose of change

The Event Log currently records individual Events but does not preserve enough information to reconstruct the direct causal path between an Event, the Action it triggered, and the Events emitted by that Action.

This PR implements the minimal Event-lineage phase discussed in #710.

It adds two framework-managed fields to each Event emitted by an Action:

  • upstreamEventId: the ID of the Event consumed by the emitting Action;
  • upstreamActionName: the name of the Action that emitted the current Event.

Together with the existing per-occurrence Event ID, these fields are sufficient to reconstruct an InputEvent-rooted Event-Action-Event causal tree from the available Event Log records.

This PR deliberately does not implement the complete execution-tracing model proposed in #900. Run identity, concrete Action invocation identity, execution hierarchy, lifecycle, retry/replay semantics, ordering, and failure details remain part of that separate design.

A real Runtime execution was verified with the following causal path:

InputEvent(input=1)
  -> action1
  -> MiddleEvent(num=2)
  -> action2
  -> OutputEvent(output=4)

The File Event Log contained three physical Event records:

Event upstreamEventId upstreamActionName
_input_event absent absent
MiddleEvent InputEvent ID action1
_output_event MiddleEvent ID action2

Running tools/reconstruct_trace_tree.py against that Runtime log produced:

Trace Tree 1
  _input_event
    [Action: action1]
      MiddleEvent
        [Action: action2]
          _output_event

The JSON reconstruction contained one root, all three Event nodes, the two expected virtual Action edges, and no warnings.

Tests

Add e2e test to review Trace Tree python/flink_agents/e2e_tests/e2e_tests_integration/python_event_logging_test.py

API

This PR changes the default Python Event ID generation strategy from content-derived IDs to UUIDv4.

Documentation

  • doc-needed
  • doc-not-needed
  • doc-included

@github-actions github-actions Bot added doc-label-missing The Bot applies this label either because none or multiple labels were provided. fixVersion/0.4.0 priority/major Default priority of the PR or issue. doc-included Your PR already contains the necessary documentation updates. and removed doc-label-missing The Bot applies this label either because none or multiple labels were provided. labels Jul 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

doc-included Your PR already contains the necessary documentation updates. fixVersion/0.4.0 priority/major Default priority of the PR or issue.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Add lineage fields to the event log for per-run trace reconstruction

1 participant