Skip to content

feat(ci): add e2e duration charts#2361

Open
universal-itengineer wants to merge 20 commits into
mainfrom
chore/ci/e2e-report-add-graphs
Open

feat(ci): add e2e duration charts#2361
universal-itengineer wants to merge 20 commits into
mainfrom
chore/ci/e2e-report-add-graphs

Conversation

@universal-itengineer
Copy link
Copy Markdown
Member

@universal-itengineer universal-itengineer commented May 18, 2026 •
edited
Loading

Description

Move E2E report chart generation from the previous JavaScript/Chart.js implementation to a Python renderer based on matplotlib.

The E2E report pipeline now renders chart PNG files in Python before the Loop messenger report is built. Python writes a manifest that maps each cluster to generated chart files, and the JavaScript messenger path only reads that manifest and attaches existing PNG files to Loop thread replies.

Main changes:

  • Generate the Loop feature-duration-status chart in Python with a stacked per-Describe duration/status layout, 60-second x-axis ticks, value labels, and compact chart margins.
  • Generate top-N slowest-specs charts in Python for local triage and CI workflow artifacts.
  • Remove the JavaScript Chart.js renderer, chart configs, builders, snapshots, and chartjs-node-canvas dependency.
  • Keep Loop chart attachments best-effort: failed file uploads do not drop the report thread text.
  • Keep local tasks report:render:slowest and report:render:top-slowest writing PNG output under tmp/charts/.
  • Update the local tmp/test-ci/report/run.sh runner to create a Python venv, install chart dependencies there, generate messenger chart manifests, and attach generated files locally.
  • Add Python unit tests for aggregation, sorting, manifest generation, artifact names, no-match errors, and chart output.

Why do we need it, and what problem does it solve?

E2E failures in the matrix workflow were hard to triage from the messenger summary alone. The text report showed pass/fail counts and failed tests, but did not show where the suite spent time or which feature areas were close to timeout/flakiness.

This change keeps the Loop thread focused while adding visual duration context. The messenger report shows the high-level feature duration/status chart, and deeper slowest-specs drill-downs remain available as local output and workflow artifacts.

Moving chart generation to Python also removes the native canvas dependency from the JavaScript report path and makes the messenger JS code responsible only for message assembly and file attachment.

What is the expected result?

  1. Run the E2E matrix workflow and open the report-to-channel job.
  2. Verify the workflow generates tmp/messenger-charts/manifest.json and feature-duration-status PNG files before the Loop report is sent.
  3. Verify the Loop report thread contains the usual summary plus feature-duration-status PNG attachments for clusters with report data.
  4. Verify the workflow uploads the e2e-report-slowest-by-describe artifact with top-5 slowest Describe charts per cluster from tmp/charts/.
  5. For local debugging, run: 
    bash tmp/test-ci/report/run.sh --json-dir tmp/test-ci/report
task report:render:top-slowest REPORTS_DIR=tmp/test-ci/report/out TOP_N=5
    and verify generated report markdown, chart manifests, and PNG files are produced.

Checklist

  • The code is covered by unit tests.
  • e2e tests passed.
  • Documentation updated according to the changes.
  • Changes were tested in the Kubernetes cluster manually.

Changelog entries

section: ci
type: feature
summary: Add Python-generated E2E duration charts and slowest-specs workflow artifacts.
impact_level: low

@universal-itengineer universal-itengineer force-pushed the chore/ci/e2e-report-add-graphs branch from 4576709 to 7f07b36 Compare May 18, 2026 14:58
@universal-itengineer
feat(ci): add e2e duration charts
f88ce29 
Add per-spec timing propagation, Chart.js report rendering, Loop file uploads, and local report artifacts so E2E failures can include duration context.

Signed-off-by: Nikita Korolev <[email protected]>
@universal-itengineer
fix(ci): log chart render errors and tighten e2e report install
0d7b308 
- Propagate core.warning from messenger-report.js into buildThreadMessages
  so duration-chart rendering failures are visible in CI logs instead of
  being silently masked by the "Charts unavailable." placeholder.
- Cache ChartJSNodeCanvas as a module-level singleton; chart rendering
  is reused across clusters instead of allocating a fresh canvas each time.
- Sanitize cluster name when composing chart file names so unexpected
  characters cannot leak into uploaded Loop attachment names.
- Switch e2e-matrix workflow to npm ci with cache and document why
  node-version stays pinned to 20 (actions/github-script@v7 ABI).
- Add jest case asserting core.warning is emitted when chart rendering
  rejects.

Signed-off-by: Nikita Korolev <[email protected]>
@universal-itengineer
fix(ci, observability): keep Loop thread reply when chart upload fails
1bf9b0b 
If the Loop bot token lacks the upload_file permission, /api/v4/files
returns HTTP 403 and the per-file uploadFileToLoop call throws. The
exception used to propagate out of makeThreadedReportInLoop, aborting
the whole thread and dropping the failed-tests reply that we delivered
before charts were introduced.

Wrap the per-reply upload in try/catch, log a warning explaining that
attachments are skipped, and still post the reply text. The behaviour
is now strictly additive to the pre-charts implementation: chart
attachments are best-effort, the failed-tests table is guaranteed.

Add a jest case that mocks /api/v4/files with a 403 and asserts the
thread reply is posted without file_ids.

Signed-off-by: Nikita Korolev <[email protected]>
@universal-itengineer
refactor(ci, observability): align messenger report with main + chart…
05367f1 
…s only

Per user feedback after a local run review: the main message has to
match the pre-charts main-branch layout, and the per-cluster thread
reply should attach only the chart PNGs without the extra "### Test
durations / Attached charts: ..." caption block.

- markdown.js: drop renderTopSlowestSection (and the
  formatDuration/renderDurationBar helpers it owned) and the call
  from buildMainMessage so "Top slowest tests" no longer appears in
  the main message.
- markdown.js: shrink renderChartCaption to "Charts unavailable."
  only on chart-render error and an empty string otherwise, so a
  cluster with successful charts gets a clean
  **[cluster](url)** header plus file attachments.
- messenger-report.test.js: refresh the chart and Loop scenarios to
  assert the new minimal thread layout and the absence of the
  Top-3 section in the main message.

Failed-tests thread reply behavior is preserved: clusters with
failures still get the "### Failed tests" table, just without the
chart caption block underneath it.

Signed-off-by: Nikita Korolev <[email protected]>
@universal-itengineer
fix adding graph for report
63d3cc7 
Signed-off-by: Nikita Korolev <[email protected]>
@universal-itengineer universal-itengineer force-pushed the chore/ci/e2e-report-add-graphs branch from 6aac4a5 to 63d3cc7 Compare May 19, 2026 09:16
@universal-itengineer
feat(ci, observability): replace e2e cluster chart set with five new …
bcd4e78 
…visualizations

Replaces the four ad-hoc cluster charts (top-slowest, duration-histogram,
feature-totals, status-stacked) with a curated set of five:

1. status-doughnut         - passed/failed/errors/skipped counts at a glance
2. pareto-slowest          - top-N slowest specs + cumulative % of suite time
3. pass-rate-per-feature   - 100% stacked horizontal bar, most-broken on top
4. quantiles-per-feature   - p50/p90/max duration per feature
5. feature-totals          - total duration per feature

Also simplifies chart-config.js: a single pass aggregator feeds every chart,
shared status and palette constants live at the top of the module, and only
buildClusterChartConfigs(specTimings) is exported. The renderer now drives
chart names from the builder list instead of hardcoding them.

Signed-off-by: Nikita Korolev <[email protected]>
@universal-itengineer
refactor(ci, observability): focus e2e charts on failures and durations
1b48856 
Reworks the experimental five-chart set into four triage-oriented charts:
feature duration by status, slowest specs, duration buckets, and failed plus
slow specs. The previous doughnut, quantile chart, and pareto percentage line
are removed because they added noise without improving failure diagnosis.

Adds local value labels to the chart configs so generated PNGs show concrete
seconds/counts without extra dependencies.

Signed-off-by: Nikita Korolev <[email protected]>
@universal-itengineer
refactor(ci, observability): remove duplicate e2e failed-slow chart
d0ac33d 
Drops the failed-and-slow-specs chart because it overlaps with slowest-specs
when a run has few failures. The messenger report now renders three charts per
cluster: feature duration by status, slowest specs, and duration buckets.

Signed-off-by: Nikita Korolev <[email protected]>
@universal-itengineer
refactor(ci, observability): use duration gradient for slowest specs …
ffbb209 
…chart

Changes slowest-specs from status-filled bars to a duration bucket gradient:
cyan for fast specs, blue for medium specs, and purple for slow specs. Failed
and errored specs remain distinct via red/amber borders and value-label suffixes.

Signed-off-by: Nikita Korolev <[email protected]>
@universal-itengineer
refactor(ci, observability): clarify e2e chart titles and layout
c2a3289 
Aligns chart titles with Ginkgo hierarchy terms, adds a drill-down legend and
minute-based duration ticks to slowest-specs, and trims the chart canvas height.
Duration bucket bars are made thinner so the three bucket rows read less heavy.

Signed-off-by: Nikita Korolev <[email protected]>
@universal-itengineer
refactor(ci, observability): widen slowest specs chart output
1d6ddc8 
Adds per-chart canvas sizing and renders slowest-specs at 1920x720 so long
It/Entry drill-down labels fit without clipping. Other E2E report charts keep
the default 1280x640 size.

Signed-off-by: Nikita Korolev <[email protected]>
@universal-itengineer
refactor(ci, observability): add margin to slowest specs chart
d9dc6f9 
Widens the slowest-specs output from 1920px to 2048px so long drill-down labels
and value labels have a little more space from the image edge.

Signed-off-by: Nikita Korolev <[email protected]>
@universal-itengineer
fix(ci, observability): keep small duration bucket counts visible
0ac6e2a 
Draws count labels for tiny stacked duration-bucket segments as offset callouts
instead of hiding them or placing overlapping text on the white background.

Signed-off-by: Nikita Korolev <[email protected]>
@universal-itengineer
refactor(ci, observability): separate slowest e2e charts from messenger
8d61c98 
Keep the Loop report focused on feature duration status while moving slowest-specs into local and CI artifact renderers for deeper triage.

Signed-off-by: Nikita Korolev <[email protected]>
@universal-itengineer
fix(ci, observability): write slowest charts to tmp charts
7a20dd1 
Use tmp/charts as the local and CI output directory for generated slowest-specs PNG artifacts.

Signed-off-by: Nikita Korolev <[email protected]>
@universal-itengineer universal-itengineer force-pushed the chore/ci/e2e-report-add-graphs branch from 9935f11 to 7a20dd1 Compare May 21, 2026 17:35
@universal-itengineer universal-itengineer added this to the v1.9.0 milestone May 21, 2026
@universal-itengineer universal-itengineer marked this pull request as ready for review May 21, 2026 18:33
@universal-itengineer universal-itengineer force-pushed the chore/ci/e2e-report-add-graphs branch from fd73274 to d8f3f5a Compare May 22, 2026 14:10
diafour
diafour reviewed May 22, 2026
View reviewed changes
Comment thread .github/workflows/e2e-matrix.yml
- uses: actions/checkout@v4

- name: Download E2E report artifacts
# v8 exists upstream and keeps report artifact downloads on the current action major.
Copy link
Copy Markdown
Member

@diafour diafour May 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it is safe to remove this comment. It says nothing specific for actions/download-artifact.

@universal-itengineer
fix(ci, observability): apply e2e chart review fixes
3b9a115 
Align Python chart rendering with the JS report contract, make chart output directories explicit, harden top chart rendering, and expand coverage for parser and CLI behavior.

Signed-off-by: Nikita Korolev <[email protected]>
@universal-itengineer universal-itengineer force-pushed the chore/ci/e2e-report-add-graphs branch 2 times, most recently from a8a71dc to fa2486a Compare May 22, 2026 16:42
@universal-itengineer
fix formatting
25ac64c 
Signed-off-by: Nikita Korolev <[email protected]>
@universal-itengineer universal-itengineer force-pushed the chore/ci/e2e-report-add-graphs branch from fa2486a to 25ac64c Compare May 22, 2026 16:45
@universal-itengineer
fix formatting 2
f0ac3a1 
Signed-off-by: Nikita Korolev <[email protected]>
@universal-itengineer
fix formatting 3
9d11bf8 
Signed-off-by: Nikita Korolev <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@diafour diafour diafour left review comments

@nevermarine nevermarine Awaiting requested review from nevermarine nevermarine is a code owner

@Isteb4k Isteb4k Awaiting requested review from Isteb4k Isteb4k is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

@universal-itengineer universal-itengineer

Labels

None yet

Projects

None yet

Milestone

v1.9.0

Development

Successfully merging this pull request may close these issues.

2 participants

@universal-itengineer @diafour