Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -288,6 +288,24 @@ the polygon in the same document. The adapter links to the alert's own endpoint,
workaround rather than a mapping. A publisher that inlines geometry currently has nowhere to
put it. See [examples/pilot-nws-alerts/README.md](examples/pilot-nws-alerts/README.md).

## Eurostat COFOG adapter — DONE

`tools/adapters/eurostat-cofog.py` converts Eurostat general government expenditure by
function. Ireland 2022: 79 lines across 10 COFOG divisions.

One defect, the same shape as the UK Contracts Finder finding: the budget arithmetic check
demanded exact equality between a parent line and the sum of its parts, which **no statistical
publisher can satisfy**. Four of ten Irish divisions failed by 0.002% - rounding, because
Eurostat reports in millions to one decimal place. The check now infers the rounding unit from
the greatest common divisor of the amounts and allows half a unit per figure; real
discrepancies are still caught.

Deliberately not added: a precision or unit-multiplier field on `amount`. The granularity is
already recoverable from the data, so a declared field would restate what the numbers say and
add something a publisher can get wrong.

See [examples/pilot-eurostat-budget/README.md](examples/pilot-eurostat-budget/README.md).

## The honest risk

Seven profiles with one fictional example city is a demo, not an adopted standard. Profiles
Expand Down
76 changes: 76 additions & 0 deletions examples/pilot-eurostat-budget/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
# Pilot — Eurostat COFOG (Ireland)

**Real data.** Generated by `tools/adapters/eurostat-cofog.py` from the Eurostat
dissemination API, which publishes general government expenditure by function for every EU
member state.

```bash
python3 tools/adapters/eurostat-cofog.py --geo IE --year 2022 \
--out examples/pilot-eurostat-budget
```

79 budget lines across 10 COFOG divisions, €120bn of Irish general government expenditure.

This is the most direct test available of the `budget` profile: Eurostat classifies by COFOG,
which is the classification the profile adopted, and it is the authority that maintains it.

## The defect it found

**The arithmetic check demanded exact equality, and no statistical publisher can satisfy it.**

Four of Ireland's ten COFOG divisions failed the parent-equals-sum-of-children rule — by
€100,000 to €200,000 on figures of €10 billion. That is 0.002%, and it is **rounding, not
error**: Eurostat reports in millions to one decimal place, so independently rounded
sub-groups cannot sum exactly to an independently rounded division.

The rule was right in principle and wrong in practice, the same shape of defect the UK
Contracts Finder pilot found in `areaServed`. The check now infers the publisher's rounding
unit from the greatest common divisor of the amounts and allows half a unit per figure:

```
parent 1,000,000 but children total 250,000 (difference 750,000; rounding tolerance 250,000)
```

Real discrepancies are still caught — verified against a fixture 75% out — and the tolerance
is reported so the reader can judge it.

## Why no precision field was added

Eurostat reports in **millions**; the adapter multiplies to whole euro, which implies
euro-level precision that the source does not have. The obvious response is a
`precision` or `unitMultiplier` field on `amount`.

It was not added, because **the granularity is already recoverable from the data**. Every
figure in this dataset is a multiple of €100,000, which is exactly what the tolerance
inference reads. A declared field would restate what the numbers already say, and would be one
more thing a publisher can get wrong or omit.

## Notation, not semantics

Eurostat writes COFOG as `GF01` and `GF0101`; the profile uses `01` and `01.1`. A pure
notation difference, but one that has to be handled: a consumer matching `GF0101` against a
code list of `01` finds nothing. The adapter normalises.

## Two structural traps

**A `TOTAL` row sits alongside the breakdown.** Imported naively it would double every figure.
It becomes the budget's `totalAmount` rather than a line.

**Divisions and their sub-groups arrive in the same response.** Emitting both as flat lines
would also double-count, so sub-groups nest under their division through `isPartOf` /
`hasPart` — which is what surfaced the rounding defect in the first place.

## What Eurostat does not publish

**Named organizations.** The publisher is ESA 2010 sector S13, "general government" — a
statistical sector, not a body. There is no ministry breakdown here, so
`administrativeClassification` is empty across all 79 lines. The profile treats that field as
the thing that joins a budget to the org chart; against statistical sources there is nothing
to join to.

**Economic classification.** This dataset is functional only. GFSM economic codes would need a
different Eurostat table.

**Phase.** These are outturn statistics, so everything maps to `executed`. A proposed or
approved budget for the same country would come from the national ministry of finance, not
from Eurostat — which is precisely why `budgetPhase` is required.
Loading
Loading