Lowers Fortran HPC kernels to optimisable DaCe SDFGs; hands back a Fortran-callable shared library preserving the caller's original interface (SDFG drops in for the kernel it replaces).
Each block builds the kernel's SDFG from its Fortran sources, runs DaCe's default parallelization pipeline on it
(dace_fortran.pipelines.optimize, below), runs the optimized program on physical inputs
from the test harnesses under tests/ and checks it twice: the pre- and post-optimization SDFGs must agree bit for
bit, and the outputs must match the gfortran build of the same source. Run them from the repository root after
installing; tests/readme_quickstart_test.py executes exactly these blocks.
samples/ holds the complete reproducers, one script per kernel that ingests, optimizes, runs and verifies it, and
checks how many maps the pipeline forms: python -m samples.graupel, python -m samples.cloudsc_monolith (the
whole scheme in one file, tests/cloudsc/full/cloudsc.F90) and python -m samples.velocity_tendencies (ICON's
dycore kernel, through its generated Fortran binding). The e2e tests under tests/e2e/ run the same scripts.
import tempfile
from pathlib import Path
import dace
import dace_fortran
from dace_fortran.pipelines import num_maps, optimize
from tests._util import BITEXACT_CPU_ARGS
from tests.icon.graupel._graupel_harness import (
SCENARIOS, Config, assert_match, compile_reference, physical_columns, run_reference, sdfg_args,
)
graupel = Path("tests/icon/graupel")
sources = [graupel / "aes_graupel" / f for f in ("mo_kind.f90", "mo_physical_constants.f90", "mo_aes_thermo.f90")]
sources.append(graupel / "aes_graupel_fused" / "graupel.f90")
sdfg = dace_fortran.build_sdfg_from_files(sources, entry="mo_aes_graupel::graupel_run", name="graupel_opt")
# DaCe's default flags let gcc contract and rewrite floating-point arithmetic, differently in the unoptimized and
# the optimized program; comparing them bit for bit needs value-preserving flags (-O3, no FMA contraction).
dace.Config.set("compiler", "cpu", "args", value=BITEXACT_CPU_ARGS)
# One 20-level column per regime (ice and snow, mixed-phase rain, melting layer, warm rain, dry), all of them.
cfg = Config(ivstart=1, ivend=len(SCENARIOS), kstart=1)
optimize(sdfg, verify_inputs=sdfg_args(physical_columns(), cfg)) # in place; validates every structural stage
assert num_maps(sdfg) > 0
reference, columns = physical_columns(), physical_columns()
run_reference(compile_reference(Path(tempfile.mkdtemp()), [sources[-1]]), reference, cfg) # gfortran -O0
sdfg(**sdfg_args(columns, cfg)) # compiles once; the array extents (nvec x ke) are read off the arguments
assert_match(reference, columns, rtol=1e-10, atol=1e-14)CloudSC bakes its species count first. Only NCLV reaches the SDFG as a free symbol (CLOUDSCOUTER sizes its
arrays by its NCLV dummy; flang lowercases it): the NCLDQ* species indices CLOUDSC reads are YOECLDP
PARAMETERs, already folded to literals by the build. tests/e2e/test_cloudsc.py runs the same check.
import copy
import tempfile
from pathlib import Path
import dace
import numpy as np
import dace_fortran
from dace_fortran.pipelines import num_maps, optimize
from tests._util import BITEXACT_CPU_ARGS
from tests.cloudsc.full._harness import f2py_reference, run_against_reference
from tests.cloudsc.full._registries import program_outputs
source = Path("tests/cloudsc/full/cloudsc.F90").read_text()
sdfg = dace_fortran.build_sdfg(source, entry="cloudscouter", name="cloudsc_opt")
unoptimized = copy.deepcopy(sdfg)
optimize(sdfg, symbols={"nclv": 5})
assert num_maps(sdfg) > 0
# DaCe's default flags let gcc contract and rewrite floating-point arithmetic, differently in the unoptimized and
# the optimized program; comparing them bit for bit needs value-preserving flags (-O3, no FMA contraction).
dace.Config.set("compiler", "cpu", "args", value=BITEXACT_CPU_ARGS)
# One seeded physical state through gfortran -O0, the optimized SDFG and, bit for bit against it, the unoptimized one.
got, want = run_against_reference(sdfg, f2py_reference(Path(tempfile.mkdtemp())), unoptimized=unoptimized)
for name in program_outputs:
np.testing.assert_allclose(got[name.lower()], want[name.lower()], rtol=1e-11, atol=1e-11, err_msg=name)- Flang (
flang-new) owns the front-end (parse / name-bind / type-infer / intrinsic-lower). - Consumes Flang's elaborated HLFIR (MLIR dialect) → MLIR pass pipeline normalises to one narrow IR shape → walks into
dace.SDFG→ regeneratesbind(c)Fortran wrapper. - Targets: ICON (ocean + atmosphere dycore, graupel, velocity advection), Quantum ESPRESSO (
exx), CLOUDSC, NPB, FV3, LULESH, + a large construct-level test corpus.
kernel.f90
│ (0) preprocess (source-text rewrites) + flang -fc1 -emit-hlfir
▼
kernel.hlfir MLIR (HLFIR dialect) — flang did the parsing,
│ (1) C++/MLIR bridge name binding, type inference, intrinsic lowering
▼
normalised single-TU IR (struct-flattened, inlined, shape-propagated, …)
│ (2) walk → DaCe
▼
dace.SDFG ──(you optimise it with any DaCe transformation)──▶
│ (3) emit binding
▼
<entry>_bindings.f90 + lib<entry>.so ◀── the caller links here
- Flang-authoritative: no Fortran re-parsing in Python; bridge consumes flang-elaborated HLFIR.
- MLIR pass pipeline: struct flatten (AoS→SoA), whole-kernel inline, pointer-assign rewrite, static devirtualisation + loud-reject surviving polymorphism, shape propagation, reduction lifting, non-numerical-noise deletion (error helpers / runtime I/O / character runtime).
- AoS / nested derived types: path-flattened to per-member arrays; array-of-struct (array + allocatable members), ICON's array-of-pointer-records (Graupel) pattern.
- Fortran binding generation: emits
<entry>_bindings.f90, preserves caller interface, zero-copy alias where layouts agree, copy-in/copy-out do-loops otherwise. Optional flat-C-ABI shim (bind(c)) for a stable C entry point. - External-call policy: a kernel
CALLto a separately-compiledbind(c)procedure (e.g. ICON's MPI halo exchangesync_patch_array) stays external → DaCe library node, or is dropped — one declaration drives both the inliner and the bridge. - GPU / OpenACC callers: an ICON-style caller that keeps arguments on the device is handled at the binding boundary — a device-resident argument feeding a host SDFG is staged with
!$ACC UPDATE HOST/UPDATE DEVICE, and one feeding a GPU SDFG is passed straight through inside!$ACC HOST_DATA USE_DEVICE(no host round-trip). Residency comes from a parse-time sidecar; an undecidable argument raises rather than guessing (dace_fortran/bindings/acc_transfers.py). - Build-system integration: standalone preprocess CLI + CMake module + Autotools macros run the source-text rewrites in place — existing build emits flang-consumable Fortran with no other changes.
Run before flang, on raw source. SED-style regex transforms with shared comment/string awareness (not a Fortran parser); narrow + idempotent. Importable standalone from dace_fortran.preprocess:
| Pass | Default | What it does |
|---|---|---|
merge_used_modules |
on | Inlines USE-d module sources so flang sees one self-contained TU. Regex text-splice by default; an fparser AST engine is available via merge_engine=MergeEngine.FPARSER. |
strip_openmp_directives |
on | Drops !$OMP / !$ACC / !$ sentinels and #ifdef _OPENMP / _OPENACC blocks. |
normalize_kind_parameters |
on | Substitutes precision aliases (wp, sp, dp, qp) with literal kind integers when the alias isn't locally bound. |
rewrite_integer_powers |
on | Expands integer-valued REAL-literal powers (x**2.0 → x*x). |
replace_external_with_modules |
opt-in | Resolves EXTERNAL :: name to USE mod, ONLY: name against search_dirs. |
rewrite_string_enum_to_integer |
opt-in | Converts CHARACTER enum-style dummies into INTEGER, returning a map for binding generation. |
preprocess_fortran (IF-intvar) |
opt-in | Rewrites IF (intvar) → IF (intvar /= 0) for INTEGER scalars (flang-21 accepts only LOGICAL). |
dace_fortran.fparser_inliner — alternative single-TU inliner on an fparser AST: parses whole project, resolves USE/ONLY:/=> renames, prunes to the entry's reachability, consolidates surviving USE clauses, desugars (ASSOCIATE/GOTO/statement-functions, removes interfaces), restores cross-module USE so the output is legal single-file Fortran, re-emits one .f90. Requires fparser > 0.2.
Order = DEFAULT_PIPELINE in dace_fortran/builder/__init__.py (MULTI_FILE_PIPELINE = multi-file variant). Runs on a dedicated 2 GB-stack worker thread, MLIRContext multithreading disabled.
| # | Pass | Purpose |
|---|---|---|
| 1 | hlfir-drop-stub-calls |
Erase calls to do_not_emit stub procedures; a consumed result is zeroed with a warning. |
| 2 | hlfir-prune-unreachable |
Erase dispatch-table bindings the entry never dynamically invokes. |
| 3 | symbol-dce |
Drop private functions the entry never reaches. |
| 4 | lower-fir-select-case |
fir.select_case → cf.cond_br before inlining (the inliner cannot clone select-case callees). |
| 5 | lift-cf-to-scf |
Structurise callees (fold early RETURN / CFG into scf.if) so inlining cannot corrupt a structured region. |
| 6 | hlfir-strip-error-helpers |
Delete CALL errore / finish / abor1 etc.: their STOP-terminated shape stays multi-block. |
| 7 | hlfir-strip-runtime-io |
Delete diagnostic _FortranAio* calls; file-bound chains become dace_fortran.libraries.fortran_io nodes. |
| 8 | hlfir-strip-character-runtime |
Delete _FortranACharacter* calls (compare/Trim/Adjust): the bridge models no character data. |
| 9 | hlfir-inline-all |
Splice every callee body into the entry; refuses multi-block callees. |
| 10 | hlfir-unwrap-eval-in-mem |
hlfir.eval_in_mem → fir.alloca + body + plain reads. |
| 11 | hlfir-fold-element-aliases |
Erase element-scoped alias declares left by inlined elementals. |
| 12–13 | hlfir-expand-vector-subscript-{gather,scatter} |
Noncontiguous gather temporaries / scatter destinations → explicit do loops. |
| 14 | symbol-dce |
Drop private callees once inlined. |
| 15 | fir-polymorphic-op |
Statically devirtualise resolvable fir.dispatch / fir.select_type. |
| 16 | hlfir-reject-polymorphism |
Fail loudly on surviving virtual dispatch (CLASS as a monomorphic box only). |
| 17 | hlfir-rewrite-sequence-association |
Collapse sequence-association adapters into section designates. |
| 18 | hlfir-fold-copy-in-out |
Fold flang's copy-in/copy-out temporaries. |
| 19 | hlfir-lift-alloc-array-of-records |
Lift type(t), allocatable :: f(:) into top-level companions. |
| 20 | hlfir-lift-aos-pointer-records |
Materialise concat companions for ICON's array of pointer records (Graupel). |
| 21 | hlfir-eliminate-double-buffer-toggle |
Unroll the loop that reassigns a double-buffer time-level toggle (nvar = nnow / nnew) and substitute it away. |
| 22 | hlfir-split-aor-dummies |
Split allocatable-array-of-records dummies into per-member descriptors. |
| 23 | hlfir-marshal-external-structs |
Expand registered-external aos calls into per-member arguments. |
| 24 | hlfir-prune-never-allocated-member-deref |
Erase a guard that dereferences a record member the module never allocates and never lets escape. |
| 25 | hlfir-flatten-structs |
AoS → SoA; emits the hlfir.flatten_plan attribute. |
| 26 | hlfir-flatten-global-scalar-reads |
Read a never-written scalar member of a module-global record through a synthetic global of its own. |
| 27 | hlfir-mark-bounds-remap-views |
Tag F2003 bounds-remapping pointer assigns so a DaCe View is emitted. |
| 28 | hlfir-rewrite-pointer-assigns |
Collapse plain ptr => target rebinds under strict no-alias. |
| 29 | hlfir-propagate-shapes |
Assumed-shape dummies acquire real extent symbols. |
| 30 | hlfir-version-shape-scalars |
SSA-version a straight-line reassigned scalar used as an array extent. |
| 31 | hlfir-lift-reduction-operands |
Lift inline reductions (max(x, MAXVAL(slice))) into a preceding scalar temporary. |
| 32 | hlfir-default-intent |
Intent-less dummies default to intent_inout. |
| 33 | lift-cf-to-scf |
Raw-CFG loops (DO WHILE, DO…EXIT) → scf.while + scf.if. |
| 34 | hlfir-preserve-mutable-globals |
Clear init bodies of caller-mutable globals so sccp cannot fold their loads. |
| 35 | hlfir-fold-assumed-rank-queries |
Fold fir.box_rank / fir.is_assumed_size when the rank / shape is statically known. |
| 36 | hlfir-fold-constant-scalars |
Promote scalar stack variables with a single constant store to SSA constants. |
| 37–39 | sccp,canonicalize,cse |
Fold, simplify, deduplicate. |
dace_fortran/bridge/— nanobind Python extension (hlfir_bridge).bridge.cppowns anMLIRContext+ModuleOp, delegates totrace_utils.cpp(declaration tracing),extract_vars.cpp(variable/descriptor extraction),extract_ast.cpp+bridge/ast/(expressions,assigns,elementals,control_flow,dispatch) for the IR walk.- Passes live under
dace_fortran/passes/, link into thehlfir_bridge_passesstatic library. - Python side:
dace_fortran/builder/(SDFGBuilder) constructs the SDFG;dace_fortran/intrinsics/maps the intrinsics that become DaCe library nodes (matmul, transpose, dot_product, count, merge, ...).
dace_fortran/bindings/ runs after the SDFG is built. Inputs: a FrozenSignature (SDFG arglist snapshot, drift-checked at codegen), an OriginalInterface (caller-facing Fortran surface), a FlattenPlan (AoS→SoA record from hlfir-flatten-structs). Emits <entry>_bindings.f90, zero-copy alias where layouts agree, copy-in/copy-out loops otherwise, optionally a bind(c) flat-C-ABI shim, then compiles + links a .so.
Mechanisms turning idiomatic Fortran into a flat, monomorphic SDFG:
DaCe SDFGs are monomorphic — no runtime type dispatch. Two layers guarantee every call site is statically resolved before SDFG construction.
Source-level — dace_fortran/inliner/ast_desugaring/monomorphize_rewrite.py (+ analyzer monomorphize.py):
- A polymorphic
CLASS(base)slot (local var, or a container-type component e.g.t_ocean_solve%act) expands intoINTEGER :: <var>__tagdiscriminator + one concrete companion per armTYPE(arm) [, ALLOCATABLE] :: <var>__<arm>. ALLOCATE(concrete :: v)→v__tag = <k>(+allocate(v__<arm>)).- Virtual dispatch
CALL v%binding(args)→ static emit-all-always ladder:IF (v__tag==k) THEN; CALL <arm-proc>(v__<arm>, args); ELSE IF .... - Each arm calls a concrete proc on a concrete
TYPE→ only directfir.calls;<var>__tagreads become free SDFG symbols (e.g.this_act__tag). - 4 composable primitives over the per-TU
MonomorphizationSpec: local-dispatch ladder, component-dispatch ladder, shared-interposer cloning (specialise an inheritedsolve/constructper arm),RETYPE(axis pinned to one concrete type at its construction site → rewriteCLASS(base)→TYPE(concrete)on declarations, no tag needed). stack_slots=True: arms become plain stack objects, not allocatable — the bridge cannot lower an allocatable derived-type scalar.
Bridge-level guard — dace_fortran/passes/RejectPolymorphism.cpp (hlfir-reject-polymorphism): after flang's own fir-polymorphic-op statically devirtualises the resolvable cases, walks for any surviving fir.dispatch / fir.select_type / fir.box_tdesc (type-info read) and loud-fails with a source-located error. Non-polymorphic CLASS(t) boxes (member access, no virtual dispatch) are supported, peeled like fir.box<T> — only genuine runtime type discrimination is rejected.
dace_fortran/passes/FlattenStructs.cpp (hlfir-flatten-structs) eliminates Fortran derived types from the IR before SDFG construction — DaCe handles flat arrays well, structs awkwardly. Post-SDFG mirror of DaCe-core's StructToContainerGroups: recursive walk over record members → one flat per-member companion array, SoA naming, outer-shape concatenation.
Three shapes handled:
- scalar struct w/ flat members:
t%u(M)→t_u(M) - array-of-struct w/ array members (outer+inner extents concatenate):
type(t), dimension(K) :: A→A_u(K,M);A(i)%u(j)→A_u(i,j) - nested records unfold recursively to flat leaf:
o%inner%x(j)→o_inner_x(j)
Struct dummy arguments get the same treatment — replaceStructArg inserts one block arg per member/leaf, _soa-suffixes the function; inlined alias chains from hlfir-inline-all are followed transparently. The pass records a hlfir.flatten_plan attribute consumed by the bindings emitter.
The bind(c) wrapper marshals host AoS ⇄ SDFG SoA with copy-in/out gather loops — _render_aos_copy_in / _render_aos_copy_out / _aos_loop_pieces in dace_fortran/bindings/block_builders.py. SoA buffer layout: [element-dims…, member-dims…] (N-D record array → N leading element-index loops, then the member's own dims). Handles N-dim record arrays and both member kinds: allocatable/pointer (extent = per-element max cap, guarded by allocated/associated, zero-filled where unallocated) and fixed-shape value members (e.g. t_cartesian_coordinates%x(3) — literal extents, always present, cap-scan/presence-guard skipped). Copy-out scatters back only when the argument is written.
The bridge's model for ALLOCATABLE arrays under arbitrary ALLOCATE/DEALLOCATE/conditional-allocate.
Semantics modelled: an ALLOCATABLE at routine/BLOCK scope has one name bound to ≤1 current buffer; allocation status persists across control flow within scope (ALLOCATE in a taken IF branch stays allocated after the IF); referencing an unallocated allocatable is prohibited. The bridge never proves allocation — it models "the current buffer at each point", trusting the program conforms, and may safely over-allocate on a path where Fortran would leave the name unallocated (a conforming program never reads it there).
Abstraction — buffer reaching-definitions: each ALLOCATE site is a buffer definition, each DEALLOCATE a kill. Two sites belong to the same DaCe transient iff their buffers can reach a common use/join as alternatives (both IF arms allocate → live to the join); sites never simultaneously reaching are distinct transients (sequential re-allocation). Build a merge relation s ~ t (both reaching at some join/use), take its union-find equivalence classes — each class is one DaCe transient. Reproduces every pattern: IF/ELSE both-allocate → one buffer; sequential alloc;dealloc;alloc → two buffers; conditional + later realloc → two classes; realloc-chain inside one branch → the right split.
Per-class shape (the PHI): a class whose sites share an extent gets a concrete shape; a class whose sites differ (a real conditional) gets a branch-dependent extent symbol <buf>_d<i> (each site assigns it on its own path, DaCe binds it from whichever path ran). Classes named in first-definition order: class 0 keeps the base name, later classes get <name>_alloc1, <name>_alloc2, …; the bridge's alias map routes reads/writes to the current class buffer as it walks the IR, and both IF arms set the same merged-class buffer so post-join reads need no special handling. Grouping = a structured recursive walk over scf/fir.if regions (no general iterative dataflow), lives in the bridge's extract_vars allocation-site analysis.
Out of scope: MOVE_ALLOC/allocatable-assignment auto-realloc; buffer-reuse storage aliasing (a DaCe-core concern).
Related hazard — shape-scalar versioning — dace_fortran/passes/VersionShapeScalars.cpp (hlfir-version-shape-scalars). A local integer scalar used as an array extent (ALLOCATE(x(m))) may be reassigned (m = m + 3) before another array is sized from it — both extents resolve to the bare name m (a mutable SDFG symbol), so a whole-array op over x's shape after the reassignment iterates m's new value over a buffer allocated to its old one (OOB / heap corruption). Fix: for a scalar feeding an fir.allocmem extent, reassigned after that allocation in straight-line code, the pass SSA-versions it (m, m_2, …) so each array binds the version live at its allocation. Reassignment inside a loop or branch is refused with a clear error rather than silently emitting a mutable shape. Non-hazards left untouched: accumulate-then-allocate-once; loop bounds/subscripts that mint a fir.shape but never an fir.allocmem extent.
Some CALLs should not be pulled into the TU — internals unlowerable (MPI, polymorphic dispatch, string scans) or target a separately-compiled bind(c) library. One declaration drives both halves: the inliner stubs the named procedure's body (keeps its interface) so its internals never enter the TU; the bridge then either emits the surviving CALL as an ExternalCall library node (dace_fortran/external.py) bound to a C-ABI symbol, or drops it. Three behaviours: inline (default), don't-inline + emit, don't-inline + ignore (invariant ignore ⊆ don't-inline).
Declare once with apply_external_functions(EXTERNAL, IGNORE): EXTERNAL = list of ExternalFunction(name, c_function=…, library=…), IGNORE = drop list (finish, message, timers, …). The emitted call's argument plan is derived from the HLFIR call site (array → inout pointer, scalar/free-symbol → by-value) — you supply only the extern "C" symbol and the .so that exports it (linked via rpath, resolved at load time). Contract: an emitted target must be bind(c, name="…") — Fortran name mangling is compiler-specific and a .mod is not C-consumable, so a stable C symbol (native or a thin forwarding shim) is the only portable handle. When the C ABI carries facts HLFIR cannot infer (whole derived-type/AoS struct args, an MPI_Comm handle, per-leaf dynamic extents, cross-library module-global forwarding, intent narrowing) — register an authored ExternalSignature via keep_external(name, c_name=…, args=…, libraries=…) (same registry apply_external_functions uses). See tests/external_call/ and tests/external_aos_test.py.
- LLVM/Flang 21 or 22 — auto-detected:
dace_fortran/llvm_toolchain.py(SUPPORTED_LLVM_VERSIONS) probesflang-new-<major>/flang-<major>and then unversionedflang-new/flang(which must self-identify via--version), taking the first installed major;LLVM_VERSIONpins one,LLVM_DIRoverrides discovery outright. Validated on LLVM 21.1.8 and 22.1.5. Debian/Ubuntu:llvm-N-dev libmlir-N-dev mlir-N-tools libflang-N-dev flang-N clang-Nfrom apt.llvm.org (N = 21 or 22).libflang-N-devprovides both the FIR/HLFIR static libs the bridge links and the flang headers it includes. The C++ side keeps both majors' spellings behindLLVM_VERSION_MAJORindace_fortran/llvm_compat.h. - Python 3.10–3.14 (CI runs 3.12).
- DaCe — pinned to a commit of the FaCe branch (
pyproject.toml), which carries the DaCe-core pieces the frontend needs. Plusfparser > 0.2,networkx,numpy. - nanobind — bridge is a nanobind extension (in the
testingdependency group). - CMake ≥ 3.18, C++17 compiler (the detected LLVM's own
clang++/clang++-<major>auto-selected if present), gfortran (binding tests + numerical references compile with gfortran; Ubuntu'sflang-new-21ships withoutlibflang_rt→ flang is emit-HLFIR-only).
Bridge locates LLVM/MLIR by deriving the install prefix from the flang it detected, using find_package(LLVM) only (avoids the often-broken Debian find_package(MLIR) cmake config; finds MLIR headers/libs via LLVM's prefix).
Dependencies are managed with uv and locked in uv.lock:
uv sync # editable install + the dev group (testing + linting)
MPICC=mpicc uv sync --group mpi # also mpi4py, built against the system MPI (multi-rank tests)
uv run pytest -n 16 tests # or: source .venv/bin/activate
uv run pre-commit run --all-files # the CI lint gate: ruff check + format, mypy, pyright, clang-format, fprettifyAgainst a local DaCe checkout, install it over the locked one with uv pip install -e <dace>; [tool.uv] sets the
compat editable mode so mypy and pyright resolve it, and the hooks run with uv run --no-sync so they keep it.
The C++ bridge compiles on first use (first reference of SDFGBuilder / build_sdfg), not at import — dace_fortran/build_bridge.py runs cmake + build and symlinks the resulting hlfir_bridge*.so into the package. To build it explicitly, or force a clean rebuild:
# Auto-detect LLVM, configure, and build:
python -m dace_fortran.build_bridge # build if stale
python -m dace_fortran.build_bridge --clean # wipe build dir and rebuild
# Or drive cmake directly:
cd dace_fortran/build
cmake .. -DCMAKE_BUILD_TYPE=Release # add -DLLVM_VERSION=22 to pin a major
make -j8Override LLVM discovery with the LLVM_VERSION / LLVM_DIR env vars if auto-detection misses; LLVM_VERSION restricts the probe to that major, LLVM_DIR names an LLVMConfig.cmake directory directly (the major is then read out of that config).
Read the ICON AES graupel microphysics (tests/icon/graupel/aes_graupel_fused/graupel.f90 and the three modules it USEs) into one SDFG, naming the entry as in Fortran, then validate and compile it. Run from the repository root; tests/readme_quickstart_test.py executes exactly this block.
from pathlib import Path
import dace_fortran
graupel = Path("tests/icon/graupel")
sources = [graupel / "aes_graupel" / f for f in ("mo_kind.f90", "mo_physical_constants.f90", "mo_aes_thermo.f90")]
sources.append(graupel / "aes_graupel_fused" / "graupel.f90")
# ``entry`` is the Fortran name of the subroutine: ``module::procedure``.
sdfg = dace_fortran.build_sdfg_from_files(sources, entry="mo_aes_graupel::graupel_run", name="graupel")
sdfg.validate()
graupel_run = sdfg.compile() # callable with the Fortran dummy arguments as keywordsdace_fortran.pipelines.optimize is the parallelization pipeline the CI e2e lanes run (tests/e2e/). It is the
recipe of DaCe's ParallelizePipeline, which DaCe's own CI runs on the Python CloudSC
(tests/corpus/cloudsc/cloudsc_parallel_pipeline_test.py), built from the passes the pinned DaCe provides:
- specialize -- bake the given symbols / scalars to constants (CloudSC's species counts: without them no species loop has a constant trip count), and read length-one arrays as scalars;
- ShortLoopUnroll -- fully unroll constant-trip loops of at most
unroll_limit(8) iterations; - UniqueLoopIterators -- one iterator name per loop, so unrolled copies of a loop do not share it;
- scalar fission -- a private copy of each scalar temporary per use (the frontend emits one per local);
- simplify, then StateFusionExtended;
- LoopToMap -- every loop free of loop-carried dependencies becomes a parallel map;
- StateFusionExtended, FullMapFusion, MapCollapse, FullMapFusion -- fuse what the lift exposed;
- MakeTransientsPersistent, BindOmpThreadCount -- allocate once, size the OpenMP team.
The ICON AES graupel has nothing to specialize (its constants are Fortran PARAMETERs, folded by the build); the
column loop DO iv = ivstart, ivend becomes a map, the level loops inside it stay sequential. CloudSC bakes its species
count nclv. The two optimize blocks at the top of this README run exactly this pipeline.
Building from one self-contained source, plus a Fortran-callable library:
import dace_fortran
from dace_fortran.bindings import build_fortran_library
src = open("kernel.f90").read()
# Build the SDFG. ``entry`` selects the target procedure; it accepts a
# plain Fortran name (``kernel``) or a ``module::proc`` qualifier
# (``mo_x::kernel``). Omit it only when the source has exactly one
# procedure.
sdfg = dace_fortran.build_sdfg(src, entry="kernel", name="kernel")
# ... optimise the SDFG here with any DaCe transformation ...
# Emit + drift-verify + link a Fortran-callable .so. The caller-facing
# interface and AoS→SoA flatten plan are auto-derived from the SDFG.
lib = build_fortran_library(sdfg, out_dir="build", name="kernel")build_sdfg takes ONE self-contained source. A kernel whose modules live in other files goes through build_sdfg_from_files below — pass every .f90, in any order; there is nothing to add to src.
Other entry points (all return a built, validated dace.SDFG):
# A multi-file project (driver + the modules it USEs, in any order). These are
# ``.f90`` PATHS, not source text and not HLFIR. Do not confuse it with
# ``SDFGBuilder.from_files``, which consumes already-emitted ``.hlfir`` files.
# ``entry`` is required here: it is what selects the root file.
sdfg = dace_fortran.build_sdfg_from_files([driver, mod], entry="mo_x::kernel")
# The same, with fparser doing the ``USE``-inlining instead of the default
# text-splicer -- it resolves the ``ONLY:`` / ``=>`` renames the splicer cannot
# see. Needs ``fparser > 0.2``.
sdfg = dace_fortran.build_sdfg_from_files([driver, mod], entry="mo_x::kernel",
merge_engine=dace_fortran.MergeEngine.FPARSER)
# A large / dependency-tangled project: emit .hlfir from your own build,
# then consume compile_commands.json directly (tier 3):
sdfg = dace_fortran.build_sdfg_from_project(
"build/compile_commands.json", entry="mymod::mysub")
# A kernel that CALLs a separately-compiled bind(c) function — keep it
# external and bind it to a C-ABI symbol / library:
dace_fortran.register_external("foo", dace_fortran.ExternalSignature(
c_name="foo",
args=[dace_fortran.Arg("array", "float64")], # intent defaults to inout
libraries=["/path/libfoo.so"]))Real-codebase recipes (ICON from source, Quantum ESPRESSO exx): docs/ICON_INTEGRATION.md, docs/CODEBASE_HELPERS.md, the external-call policy above (§4); worked examples under tests/external_call/, tests/icon/full/, tests/qe/.
Three integration paths run the source-text preprocess passes in place so your existing compiler builds the result:
# Standalone CLI — atomic in-place rewrite, no build-file changes:
python -m dace_fortran.preprocess_cli \
--all-defaults --rewrite-external --rewrite-string-enum \
--search-dir src/utils --inplace \
--in src/kernel.f90 --in src/helper.f90# CMake (cmake/DaceFortran.cmake):
include(DaceFortran)
dace_fortran_preprocess(
TARGET mylib SOURCES src/kernel.f90 src/helper.f90
SEARCH_DIRS src/utils
PASSES all_defaults rewrite_external rewrite_string_enum)
add_library(mylib ${mylib_PREPROCESSED_SOURCES})Autotools is supported via autotools/dace_fortran.m4 + an included dace_fortran.mk.
CI (.github/workflows/fortran-ci.yml) runs five lanes on LLVM 21 and 22, all from one job definition and the
shared setup action .github/actions/setup (toolchain, locked uv environment, HLFIR bridge restored from the cache).
The per-lane commands, runnable locally:
# fast -- everything outside the lanes below:
pytest -n auto --dist loadgroup -m "not mpi and not long and not integration and not e2e" \
--ignore=tests/cloudsc/full --ignore=tests/cloudsc/variants --ignore=tests/qe/h_psi tests/
# e2e -- multi-rank MPI tests, integration tests, then QE vexx + ICON velocity_tendencies through the pipeline:
mpirun --oversubscribe -n 4 python -m pytest -m mpi tests/
python -m pytest -m "integration and not mpi" tests/
python -m pytest -m "e2e and not mpi" --ignore=tests/e2e/test_cloudsc.py --ignore=tests/e2e/test_graupel.py tests/
python -m pytest -m "not mpi and not long and not integration and not e2e" tests/qe/h_psi
# e2e-cloudsc -- CloudSC through the pipeline, then the CloudSC numerical tests one at a time (each -O3 build
# peaks at several GB):
python -m pytest -m e2e tests/e2e/test_cloudsc.py
python -m pytest -m "not mpi and not long and not integration and not e2e" tests/cloudsc/full tests/cloudsc/variants
# heavy-icon -- long tests outside tests/icon/{atmosphere,ocean,graupel} and tests/qe/h_psi; some need ICON built from
# source (tests/icon/dycore/setup_icon_dycore.sh and the icon_build fixture, whose tree ICON_BUILD names):
pytest -n auto --dist loadgroup -m "long and not mpi and not integration and not e2e" \
--ignore=tests/icon/atmosphere --ignore=tests/icon/ocean --ignore=tests/icon/graupel --ignore=tests/qe/h_psi tests/
# heavy-qe -- the two full QE h_psi builds:
pytest -n 2 -m "long and not mpi and not integration and not e2e" tests/qe/h_psi
# heavy-atmo, heavy-ocean -- the long atmosphere and ocean tests:
pytest -n auto --dist loadgroup -m "long and not mpi and not integration and not e2e" tests/icon/atmosphere
pytest -n auto --dist loadgroup -m "long and not mpi and not integration and not e2e" tests/icon/ocean
# Dump built SDFGs for inspection:
__DACE_HLFIR_GEN_TEST_SDFGS=1 python -m pytest tests/tests/conftest.py sets test-env defaults automatically (via setdefault — explicit override still wins): HWLOC_COMPONENTS=-gl (stop hwloc's GL/X11 probe hanging MPI_Init on a desktop X display), UCX_VFS_ENABLE=n + OMPI_MCA_pml=ob1/OMPI_MCA_btl=self,vader (in-node transports, so UCX/PMIx finalize can't abort xdist workers), raises the stack soft-limit to its hard limit for deeply-inlined kernels. Pytest markers: long, sequential, mpi, integration, e2e, xdist_group, fftw (see pyproject.toml).
TMPDIR controls where scratch .f90/.hlfir/.dacecache build artifacts land. Executable-Fortran tests compile+run with gfortran/f2py against a seeded numerical reference.
tests/e2e/ drives three production kernels (four configurations) through the full parallelization pipeline (dace_fortran/pipelines.optimize: specialize → short-unroll → unique-loop-iterators → scalar fission → simplify → state fusion → loop-to-map → state fusion → map fusion) — a transformation that changes a value reds these lanes even when every unit test passes.
Every configuration is built twice, with the pipeline and without, and run on one seeded random input set, so each carries two independent bars:
- pre- vs post-optimize, BIT-EXACT — the pipeline differential, and the bar that actually isolates a transformation bug.
- post-optimize vs the untouched gfortran/f2py reference — the frontend check. Frontend and gfortran differ in evaluation order, so for CloudSC and vexx this can only ever be a
1e-11comparison; on its own it lets a pipeline bug worth less than1e-11through. That gap is what the differential closes.
| Test | Kernel | Differential (pre vs post) | Reference check | Wall time | Peak RAM |
|---|---|---|---|---|---|
test_cloudsc.py |
ECMWF CLOUDSC | bit-exact on every call argument | rtol=atol=1e-11 on all program outputs, num_maps > 0 |
~32 min | ~1.2 GB |
test_velocity.py |
ICON atmosphere velocity_tendencies, both __LOOP_EXCHANGE variants |
bit-exact, via both legs matching the reference at exactly 0.0 | bit-exact (max_diff == 0.0), n_changed > 0 |
~5 min (measured 4:52, both variants) | ~1 GB |
test_vexx.py |
QE exx_bp::vexx_bp_k_gpu |
bit-exact on hpsi, through two independently built bindings |
rtol=atol=1e-11, num_maps > 0 |
~8 min (measured 7:26) | ~0.9 GB |
The two __LOOP_EXCHANGE variants get separate RNG streams (seeds 1001 / 1002). The define swaps the automatic transients' data layout, so they are different kernels that merely share a dummy-argument ABI; one shared stream would let a layout-sensitive bug hide behind inputs that happen to suit the other variant.
Running it locally:
- Sequential, one build at a time.
tests/e2e/conftest.pypinsDACE_compiler_build_jobs=1: each kernel is a single very large TU (CloudSC ~25k lines) whose-O3compile peaks in the multi-GB range, and concurrent compiles OOM the box. Never run the e2e lane under xdist. - Compile flags. The lanes build with
BITEXACT_CPU_ARGS(tests/_util.py):-O3 -march=native -fno-fast-math -ffp-contract=off -fno-math-errno -fno-trapping-math(no reassociation, no FMA contraction — that's what makes bit-exactness achievable), plus a memory budget of roughly 6–9 GB built from four params.--param ggc-min-expand=100 --param ggc-min-heapsize=6291456sets the GC budget: gcc collects once the heap passesmax(heapsize, live * (1 + expand/100)), so every kernel but CloudSC (live set well under a GB) never collects at all, while CloudSC stays bounded instead of reaching ~12 GB.--param max-inline-recursive-depth=4 --param max-inline-recursive-depth-auto=4(gcc default 8) halves recursive inlining, the main driver of function-size blowup on a 25k-line TU, and--param max-gcse-memory=524288(KB, so 512 MB; default 128 MB) is a hard budget past which global CSE bails instead of growing without bound. The ggc pair does not affect codegen at all; the inline and gcse caps do, but neither licenses FP reassociation, so both numerical bars still hold. ⛔ Don't driveggc-min-expandtoward 0 to "save memory" — measured,expand=0left vexx peaking at 871 MB against a 9 GB cap (it bought nothing, only CloudSC is near the limit) while onecc1plusburned 41 min of CPU on a single TU without finishing. - Environment. Set
PYTHONHASHSEED=0(deterministic SDFG hashing) and, if anything in the run imports mpi4py, the anti-hang varsHWLOC_COMPONENTS=-gl UCX_VFS_ENABLE=n OMPI_MCA_pml=ob1 OMPI_MCA_btl=self,vader,tcp PMIX_MCA_gds=hash MPI4PY_RC_INITIALIZE=0. - Memory-tight boxes. Wrap the run in a cgroup so a runaway peak can't take the session down, e.g.
systemd-run --user --service-type=oneshot --wait -p MemoryMax=10G --working-directory=$PWD -- python3 -m pytest -m e2e .... Note: with--wait, pytest's stdout goes to the journal (journalctl --user -u <unit>), not to a shell redirect.
Validated corpora: construct-level suite (types, control flow, allocatable/pointer, slicing/intrinsics, reductions, BLAS/LAPACK, derived types, MPI send/recv); ICON ocean + atmosphere dycore/graupel/velocity-advection + full ICON-from-source integration (tests/icon/); Quantum ESPRESSO exx (tests/qe/); CLOUDSC (tests/cloudsc/); NPB LU (tests/npb/); FV3; LULESH; build-system integration (tests/buildsys_integration/); binding-specific tests (tests/bindings/).
dace_fortran/
build.py public build_sdfg* entry points
build_bridge.py auto-build + import the C++ bridge
preprocess.py source-text preprocess passes
preprocess_cli.py CLI for the preprocess passes
fparser_inliner.py fparser-AST single-TU inliner
flang_codebase.py real-codebase flang driver helpers (ICON/IFS/…)
external.py / external_functions.py external-call policy + registry
emit_hlfir.py tier-3 .hlfir emission helper
pipelines.py whole-SDFG optimization recipes (optimize: the e2e parallelization pipeline)
acc_residency.py per-argument OpenACC data residency sidecar
integer_power_exponents.py integer-valued ** exponents -> ipow
omp_threads.py OpenMP thread-count symbol for persistent maps
entry_names.py module::proc entry spelling
bridge_types.py / dace_types.py static types of the bridge extension and DaCe signatures
CMakeLists.txt bridge build (LLVM 21/22, nanobind)
llvm_toolchain.py LLVM/flang discovery over the supported majors
llvm_compat.h LLVM_VERSION_MAJOR shims for the 21/22 C++ API delta
bridge/ C++/MLIR bridge (nanobind ext: hlfir_bridge)
bridge.cpp nanobind boundary
extract_vars.cpp / extract_ast.cpp / trace_utils.cpp
ast/ expressions, assigns, elementals, control_flow, dispatch
passes/ the MLIR passes (one .cpp per pass + Passes.cpp)
builder/ SDFG construction (SDFGBuilder, DEFAULT_PIPELINE, access, descriptors, emit_*)
libraries/ DaCe library nodes: Fortran file I/O, FFT interpolation
bindings/ Fortran bind(c) binding generator + C-ABI shim
intrinsics/ intrinsics lowered to DaCe library nodes
inliner/ fparser-based module inliner / ast_desugaring
data/ distributed-data helpers
cmake/ DaceFortran.cmake (CMake integration)
autotools/ dace_fortran.m4 + dace_fortran.mk
scripts/ ICON build/configure helpers, pre-commit guards,
generated-kernel static analysis
docs/ CODEBASE_HELPERS (flang on a real codebase),
ICON_INTEGRATION (full ICON CPU recipe)
tests/ test corpora (see Testing)
DIM=over an expression source — WRONG RESULT, not an error.SUM(arr, DIM=k)over a named array is correct (rank-reducing, numerically checked). With a computed source —SUM(arr**2, DIM=1), QE'sSQRT(SUM(matrix**2, DIM=1))—emit_reducedrops the axis and reduces the whole array, broadcasting one scalar over the result. It compiles and runs; only the values are wrong.CHARACTERstring content — only the enum-as-integer pattern (rewrite_string_enum_to_integer) is supported; arbitrary character data is not modelled.- Polymorphism beyond monomorphic CLASS —
SELECT TYPE/CLASS(*)with genuine runtime type discrimination is rejected; static devirtualisation handles only the resolvable case. - Caller-mutable global classification —
PreserveMutableGlobalstreats everyfir.zero_bits(BSS-default) global as caller-mutable; a genuinely zero-initialised mutable global is indistinguishable from an uninitialised input in the IR.
- Re-parsing Fortran in Python — Flang is authoritative.
- Cross-kernel fusion across translation-unit boundaries (inline-all handles intra-TU fusion; cross-TU is the binding emitter's concern).
- Unstructured
GOTO(it does not lift toscf).
Before→after for the key source-text transforms (conceptual tables above; every snippet below is exercised by a test in tests/inliner/).
Type-bound call → free call (deconstruct_procedure_calls) — the type loses its CONTAINS block (becomes pure data); each obj%bind(a) becomes a free call with obj threaded as the first argument:
type Square ; real::side ; contains ; procedure::area ; end type ! before
a = s%area(1.0)
! after: TYPE :: Square ; REAL :: side ; END TYPE (no CONTAINS)
a = area(s, 1.0)Local constant propagation (exploit_locally_constant_variables) — constant and pointer values are folded into later uses, but a pointer passed as an actual argument is left intact (a POINTER dummy needs a pointer actual, not its target expression):
ptr => data ; x = ptr + 1. → x = data + 1. ! folded in an expression
ptr => data ; call f(ptr) → call f(ptr) ! NOT f(data) — pointer keptStubbing & the external-call / IO-binding policy — the callee shell (and its call sites) always survives; only the body is rewritten. Pick the variant by what the procedure is:
make_noop / do_not_emit : SUBROUTINE log(x) ; END ! body emptied; bridge keeps/DROPs the call
make_return_false : LOGICAL FUNCTION isRestart() ; isRestart = .FALSE. ; END
ExternalFunction("sync_patch_array", c_function=…, library=…) ! body emptied; bridge EMITS a bind(c) calldo_not_emit: pure side-effects (IO, timers, finish). ExternalFunction: calls the runtime must still make (MPI halo) — bound to a bind(c) symbol, typically via a thin hand-written shim. One generic name stubs the whole specific family (sync_patch_array ⇒ sync_patch_array_3d_dp, …).
AoS → SoA flatten (hlfir-flatten-structs) — a derived-type access becomes a plain per-member array indexed by the record index (7 variants: scalar dummy, array-of-records, multi-dim, nested connectivity, pointer/allocatable box-section, local never-allocated, double-buffer):
type(t) :: p(:) ; ... = p(i)%w → ... = p_w(i)Monomorphization (monomorphize(program, spec)) — removes Fortran virtual dispatch (fir.dispatch, which has no SDFG node) so a CLASS-heavy kernel like the ICON-O solver becomes lowerable. Two strategies:
! retype: pin a CLASS to one concrete type at its construction site
CLASS(t_transfer), POINTER :: trans → TYPE(t_trivial_transfer), POINTER :: trans
! ladder: a runtime factory over a closed arm-set → static type-tag if-ladder
call s%apply(x) → IF (s__tag==1) CALL gmres_apply(s__t_gmres, x)
ELSE IF (s__tag==2) CALL cg_apply(s__t_cg, x) ...GPL-3.0-or-later. Copyright ETH Zurich and the dace-fortran authors (see AUTHORS / LICENSE / NOTICE).