Skip to content

About

No description, website, or topics provided.

Resources

Stars

2 stars

Watchers

0 watching

Forks

Repository files navigation

dace-fortran — a Fortran (HLFIR) frontend for DaCe

Lowers Fortran HPC kernels to optimisable DaCe SDFGs; hands back a Fortran-callable shared library preserving the caller's original interface (SDFG drops in for the kernel it replaces).

Quick start: graupel and CloudSC, optimized end to end

Each block builds the kernel's SDFG from its Fortran sources, runs DaCe's default parallelization pipeline on it (dace_fortran.pipelines.optimize, below), runs the optimized program on physical inputs from the test harnesses under tests/ and checks it twice: the pre- and post-optimization SDFGs must agree bit for bit, and the outputs must match the gfortran build of the same source. Run them from the repository root after installing; tests/readme_quickstart_test.py executes exactly these blocks.

samples/ holds the complete reproducers, one script per kernel that ingests, optimizes, runs and verifies it, and checks how many maps the pipeline forms: python -m samples.graupel, python -m samples.cloudsc_monolith (the whole scheme in one file, tests/cloudsc/full/cloudsc.F90) and python -m samples.velocity_tendencies (ICON's dycore kernel, through its generated Fortran binding). The e2e tests under tests/e2e/ run the same scripts.

import tempfile
from pathlib import Path

import dace

import dace_fortran
from dace_fortran.pipelines import num_maps, optimize
from tests._util import BITEXACT_CPU_ARGS
from tests.icon.graupel._graupel_harness import (
    SCENARIOS, Config, assert_match, compile_reference, physical_columns, run_reference, sdfg_args,
)

graupel = Path("tests/icon/graupel")
sources = [graupel / "aes_graupel" / f for f in ("mo_kind.f90", "mo_physical_constants.f90", "mo_aes_thermo.f90")]
sources.append(graupel / "aes_graupel_fused" / "graupel.f90")
sdfg = dace_fortran.build_sdfg_from_files(sources, entry="mo_aes_graupel::graupel_run", name="graupel_opt")

# DaCe's default flags let gcc contract and rewrite floating-point arithmetic, differently in the unoptimized and
# the optimized program; comparing them bit for bit needs value-preserving flags (-O3, no FMA contraction).
dace.Config.set("compiler", "cpu", "args", value=BITEXACT_CPU_ARGS)

# One 20-level column per regime (ice and snow, mixed-phase rain, melting layer, warm rain, dry), all of them.
cfg = Config(ivstart=1, ivend=len(SCENARIOS), kstart=1)
optimize(sdfg, verify_inputs=sdfg_args(physical_columns(), cfg))  # in place; validates every structural stage
assert num_maps(sdfg) > 0

reference, columns = physical_columns(), physical_columns()
run_reference(compile_reference(Path(tempfile.mkdtemp()), [sources[-1]]), reference, cfg)  # gfortran -O0
sdfg(**sdfg_args(columns, cfg))  # compiles once; the array extents (nvec x ke) are read off the arguments
assert_match(reference, columns, rtol=1e-10, atol=1e-14)

CloudSC bakes its species count first. Only NCLV reaches the SDFG as a free symbol (CLOUDSCOUTER sizes its arrays by its NCLV dummy; flang lowercases it): the NCLDQ* species indices CLOUDSC reads are YOECLDP PARAMETERs, already folded to literals by the build. tests/e2e/test_cloudsc.py runs the same check.

import copy
import tempfile
from pathlib import Path

import dace
import numpy as np

import dace_fortran
from dace_fortran.pipelines import num_maps, optimize
from tests._util import BITEXACT_CPU_ARGS
from tests.cloudsc.full._harness import f2py_reference, run_against_reference
from tests.cloudsc.full._registries import program_outputs

source = Path("tests/cloudsc/full/cloudsc.F90").read_text()
sdfg = dace_fortran.build_sdfg(source, entry="cloudscouter", name="cloudsc_opt")
unoptimized = copy.deepcopy(sdfg)
optimize(sdfg, symbols={"nclv": 5})
assert num_maps(sdfg) > 0

# DaCe's default flags let gcc contract and rewrite floating-point arithmetic, differently in the unoptimized and
# the optimized program; comparing them bit for bit needs value-preserving flags (-O3, no FMA contraction).
dace.Config.set("compiler", "cpu", "args", value=BITEXACT_CPU_ARGS)

# One seeded physical state through gfortran -O0, the optimized SDFG and, bit for bit against it, the unoptimized one.
got, want = run_against_reference(sdfg, f2py_reference(Path(tempfile.mkdtemp())), unoptimized=unoptimized)
for name in program_outputs:
    np.testing.assert_allclose(got[name.lower()], want[name.lower()], rtol=1e-11, atol=1e-11, err_msg=name)

Overview

  • Flang (flang-new) owns the front-end (parse / name-bind / type-infer / intrinsic-lower).
  • Consumes Flang's elaborated HLFIR (MLIR dialect) → MLIR pass pipeline normalises to one narrow IR shape → walks into dace.SDFG → regenerates bind(c) Fortran wrapper.
  • Targets: ICON (ocean + atmosphere dycore, graupel, velocity advection), Quantum ESPRESSO (exx), CLOUDSC, NPB, FV3, LULESH, + a large construct-level test corpus.
 kernel.f90
     │  (0) preprocess (source-text rewrites) + flang -fc1 -emit-hlfir
     ▼
 kernel.hlfir            MLIR (HLFIR dialect) — flang did the parsing,
     │  (1) C++/MLIR bridge   name binding, type inference, intrinsic lowering
     ▼
 normalised single-TU IR (struct-flattened, inlined, shape-propagated, …)
     │  (2) walk → DaCe
     ▼
 dace.SDFG  ──(you optimise it with any DaCe transformation)──▶
     │  (3) emit binding
     ▼
 <entry>_bindings.f90  +  lib<entry>.so   ◀── the caller links here

Key features

  • Flang-authoritative: no Fortran re-parsing in Python; bridge consumes flang-elaborated HLFIR.
  • MLIR pass pipeline: struct flatten (AoS→SoA), whole-kernel inline, pointer-assign rewrite, static devirtualisation + loud-reject surviving polymorphism, shape propagation, reduction lifting, non-numerical-noise deletion (error helpers / runtime I/O / character runtime).
  • AoS / nested derived types: path-flattened to per-member arrays; array-of-struct (array + allocatable members), ICON's array-of-pointer-records (Graupel) pattern.
  • Fortran binding generation: emits <entry>_bindings.f90, preserves caller interface, zero-copy alias where layouts agree, copy-in/copy-out do-loops otherwise. Optional flat-C-ABI shim (bind(c)) for a stable C entry point.
  • External-call policy: a kernel CALL to a separately-compiled bind(c) procedure (e.g. ICON's MPI halo exchange sync_patch_array) stays external → DaCe library node, or is dropped — one declaration drives both the inliner and the bridge.
  • GPU / OpenACC callers: an ICON-style caller that keeps arguments on the device is handled at the binding boundary — a device-resident argument feeding a host SDFG is staged with !$ACC UPDATE HOST/UPDATE DEVICE, and one feeding a GPU SDFG is passed straight through inside !$ACC HOST_DATA USE_DEVICE (no host round-trip). Residency comes from a parse-time sidecar; an undecidable argument raises rather than guessing (dace_fortran/bindings/acc_transfers.py).
  • Build-system integration: standalone preprocess CLI + CMake module + Autotools macros run the source-text rewrites in place — existing build emits flang-consumable Fortran with no other changes.

Architecture / pipeline

Source-text preprocess passes (before flang)

Run before flang, on raw source. SED-style regex transforms with shared comment/string awareness (not a Fortran parser); narrow + idempotent. Importable standalone from dace_fortran.preprocess:

Pass Default What it does
merge_used_modules on Inlines USE-d module sources so flang sees one self-contained TU. Regex text-splice by default; an fparser AST engine is available via merge_engine=MergeEngine.FPARSER.
strip_openmp_directives on Drops !$OMP / !$ACC / !$ sentinels and #ifdef _OPENMP / _OPENACC blocks.
normalize_kind_parameters on Substitutes precision aliases (wp, sp, dp, qp) with literal kind integers when the alias isn't locally bound.
rewrite_integer_powers on Expands integer-valued REAL-literal powers (x**2.0 → x*x).
replace_external_with_modules opt-in Resolves EXTERNAL :: name to USE mod, ONLY: name against search_dirs.
rewrite_string_enum_to_integer opt-in Converts CHARACTER enum-style dummies into INTEGER, returning a map for binding generation.
preprocess_fortran (IF-intvar) opt-in Rewrites IF (intvar) → IF (intvar /= 0) for INTEGER scalars (flang-21 accepts only LOGICAL).

dace_fortran.fparser_inliner — alternative single-TU inliner on an fparser AST: parses whole project, resolves USE/ONLY:/=> renames, prunes to the entry's reachability, consolidates surviving USE clauses, desugars (ASSOCIATE/GOTO/statement-functions, removes interfaces), restores cross-module USE so the output is legal single-file Fortran, re-emits one .f90. Requires fparser > 0.2.

HLFIR pass pipeline (inside the bridge)

Order = DEFAULT_PIPELINE in dace_fortran/builder/__init__.py (MULTI_FILE_PIPELINE = multi-file variant). Runs on a dedicated 2 GB-stack worker thread, MLIRContext multithreading disabled.

# Pass Purpose
1 hlfir-drop-stub-calls Erase calls to do_not_emit stub procedures; a consumed result is zeroed with a warning.
2 hlfir-prune-unreachable Erase dispatch-table bindings the entry never dynamically invokes.
3 symbol-dce Drop private functions the entry never reaches.
4 lower-fir-select-case fir.select_case → cf.cond_br before inlining (the inliner cannot clone select-case callees).
5 lift-cf-to-scf Structurise callees (fold early RETURN / CFG into scf.if) so inlining cannot corrupt a structured region.
6 hlfir-strip-error-helpers Delete CALL errore / finish / abor1 etc.: their STOP-terminated shape stays multi-block.
7 hlfir-strip-runtime-io Delete diagnostic _FortranAio* calls; file-bound chains become dace_fortran.libraries.fortran_io nodes.
8 hlfir-strip-character-runtime Delete _FortranACharacter* calls (compare/Trim/Adjust): the bridge models no character data.
9 hlfir-inline-all Splice every callee body into the entry; refuses multi-block callees.
10 hlfir-unwrap-eval-in-mem hlfir.eval_in_mem → fir.alloca + body + plain reads.
11 hlfir-fold-element-aliases Erase element-scoped alias declares left by inlined elementals.
12–13 hlfir-expand-vector-subscript-{gather,scatter} Noncontiguous gather temporaries / scatter destinations → explicit do loops.
14 symbol-dce Drop private callees once inlined.
15 fir-polymorphic-op Statically devirtualise resolvable fir.dispatch / fir.select_type.
16 hlfir-reject-polymorphism Fail loudly on surviving virtual dispatch (CLASS as a monomorphic box only).
17 hlfir-rewrite-sequence-association Collapse sequence-association adapters into section designates.
18 hlfir-fold-copy-in-out Fold flang's copy-in/copy-out temporaries.
19 hlfir-lift-alloc-array-of-records Lift type(t), allocatable :: f(:) into top-level companions.
20 hlfir-lift-aos-pointer-records Materialise concat companions for ICON's array of pointer records (Graupel).
21 hlfir-eliminate-double-buffer-toggle Unroll the loop that reassigns a double-buffer time-level toggle (nvar = nnow / nnew) and substitute it away.
22 hlfir-split-aor-dummies Split allocatable-array-of-records dummies into per-member descriptors.
23 hlfir-marshal-external-structs Expand registered-external aos calls into per-member arguments.
24 hlfir-prune-never-allocated-member-deref Erase a guard that dereferences a record member the module never allocates and never lets escape.
25 hlfir-flatten-structs AoS → SoA; emits the hlfir.flatten_plan attribute.
26 hlfir-flatten-global-scalar-reads Read a never-written scalar member of a module-global record through a synthetic global of its own.
27 hlfir-mark-bounds-remap-views Tag F2003 bounds-remapping pointer assigns so a DaCe View is emitted.
28 hlfir-rewrite-pointer-assigns Collapse plain ptr => target rebinds under strict no-alias.
29 hlfir-propagate-shapes Assumed-shape dummies acquire real extent symbols.
30 hlfir-version-shape-scalars SSA-version a straight-line reassigned scalar used as an array extent.
31 hlfir-lift-reduction-operands Lift inline reductions (max(x, MAXVAL(slice))) into a preceding scalar temporary.
32 hlfir-default-intent Intent-less dummies default to intent_inout.
33 lift-cf-to-scf Raw-CFG loops (DO WHILE, DO…EXIT) → scf.while + scf.if.
34 hlfir-preserve-mutable-globals Clear init bodies of caller-mutable globals so sccp cannot fold their loads.
35 hlfir-fold-assumed-rank-queries Fold fir.box_rank / fir.is_assumed_size when the rank / shape is statically known.
36 hlfir-fold-constant-scalars Promote scalar stack variables with a single constant store to SSA constants.
37–39 sccp,canonicalize,cse Fold, simplify, deduplicate.

Bridge (HLFIR → SDFG)

  • dace_fortran/bridge/ — nanobind Python extension (hlfir_bridge). bridge.cpp owns an MLIRContext+ModuleOp, delegates to trace_utils.cpp (declaration tracing), extract_vars.cpp (variable/descriptor extraction), extract_ast.cpp + bridge/ast/ (expressions, assigns, elementals, control_flow, dispatch) for the IR walk.
  • Passes live under dace_fortran/passes/, link into the hlfir_bridge_passes static library.
  • Python side: dace_fortran/builder/ (SDFGBuilder) constructs the SDFG; dace_fortran/intrinsics/ maps the intrinsics that become DaCe library nodes (matmul, transpose, dot_product, count, merge, ...).

Binding generation (SDFG → Fortran-callable .so)

dace_fortran/bindings/ runs after the SDFG is built. Inputs: a FrozenSignature (SDFG arglist snapshot, drift-checked at codegen), an OriginalInterface (caller-facing Fortran surface), a FlattenPlan (AoS→SoA record from hlfir-flatten-structs). Emits <entry>_bindings.f90, zero-copy alias where layouts agree, copy-in/copy-out loops otherwise, optionally a bind(c) flat-C-ABI shim, then compiles + links a .so.

Key design decisions

Mechanisms turning idiomatic Fortran into a flat, monomorphic SDFG:

1. Devirtualisation / monomorphisation (removing CLASS dispatch)

DaCe SDFGs are monomorphic — no runtime type dispatch. Two layers guarantee every call site is statically resolved before SDFG construction.

Source-level — dace_fortran/inliner/ast_desugaring/monomorphize_rewrite.py (+ analyzer monomorphize.py):

  • A polymorphic CLASS(base) slot (local var, or a container-type component e.g. t_ocean_solve%act) expands into INTEGER :: <var>__tag discriminator + one concrete companion per arm TYPE(arm) [, ALLOCATABLE] :: <var>__<arm>.
  • ALLOCATE(concrete :: v) → v__tag = <k> (+ allocate(v__<arm>)).
  • Virtual dispatch CALL v%binding(args) → static emit-all-always ladder: IF (v__tag==k) THEN; CALL <arm-proc>(v__<arm>, args); ELSE IF ....
  • Each arm calls a concrete proc on a concrete TYPE → only direct fir.calls; <var>__tag reads become free SDFG symbols (e.g. this_act__tag).
  • 4 composable primitives over the per-TU MonomorphizationSpec: local-dispatch ladder, component-dispatch ladder, shared-interposer cloning (specialise an inherited solve/construct per arm), RETYPE (axis pinned to one concrete type at its construction site → rewrite CLASS(base)→TYPE(concrete) on declarations, no tag needed).
  • stack_slots=True: arms become plain stack objects, not allocatable — the bridge cannot lower an allocatable derived-type scalar.

Bridge-level guard — dace_fortran/passes/RejectPolymorphism.cpp (hlfir-reject-polymorphism): after flang's own fir-polymorphic-op statically devirtualises the resolvable cases, walks for any surviving fir.dispatch / fir.select_type / fir.box_tdesc (type-info read) and loud-fails with a source-located error. Non-polymorphic CLASS(t) boxes (member access, no virtual dispatch) are supported, peeled like fir.box<T> — only genuine runtime type discrimination is rejected.

2. Struct flattening (AoS → SoA)

dace_fortran/passes/FlattenStructs.cpp (hlfir-flatten-structs) eliminates Fortran derived types from the IR before SDFG construction — DaCe handles flat arrays well, structs awkwardly. Post-SDFG mirror of DaCe-core's StructToContainerGroups: recursive walk over record members → one flat per-member companion array, SoA naming, outer-shape concatenation.

Three shapes handled:

  • scalar struct w/ flat members: t%u(M) → t_u(M)
  • array-of-struct w/ array members (outer+inner extents concatenate): type(t), dimension(K) :: A → A_u(K,M); A(i)%u(j) → A_u(i,j)
  • nested records unfold recursively to flat leaf: o%inner%x(j) → o_inner_x(j)

Struct dummy arguments get the same treatment — replaceStructArg inserts one block arg per member/leaf, _soa-suffixes the function; inlined alias chains from hlfir-inline-all are followed transparently. The pass records a hlfir.flatten_plan attribute consumed by the bindings emitter.

The bind(c) wrapper marshals host AoS ⇄ SDFG SoA with copy-in/out gather loops — _render_aos_copy_in / _render_aos_copy_out / _aos_loop_pieces in dace_fortran/bindings/block_builders.py. SoA buffer layout: [element-dims…, member-dims…] (N-D record array → N leading element-index loops, then the member's own dims). Handles N-dim record arrays and both member kinds: allocatable/pointer (extent = per-element max cap, guarded by allocated/associated, zero-filled where unallocated) and fixed-shape value members (e.g. t_cartesian_coordinates%x(3) — literal extents, always present, cap-scan/presence-guard skipped). Copy-out scatters back only when the argument is written.

3. Allocation-buffer SSA (the unifying ALLOCATABLE model)

The bridge's model for ALLOCATABLE arrays under arbitrary ALLOCATE/DEALLOCATE/conditional-allocate.

Semantics modelled: an ALLOCATABLE at routine/BLOCK scope has one name bound to ≤1 current buffer; allocation status persists across control flow within scope (ALLOCATE in a taken IF branch stays allocated after the IF); referencing an unallocated allocatable is prohibited. The bridge never proves allocation — it models "the current buffer at each point", trusting the program conforms, and may safely over-allocate on a path where Fortran would leave the name unallocated (a conforming program never reads it there).

Abstraction — buffer reaching-definitions: each ALLOCATE site is a buffer definition, each DEALLOCATE a kill. Two sites belong to the same DaCe transient iff their buffers can reach a common use/join as alternatives (both IF arms allocate → live to the join); sites never simultaneously reaching are distinct transients (sequential re-allocation). Build a merge relation s ~ t (both reaching at some join/use), take its union-find equivalence classes — each class is one DaCe transient. Reproduces every pattern: IF/ELSE both-allocate → one buffer; sequential alloc;dealloc;alloc → two buffers; conditional + later realloc → two classes; realloc-chain inside one branch → the right split.

Per-class shape (the PHI): a class whose sites share an extent gets a concrete shape; a class whose sites differ (a real conditional) gets a branch-dependent extent symbol <buf>_d<i> (each site assigns it on its own path, DaCe binds it from whichever path ran). Classes named in first-definition order: class 0 keeps the base name, later classes get <name>_alloc1, <name>_alloc2, …; the bridge's alias map routes reads/writes to the current class buffer as it walks the IR, and both IF arms set the same merged-class buffer so post-join reads need no special handling. Grouping = a structured recursive walk over scf/fir.if regions (no general iterative dataflow), lives in the bridge's extract_vars allocation-site analysis.

Out of scope: MOVE_ALLOC/allocatable-assignment auto-realloc; buffer-reuse storage aliasing (a DaCe-core concern).

Related hazard — shape-scalar versioning — dace_fortran/passes/VersionShapeScalars.cpp (hlfir-version-shape-scalars). A local integer scalar used as an array extent (ALLOCATE(x(m))) may be reassigned (m = m + 3) before another array is sized from it — both extents resolve to the bare name m (a mutable SDFG symbol), so a whole-array op over x's shape after the reassignment iterates m's new value over a buffer allocated to its old one (OOB / heap corruption). Fix: for a scalar feeding an fir.allocmem extent, reassigned after that allocation in straight-line code, the pass SSA-versions it (m, m_2, …) so each array binds the version live at its allocation. Reassignment inside a loop or branch is refused with a clear error rather than silently emitting a mutable shape. Non-hazards left untouched: accumulate-then-allocate-once; loop bounds/subscripts that mint a fir.shape but never an fir.allocmem extent.

4. External-call policy (calls left un-inlined)

Some CALLs should not be pulled into the TU — internals unlowerable (MPI, polymorphic dispatch, string scans) or target a separately-compiled bind(c) library. One declaration drives both halves: the inliner stubs the named procedure's body (keeps its interface) so its internals never enter the TU; the bridge then either emits the surviving CALL as an ExternalCall library node (dace_fortran/external.py) bound to a C-ABI symbol, or drops it. Three behaviours: inline (default), don't-inline + emit, don't-inline + ignore (invariant ignore ⊆ don't-inline).

Declare once with apply_external_functions(EXTERNAL, IGNORE): EXTERNAL = list of ExternalFunction(name, c_function=…, library=…), IGNORE = drop list (finish, message, timers, …). The emitted call's argument plan is derived from the HLFIR call site (array → inout pointer, scalar/free-symbol → by-value) — you supply only the extern "C" symbol and the .so that exports it (linked via rpath, resolved at load time). Contract: an emitted target must be bind(c, name="…") — Fortran name mangling is compiler-specific and a .mod is not C-consumable, so a stable C symbol (native or a thin forwarding shim) is the only portable handle. When the C ABI carries facts HLFIR cannot infer (whole derived-type/AoS struct args, an MPI_Comm handle, per-leaf dynamic extents, cross-library module-global forwarding, intent narrowing) — register an authored ExternalSignature via keep_external(name, c_name=…, args=…, libraries=…) (same registry apply_external_functions uses). See tests/external_call/ and tests/external_aos_test.py.

Prerequisites

  • LLVM/Flang 21 or 22 — auto-detected: dace_fortran/llvm_toolchain.py (SUPPORTED_LLVM_VERSIONS) probes flang-new-<major> / flang-<major> and then unversioned flang-new / flang (which must self-identify via --version), taking the first installed major; LLVM_VERSION pins one, LLVM_DIR overrides discovery outright. Validated on LLVM 21.1.8 and 22.1.5. Debian/Ubuntu: llvm-N-dev libmlir-N-dev mlir-N-tools libflang-N-dev flang-N clang-N from apt.llvm.org (N = 21 or 22). libflang-N-dev provides both the FIR/HLFIR static libs the bridge links and the flang headers it includes. The C++ side keeps both majors' spellings behind LLVM_VERSION_MAJOR in dace_fortran/llvm_compat.h.
  • Python 3.10–3.14 (CI runs 3.12).
  • DaCe — pinned to a commit of the FaCe branch (pyproject.toml), which carries the DaCe-core pieces the frontend needs. Plus fparser > 0.2, networkx, numpy.
  • nanobind — bridge is a nanobind extension (in the testing dependency group).
  • CMake ≥ 3.18, C++17 compiler (the detected LLVM's own clang++/clang++-<major> auto-selected if present), gfortran (binding tests + numerical references compile with gfortran; Ubuntu's flang-new-21 ships without libflang_rt → flang is emit-HLFIR-only).

Bridge locates LLVM/MLIR by deriving the install prefix from the flang it detected, using find_package(LLVM) only (avoids the often-broken Debian find_package(MLIR) cmake config; finds MLIR headers/libs via LLVM's prefix).

Install & build

Dependencies are managed with uv and locked in uv.lock:

uv sync                          # editable install + the dev group (testing + linting)
MPICC=mpicc uv sync --group mpi  # also mpi4py, built against the system MPI (multi-rank tests)
uv run pytest -n 16 tests        # or: source .venv/bin/activate
uv run pre-commit run --all-files  # the CI lint gate: ruff check + format, mypy, pyright, clang-format, fprettify

Against a local DaCe checkout, install it over the locked one with uv pip install -e <dace>; [tool.uv] sets the compat editable mode so mypy and pyright resolve it, and the hooks run with uv run --no-sync so they keep it.

The C++ bridge compiles on first use (first reference of SDFGBuilder / build_sdfg), not at import — dace_fortran/build_bridge.py runs cmake + build and symlinks the resulting hlfir_bridge*.so into the package. To build it explicitly, or force a clean rebuild:

# Auto-detect LLVM, configure, and build:
python -m dace_fortran.build_bridge            # build if stale
python -m dace_fortran.build_bridge --clean    # wipe build dir and rebuild

# Or drive cmake directly:
cd dace_fortran/build
cmake .. -DCMAKE_BUILD_TYPE=Release          # add -DLLVM_VERSION=22 to pin a major
make -j8

Override LLVM discovery with the LLVM_VERSION / LLVM_DIR env vars if auto-detection misses; LLVM_VERSION restricts the probe to that major, LLVM_DIR names an LLVMConfig.cmake directory directly (the major is then read out of that config).

Building an SDFG

Read the ICON AES graupel microphysics (tests/icon/graupel/aes_graupel_fused/graupel.f90 and the three modules it USEs) into one SDFG, naming the entry as in Fortran, then validate and compile it. Run from the repository root; tests/readme_quickstart_test.py executes exactly this block.

from pathlib import Path

import dace_fortran

graupel = Path("tests/icon/graupel")
sources = [graupel / "aes_graupel" / f for f in ("mo_kind.f90", "mo_physical_constants.f90", "mo_aes_thermo.f90")]
sources.append(graupel / "aes_graupel_fused" / "graupel.f90")

# ``entry`` is the Fortran name of the subroutine: ``module::procedure``.
sdfg = dace_fortran.build_sdfg_from_files(sources, entry="mo_aes_graupel::graupel_run", name="graupel")
sdfg.validate()
graupel_run = sdfg.compile()  # callable with the Fortran dummy arguments as keywords

The optimize pipeline

dace_fortran.pipelines.optimize is the parallelization pipeline the CI e2e lanes run (tests/e2e/). It is the recipe of DaCe's ParallelizePipeline, which DaCe's own CI runs on the Python CloudSC (tests/corpus/cloudsc/cloudsc_parallel_pipeline_test.py), built from the passes the pinned DaCe provides:

  1. specialize -- bake the given symbols / scalars to constants (CloudSC's species counts: without them no species loop has a constant trip count), and read length-one arrays as scalars;
  2. ShortLoopUnroll -- fully unroll constant-trip loops of at most unroll_limit (8) iterations;
  3. UniqueLoopIterators -- one iterator name per loop, so unrolled copies of a loop do not share it;
  4. scalar fission -- a private copy of each scalar temporary per use (the frontend emits one per local);
  5. simplify, then StateFusionExtended;
  6. LoopToMap -- every loop free of loop-carried dependencies becomes a parallel map;
  7. StateFusionExtended, FullMapFusion, MapCollapse, FullMapFusion -- fuse what the lift exposed;
  8. MakeTransientsPersistent, BindOmpThreadCount -- allocate once, size the OpenMP team.

The ICON AES graupel has nothing to specialize (its constants are Fortran PARAMETERs, folded by the build); the column loop DO iv = ivstart, ivend becomes a map, the level loops inside it stay sequential. CloudSC bakes its species count nclv. The two optimize blocks at the top of this README run exactly this pipeline.

Building from one self-contained source, plus a Fortran-callable library:

import dace_fortran
from dace_fortran.bindings import build_fortran_library

src = open("kernel.f90").read()

# Build the SDFG.  ``entry`` selects the target procedure; it accepts a
# plain Fortran name (``kernel``) or a ``module::proc`` qualifier
# (``mo_x::kernel``).  Omit it only when the source has exactly one
# procedure.
sdfg = dace_fortran.build_sdfg(src, entry="kernel", name="kernel")

# ... optimise the SDFG here with any DaCe transformation ...

# Emit + drift-verify + link a Fortran-callable .so.  The caller-facing
# interface and AoS→SoA flatten plan are auto-derived from the SDFG.
lib = build_fortran_library(sdfg, out_dir="build", name="kernel")

build_sdfg takes ONE self-contained source. A kernel whose modules live in other files goes through build_sdfg_from_files below — pass every .f90, in any order; there is nothing to add to src.

Other entry points (all return a built, validated dace.SDFG):

# A multi-file project (driver + the modules it USEs, in any order).  These are
# ``.f90`` PATHS, not source text and not HLFIR.  Do not confuse it with
# ``SDFGBuilder.from_files``, which consumes already-emitted ``.hlfir`` files.
# ``entry`` is required here: it is what selects the root file.
sdfg = dace_fortran.build_sdfg_from_files([driver, mod], entry="mo_x::kernel")

# The same, with fparser doing the ``USE``-inlining instead of the default
# text-splicer -- it resolves the ``ONLY:`` / ``=>`` renames the splicer cannot
# see.  Needs ``fparser > 0.2``.
sdfg = dace_fortran.build_sdfg_from_files([driver, mod], entry="mo_x::kernel",
                                          merge_engine=dace_fortran.MergeEngine.FPARSER)

# A large / dependency-tangled project: emit .hlfir from your own build,
# then consume compile_commands.json directly (tier 3):
sdfg = dace_fortran.build_sdfg_from_project(
    "build/compile_commands.json", entry="mymod::mysub")

# A kernel that CALLs a separately-compiled bind(c) function — keep it
# external and bind it to a C-ABI symbol / library:
dace_fortran.register_external("foo", dace_fortran.ExternalSignature(
    c_name="foo",
    args=[dace_fortran.Arg("array", "float64")],   # intent defaults to inout
    libraries=["/path/libfoo.so"]))

Real-codebase recipes (ICON from source, Quantum ESPRESSO exx): docs/ICON_INTEGRATION.md, docs/CODEBASE_HELPERS.md, the external-call policy above (§4); worked examples under tests/external_call/, tests/icon/full/, tests/qe/.

Build-system integration

Three integration paths run the source-text preprocess passes in place so your existing compiler builds the result:

# Standalone CLI — atomic in-place rewrite, no build-file changes:
python -m dace_fortran.preprocess_cli \
    --all-defaults --rewrite-external --rewrite-string-enum \
    --search-dir src/utils --inplace \
    --in src/kernel.f90 --in src/helper.f90
# CMake (cmake/DaceFortran.cmake):
include(DaceFortran)
dace_fortran_preprocess(
    TARGET mylib SOURCES src/kernel.f90 src/helper.f90
    SEARCH_DIRS src/utils
    PASSES all_defaults rewrite_external rewrite_string_enum)
add_library(mylib ${mylib_PREPROCESSED_SOURCES})

Autotools is supported via autotools/dace_fortran.m4 + an included dace_fortran.mk.

Testing

CI (.github/workflows/fortran-ci.yml) runs five lanes on LLVM 21 and 22, all from one job definition and the shared setup action .github/actions/setup (toolchain, locked uv environment, HLFIR bridge restored from the cache). The per-lane commands, runnable locally:

# fast -- everything outside the lanes below:
pytest -n auto --dist loadgroup -m "not mpi and not long and not integration and not e2e" \
    --ignore=tests/cloudsc/full --ignore=tests/cloudsc/variants --ignore=tests/qe/h_psi tests/

# e2e -- multi-rank MPI tests, integration tests, then QE vexx + ICON velocity_tendencies through the pipeline:
mpirun --oversubscribe -n 4 python -m pytest -m mpi tests/
python -m pytest -m "integration and not mpi" tests/
python -m pytest -m "e2e and not mpi" --ignore=tests/e2e/test_cloudsc.py --ignore=tests/e2e/test_graupel.py tests/
python -m pytest -m "not mpi and not long and not integration and not e2e" tests/qe/h_psi

# e2e-cloudsc -- CloudSC through the pipeline, then the CloudSC numerical tests one at a time (each -O3 build
# peaks at several GB):
python -m pytest -m e2e tests/e2e/test_cloudsc.py
python -m pytest -m "not mpi and not long and not integration and not e2e" tests/cloudsc/full tests/cloudsc/variants

# heavy-icon -- long tests outside tests/icon/{atmosphere,ocean,graupel} and tests/qe/h_psi; some need ICON built from
# source (tests/icon/dycore/setup_icon_dycore.sh and the icon_build fixture, whose tree ICON_BUILD names):
pytest -n auto --dist loadgroup -m "long and not mpi and not integration and not e2e" \
    --ignore=tests/icon/atmosphere --ignore=tests/icon/ocean --ignore=tests/icon/graupel --ignore=tests/qe/h_psi tests/

# heavy-qe -- the two full QE h_psi builds:
pytest -n 2 -m "long and not mpi and not integration and not e2e" tests/qe/h_psi

# heavy-atmo, heavy-ocean -- the long atmosphere and ocean tests:
pytest -n auto --dist loadgroup -m "long and not mpi and not integration and not e2e" tests/icon/atmosphere
pytest -n auto --dist loadgroup -m "long and not mpi and not integration and not e2e" tests/icon/ocean

# Dump built SDFGs for inspection:
__DACE_HLFIR_GEN_TEST_SDFGS=1 python -m pytest tests/

tests/conftest.py sets test-env defaults automatically (via setdefault — explicit override still wins): HWLOC_COMPONENTS=-gl (stop hwloc's GL/X11 probe hanging MPI_Init on a desktop X display), UCX_VFS_ENABLE=n + OMPI_MCA_pml=ob1/OMPI_MCA_btl=self,vader (in-node transports, so UCX/PMIx finalize can't abort xdist workers), raises the stack soft-limit to its hard limit for deeply-inlined kernels. Pytest markers: long, sequential, mpi, integration, e2e, xdist_group, fftw (see pyproject.toml).

TMPDIR controls where scratch .f90/.hlfir/.dacecache build artifacts land. Executable-Fortran tests compile+run with gfortran/f2py against a seeded numerical reference.

e2e tests

tests/e2e/ drives three production kernels (four configurations) through the full parallelization pipeline (dace_fortran/pipelines.optimize: specialize → short-unroll → unique-loop-iterators → scalar fission → simplify → state fusion → loop-to-map → state fusion → map fusion) — a transformation that changes a value reds these lanes even when every unit test passes.

Every configuration is built twice, with the pipeline and without, and run on one seeded random input set, so each carries two independent bars:

  • pre- vs post-optimize, BIT-EXACT — the pipeline differential, and the bar that actually isolates a transformation bug.
  • post-optimize vs the untouched gfortran/f2py reference — the frontend check. Frontend and gfortran differ in evaluation order, so for CloudSC and vexx this can only ever be a 1e-11 comparison; on its own it lets a pipeline bug worth less than 1e-11 through. That gap is what the differential closes.
Test Kernel Differential (pre vs post) Reference check Wall time Peak RAM
test_cloudsc.py ECMWF CLOUDSC bit-exact on every call argument rtol=atol=1e-11 on all program outputs, num_maps > 0 ~32 min ~1.2 GB
test_velocity.py ICON atmosphere velocity_tendencies, both __LOOP_EXCHANGE variants bit-exact, via both legs matching the reference at exactly 0.0 bit-exact (max_diff == 0.0), n_changed > 0 ~5 min (measured 4:52, both variants) ~1 GB
test_vexx.py QE exx_bp::vexx_bp_k_gpu bit-exact on hpsi, through two independently built bindings rtol=atol=1e-11, num_maps > 0 ~8 min (measured 7:26) ~0.9 GB

The two __LOOP_EXCHANGE variants get separate RNG streams (seeds 1001 / 1002). The define swaps the automatic transients' data layout, so they are different kernels that merely share a dummy-argument ABI; one shared stream would let a layout-sensitive bug hide behind inputs that happen to suit the other variant.

Running it locally:

  • Sequential, one build at a time. tests/e2e/conftest.py pins DACE_compiler_build_jobs=1: each kernel is a single very large TU (CloudSC ~25k lines) whose -O3 compile peaks in the multi-GB range, and concurrent compiles OOM the box. Never run the e2e lane under xdist.
  • Compile flags. The lanes build with BITEXACT_CPU_ARGS (tests/_util.py): -O3 -march=native -fno-fast-math -ffp-contract=off -fno-math-errno -fno-trapping-math (no reassociation, no FMA contraction — that's what makes bit-exactness achievable), plus a memory budget of roughly 6–9 GB built from four params. --param ggc-min-expand=100 --param ggc-min-heapsize=6291456 sets the GC budget: gcc collects once the heap passes max(heapsize, live * (1 + expand/100)), so every kernel but CloudSC (live set well under a GB) never collects at all, while CloudSC stays bounded instead of reaching ~12 GB. --param max-inline-recursive-depth=4 --param max-inline-recursive-depth-auto=4 (gcc default 8) halves recursive inlining, the main driver of function-size blowup on a 25k-line TU, and --param max-gcse-memory=524288 (KB, so 512 MB; default 128 MB) is a hard budget past which global CSE bails instead of growing without bound. The ggc pair does not affect codegen at all; the inline and gcse caps do, but neither licenses FP reassociation, so both numerical bars still hold. ⛔ Don't drive ggc-min-expand toward 0 to "save memory" — measured, expand=0 left vexx peaking at 871 MB against a 9 GB cap (it bought nothing, only CloudSC is near the limit) while one cc1plus burned 41 min of CPU on a single TU without finishing.
  • Environment. Set PYTHONHASHSEED=0 (deterministic SDFG hashing) and, if anything in the run imports mpi4py, the anti-hang vars HWLOC_COMPONENTS=-gl UCX_VFS_ENABLE=n OMPI_MCA_pml=ob1 OMPI_MCA_btl=self,vader,tcp PMIX_MCA_gds=hash MPI4PY_RC_INITIALIZE=0.
  • Memory-tight boxes. Wrap the run in a cgroup so a runaway peak can't take the session down, e.g. systemd-run --user --service-type=oneshot --wait -p MemoryMax=10G --working-directory=$PWD -- python3 -m pytest -m e2e .... Note: with --wait, pytest's stdout goes to the journal (journalctl --user -u <unit>), not to a shell redirect.

Validated corpora: construct-level suite (types, control flow, allocatable/pointer, slicing/intrinsics, reductions, BLAS/LAPACK, derived types, MPI send/recv); ICON ocean + atmosphere dycore/graupel/velocity-advection + full ICON-from-source integration (tests/icon/); Quantum ESPRESSO exx (tests/qe/); CLOUDSC (tests/cloudsc/); NPB LU (tests/npb/); FV3; LULESH; build-system integration (tests/buildsys_integration/); binding-specific tests (tests/bindings/).

Repository layout

dace_fortran/
  build.py                 public build_sdfg* entry points
  build_bridge.py          auto-build + import the C++ bridge
  preprocess.py            source-text preprocess passes
  preprocess_cli.py        CLI for the preprocess passes
  fparser_inliner.py       fparser-AST single-TU inliner
  flang_codebase.py        real-codebase flang driver helpers (ICON/IFS/…)
  external.py / external_functions.py  external-call policy + registry
  emit_hlfir.py            tier-3 .hlfir emission helper
  pipelines.py             whole-SDFG optimization recipes (optimize: the e2e parallelization pipeline)
  acc_residency.py         per-argument OpenACC data residency sidecar
  integer_power_exponents.py  integer-valued ** exponents -> ipow
  omp_threads.py           OpenMP thread-count symbol for persistent maps
  entry_names.py           module::proc entry spelling
  bridge_types.py / dace_types.py  static types of the bridge extension and DaCe signatures
  CMakeLists.txt           bridge build (LLVM 21/22, nanobind)
  llvm_toolchain.py        LLVM/flang discovery over the supported majors
  llvm_compat.h            LLVM_VERSION_MAJOR shims for the 21/22 C++ API delta
  bridge/                  C++/MLIR bridge (nanobind ext: hlfir_bridge)
    bridge.cpp             nanobind boundary
    extract_vars.cpp / extract_ast.cpp / trace_utils.cpp
    ast/                   expressions, assigns, elementals, control_flow, dispatch
  passes/                  the MLIR passes (one .cpp per pass + Passes.cpp)
  builder/                 SDFG construction (SDFGBuilder, DEFAULT_PIPELINE, access, descriptors, emit_*)
  libraries/               DaCe library nodes: Fortran file I/O, FFT interpolation
  bindings/                Fortran bind(c) binding generator + C-ABI shim
  intrinsics/              intrinsics lowered to DaCe library nodes
  inliner/                 fparser-based module inliner / ast_desugaring
  data/                    distributed-data helpers
cmake/                     DaceFortran.cmake (CMake integration)
autotools/                 dace_fortran.m4 + dace_fortran.mk
scripts/                   ICON build/configure helpers, pre-commit guards,
                           generated-kernel static analysis
docs/                      CODEBASE_HELPERS (flang on a real codebase),
                           ICON_INTEGRATION (full ICON CPU recipe)
tests/                     test corpora (see Testing)

Future work / roadmap

  • DIM= over an expression source — WRONG RESULT, not an error. SUM(arr, DIM=k) over a named array is correct (rank-reducing, numerically checked). With a computed source — SUM(arr**2, DIM=1), QE's SQRT(SUM(matrix**2, DIM=1)) — emit_reduce drops the axis and reduces the whole array, broadcasting one scalar over the result. It compiles and runs; only the values are wrong.
  • CHARACTER string content — only the enum-as-integer pattern (rewrite_string_enum_to_integer) is supported; arbitrary character data is not modelled.
  • Polymorphism beyond monomorphic CLASS — SELECT TYPE/CLASS(*) with genuine runtime type discrimination is rejected; static devirtualisation handles only the resolvable case.
  • Caller-mutable global classification — PreserveMutableGlobals treats every fir.zero_bits (BSS-default) global as caller-mutable; a genuinely zero-initialised mutable global is indistinguishable from an uninitialised input in the IR.

Non-goals

  • Re-parsing Fortran in Python — Flang is authoritative.
  • Cross-kernel fusion across translation-unit boundaries (inline-all handles intra-TU fusion; cross-TU is the binding emitter's concern).
  • Unstructured GOTO (it does not lift to scf).

Transformation examples (Fortran→Fortran)

Before→after for the key source-text transforms (conceptual tables above; every snippet below is exercised by a test in tests/inliner/).

Type-bound call → free call (deconstruct_procedure_calls) — the type loses its CONTAINS block (becomes pure data); each obj%bind(a) becomes a free call with obj threaded as the first argument:

type Square ; real::side ; contains ; procedure::area ; end type   ! before
a = s%area(1.0)
! after:  TYPE :: Square ; REAL :: side ; END TYPE   (no CONTAINS)
a = area(s, 1.0)

Local constant propagation (exploit_locally_constant_variables) — constant and pointer values are folded into later uses, but a pointer passed as an actual argument is left intact (a POINTER dummy needs a pointer actual, not its target expression):

ptr => data ; x = ptr + 1.    →    x = data + 1.        ! folded in an expression
ptr => data ; call f(ptr)      →    call f(ptr)          ! NOT f(data) — pointer kept

Stubbing & the external-call / IO-binding policy — the callee shell (and its call sites) always survives; only the body is rewritten. Pick the variant by what the procedure is:

make_noop / do_not_emit  : SUBROUTINE log(x) ; END           ! body emptied; bridge keeps/DROPs the call
make_return_false        : LOGICAL FUNCTION isRestart() ; isRestart = .FALSE. ; END
ExternalFunction("sync_patch_array", c_function=…, library=…) ! body emptied; bridge EMITS a bind(c) call

do_not_emit: pure side-effects (IO, timers, finish). ExternalFunction: calls the runtime must still make (MPI halo) — bound to a bind(c) symbol, typically via a thin hand-written shim. One generic name stubs the whole specific family (sync_patch_array ⇒ sync_patch_array_3d_dp, …).

AoS → SoA flatten (hlfir-flatten-structs) — a derived-type access becomes a plain per-member array indexed by the record index (7 variants: scalar dummy, array-of-records, multi-dim, nested connectivity, pointer/allocatable box-section, local never-allocated, double-buffer):

type(t) :: p(:) ; ... = p(i)%w     →     ... = p_w(i)

Monomorphization (monomorphize(program, spec)) — removes Fortran virtual dispatch (fir.dispatch, which has no SDFG node) so a CLASS-heavy kernel like the ICON-O solver becomes lowerable. Two strategies:

! retype: pin a CLASS to one concrete type at its construction site
CLASS(t_transfer), POINTER :: trans   →   TYPE(t_trivial_transfer), POINTER :: trans

! ladder: a runtime factory over a closed arm-set → static type-tag if-ladder
call s%apply(x)   →   IF (s__tag==1) CALL gmres_apply(s__t_gmres, x)
                      ELSE IF (s__tag==2) CALL cg_apply(s__t_cg, x) ...

License

GPL-3.0-or-later. Copyright ETH Zurich and the dace-fortran authors (see AUTHORS / LICENSE / NOTICE).

About

No description, website, or topics provided.

Resources

Stars

2 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages