Workflows: add more eamxx-v1 tests on ghci-snl runners by bartgol · Pull Request #8564 · E3SM-Project/E3SM

bartgol · 2026-07-15T17:15:17Z

Add an eamxx-prod case with cuda and ~~oneapi~~intel compilers.

[non-BFB] for EAM cases on chrysalis

Some comments:

renamed oneapi->intel (following the overall e3sm transition) in eamxx machine specs as well as in the workflows
new eamxx-v1 tests: they should hopefully help spot issues with CUDA and OneAPI in cime cases before we merge.
moved ctest custom exception (for compiler output) to a file in our repo: gemini suggests it's the industry preferred way. And it's easier to maintain, imho.
upload ALL content of Testing for eamxx-sa: this should help spot the actual lines triggering oneapi fails. They are usually b/c of some bad string that pops out in the compiler log, but it's hard to know which. Inside Testing/YYYYMMDD-HHMM there are all the xml files (that ctest uploads to cdash when submitting), and each error warning is logged in Build.xml. This should allow us to modify CTestCustom.cmake if needed.

bartgol · 2026-07-15T17:20:30Z

@jgfouca do you know what this means?

ERROR: Command: '/spack-installs/libxml2/2.13.4/gcc/13.3.0/base/dfd2xk4/bin/xmllint --xinclude --noout --schema /home/runner/_work/E3SM/E3SM/cime/CIME/data/config/xml_schemas/config_machines.xsd /home/runner/_work/E3SM/E3SM/cime_config/machines/config_machines.xml' failed with error '/home/runner/_work/E3SM/E3SM/cime_config/machines/config_machines.xml:1938: Schemas validity error : Element 'GMAKE_J': This element is not expected. Expected is one of ( BATCHED_BUILD, TESTS, NTEST_PARALLEL_JOBS, BATCH_SYSTEM ).
9
/home/runner/_work/E3SM/E3SM/cime_config/machines/config_machines.xml fails to validate' from dir '/home/runner/_work/E3SM/E3SM'

I left a "default" GMAKE_J, but maybe that's not allowed, and they all need a selector? Or maybe GMAKE_J does not allow to have selectors?

Edit: ah, I think this is the issue

 <xs:element ref="GMAKE_J" minOccurs="0" maxOccurs="1"/>

So GMAKE_J can only appear once. Ok, I'll just set 32 on all machines.

jgfouca · 2026-07-15T17:25:20Z

@bartgol , yes, you figured it out. This is perhaps something we could address in CIME if needed.

bartgol · 2026-07-15T17:26:26Z

@bartgol , yes, you figured it out. This is perhaps something we could address in CIME if needed.

No need. I think this is a corner case, where different compiler are hosted on different containers (I just used ONE machine with different compiler names to avoid repeating a lot of code). Usually, a machine should have ONE value for GMAKE_J.

bartgol · 2026-07-15T20:34:26Z

I need to install moab on the cuda/oneapi containers before the v1 tests can run...

bartgol · 2026-07-15T23:34:18Z

I'm going to pause this. I want to first integrate a PR that changes oneapi->intel on ghci, which makes it easier to recycle existing macro files settings for intel. I'm going to do it in EKAT too, but before I can update EKAT, I need to wait for the C++20 PR to be merged.

rljacob · 2026-09-09T20:45:55Z

@bartgol what is the status of this PR ?

bartgol · 2026-09-09T21:11:07Z

@bartgol what is the status of this PR ?

I wanted to first merge the PR that does oneapi->intel. I tested that on chrys, but still need to do pm.

bartgol · 2026-09-12T00:06:33Z

The eamxx-v1 fail for intel may be due to our runners having the env var COMPILER=oneapi defined (which is used during the image build to tell spack which compiler to use). The test fails b/c oneapi.cmake is not found in the cmake_macros folder. But nobody claims to use oneapi compiler (only intel). So the only suspect is that env var...

If so, I need to change our images, perhaps using SPACK_COMPILER instead of COMPILER...

Edit: no, the reason must be another. eamxx-v1 / gnu also fails, with the same issue, which points to an issue maybe with config_machines.xml. I need to dig.

bartgol · 2026-09-14T22:19:29Z

I factored out ghci-snl fixes for GNU compiler in #8726, to limit CI fails in this PR to the new compilers added. I will rebase/retest once that one is merged.

bartgol · 2026-09-15T03:41:22Z

The intel DIFF is expected (there are no baselines), but the RUN fail on gnugpu isn't. I need to investigate.

The value 'source' is no longer supported in CXX, although it was kept for F90 for bwd compatibility. The new correct value is 'precise'.

bartgol · 2026-09-17T21:26:39Z

DIFFs are expected, since v1 baselines for intel and gnugpu don't exist yet. Merging.

…8564) Add an eamxx-prod case with cuda and intel compilers. [BFB]

rljacob · 2026-09-20T18:17:08Z

 string(APPEND CMAKE_CXX_FLAGS_DEBUG " -O0 -g")
 string(APPEND CMAKE_Fortran_FLAGS_DEBUG   " -O0 -g -check uninit -check bounds -check pointers -fpe0 -check noarg_temp_created -init=snan,arrays")
-string(APPEND CMAKE_CXX_FLAGS " -fp-model source")
+string(APPEND CMAKE_CXX_FLAGS " -fp-model precise")


This changed answers for EAM cases on chrysalis.

I had to add this, as -fp-model source is no longer a valid intel flag. @amametjanov has a similar mod on his PR to fix aurora.

That said, I don't think it explains anything. The file <compiler>.cmake is included by cmake before <machine_compiler>.cmake. And chrysalis_intel.cmake already has

string(APPEND CMAKE_CXX_FLAGS " -fp-model precise")

So this change ONLY affects builds that did NOT set the fp model in <machine_compiler>.cmake. And if they didn't, oneapi should have errored out. Like it did on Aurora (hence, Az's PR).

rljacob · 2026-09-20T18:44:26Z

    <SUPPORTED_BY>lbertag at sandia dot gov</SUPPORTED_BY>
    <MAX_TASKS_PER_NODE>32</MAX_TASKS_PER_NODE>
    <MAX_MPITASKS_PER_NODE>32</MAX_MPITASKS_PER_NODE>
+    <MAX_MPITASKS_PER_NODE compiler="gnugpu">2</MAX_MPITASKS_PER_NODE>


why was this PR messing with aurora?

This is the ghci-snl entry of config_machines though. There's one mod to aurora.cmake (not sure why), but only in comments, which should not affect anything.

rljacob · 2026-09-20T18:45:30Z

This PR should have been tested overnight on next since it changed general macros like intel.cmake

bartgol requested a review from jgfouca July 15, 2026 17:15

bartgol self-assigned this Jul 15, 2026

bartgol added Machine Files BFB PR leaves answers BFB Testing Anything related to unit/system tests EAMxx C++ based E3SM atmosphere model (aka SCREAM) CI: workflow change approved Allow gh action PR testing on ghci-snl-* machines for PRs that alter a worfklow file workflows labels Jul 15, 2026

bartgol force-pushed the bartgol/machines/more-ghci-snl-testing branch from a02e722 to f47feae Compare July 15, 2026 17:25

jgfouca approved these changes Jul 15, 2026

View reviewed changes

bartgol force-pushed the bartgol/machines/more-ghci-snl-testing branch from 9954912 to 61001e3 Compare July 15, 2026 23:14

bartgol marked this pull request as draft July 15, 2026 23:33

bartgol force-pushed the bartgol/machines/more-ghci-snl-testing branch from 61001e3 to f64b3c6 Compare August 18, 2026 16:09

bartgol added CI: workflow change approved Allow gh action PR testing on ghci-snl-* machines for PRs that alter a worfklow file and removed CI: workflow change approved Allow gh action PR testing on ghci-snl-* machines for PRs that alter a worfklow file labels Aug 18, 2026

bartgol force-pushed the bartgol/machines/more-ghci-snl-testing branch 3 times, most recently from 3ebac1d to 3968ce7 Compare August 18, 2026 20:06

bartgol marked this pull request as ready for review August 18, 2026 20:06

bartgol force-pushed the bartgol/machines/more-ghci-snl-testing branch 2 times, most recently from a46122d to f180e1e Compare August 18, 2026 21:01

bartgol mentioned this pull request Aug 19, 2026

Add sponge-layer tensor viscosity to theta-l_kokkos and EAMxx #8642

Merged

bartgol force-pushed the bartgol/machines/more-ghci-snl-testing branch 2 times, most recently from 5d30cb2 to 8663895 Compare August 20, 2026 17:32

bartgol force-pushed the bartgol/machines/more-ghci-snl-testing branch 2 times, most recently from 4cbfb3a to 74af5e5 Compare September 11, 2026 23:48

bartgol force-pushed the bartgol/machines/more-ghci-snl-testing branch from 74af5e5 to 9109e4a Compare September 14, 2026 20:20

bartgol mentioned this pull request Sep 14, 2026

Machines: small fix to ghci-snl cmake macro files #8726

Merged

bartgol force-pushed the bartgol/machines/more-ghci-snl-testing branch from 968de76 to c7f88a0 Compare September 14, 2026 21:47

bartgol force-pushed the bartgol/machines/more-ghci-snl-testing branch from c7f88a0 to feaff39 Compare September 14, 2026 23:42

bartgol added 5 commits September 17, 2026 10:29

EAMxx: rename oneapi->intel for ghci-snl

63d1992

EAMxx: nano-fix in ghci-snl-cuda eamxx mach file

8191ba7

Machines: expand congig files for ghci-snl machine

b96c75f

Workflows: add eamxx-v1 testing with CUDA and intel compilers

796696a

Machines: fix fp-model flag for intel OneAPI

0119149

The value 'source' is no longer supported in CXX, although it was kept for F90 for bwd compatibility. The new correct value is 'precise'.

bartgol force-pushed the bartgol/machines/more-ghci-snl-testing branch from feaff39 to 0119149 Compare September 17, 2026 16:29

Machines: disable cuda-aware MPI for ghci-snl_gnugpu

e4e60d0

tcclevenger approved these changes Sep 17, 2026

View reviewed changes

bartgol added a commit that referenced this pull request Sep 17, 2026

Merge branch 'bartgol/machines/more-ghci-snl-testing' into next (PR #…

4792a79

…8564) Add an eamxx-prod case with cuda and intel compilers. [BFB]

bartgol merged commit 9e63d09 into master Sep 17, 2026
14 of 16 checks passed

bartgol deleted the bartgol/machines/more-ghci-snl-testing branch September 17, 2026 21:28

rljacob reviewed Sep 20, 2026

View reviewed changes

rljacob added non-BFB PR makes roundoff changes to answers. and removed BFB PR leaves answers BFB labels Sep 20, 2026

rljacob reviewed Sep 20, 2026

View reviewed changes

Conversation

bartgol commented Jul 15, 2026 • edited by rljacob Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bartgol commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jgfouca commented Jul 15, 2026

Uh oh!

bartgol commented Jul 15, 2026

Uh oh!

bartgol commented Jul 15, 2026

Uh oh!

bartgol commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rljacob commented Sep 9, 2026

Uh oh!

bartgol commented Sep 9, 2026

Uh oh!

bartgol commented Sep 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bartgol commented Sep 14, 2026

Uh oh!

bartgol commented Sep 15, 2026

Uh oh!

bartgol commented Sep 17, 2026

Uh oh!

Uh oh!

rljacob Sep 20, 2026

Choose a reason for hiding this comment

Uh oh!

bartgol Sep 21, 2026

Choose a reason for hiding this comment

Uh oh!

rljacob Sep 20, 2026

Choose a reason for hiding this comment

Uh oh!

bartgol Sep 21, 2026

Choose a reason for hiding this comment

Uh oh!

rljacob commented Sep 20, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

bartgol commented Jul 15, 2026 •

edited by rljacob

Loading

bartgol commented Jul 15, 2026 •

edited

Loading

bartgol commented Jul 15, 2026 •

edited

Loading

bartgol commented Sep 12, 2026 •

edited

Loading