Possible NetCDF and/or MPI error running on Perlmutter

[danielletijerina]

Hi,
I am trying to debug my initial ELM-FATES runs. I am encountering a couple of issues, outlined here since I am not sure which of these are actually problematic and which may be just warnings.

1. I am able to (seemingly) get the model to run. In one log, I see there are timesteps being completed. Then a NetCDF is created, but the run appears to fail just before any of the outputs are written to the NetCDF file.
2. MPICH Warnings - I am getting multiple MPICH warnings indicating something about the DVS stripe width requested was too large:

1: MPIIO WARNING: DVS stripe width of 24 was requested but DVS set it to 1
1: See MPICH_MPIIO_DVS_MAXNODES in the intro_mpi man page.

3. MPICH Error - The run is aborted when the NetCDF file is opened and I get the following error:

1: pio_support::pio_die:: myrank= -1 : ERROR: nf_mod.F90: 1293 : NetCDF: Invalid dimension ID or name

After this, there are MPICH errors that appear to be aborting at every 10 processes (lines 438-453 of the e3sm.log, attached)

I have tried multiple iterations of the run and I have been getting the NetCDF invalid dimensions warnings, but in this run, the model appears to be running (one month of daily timesteps) and I am not sure why it is not writing to the .nc file.

(FYI, I have already reached out to NERSC and they suggested asking within the FATES code base community)

Thanks!
e3sm.log.32521341.241104-130631.txt

[ckoven]

Hi @danielletijerina I don't see anything in your output that is really FATES-specific, it seems lower-level than that; either with ELM or an I/O library?

[danielletijerina]

Ok, thanks for checking it out. I will try to post this with NERSC and/or on the E3SM forums.