NOAA CPC NMME Phase 3 (NMME3) Operational User Manual

1. System Overview & Architecture

The National Multi-Model Ensemble Phase 3 (NMME3) operational system at NOAA Climate Prediction Center (CPC) automates the end-to-end ingestion, standardization, post-processing, and graphical product generation for subseasonal and seasonal climate forecasts.

1.1 Participating Forecast Models

1.2 Forecast Variables

• tmp2m: 2-meter air temperature ($^\circ\text{C}$ or $\text{K}$)
• prate: Precipitation rate ($\text{mm/day}$ or $\text{kg}\,\text{m}^{-2}\,\text{s}^{-1}$)
• tmpsfc: Sea Surface Temperature (SST / $^\circ\text{C}$)
• tmax: Daily maximum temperature ($^\circ\text{C}$)
• tmin: Daily minimum temperature ($^\circ\text{C}$)
• z200: 200 hPa geopotential height ($\text{gpm}$)

1.3 Climatology Baseline

The standard climatological normal across all models is 1991–2020 (CLIM_PERIOD = "1991_2020").

2. Environment Setup & Configuration

2.1 Key Environment Variables

2.2 Python Environment

The operational system runs on a dedicated Conda virtual environment located at:

$NMME_PROC/venv_conda

To build or refresh the environment from scratch:

cd $NMME_PROC
make -f Makefile.ops

This automatically configures Python 3.11 with numpy, pandas, xarray, netcdf4, dask, scipy, cartopy, xesmf, pygrib, click, and installs the nmme-processing CLI suite in editable mode.

2.3 Local Development Overrides

For local offline testing, place overrides in $NMME_PROC/library/local_dev_paths.env (this file is ignored by Git):

NMME_PROC=/cpc/nmme_dev/Projects/NMME3
DATA_OUT=/cpc/nmme_dev/Projects/NMME3/output
MODEL_REFORECASTS=/cpc/model_reforecasts/

3. Operational Cron Jobs (cron.txt)

All operational cron entries are managed under /situation/bin/primary.sh PYTHON to ensure cluster primary failover compatibility.

• Process Owner: Li Xu (li.xu@noaa.gov, 301-683-1548) / Johnna Infanti
• Process Backup: Ginger Zhang / Johnna Infanti
• Process Buddy: Ginger Zhang
• RFI ID: 405
• Operational Documentation Links:
• CPC NMME3 Operational Overview Slides
• CPC NMME3 Processing Architecture Doc

3.1 Cron Schedule Summary Table

3.2 Cron Job Details

Job 1: Raw Data Retrieval (cron_get_data.sh)

• Crontab: 00 06 1-7 * * and 00 18 1-7 * *
• Script: $NMME_PROC/scripts/cron_get_data.sh
• Log Files:
• Standard output: $NMME_PROC/logs/nmme_get_data.log
• Timestamped history: $NMME_PROC/logs/retrieval/cron_fetch_${YYYYMMDD}.log
• Actions:
• Activates $NMME_PROC/venv_conda.
• Runs model-specific fetchers:
  • get_CFSv2.py $yyyy $mm (runs on days 1–7)
  • get_CMC.py $rundate (retrieves CanESM5 and GEM5.2_NEMO)
  • get_CESM1.py $rundate
  • get_CCSM4.py $rundate
  • get_NASA.py $rundate
  • get_ECMWF.py $rundate (triggered on days 6 and 7)
• Scans output for errors/failures. Sends automated status email to li.xu@noaa.gov.

Job 2: NetCDF Conversion (cron_read2nc.sh)

• Crontab: 30 06 1-7 * * and 30 18 1-7 * * (30 minutes after retrieval)
• Script: $NMME_PROC/scripts/cron_read2nc.sh
• Log Files:
• Standard output: $NMME_PROC/logs/nmme_read2nc.log
• Timestamped history: $NMME_PROC/logs/retrieval/cron_read2nc_${YYYYMMDD}.log
• Actions:
• Activates $NMME_PROC/venv_conda.
• Executes $NMME_PROC/source/NMME_read2nc.py "$today".
• Formats heterogeneous raw grids into standard CF-compliant NetCDF files in: $DATA_OUT/{model}/fcst/{YYYYMM}0800/
• Emails status report with error summary.

Job 3: Single-Model Real-Time Processing (run_nmme_single_ops.sh)

• Crontab: 30 07 1-7 * * and 30 19 1-7 * *
• Script: $NMME_PROC/scripts/nmme_processing/run_nmme_single_ops.sh
• Log Files:
• Standard output: $NMME_PROC/logs/nmme-processing/cron_nmme_single_ops.log
• Execution log: $NMME_PROC/logs/nmme-processing/single_model_${YYYYMM}.log
• Actions:
• Evaluates all models: CanESM5 CFSv2 GEM5.2_NEMO NASA_GEOS5v2 NCAR_CCSM4 NCAR_CESM1 SFS_Beta ECMWF.
• Evaluates all variables: tmp2m prate tmpsfc.
• Calls nmme-check input to ensure raw forecast files and reference statistics exist before launching.
• Skips already completed combinations unless FORCE_OVERWRITE=true.
• Computes deterministic anomalies (determ) and probabilities (prob) via nmme-processing.
• For tmpsfc, verifies complete rows and computes ENSO SST plume data via nmme-export-enso.

Job 4: Multi-Model Ensemble Processing (run_nmme_multi_ops.sh)

• Crontab: 30 08 1-7 * * and 30 20 1-7 * * (1 hour after single-model processing)
• Script: $NMME_PROC/scripts/nmme_processing/run_nmme_multi_ops.sh
• Log Files:
• Standard output: $NMME_PROC/logs/nmme-processing/cron_nmme_multi_ops.log
• Execution log: $NMME_PROC/logs/nmme-processing/multi_model_${YYYYMM}.log
• Actions:
• Uses nmme-check input -m NMME -v <var> -o <output> to ensure all constituent single models have completed.
• Synthesizes constituent forecasts into the official NMME multi-model ensemble (MME) for determ and prob.
• Writes outputs to $DATA_OUT/NMME/anom/{YYYYMM}0800/ and $DATA_OUT/NMME/prob/{YYYYMM}0800/.

Job 5: Monthly Production Master Driver (driver.sh)

• Crontab: 00 08 7 * * (Monthly on the 7th at 08:00 UTC)
• Script: $NMME_PROC/scripts/driver.sh
• Log Files:
• Master log: $NMME_PROC/logs/nmme_driver.log
• Error capture: $NMME_PROC/scripts/driver.err
• Actions:
• Runs run_nmme_single_ops.sh $mm $yyyy.
• Runs run_nmme_multi_ops.sh $mm $yyyy.
• Runs Pacific calibrated probabilities via $NMME_PROC/scripts/driver_realtime_probs.sh $yyyy$mm.
• Clears previous figure logs and executes the complete plotting suite: $NMME_PROC/scripts/NMMEv3_plotting/MakeAllPlots.sh $yyyy $mm.
• Pushes figures to cpcrzdm web server via push_figs.sh.
• Greps ../logs for errors and emails a diagnostic alert to li.xu@noaa.gov.

4. CLI Utilities Reference

The NMME3 suite provides three unified command-line entry points in the conda environment:

4.1 nmme-processing (Main Processing Engine)

Executes climatology, anomaly, and probabilistic calculations.

# General Syntax
nmme-processing -y <YYYY> -init <MM> -v <VAR> -m <MODEL> -o <OUTPUT_TYPE> [--force true/false]

• Options:
• -y, --yyyy: Forecast target year (YYYY).
• -init, --init_mon: Initial forecast month (MM, e.g., 03 or 10).
• -v, --var: Variable (tmp2m, prate, tmpsfc).
• -m, --model: Target model (e.g., CFSv2, CanESM5, NMME).
• -o, --output_type: Operation type (determ, prob, stats).
• --force: Set true to overwrite existing files (default: false).

Examples:

# 1. Run single-model deterministic anomalies
nmme-processing -y 2026 -init 07 -v tmp2m -m CFSv2 -o determ

# 2. Run MME probabilistic forecast
nmme-processing -y 2026 -init 07 -v prate -m NMME -o prob

# 3. Generate reference hindcast statistics (year not required)
nmme-processing -init 07 -v tmpsfc -m CanESM5 -o stats

4.2 nmme-check (I/O Status & Gatekeeper)

Verifies input readiness or confirms whether outputs are already complete.

# Check if all required raw inputs exist before running:
nmme-check input -m CFSv2 -v tmp2m -y 2026 -init 07

# Check if multi-model ensemble dependencies are satisfied:
nmme-check input -m NMME -v tmp2m -y 2026 -init 07 -o determ

# Verify if processing output is already complete:
nmme-check output -m CanESM5 -v prate -y 2026 -init 07 -o prob

4.3 nmme-export-enso (ENSO Index Export)

Extracts SST Nino indices (Nino 1.2, 3, 3.4, 4) from tmpsfc operational forecasts:

nmme-export-enso -m CFSv2 -init 07 -clim 1991_2020

5. Directory Structure & Conventions

$NMME_PROC/
├── config.py                     # Central configuration & Paths manager
├── Makefile.ops                  # Operational environment build & package install
├── cron.txt                      # Crontab schedule configuration
├── venv_conda/                   # Active Conda virtual environment
├── scripts/                      # Operational shell drivers & wrappers
│   ├── driver.sh                 # Master monthly production release script
│   ├── cron_get_data.sh          # Cron wrapper for data ingestion
│   ├── cron_read2nc.sh           # Cron wrapper for NetCDF standardization
│   ├── driver_realtime_probs.sh  # Pacific calibrated probabilities
│   ├── push_figs.sh              # Web server rsync script
│   ├── push_ftp.sh               # Anonymous FTP distribution script
│   ├── nmme_processing/          # Ops & rerun drivers for single/multi/stats
│   │   ├── run_nmme_single_ops.sh
│   │   ├── run_nmme_multi_ops.sh
│   │   ├── rerun_nmme_single_ops.sh
│   │   └── rerun_nmme_multi_ops.sh
│   └── NMMEv3_plotting/          # Graphics and plotting suite
│       ├── MakeAllPlots.sh       # Main batch plotting orchestrator
│       └── rerun_MakeAllPlots.sh # Plotting rerun script
├── source/                       # Core python modules
│   ├── NMME_config.py            # Model lists, variables, regional bounds
│   ├── NMME_read2nc.py           # GRIB2/NC to NetCDF standardization
│   ├── get_*.py                  # Automated model fetchers (CFSv2, CMC, NCAR, etc.)
│   └── nmme_processing/          # Python package source code
├── library/                      # Static reference files, shapefiles, masks
├── logs/                         # Execution logs
│   ├── retrieval/                # Fetch and NetCDF conversion logs
│   ├── nmme-processing/          # Single and multi-model operational logs
│   └── NMME/                     # Plotting and ensemble logs
└── output/                       # Operational output repository
    ├── {MODEL}/
    │   ├── fcst/{YYYYMM}0800/    # Standardized operational forecasts
    │   ├── anom/{YYYYMM}0800/    # Deterministic anomaly NetCDF files
    │   ├── prob/{YYYYMM}0800/    # Probabilistic forecast NetCDF files
    │   ├── stats/{MM}0100/       # Hindcast climatology means & standard deviations
    │   └── enso/                 # Plume data & tabular ENSO text indices
    └── figs/                     # PNG graphic maps, probabilities, and plumes

6. Plotting Pipeline (NMMEv3_plotting)

Graphics generation is triggered either automatically by driver.sh on Day 7, or manually via MakeAllPlots.sh.

# Usage:
cd $NMME_PROC/scripts/NMMEv3_plotting
./MakeAllPlots.sh <YYYY> <MM> [force]

# Example:
./MakeAllPlots.sh 2026 07 force

Generated Products:

1. Standardized Anomalies (plotNMMEStdAnoms.py, plotNMMEStdAnoms_wSkillMask.py): Normalized by model climatological standard deviation.
2. Deterministic Anomalies (plotNMMEanoms.py): Temperature ($^\circ\text{C}$), precipitation rate ($\text{mm/day}$), and SST anomalies.
3. Probabilistic Maps (plotNMMEprobs.py, plotNMMEprobs_wDryMask.py): Tercile probabilities (Below, Normal, Above) with masked dry climatological areas.
4. Regional Subsets:
5. us (United States contiguous + Alaska)
6. usapi (US-Affiliated Pacific Islands)
7. South_America, Central_America_Caribbean, Africa, East_Asia, South_Asia, Global_land
8. ENSO Plumes (NMME_plumes.py): Forecast plumes for Nino 3.4 SST index compared with OISST observations.

7. Manual Rerun & Backfill Procedures

When upstream model transfers are delayed, network connections timeout, or a reprocessing cycle is required:

7.1 Manual Data Fetching

cd $NMME_PROC/scripts
# Fetch data for a specific date (YYYYMMDD):
./cron_get_data.sh 20260706

# Standardize to NetCDF:
./cron_read2nc.sh 20260706

7.2 Rerunning Single-Model Calculations

To force re-calculation and overwrite existing files for month MM and year YYYY:

cd $NMME_PROC/scripts/nmme_processing
./rerun_nmme_single_ops.sh 07 2026

7.3 Rerunning Multi-Model Ensemble

After all constituent models have completed:

cd $NMME_PROC/scripts/nmme_processing
./rerun_nmme_multi_ops.sh 07 2026

7.4 Backfilling Pacific Calibrated Probabilities

cd $NMME_PROC/scripts
# Target specific cycle: YYYYMM
./driver_realtime_probs.sh 202607

7.5 Rerunning Plots

cd $NMME_PROC/scripts/NMMEv3_plotting
./rerun_MakeAllPlots.sh 2026 07

7.6 Re-syncing Web Graphics

cd $NMME_PROC/scripts
./push_figs.sh

8. Operational Troubleshooting & Checklists

8.1 Verification Commands

Before escalating issues, verify state using nmme-check:

# 1. Activate environment
conda activate $NMME_PROC/venv_conda

# 2. Check single-model input completeness:
nmme-check input -m ECMWF -v tmp2m -y 2026 -init 07

# 3. Check multi-model dependencies:
nmme-check input -m NMME -v tmp2m -y 2026 -init 07 -o determ

# 4. Check ENSO text file status (detects if padded with zeros):
nmme-check output -m CFSv2 -v tmpsfc -y 2026 -init 07 -o enso

8.2 Common Error Scenarios

9. Contacts & Escalation

• Lead Developer / Process Owner: Li Xu (li.xu@noaa.gov, 301-683-1548)
• Co-Developer / Process Owner: Johnna Infanti (johnna.infanti@noaa.gov)
• Process Backup & Buddy: Ginger Zhang (ginger.zhang@noaa.gov)