get_core.py 9/5/2025 The python script "get_core.py" allows you to download selected fields from the CORe (Conventional Observation Reanalysis) archive that is stored by NODD (NOAA Open Data Dissemination) on Google Cloud. The archive can be accessed by programs running on Google Cloud, or by downloading the data using https. This program allows you to select the fields for downloading. This saves time, and helps to avoid reaching the monthly quota for free downloading. The CORe archive consists of meteorological fields stored in grib v2 format. Grib is format for storing meteorological and oceanic fields sponsored by the World Meteorological Organization (WMO). There are many programs that understand grib v2, and several programs that will convert the files into other formats such as netcdf. Much of grib processing was done using wgrib2 which is freely available for Windows, Mac and Linux. https://github.com/NOAA-EMC/wgrib2 Names of Files As of 8/2025, the CORe archive consists of files with the naming structure TYPE.DATE.typ TYPE = pgb or flx DATE = YYYYMM, YYYYMMDD or YYYYMMDDHH typ = grb grib version 2 idx index file of the grb file The naming convention may be extended to include ensemble members, ensemble statistics and short range forecasts. The CORe data in NODD are stored in files with the names xxx.grb and xxx.idx. The *.idx files are index files created by $ wgrib2 xxx.grb > xxx.idx The *.idx files show the contents of the grib file. For example, 1:0:d=1950010100:PRES:mean sea level:anl: 2:128728:d=1950010100:VIS:surface:anl: 3:192074:d=1950010100:UGRD:planetary boundary layer:anl: 4:252978:d=1950010100:VGRD:planetary boundary layer:anl: 5:313928:d=1950010100:VRATE:planetary boundary layer:anl: ... Field 1: 1..N the record number (grib message) Field 2: byte location of record (grib message) starting from 0 Field 3: d=(year)(month)(day)(hour) of the analysis or start of the forecast Field 4: field name (NCEP convention) See below for more information Field 5: level or layer Field 6: timing information, anl=analysis Field 7+: extra information Types of Grib Files, pgb and flx The NODD CORe archive (at present) consists of pgb and flx files. The pgb files are atmospheric analyses created using the Unified Post Processor (UPP) or commonly called the ncep post. The flx file is directly created by the FV3GFS model and has various fluxes like the radiative, and heat fluxes. The flx file contains the land surface, soil conditions and cloud layers. Monthly means, daily means and 3 hourly output. CORe creates output every 3 hours, and these files include a YYYYMMDDHH date code in their names. The daily means files have a YYYYMMDD date code in their names. The Monthly mean files include a YYYYMM date code in their names. The "get_core.py" program requires a date code, and you use a YYYYMM date code to get the monthly, YYYYMMDD date code to get the daily and YYYYMMDDHH date code to get the 3 hourly files. Directory Structure on NODD The CORe files are stored in a bucket called noaa-nws-ncep-core. The URLs of the files are: monthly flx and pgb https://storage.googleapis.com/noaa-nws-ncep-core/grib/month/flx/$YYYY/flx.$YYYY$MM https://storage.googleapis.com/noaa-nws-ncep-core/grib/month/pgb/$YYYY/pgb.$YYYY$MM daily flx and pgb https://storage.googleapis.com/noaa-nws-ncep-core/grib/day/flx/$YYYY/$MM/flx.$YYYY$MM$DD https://storage.googleapis.com/noaa-nws-ncep-core/grib/day/pgb/$YYYY/$MM/pgb.$YYYY$MM$DD 3 hourly flx and pgb https://storage.googleapis.com/noaa-nws-ncep-core/grib/3hour/flx/$YYYY/$MM/flx.$YYYY$MM$DD$HH https://storage.googleapis.com/noaa-nws-ncep-core/grib/3hour/pgb/$YYYY/$MM/pgb.$YYYY$MM$DD$HH Using get_core.py get_core.py FILETYPE DATE NLOOP DT REGEX DIR FILETYPE = flx flx flx.idx flx index (displays on the screen) pgb pgb pgb.idx pgb index (displays on the screen) DATE = YYYYMM monthly file = YYYYMMDD daily file = YYYYMMDDHH 3 hourly file NLOOP = number of files to download/process (minimum 1) DT = integer, delta time for monthly, delta time is in months for daily, delta time is in days for 3 hourly, delta time is in hours (3 hour minimum) REGEX = regular expression all for entire file DIR = directory for the output files ex. get_core.py flx 195001 1 1 all . write flx.195001.grb to current directory ex. get_core.py flx 195001 12 1 PRATE . write flx.195001.grb, .., flx.195012.grb to current directory flx.1950xx.grb only contains the PRATE Requirements: get_core.py 1. Python v3.6+ has to be installed. 2. You are not in China because the data stored on Google Cloud. You assume all risks in circumventing local laws. 3. The first line of get_core.py may have to be changed to reflect how to invoke python v3.6+. Status Get_core.py is in the public domain. The program downloads data from the NODD archive which may under go changes that render the program non-functional. The NODD archive allows free downloads up to a monthly quota but the policies may change. The analyses from CORe are in the public domain. The are based on a data assimilation system like used for numerical weather prediction. Like weather forecasts, the results are not always accurate. The model may predict rain but you only saw clear sky. Future efforts are expect to improve the results. NWS fair usage suggests attribution which makes makes future funding easier to obtain. Good and bad feedback is welcome. Bad feedback is useful in planning future reanalyses. Regular Expressions The program, get_core.py, allows you to enter a regular expression (regex). The regex will be compared the index file, and all records that match the regex will be downloaded. Some examples using the above index file 1. regex = 'PRES' matches any line with the word PRES 2. regex = '(PRES|UGRD|VGRD)' matches any line that has the words PRES, UGRD or VGRD 3. regex = ':(200|300) mb:' matches ":200 mb:" and ":300 mb:" There are many web pages that define regular expressions. Pgb Field (Grib) Names 4LFTX Best (4 layer) Lifted Index [K] ABSV Absolute Vorticity [1/s] BRTEMP Brightness Temperature [K] note: this is equivalent to BRTMP in files created by UPP (9/2025) CAPE Convective Available Potential Energy [J/kg] CIN Convective Inhibition [J/kg] CLMR Cloud Mixing Ratio [kg/kg] CWAT Cloud Water [kg/m^2] DPT Dew Point Temperature [K] GUST Wind Speed (Gust) [m/s] HGT Geopotential Height [gpm] HLCY Storm Relative Helicity [m^2/s^2] HPBL Planetary Boundary Layer Height [m] ICAHT ICAO Standard Atmosphere Reference Height [m] ICEC Ice Cover [Proportion] LAND Land Cover (0=sea, 1=land) [Proportion] LFTX Surface Lifted Index [K] MNTSF Montgomery Stream Function [m^2/s^2] MSLET MSLP (Eta model reduction) [Pa] O3MR Ozone Mixing Ratio [kg/kg] PLI Parcel Lifted Index (to 500 hPa) [K] PLPL Pressure of level from which parcel was lifted [Pa] POT Potential Temperature [K] PRES Pressure [Pa] PVORT Potential Vorticity [K*m^2/kg/s] RH Relative Humidity [%] SPFH Specific Humidity [kg/kg] STRM Stream Function [m^2/s] TMP Temperature [K] TOZNE Total Ozone [DU] UGRD U-Component of Wind [m/s] USTM U-Component Storm Motion [m/s] VGRD V-Component of Wind [m/s] VIS Visibility [m] VPOT Velocity Potential [m^2/s] VRATE Ventilation Rate [m^2/s] VSTM V-Component Storm Motion [m/s] VVEL Vertical Velocity (Pressure) [Pa/s] VWSH Vertical Speed Shear [1/s] WEASD Water Equivalent of Accumulated Snow Depth [kg/m^2] Flx Field (Grib) Names ACOND Aerodynamic conductance [m/s] ALBDO Albedo [%] CDUVB Clear sky UV-B Downward Solar Flux [W/m^2] CNWAT Plant Canopy Surface Water [kg/m^2] CPOFP Percent frozen precipitation [%] CPRAT Convective Precipitation Rate [kg/m^2/s] CRAIN Categorical Rain [-] CSDLF Clear Sky Downward Long Wave Flux [W/m^2] CSDSF Clear Sky Downward Solar Flux [W/m^2] CSULF Clear Sky Upward Long Wave Flux [W/m^2] CSUSF Clear Sky Upward Solar Flux [W/m^2] CWORK Cloud Work Function [J/kg] DLWRF Downward Long-Wave Rad. Flux [W/m^2] DSWRF Downward Short-Wave Radiation Flux [W/m^2] DUVB UV-B Downward Solar Flux [W/m^2] EVBS Direct Evaporation from Bare Soil [W/m^2] EVCW Canopy water evaporation [W/m^2] FLDCP Field Capacity [Fraction] FRICV Frictional Velocity [m/s] GFLUX Ground Heat Flux [W/m^2] HGT Geopotential Height [gpm] HPBL Planetary Boundary Layer Height [m] ICEC Ice Cover [Proportion] ICETK Ice Thickness [m] LAND Land Cover (0=sea, 1=land) [Proportion] LHTFL Latent Heat Net Flux [W/m^2] NBDSF Near IR Beam Downward Solar Flux [W/m^2] NDDSF Near IR Diffuse Downward Solar Flux [W/m^2] PEVPR Potential Evaporation Rate [W/m^2] PRATE Precipitation Rate [kg/m^2/s] PRES Pressure [Pa] PWAT Precipitable Water [kg/m^2] SBSNO Sublimation (evaporation from snow) [W/m^2] SFCR Surface Roughness [m] SFEXC Exchange Coefficient [kg/m^2/s] SHTFL Sensible Heat Net Flux [W/m^2] SLTYP Surface Slope Type [Index] SNOD Snow Depth [m] SNOHF Snow Phase Change Heat Flux [W/m^2] SNOWC Snow Cover [%] SOILL Liquid Volumetric Soil Moisture (non-frozen) [Proportion] SOILM Soil Moisture Content [kg/m^2] SOILW Volumetric Soil Moisture Content [Proportion] SOTYP Soil Type [-] SPFH Specific Humidity [kg/kg] SSRUN Storm Surface Runoff [kg/m^2] SUNSD Sunshine Duration [s] TCDC Total Cloud Cover [%] TMP Temperature [K] TRANS Transpiration [W/m^2] TSOIL Soil Temperature [K] UFLX Momentum Flux, U-Component [N/m^2] UGRD U-Component of Wind [m/s] U-GWD Zonal Flux of Gravity Wave Stress [N/m^2] ULWRF Upward Long-Wave Rad. Flux [W/m^2] USWRF Upward Short-Wave Radiation Flux [W/m^2] VBDSF Visible Beam Downward Solar Flux [W/m^2] VDDSF Visible Diffuse Downward Solar Flux [W/m^2] VEG Vegetation [%] VFLX Momentum Flux, V-Component [N/m^2] VGRD V-Component of Wind [m/s] VGTYP Vegetation Type [Integer(0-13)] V-GWD Meridional Flux of Gravity Wave Stress [N/m^2] WATR Water Runoff [kg/m^2] WEASD Water Equivalent of Accumulated Snow Depth [kg/m^2] WILT Wilting Point [Proportion] WIND Wind Speed [m/s] Pgb Levels 0C isotherm 320 K isentropic level atmos col highest tropospheric freezing level max wind mean sea level planetary boundary layer PV=-1.5e-06 (Km^2/kg/s) surface PV=1.5e-06 (Km^2/kg/s) surface PV=-1e-06 (Km^2/kg/s) surface PV=1e-06 (Km^2/kg/s) surface PV=-2e-06 (Km^2/kg/s) surface PV=2e-06 (Km^2/kg/s) surface PV=-5e-07 (Km^2/kg/s) surface PV=5e-07 (Km^2/kg/s) surface surface top of atmosphere tropopause 0.33-1 sigma layer 0.44-0.72 sigma layer 0.44-1 sigma layer 0.72-0.94 sigma layer 0.995 sigma level 2 m above ground 10 m above ground 40 m above ground 50 m above ground 30-0 mb above ground 60-30 mb above ground 90-60 mb above ground 120-90 mb above ground 150-120 mb above ground 180-0 mb above ground 180-150 mb above ground 255-0 mb above ground 3000-0 m above ground 6000-0 m above ground 1 mb 2 mb 5 mb 10 mb 20 mb 30 mb 50 mb 70 mb 100 mb 150 mb 200 mb 250 mb 300 mb 400 mb 500 mb 600 mb 700 mb 750 mb 800 mb 850 mb 925 mb 1000 mb 80 m above mean sea level 100 m above mean sea level 305 m above mean sea level 457 m above mean sea level 914 m above mean sea level 1829 m above mean sea level 2134 m above mean sea level 2743 m above mean sea level Flx Levels atmos col boundary layer cloud layer convective cloud layer high cloud layer low cloud layer middle cloud layer surface top of atmosphere 1 hybrid pressure level 2 m above ground 10 m above ground 0-0.1 m below ground 0.1-0.4 m below ground 0.4-1 m below ground 1-2 m below ground 0-2 m below ground * These levels will be available in the real-time CORe convective cloud top level convective cloud bottom level high cloud top level high cloud bottom level high cloud top level middle cloud top level middle cloud bottom level middle cloud top level low cloud top level low cloud bottom level low cloud top level Timing Information 3 Hourly Files anl analysis 0-3 hour acc fcst accumulation of the 0-3 hour forecast 0-3 hour ave fcst average of the 0-3 hour forecast 0-3 hour max fcst maximum of the 0-3 hour forecast 0-3 hour min fcst minimum of the 0-3 hour forecast Daily Files 8@3 hour ave(anl),missing=0 average of 8 analyses every 3 hours 8@3 hour ave(0-3 hour acc fcst),missing=0 average of 8 accumulations of 0-3 hour forecasts 8@3 hour ave(0-3 hour ave fcst),missing=0 average of 8 averages of 0-3 hour forecasts 8@3 hour max(0-3 hour max fcst),missing=0 maximum of 8 maximums of 0-3 hour forecasts 8@3 hour min(0-3 hour min fcst),missing=0 minimum of 8 minimums of 0-3 hour forecasts Monthly Files 248@3 hour ave(anl),missing=0 average of 248 analyses every 3 hours (31 days) 248@3 hour ave(0-3 hour ave fcst) average of 248 forecasts of the average 3 hour value every 3 hours 31@1 day ave(8@3 hour max(0-3 hour max fcst)) average of 31 days of daily maximum forecast values from a 0-3 hour forecast 31@1 day ave(8@3 hour min(0-3 hour max fcst)) average of 31 days of daily minimum forecast values from a 0-3 hour forecast Note: max and min only apply to 2 m temperature and 2 m specific humidity Changes to the Real-Time System The real-time system has changes to the output files. The interpolation of the cloud top/bottom TMP/PRES from the model grid to the Gaussian grid was found to have a problem. This problem only affects the production of the flx files. The bad fields were removed before uploading the flx files to the NODD archive. When the NODD archive is updated in real-time, the fields will be fixed. Because of the fix, the flx files will have 11 more grib messages (records). The numbering of the fields in the the flx files will be different.