News

Wgrib2 development has transitioned from a makefile build to a github/CMake build. This allows future wgrib2 development to be integrated into a the software build/maintanence system that is used by NCEP and the NOAA R&D HPC systems. The current "develop" wgrib2 has the major functionality of wgrib2 v3.1.3. (Ubuntu: IP=yes, AEC=yes, OpenJPEG=yes, PNG=no, wgrib2lib=coming, jasper=soon, proj4=no). Some improvements: -new_grid handles WMO and NCEP rotated lat-lon grids with one version, PDT=49 upgrade, Making a libwgrib2 for python is doable but not automatic.

Different
1. CMake build vs gnu makefile
2. build does not include independent library sources
3. questions, bug reports go to the github site
4. moving development into EMC/NCEP mainstream

Bad
1. equal area Lambert projection on aspherical earth is not supported (to do)
2. cannot check compatibiliity for your local library
3. CMake support for unusual platforms vs gnu make?
4. learning curve to compile.
5. Updating documentation requires issue/pull request/test the code/review.

Good
1. future maintenence will be easier
2. support by multiple people
3. CMake is a commonly used build system.
4. CMake was designed to build on wide variety of platforms
5. Potential outside developers are familiar with github.
6. SPack support.
7. documentation is version specific.

At this point, we need feedback. The install documentation is weak. IF you find something is broken, add it to the issues so it can be fixed and a test added.

Introduction

Wgrib2 is a processor for grib2 files. It is a utility and library for manipulating grib files, The utility was designed to be used to reduce the need for custom Fortran programs to read, write and manipulate grib files. Wgrib2 has the following abilities.

• inventory and read grib2 files
• create subsets
• create regional subsets by cookie cutter or projections
• export to ieee, text, binary, CSV, netcdf and mysql
• import to ieee, text, binary, and netcdf
• write of new grib2 fields
• parallel processing by using threads (OpenMP)
• parallel processing by flow-based programming

Wgrib2 is versatile because it's command line is a simple language. This makes wgrib2 useful in embedding. Some programs that embed wgrib2.

• g2ctl a control maker for GraADS
• atl_g2ctl, alt_gmp alternative versions of g2ctl/gribmap for for GraADS
• grib-filter/g2subset cgi-bin programs for Nomads (NOAA OperationalModel Archive & Distribution System)
• g2grb.gs enables GrADS to write grib2 files
• grb1to2.pl grib1 to grib2 converter
• Xplane11 flight simulator
• rNOMADS R interface for NOAA weather data

OpenMP is used to speed up wgrib2 by running loops over multiple cores. OpenMP is enabled by default for most builds and you can slow up your machine if you the wrong OpenMP configuration.

The first configuration is to set up the number of threads that wgrib2 will use. The default configuration is set the number of threads equal to the number of physical cores. The default is reasonable except for when your computer becommes becomes unresonsive. For a 4 core CPU, I set the number of threads to be 3, so I another core to handle other work.

export OMP_NUM_THREADS=3

The next OpenMP configuration is how to handle unused threads (cores). By setting OMP_WAIT_POLICY to PASSIVE, unused cores are made availible to to work on other tasks. Setting OMP_WAIT_POLICY to ACTIVE, will not share the cores. In kindergarten, you learned the value of sharing.

export OMP_WAIT_POLICY=PASSIVE

News

4/14/2022: wgrib2 v3.1.1 is released

Changes for wgrib2 v3.1.1
• fixes check_pdt_len for some ECMWF and ICON files with vertial coordinates
• fixes -unix_time because of glibc
• more support for unix time: -set_date, -import_netcdf
• tested: gcc/gfortran on Ubuntu 20.04, Redhat 7
• tested: icc/ifort on Redhat 7
• tested: AOCC v3.2 (clang, flang) on Ubuntu 20.04
• minor tested: Windows 10 using cygwin64
• minor tested: gcc/gfortran on Ubuntu 18.04

Usage

• each option corresponds to a subroutine call
• type wgrib2 to see primary options
• inventory format is specified on command line by options
• if no "inv" option is specified, -s is used

Joining the development effort

• Source code: knowledge of C and some grib-2
• Changes to existing source code has to use the same licence as the original code
• New code (files) must either be GNU or public domain.
• Github? No.
• Bug reports are important
• Some inovations by first time contributors
  • first implementation of lat-lon of common grids
  • write netcdf files
  • callable wgrib2 (making wgrib2 a subroutine)
  • write to mysql files
  • AEC compression
  • python interface
• Contact wesley.ebisuzaki@noaa.gov

Contributions by

• Wesley Ebisuzaki: many modules
• Reinoud Bokhorst: tosubmsg, checksum
• DWD (Gregor Schee, Daniel Lee and others): AEC compression
• Jaakko Hyvätti: gribtab
• John Howard: callable wgrib2
• Dusan Jovic: staggered grids, proj4 code
• Kristian Nilssen: netcdf module
• Karl Pfeiffer: georeferencing
• Pablo Romero: unix_time
• Manfred Schwarb: many modules
• Arlindo da Silva: openGrADS, bbox
• Niklas Sondell: mysql module
• Sam Trahan: satellite tables
• George Trojan: python interface, improvements to wgrib2api
• Sergey Varlamov: netcdf module improvements, georeferencing updates
• thanks to the people who report the bugs and more who provide the fixes!

Documentation

• problems-error message
• Slides for talk to CUNY students 6/2016: grib1/grib2
• intro_grib2.pdf using GFS fcst
• Understanding the default inventory
• converting from wgrib to wgrib2 in scripts
• Powerpoint presentation 4/6/2011 at NCEP
• Changes in wgrib2 from 5/2012 to 1/2015 (pdf)
• submessages
• writing a simple function
• selecting fields to decode
• option types: init inv out misc
• bin, ieee, text format
• small fast databases
• usage questions
• using pipes and multiprocessing
• compile questions
• limitations
• make wgrib2 faster
• calling wgrib2 from C
• user defined grib tables
• special files
• some tricks
• some tricks for NCEP users
• one-line tricks

The options

• the common options
• all the commands

Some solutions

• Time interpolation of two grib files
• interpolation to new grid: new_grid, Lat-Lon nearest neighbor, Lat-Lon by Cressman
• CSV (comman separated value) file: -csv, -spread
• Windows_10 version of wgrib2
• Fast averaging

Selecting Fields/Records/Message

Select/Ignore by name/level/time/etc

• -match process records that match a posix extended regular expression
• -not process records not matching a regular expression
• -match_inv the inventory used by -match and -not
• -i reads inventory from stdin for record selection

Select by number (better to use previous method)

• -d dump specific record
• -for select a range of records to process (nth message)
• -for_n select a range of records to process (nth message/submessage)

Selective Processing: if options

After selecting the fields to process, you can refine the processing by the "if" options. With wgrib2 v3.0.0, the IF structure was improved. Scripts that used the older IF structure still work; however, the new IF structure is easier to read.

If possible, it is better to use the match options than the if options. The -match/-not options prevent unwanted records from being processed which saves time over the if options which process the fields.

Individual Grid Point Data

• -ij print value at grid point (i, j)
• -ijlat print lat, lon, value at grid point (i, j)
• -ilat print lat, lon, value of Nth grid point
• -lon prints the lat-lon, and value of the grid point nearest the specified lat-lon
• -max prints the maximum value
• -min prints the minimum value
• -stats prints some statistics about the fields
• -V verbose inventory (shows stats)

Regridding, Interpolating to new grids

Wgrib2 has the ability to convert grib files from one grid to another. The conversion uses a user-selected interpolation scheme: bilinear (default), bicubic, nearest neighbor, budget, and spectral. The supported grids include lat-lon, gaussian, Lambert conformal, polar stereographic, and WMO-defined rotated lat-lon grid. This capability uses the NCEP IPOLATES2 library and is an optional package.

Note: the interpolation uses scalar and vector interpolation schemes. For the vector quantities the V field must immediately follow the corresponding U field.

• new_grid
• -new_grid_interpolation set interpolation type used by -new_grid
• wgrib2m fast regrid using multiple processes

Exporting data to other programs

• -netcdf: write data in netcdf format
• -mysql: export data to a mysql database
• -mysql_speed: export data to a mysql database
• -spread: write data for spreadsheets
• -csv: write in column separated values, another one for spreadsheets
• -text: data in text format
• -bin: data native binary floating point
• -ieee: data in big endian IEEE format
• -ijbox: write a rectangular grid of data
• -AAIG: arcinfo ascii grid, GIS

Writing grib2

1. from wgrib2 command line
2. from GrADS using g2grb.gs
3. fron python using write
4. from fortran using grb2_write

Machines able to run wgrib2

64-bit with IPOLATES
• Redhat 7 Enterprise: gcc/gfortran (primary development system)
• Redhat 7 Enterprise: gcc/gfortran, icc/ifortran
• SUSE Enterprise: gcc/gfortran, icc/ifortran
• Ubuntu 20.04: gcc/gfortran (primary development system)
• Ubuntu 20.04: AOCC's clang and flang (development system) with OpenJPEG
• Ubuntu 20.04: nvidia with OpenJPEG
• ARM: needed to be compiled with USE_NETCDF4=1, USE_JASPER=0 (old report)
• Redhat linux: 32-bit with IPOLATES, not tested recently, use netcdf4 (old report)
• Mandriva linux (old report)
• AIX: use makefile, some fiddling with libraries is necessary, not tested recently
• Solaris, needs gnu make and gcc (old report)
• Solaris-10 (old report)
• HPUX, needs changes to makefiles (old report)
• Windows: using Cygwin system produces 32-bit binaries (old report)
• Windows: using Cygwin system produces 64-bit binaries
• Windows: compiled MingW (not recent), Watcom C, icc/ifort (old report)
• Windows/linux subsystem (ubuntu): compiled with gcc/gfortran (old report)
• Intel-based Mac using gcc and gfortran

System dependencies: 32 vs 64 bit, big vs little endian, Windows vs Linux/UNIX

The wgrib2 source code is written to be portable; there are no issues with big vs little endian or the size of the integer as long as it is 32 bits or more. The source code is written in ANSI/ISO C (C89), with optional features that require POSIX or POSIX-2. There has been a debate about moving the base rquirements to C99.

• big vs little endian: either works
• 32-bit machine: files limited to 2GB files, netcdf3 may not work
• Windows: 2GB+ files with 64-bit cygwin, 2GB with other windows C compilers
• Windows: only 64-bit cygwin is supported
• POSIX: all POSIX code is optional. Regex support is POSIX and useful.
• JPEG2000: OpenJPEG library or Jasper library

Source Code and Compling Hints

The wgrib2 source code is written to the POSIX-2 standard. Features requiring POSIX2, such as regular expressions, can usually be turned off in the makefile. The wgrib2 code can be compiled with 32 or 64 bit pointers and integers. However, the code has to be compiled in a like manner for all the libraries. Some packages are optional (netcdf, mysql) and enabling these options can really increase the executable size.

While compiling wgrib2, you may see warnings about unknown pragmas. Pragmas are "comments" that are used when compiling an OpenMP version of the code. The default is to compile with OpenMP turned on.

2/2019: The HDF5 library has problems being compiled with newer gcc compilers. The only consisten method of adding NetCDF4 to wgrib2 is by using old compilers (RedHat 6) or using the Intel compilers. The work around to enable NetCDF3 support and use an external utility to convert the files to Netcdf4.

NetCDF3 has problem being compiled on 32-bit machines.

• Source code
• _README
• Change log
• compiling directions and questions
• wgrib2.tgz
• OS-X installation by author of rNOMADS using gcc and OS-X

Precompiled code from External Sites

There are many sites with precompiled versions of wgrib2. This list is neither exhaustive nor an endorsement of the sites. I have not tested the wgrib2 executables from these sites and YMMV.

• OpenGrads: Darwin, Freebsd, Linux, Windows(cygwin)
• Fedora Project
• RPMs: Centros, Fedora, OpenSUSE, RedHat, SUSE
• MacPorts

Status

The wgrib2 is used operationally in the NCEP production suite.

Change logfile

Changes

Prior to wgrib2 v3.1.0, chemical and aerosol names were from the NCO web table for Code Table 4.30. For wgrib2 v3.1.0+, the names are from the WMO names for CCT14. Expect differences. For example, "Total Aerosol" became "Total aerosol", and "Carbon Monoxide" became "Carbon monoxide". If you use for "Carbon ]mM]onoxide", it will work with both -if and -match.

Bugs

Please report bugs to wesley.ebisuzaki@noaa.gov. When you report bugs, try to make them reproducible on a linux machine and include sample data.

The long time practice was not to port bug fixes to older versions of wgrib2. As of June 23, 2023, the official policy is not to backport bug fixes to older versions of wgrib2, especially versions that are over 4 year old, and 5 public releases ago. As of June 23, 2023, a recent CMake is necessary to compile some default library/libraries used by wgrib2. Wgrib2 can be compiled without the libraries and avoiding the use of CMake.

Distribution

The source code modules for wgrib2 are either in the public domain or under the GNU licence depending on the authors of the various modules. Wgrib2 uses libraries that are in the public domain, under various GNU licences, the Image Power JPEG-2000 Public Licence*, libpng licence*, the zlib licence*, the netcdf licence*, HDF5 licence*, MySQL licence* and perhaps others. How about one licence to rule them all?

* optional package