grb1to2.pl - grib1 to grib2 converter
Grb1to2.pl converts grib1 files to grib2. This perl script uses wgrib, wgrib2 and a few minor
programs that are included with the wgrib2 package (gmerge, smallest_grib2). Here are some
of the advantages of grb1to2.pl,
- Easier than some other solutions to compile.
Many repositories include wgrib2 executables
wgrib, gmerge and smallest_grib2 are very easy to compile C programs
- More fields converted than some other solutions. Easy to modify code for new fields.
- Does c0 packing, the smallest of 3 different complex packings.
- Complex packing algorithms produce fast-to-read grib2 files
which are comparable in size to jpeg2000 packing.
- Fast packing option which is optimized for 4-core CPUs
Disadvanges are
- Only partial ensemble metadata support
- POSIX required
- limited to lat-lon, polar stereographic, Lambert conformal, Mercator, and Gaussian grids
- other grids can be handled if the appropriate grib2 template is supplied.
- Does not convert from grib2 to grib1.
- Quits when finds a problem grid. You need to alter perl script to handle grid before continuing.
- Only simple scan mode options are handled, latlon: we:sn and we:ns, others: we:sn (needs to improved)
Requirements are
- wgrib2 with IPOLATES enabled (-new_grid option)
- wgrib
- gmerge and smallest_grib2, only needed by -fast and -packing c0 (default), supplied in the aux_progs directory of wgrib2
- perl
- POSIX2 environment
History
Grb1to2.pl is based on code that was written in 12/2009 at CPC and versions
of the code are being used at CPC. Variants of the code were written for (1) speed,
the (2) large files, (3) ease of updating and/or (4) potential operational usage.
The latest version (4/2014) had a name change (grib1to2 had been taken),
large file support was dropped (not needed with a 64-bit OS) and most importantly
installion was made easier by eliminating the need for cnvgrib. This
version retains radius of the earth from the grib1 file unlike the previous
versions.
Changes
How it works
grb1to2.gs works by
- interpolating the generic template file to the same grid as the grib1 file to make the grib2 template
- adding metadata (center, subcenter, process, shape of earth) to the template
- converting grib1 file to binary
- making a simple inventory of grib1 file (inv1)
- making a long inventory of grib1 file (inv2)
- grib1to2_metadata.pl combines inv1 and inv2 to make a metadata file (wgrib2 inv + extra fields)
- wgrib2 converts binary data + metadata + template into a grib2 file
Usage
grb1to2.pl [list of options] INPUT_GRIB1_FILE
Runtime Options:
-fast converts the file faster by running 3 streams of processing
harder to debug
-packing [c0|c1|c2|c3|j|s] set packing format for grib2 files
s = simple packing
j = jpeg2000 packing
c1 = complex packing of values
c2 = complex packing of increments
c3 = complex packing of increments of increments
c0 = best packing of c1, c2 and c3 (packing for each field is independent)
default
-ncep_uv U and V are saved to the same grib message (typical of NCEP operations)
-o OUTPUT_GRIB2_FILE output file, default is INPUT_GRIB1_FILE.grb2
-grid_template filename location of specific grib2 file (correct grid and metadata)
default for for grb1to2.pl to create the grid_template file
Configuration Options:
-gmerge filename set location of gmerge executable
-smallest_grib2 filename set location of smallest_grib2 executable
-wgrib filename set location of wgrib
-wgrib2 filename set location of wgrib2
-template filename set location of sample grib2 file
Note: if grb1to2.pl has been configured, the configuration options should have the
appropriate default values. To see the default values, type grb1to2.pl.
Example
This example uses the best compression (complex1,complex2, and complex3); however, it
requires more CPU time.
$ grb1to2.pl pgb.20190729
(((rec 1:0:date 2019072900 4LFTX kpds5=132 kpds6=1 kpds7=0 levels=(0,0) grid=2 sfc anl:ave@6hr:
4LFTX=Best (4-layer) lifted index [K]
timerange 113 P1 0 P2 6 TimeU 1 nx 144 ny 73 GDS grid 0 num_in_ave 4 missing 0
center 7 subcenter 1 process 180 Table 2 scan: WE:NS winds(N/S)
latlon: lat 90.000000 to -90.000000 by 2.500000 nxny 10512
long 0.000000 to -2.500000 by 2.500000, (144 x 73) scan 0 mode 128 bdsgrid 1
min/max data -8 22.6 num bits 9 BDS_Ref -80 DecScale 1 BinScale 0
)))
>> grib2_metadata --- pgb.20190729 wgrib=/export/cpc-lw-webisuzak/wd51we/bin/wgrib
wgrib2=/export/cpc-lw-webisuzak/wd51we/bin/wgrib2
code=3 n1 8763 n2 7347 n3 7045 : record 1, code=3 -> use complex packing 3
code=3 n1 11941 n2 10331 n3 9455 : record 2, n1 11941 -> complex packing 1 uses 11941 bytes
code=3 n1 12340 n2 10682 n3 9812 : record 3, n2 10682 -> complex packing 2 uses 10682 byres
code=3 n1 12309 n2 10629 n3 9773 : record 4, n3 9773 -> complex packing 3 uses 9773 bytes
...
code=2 n1 10794 n2 10206 n3 10381
records = 217 : record = 217 -> 217 fields converted
remove files
0 fields with unknown grib2 names set to IMGD : 0 fields had to be converted to IMGD (unknown var)
The new grib2 file is pgb.20190729.grb2.
$ wgrib2 pgb.20190729.grb2
1:0:d=2019072900:4LFTX:surface:4@6 hour ave(anl),missing=0:
2:7045:d=2019072900:ABSV:1000 mb:4@6 hour ave(anl),missing=0:
3:16500:d=2019072900:ABSV:925 mb:4@6 hour ave(anl),missing=0:
4:26312:d=2019072900:ABSV:850 mb:4@6 hour ave(anl),missing=0:
...
This example converts to complex packing 3. It uses less CPU and does not
require installation of gmerge and smallest_grib2. The grib2 file is larger
than the best of the complex packings.
$ grb1to2.pl -packing c3 pgb.20190729
(((rec 1:0:date 2019072900 4LFTX kpds5=132 kpds6=1 kpds7=0 levels=(0,0) grid=2 sfc anl:ave@6hr:
4LFTX=Best (4-layer) lifted index [K]
timerange 113 P1 0 P2 6 TimeU 1 nx 144 ny 73 GDS grid 0 num_in_ave 4 missing 0
center 7 subcenter 1 process 180 Table 2 scan: WE:NS winds(N/S)
latlon: lat 90.000000 to -90.000000 by 2.500000 nxny 10512
long 0.000000 to -2.500000 by 2.500000, (144 x 73) scan 0 mode 128 bdsgrid 1
min/max data -8 22.6 num bits 9 BDS_Ref -80 DecScale 1 BinScale 0
)))
>> grib2_metadata --- pgb.20190729 wgrib=/export/cpc-lw-webisuzak/wd51we/bin/wgrib wgrib2=/export/cpc-lw-webisuzak/wd51we/bin/wgrib2
1:0:d=2019072900:4LFTX:surface:4@6 hour ave(anl),missing=0:
1:0:d=2019072900:ABSV:1000 mb:4@6 hour ave(anl),missing=0:
1:0:d=2019072900:ABSV:925 mb:4@6 hour ave(anl),missing=0:
1:0:d=2019072900:ABSV:850 mb:4@6 hour ave(anl),missing=0:
...
1:0:d=2019072900:VVEL:100 mb:4@6 hour ave(anl),missing=0:
1:0:d=2019072900:VVEL:0.995 sigma level:4@6 hour ave(anl),missing=0:
remove files
0 fields with unknown grib2 names set to IMGD
The new grib2 file is pgb.20190729.grb2.
Old Installation: grb1to2.pl versions prior to 12/2015
The basic installation consists of wgrib, wgrib2+aux_progs and 3 new files. (See)
Wgrib2 needs to be compiled with the IPOLATES option. (See wgrib2 documentation.) You need to install
the various files in the $HOME/bin or $HOME/bin_data/ directories. ($HOME is a an environment variable
that refers to your home directory.)
- $HOME/bin/wgrib
- $HOME/bin/wgrib2
- $HOME/bin/gmerge (included with wgrib2), see grib2/aux_progs of the wgrib2 compilation directory
- $HOME/bin/small_grib2 (included with wgrib2), see grib2/aux_progs of the wgrib2 compilation directory
- $HOME/bin/grb1to2.pl
- $HOME/bin/grib1to2_metadata.pl
- $HOME/bin_data/global_template.g2
You can also modify global_template.g2 to add site
specific metadata.
New Installation: versions after 12/2015
The following files have to be placed in the same directory as grb1to2.pl. With the
12/2015 version, any account can run an installed version of grb1to2.pl.
- wgrib
- wgrib2
wgrib2 needs to be compiled with the IPOLATES option so that the -new_grid option works.
See the wgrib2 documentation.
- gmerge (included with wgrib2), see grib2/aux_progs of the wgrib2 compilation directory
- small_grib2 (included with wgrib2), see grib2/aux_progs of the wgrib2 compilation directory
- grb1to2.pl
- grib1to2_metadata.pl
- global_template.g2
Comments: Wesley.Ebisuzaki@noaa.gov, updated 1/27/2016, 11/04/2014, 12/3/2015
|