$Header$ -*-text-*-

netCDF Operators NCO version 5.1.2 accede to the Silicon Throne

http://nco.sf.net (Homepage, Mailing lists, Help)
http://github.com/nco (Source Code, Issues, Releases, Developers)

What's new?
Version 5.1.2 improves support for horizontal regridding, vertical
interpolation, or both, on ultra high-resolution model output.
Users of such functionality may benefit from upgrading, otherwise
this release can be skipped.

Work on NCO 5.1.3 has commenced and aims to add support for Zarr S3
stores and to polish support for new codecs.

Enjoy,
Charlie

NEW FEATURES (full details always in ChangeLog):

A. ncremap supports new options --vrt_in=vrt.nc and --ps_nm=ps_nm.
The argument to --vrt_in is a vertical coordinate file that contains
the pure or hybrid pressure information for the input data to be
vertically interpolated. Previously, this information was required
to be present in the input data file. However, SCREAM/EAMxx and
other ultra-high resolution models may choose to output the grid
information in separate files from the geophysical fields. This
can save considerable space, especially for hybrid sigma-pressure
coordinates with time-varying surface pressure. To complement the
--vrt_in option, there is now a --vrt_out option to explicitly
identify the file containing the output vertical grid. The best
practice is now to use --vrt_out (instead of --vrt or --vrt_fl,
which still work) to specify the output vertical grid and clearly
distinguish it from the input vertical grid, if any.
ncremap --vrt_out=vrt2.nc # Output vertical grid option new name
ncremap --vrt_in=vrt1.nc --vrt_out=vrt2.nc ... # Specify both grids
Thanks to Paul Ullrich (UCD) for suggesting these options.
http://nco.sf.net/nco.html#ncremap
http://nco.sf.net/nco.html#vrt_in
http://nco.sf.net/nco.html#vrt_out

B. Previously the vertical interpolation feature required that the
surface pressure field for hybrid coordinates be named PS.
Now the name of the surface pressure field can be changed using the
convenient ps_nm option. Moreover, if the argument of --ps_nm 
includes a filepath separator (slash or backslash), then the
portion preceding the final separator will be treated as a filename 
path, and the final portion will be treated as the surface pressure
variable name:
ncremap --ps_nm=ps ... # Surface pressure is ps not PS 
ncremap --ps_nm=/path/to/vrt.nc/ps ... # Use ps from vrt.nc
http://nco.sf.net/nco.html#ncremap
http://nco.sf.net/nco.html#ps_nm

C. ncclimo and ncremap support a new procedure type option, -P eamxx.
This option automatically generates ncremap flags useful for E3SM
EAMxx datasets. In addition to their typically ginormous size, these
datasets have different dimension and variable names than EAM/CAM.
Current -P eamxx causes ncremap to do two actions automatically:
(1) Permute the horizontal dimensions to be most rapidly varying by 
generating the option --pdq_opt=ilev,lev,dim2,col prior to horizontal
regridding. (2) Search for surface pressure under the name ps instead
of PS prior to vertical interpolation of hybrid-pressure grids.
ncremap --pdq_opt=ilev,lev,dim2,col # Manual dimension permutation
ncremap --ps_nm=ps # Manual surface pressure renaming from PS to ps
ncremap -P eamxx # Both options automagically
Thanks to Paul Ullrich (UCD) for suggesting these options.
http://nco.sf.net/nco.html#ncremap
http://nco.sf.net/nco.html#prc_typ

D. ncremap has changed the default behavoir of outputting the
staggered grid to finite volume (FV) destination grids from opt-out
to opt-in. This means that it is no longer necessary to specify
--no_stg_grd when regridding. Instead, it is necessary to specify
--stg_grd when the staggered grid (including variables slat and slon) 
is desired as additional information in the regridded file. 
ncremap --no_stg_grd ... # Old method of opting out
ncremap ...              # New default behavior
ncremap --stg_grd ...    # Recovers old default behavior
http://nco.sf.net/nco.html#stg_grd

E. ncremap vertical interpolation routines now recognize variables
with names T_mid and VerticalLayerMidpoint as containing temperature
and geopotential height, respectively. These variable names are used
in SCREAM/EAMxx. The default extrapolation for these variables now
behaves the same as T and Z3 for EAM/CAM datasets.
http://nco.sf.net/nco.html#vrt_ntp
http://nco.sf.net/nco.html#vrt_xtr

F. ncz2psx now has a man page.
Full documentation for ncz2psx is expected to become available
once NCO fully supports the Amazon S3 scheme for NCZarr in 2023.
http://nco.sf.net/nco.html#ncz2psx

BUG FIXES:
   
A. Some operators in NCO 5.1.1 would segfault instead of printing
usage information when called without any arguments. 5.1.2 fixes this.

B. Previous versions of ncks accidentally printed the ncgen command
to generate binary from CDL files assuming the input file was
NC_FORMAT_64BIT_OFFSET instead of its actual type. 5.1.2 fixes this.

Full release statement at http://nco.sf.net/ANNOUNCE
    
KNOWN PROBLEMS DUE TO NCO:

This section of ANNOUNCE reports and reminds users of the
existence and severity of known, not yet fixed, problems. 
These problems occur with NCO 5.1.1 built/tested under
MacOS 13.0.1 with netCDF 4.9.1-RC2 on HDF5 1.12.2 and with
Linux with netCDF 4.8.1 on HDF5 1.8.19.

A. NOT YET FIXED (NCO problem)
   Correctly read arrays of NC_STRING with embedded delimiters in ncatted arguments

   Demonstration:
   ncatted -D 5 -O -a new_string_att,att_var,c,sng,"list","of","str,ings" ~/nco/data/in_4.nc ~/foo.nc
   ncks -m -C -v att_var ~/foo.nc

   20130724: Verified problem still exists
   TODO nco1102
   Cause: NCO parsing of ncatted arguments is not sophisticated
   enough to handle arrays of NC_STRINGS with embedded delimiters.

B. NOT YET FIXED (NCO problem?)
   ncra/ncrcat (not ncks) hyperslabbing can fail on variables with multiple record dimensions

   Demonstration:
   ncrcat -O -d time,0 ~/nco/data/mrd.nc ~/foo.nc

   20140826: Verified problem still exists
   20140619: Problem reported by rmla
   Cause: Unsure. Maybe ncra.c loop structure not amenable to MRD?
   Workaround: Convert to fixed dimensions then hyperslab

KNOWN PROBLEMS DUE TO BASE LIBRARIES/PROTOCOLS:

A. NOT YET FIXED (netCDF4 or HDF5 problem?)
   Specifying strided hyperslab on large netCDF4 datasets leads
   to slowdown or failure with recent netCDF versions.

   Demonstration with NCO <= 4.4.5:
   time ncks -O -d time,0,,12 ~/ET_2000-01_2001-12.nc ~/foo.nc
   Demonstration with NCL:
   time ncl < ~/nco/data/ncl.ncl   
   20140718: Problem reported by Parker Norton
   20140826: Verified problem still exists
   20140930: Finish NCO workaround for problem
   20190201: Possibly this problem was fixed in netCDF 4.6.2 by https://github.com/Unidata/netcdf-c/pull/1001
   Cause: Slow algorithm in nc_var_gets()?
   Workaround #1: Use NCO 4.4.6 or later (avoids nc_var_gets())
   Workaround #2: Convert file to netCDF3 first, then use stride
   Workaround #3: Compile NCO with netCDF >= 4.6.2

B. NOT YET FIXED (netCDF4 library bug)
   Simultaneously renaming multiple dimensions in netCDF4 file can corrupt output

   Demonstration:
   ncrename -O -d lev,z -d lat,y -d lon,x ~/nco/data/in_grp.nc ~/foo.nc # Completes but produces unreadable file foo.nc
   ncks -v one ~/foo.nc

   20150922: Confirmed problem reported by Isabelle Dast, reported to Unidata
   20150924: Unidata confirmed problem
   20160212: Verified problem still exists in netCDF library
   20160512: Ditto
   20161028: Verified problem still exists with netCDF 4.4.1
   20170323: Verified problem still exists with netCDF 4.4.2-development
   20170323: https://github.com/Unidata/netcdf-c/issues/381
   20171102: Verified problem still exists with netCDF 4.5.1-development
   20171107: https://github.com/Unidata/netcdf-c/issues/597
   20190202: Progress has recently been made in netCDF 4.6.3-development
   More details: http://nco.sf.net/nco.html#ncrename_crd

C. NOT YET FIXED (would require DAP protocol change?)
   Unable to retrieve contents of variables including period '.' in name
   Periods are legal characters in netCDF variable names.
   Metadata are returned successfully, data are not.
   DAP non-transparency: Works locally, fails through DAP server.

   Demonstration:
   ncks -O -C -D 3 -v var_nm.dot -p http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc # Fails to find variable

   20130724: Verified problem still exists. 
   Stopped testing because inclusion of var_nm.dot broke all test scripts.
   NB: Hard to fix since DAP interprets '.' as structure delimiter in HTTP query string.

   Bug tracking: https://www.unidata.ucar.edu/jira/browse/NCF-47

D. NOT YET FIXED (would require DAP protocol change)
   Correctly read scalar characters over DAP.
   DAP non-transparency: Works locally, fails through DAP server.
   Problem, IMHO, is with DAP definition/protocol

   Demonstration:
   ncks -O -D 1 -H -C -m --md5_dgs -v md5_a -p http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc

   20120801: Verified problem still exists
   Bug report not filed
   Cause: DAP translates scalar characters into 64-element (this
   dimension is user-configurable, but still...), NUL-terminated
   strings so MD5 agreement fails 

"Sticky" reminders:

A. Reminder that NCO works on most HDF4 and HDF5 datasets, e.g., 
   HDF4: AMSR MERRA MODIS ...
   HDF5: GLAS ICESat Mabel SBUV ...
   HDF-EOS5: AURA HIRDLS OMI ...

B. Pre-built executables for many OS's at:
   http://nco.sf.net#bnr

