Modeler's Introduction to NetCDF

The Goal

So you have a model and want it to write NetCDF? Here's the page I wish I had when I started out.

Stuff you absolutely need to know if you're new to NetCDF

Sometimes there is such a thing as too much information. Trying to do simple tasks with NetCDF, as far as I can tell, is a typical instance. This page is an attempt to solve that, for a specific audience. This page is for you if you are writing your own code in C and want to write NetCDF so you can use the numerous available tools
A lot of what follows applies to FORTRAN, but of course, the specific example doesn't.
In particular, don't try to read the C programmer's manual yet. (Here's the PDF .) It's written, for the most part, for the general case of reading and processing NetCDF files. That is, it is aimed at the tool writer. If you are still here, you aren't ready for that yet. You just want to write a file that your tools can use.
It's actually fairly easy, if a bit messy. I couldn't find a complete simple example, so I made one. See below for my simple example.
So what IS NetCDF anyway? NetCDF is a way of storing data and the description of the data in the same file. If you know what XML is, you can think of NetCDF as achieving essentially the same purposes as XML, for gridded rather than heirarchical data.
The structure of the file is 1) metadata and 2) data. Simple enough.
The metadata is a list of the data fields and their dimensions, along with optional attributes. Most attributes go to a particular data field, and include such things as a short name (for plots) a long name (for documentation) and a set of units (for combining data from various sources).
The metadata also may contain global attributes, which should describe the dataset as a whole. (The name and build version of the model that crveated the data would go here in the case of model output.) It's very easy to forget to update this in development cycles, so it's a good idea to enforce updates of the global metadata somehow in your make script.
The data is in rectangular grids, with one optional "unlimited dimension". Typically, the "unlimited dimension" is a time axis. Conceivably, it could be a set of realizations of a single data set.
In practice you might want to write out a separate file for each time slice for a first pass. You can easily concatenate the files in postprocessing. However, for production code you should probably concatenate the slices on the fly.
The library provides routines for writing header information, and for writing data. In practice, there is a code generation tool that you can use to do most of the work for you. Read on.
The approach I am advocating here requires you to know the simple relationship between CDL and NetCDF. CDL is the (ASCII) human-readable version of the format. (Usually you will only be interested in CDL for metadata, except in debugging.) You can go back and forth between these two formats using the tools ncdump and ncgen. See below.

Getting familiar with NetCDF

Start by getting some NetCDF files. If you're doing climate work, you should learn how to navigate the IRI website anyway, so poke around there starting onthe IRI data library tutorial and find yourself a NetCDF dataset that interests you. I don't find the site or the tutorial to be models of clarity, either, but that's for another day. Here's a nice, useful one: Mean NCEP SLP; click on the "NetCDF" option. You will get the rather foolishly named "data.cdf"; rename it to, say, "ncep-mslp.nc"
Next understand how to use Ferret. Unlike NetCDF itself, Ferret, which is a nice NetCDF display program, also has an excellent, well-thought-out tutorial, here. If you have ferret set up properly you can also probably run this interactively, which is somehow more useful. Just enter
```
go tutorial
```
at the Ferret command prompt once you have Ferret working. Look at the SLP data you just downloaded
Another valuable NetCDF tool is ncdump We'll talk about it more below. For now, you may want to look at the metadata for some NetCDF data that you have. To see a human-readable description, enter
```
ncdump -h [filename] | more
```

Writing CDF the (sort of) easy way

Part I: Know what CDL is

Begin by looking at the CDL format. Understanding CDL incrementally is not hard, though the descriptions in the C manual are impenetrable. Here are two records of nine values in cdl:
```
netcdf magicsquare {
dimensions:
	x = 3,
	y = 3;
variables:
	float sqr1(x,y);
	int sqr2(x,y);
data: 
	sqr1 = 8.0,1.0,6.0,3.0,5.0,7.0,4.0,9.0,2.0;
	sqr2 = 8,1,6,3,5,7,4,9,2;
}
```
warning: If you mess around writing cdl by hand (which you will if you use the method I describe here), be sure to put the dimensions in brackets for each variable. This is not always caught as a syntax error and causes segmentation faults.
What the above CDL code says is I have two 3x3 arrays, one of integers and one of floats. (They both happen to be the simplest magic square, where all rows, columns and diagonals add up to 15.)
Once you understand the above (easy enough), the best place I've found to understand more details NetCDF format is the COARDS format pages. They don't describe the format explicitly, but as far as I can tell nothing actually does.
The main thing to understand that isn't fairly obvious is "attributes", which may be global (the entire NetCDF file) or variable-specific. Here's an example of a variable-specific attribute:
```
float xdim;
	xdim:units = "furlong";
```
which tells postprocessors that x is dimensioned in furlongs.
Here is an example of a global attribute:
```
	:Conventions="COARDS"
```
which represents a claim that the whole file attempts to follow COARDS standards. Both of these attribute definitions come in the data section of the CDL.

Part II: Understanding ncgen and ncdump

A valuable NetCDF tool is ncdump
You can get the CDL (human-readable form) for an existing NetCDF dataset by issuing
```
ncdump [filename]
```
Or, to just get the header information (typically more useful)
```
ncdump -h [filename]
```
You will probably want to redirect the output to a file, which you will probably want to give the extension ".cdl". For instance:
```
ncdump -h ncep-mslp.nc > ncep-mslp.cdl
```
ncgen reads CDL, (which is sometimes generated by ncdump). It optionally creates a cdf file or a source file.
If the CDL code above is saved as magsqr.cdl, the following pair of instructions returns essentially the original stuff:
```
ncgen -o magsqr.nc magsqr.cdl
ncdump magsqr.nc
```

Part III: generating header files

unfortunately, the library calls to build a NetCDF file are quite lengthy, if not especially interesting or difficult. However, do not despair!
OK, here's the interesting part: As emphasized above, ncgen can also generate your source code for you! Writing the CDL is all you need to do. For example:
```
ncgen -c magsqr.cdl > magsqr.c
```
returns this source file in magsqr.c . This is C source that would write exactly the file you started with!
You can either hand-generate the CDL, or you can use ncdump on a NetCDF file of a similar format to the one you want to make.
In practice, you will not want to put a data section in the CDL you hand-generate.
Write the data from your model. The easiest way uses
```
nc_put_var_THING
```
where THING is a recognized data type. See the example below, or the automatically generated code here for example, or see the C programmer's manual section 7.6 for more details. (Fancier write methods are shown in sections 7.5 - 7.9 .)
As mentioned above, it is not practical (as far as I know) to write multiple time slices on the fly from a model. You will want to write files with one time slice each. If you want to combine everything into one NetCDF file, do it offline with ncrcat

Part IV: Understand how to write a record

This is the easy way to write a record of floats:

When you automatically generated your code, it essentially produced a netCDF handle (here, ncid):

   int stat = nc_create("c5.nc", NC_CLOBBER, &ncid);

and a NetCDF tag for each variable (here, drom_id):

  drom_dims[0] = x_dim;
  drom_dims[1] = y_dim;
  stat = nc_def_var(ncid, "drom", NC_FLOAT, RANK_drom, drom_dims, &drom_id);
  check_err(stat,__LINE__,__FILE__);

With these two things defined, you can now write your array:
nc_put_var_float(ncid,drom_id,&(drom[0][0])); check_err(stat,__LINE__,__FILE__);
The above writes the float array where it goes. When you have written all your fields as above you This is the only code you have to write by hand!

Note, There's comparable routines for nc_put_var_type where "type" is one of byte short int float double . If you prefer you can use real as a synonym for float.
Note, in case you didn't know (I didn't) __LINE__ and __FILE__ are system constants provided by the preprocessor. This facilitates reporting of runtime errors.

Part V: OK, now you're ready

Here are the four quasi-simple steps you need to take to splice NetCDF writes into your model the easy way:

Write a CDL file by hand, or get one from a comparable file to the one you want to write using ncdump -h
Run it through ncgen -c to get the endless setup code automatically.
Splice the headers and the executable into your code appropriately
Write the data fields as needed following the model in bold above.

See the example below and follow along, keeping the four steps in mind.

Is it worth it?

If you want to use the tools that analyze and display NetCDF, yes. If your output is real science that you intend to keep around and publish, yes. If you're just messing around, probably not.

A Simple Example (which is something that, as far as I can tell you can't find anywhere else).

What Gives

I'm a fan of code generation in general. I don't think humans should write tedious code at all. So I'm using the ncgen tool here.

However, let me emphasize that the case below is simple enough that you can follow it as a prototype. You don't need to use ncgen if you prefer writing C directly. That said, I can't resist asserting that this is a perfect example of a case where using a code generator is a big win.

Step 0: Pre-existing model or analysis code.

Here I build a couple of arrays. (Ignore the variable names, they are for my own peculiar amusement.)

float drom[4][4];
float bact[4][4];

main()
{
int i,j;

printf("hello\n");

for (i=0;i<4;++i)
        for (j=0;j<4;++j)
        {
                drom[i][j] = 10. * i + j;
                bact[i][j] = 10. * j + i;
        }
}

Step 1: I build this CDL file to match

I call it c5.cdl

netcdf camelsrc{
dimensions:
        x = 4,
        y = 4;
variables:
        real drom(x,y);
        real bact(x,y);
}

Step 2: Generate code

ncgen -c c5.cdl > c5src.c

This gives me the automatically generated c code:

/* Include standard and NetCDF libraries */

#include <stdio.h>
#include <stdlib.h>
#include <netcdf.h>

/* Error handling routine */

void
check_err(const int stat, const int line, const char *file) {
    if (stat != NC_NOERR) {
	   (void) fprintf(stderr, "line %d of %s: %s\n", line, file, nc_strerror(stat));
        exit(1);
    }
}

int
main() {			/* create camelsrc.nc */

/* Set up all sorts of tags and handles. Notice there is no actual data in this program! This just creates the CDL header. */

   int  ncid;			/* netCDF id */

   /* dimension ids */
   int x_dim;
   int y_dim;

   /* dimension lengths */
   size_t x_len = 4;
   size_t y_len = 4;

   /* variable ids */
   int drom_id;
   int bact_id;

   /* rank (number of dimensions) for each variable */
#  define RANK_drom 2
#  define RANK_bact 2

   /* variable shapes */
   int drom_dims[RANK_drom];
   int bact_dims[RANK_bact];

/* create the NetCDF handle, akin to opening the file */

   
   /* enter define mode */
   
   int stat = nc_create("camelsrc.nc", NC_CLOBBER, &ncid);
   check_err(stat,__LINE__,__FILE__);

/* declare the variables */

   
   /* define dimensions */

   stat = nc_def_dim(ncid, "x", x_len, &x_dim);
   check_err(stat,__LINE__,__FILE__);
   stat = nc_def_dim(ncid, "y", y_len, &y_dim);
   check_err(stat,__LINE__,__FILE__);

   /* define variables */

   drom_dims[0] = x_dim;
   drom_dims[1] = y_dim;
   stat = nc_def_var(ncid, "drom", NC_FLOAT, RANK_drom, drom_dims, &drom_id);
   check_err(stat,__LINE__,__FILE__);

   bact_dims[0] = x_dim;
   bact_dims[1] = y_dim;
   stat = nc_def_var(ncid, "bact", NC_FLOAT, RANK_bact, bact_dims, &bact_id);
   check_err(stat,__LINE__,__FILE__);

   /* leave define mode */

   stat = nc_enddef (ncid);
   check_err(stat,__LINE__,__FILE__);

/* now close the file and exit */

   stat = nc_close(ncid);
   check_err(stat,__LINE__,__FILE__);
   return 0;
}

Step 3: Write the code to write the fields

I'm writing two fields:

   nc_put_var_float(ncid,drom_id,&(drom[0][0]));
   check_err(stat,__LINE__,__FILE__);
   
   nc_put_var_float(ncid,bact_id,&(bact[0][0]));
   check_err(stat,__LINE__,__FILE__);

The above is the only new code I wrote by hand (not counting the original model)!

Step 4: Now splice it all together.

KEY:
Original file (black) Autogenerated code (red) New code to write variables and close file (blue)


#include <stdio.h>
#include <stdlib.h>
#include <netcdf.h>

float drom[4][4];
float bact[4][4];

void
check_err(const int stat, const int line, const char *file) {
    if (stat != NC_NOERR) {
	   (void) fprintf(stderr, "line %d of %s: %s\n", line, file, nc_strerror(stat));
        exit(1);
    }
}


main()
{
int i,j;

for (i=0;i<4;++i)
	for (j=0;j<4;++j)
	{
		drom[i][j] = 10.*i + j;
		bact[i][j] = 10.*j + i;
	}

   int  ncid;			/* netCDF id */

   /* dimension ids */
   int x_dim;
   int y_dim;

   /* dimension lengths */
   size_t x_len = 4;
   size_t y_len = 4;

   /* variable ids */
   int drom_id;
   int bact_id;

   /* rank (number of dimensions) for each variable */
#  define RANK_drom 2
#  define RANK_bact 2

   /* variable shapes */
   int drom_dims[RANK_drom];
   int bact_dims[RANK_bact];

   /* enter define mode */
   int stat = nc_create("camelsrc.nc", NC_CLOBBER, &ncid);
   check_err(stat,__LINE__,__FILE__);

   /* define dimensions */
   stat = nc_def_dim(ncid, "x", x_len, &x_dim);
   check_err(stat,__LINE__,__FILE__);
   stat = nc_def_dim(ncid, "y", y_len, &y_dim);
   check_err(stat,__LINE__,__FILE__);

   /* define variables */

   drom_dims[0] = x_dim;
   drom_dims[1] = y_dim;
   stat = nc_def_var(ncid, "drom", NC_FLOAT, RANK_drom, drom_dims, &drom_id);
   check_err(stat,__LINE__,__FILE__);

   bact_dims[0] = x_dim;
   bact_dims[1] = y_dim;
   stat = nc_def_var(ncid, "bact", NC_FLOAT, RANK_bact, bact_dims, &bact_id);
   check_err(stat,__LINE__,__FILE__);

   /* leave define mode */
   stat = nc_enddef(ncid);
   check_err(stat,__LINE__,__FILE__);



   /* now write them */
   
   nc_put_var_float(ncid,drom_id,&(drom[0][0]));
   check_err(stat,__LINE__,__FILE__);
   
   nc_put_var_float(ncid,bact_id,&(bact[0][0]));
   check_err(stat,__LINE__,__FILE__);



   /* close the file */
      
   stat = nc_close(ncid);
   check_err(stat,__LINE__,__FILE__);

}

Here's a tarball of the whole kaboodle

tarball

More Things About NetCDF that Might Prove Useful

NetCDF has altogether too many supporting programs. Here is a collection of some of them. Don't ask me which ones are useful, unless you decide to pay me to figure it out. I actually think the number of items here is symptomatic of a problem in contemporary science, but that's for another time and place.
The official NetCDF C user's guide is useful in an overwhelming sort of way. It's really a reference manual, not a user's guide. If somebody had written a proper user's guide I wouldn't have spent a couple of days scratching this together.
This intro to NetCDF may also prove useful, though in some ways it is a bit specific for a particular installation.
You may find this effort at a user-friendly introduction helpful, but perhaps it goes too far in the direction of friendliness, as opposed to, er, actual use. (Flash 6 required.)
I did find this, which is actually a helpful simple example. However, it calls an older netcdf library so will not work out of the box.
In the same way that Javascript has nothing whatsoever to do with Java, NetCDF is not the same thing as CDF which is a different data format. Unfortunately, Many NetCDF files have a ".cdf" extension. An alternative is to use ".nc", which is, after all, much more in the spirit of metadata.
You may wish to comply with the CF standards for data formats. (This is a sort of NetCDF dialect.)
The CF standards attempt to supercede the COARDS standards. One advantage of CF is the existence of a CF validator (a Python-based CGI, interestingly). Thanks to Rob Jacob for this link.

Comments

Dessert

I needed a code-to-html filter to keep these notes going. Here it is.

My home page such as it is.