[netcdfgroup] Bug in nc4file.c

• To: "Netcdf group" <netcdfgroup@xxxxxxxxxxxxxxxx>
• Subject: [netcdfgroup] Bug in nc4file.c
• From: "Philippe Poilbarbe" <Philippe.Poilbarbe@xxxxxx>
• Date: Thu, 30 Sep 2010 14:56:58 +0200

 Hello,

I think I have found a bug in nc4file.c of NetCDF version 4.1.1.

It is the way the external file id is created (in nc_open_file and
nc_create_file functions).
The formula to obtain a new external file id (the one seen by programs
calling the library NetCDF) is:
    ++last_file_id << ID_SHIFT
where last_file_id is initialized with 0.

The problem comes when a program open/close many files, the external ids
are not reused so last_file_id grows each time a nc_open or nc_create is
called.

When last_file_id reaches 65536, the external file id is 0 (65536
shifted by ID_SHIFT (16) bits is 0).
The next call returns 65536 as an external file id which is the very
first one returned. If the first opened file is already opened there is
a confusion because files ids are reused ignoring if they have been
closed or not...

Below is a part off calling sequence done by one of our programs which
opens/close many files (there is no more than 7 files opened at the same
time).
  - CDF_OpenExt is our routine to open files (which calls nc_open)
  - CDF_NetCdf::CloseOpenFile is the one which closes the files (and
calls nc_close)
  - Handle is our internal handle
  - NcFileId is the external_file_handle returned by ncopen (written in
hexadecimal)
Here are the extracted lines for open/close written by the debug mode of
the program (prefixed by a sequence number):

171:CDF_OpenExt: Opening Bathy_1_12.nc
172:CDF_OpenExt:      Handle=19 NcFileId=00010000
173:CDF_OpenExt: Opening
CO_PR_BA_A_20100123_0000_20100111_20100121_MERC.nc, Mode=0, Flags=0x0
174:CDF_OpenExt:      Handle=1 NcFileId=00020000
175:CDF_OpenExt: Opening t01an1.nc
176:CDF_OpenExt:      Handle=2 NcFileId=00030000
177:CDF_OpenExt: Opening s01an1.nc
178:CDF_OpenExt:      Handle=3 NcFileId=00040000
187:CDF_NetCdf::CloseOpenedFile: Handle=2 NcFileId=00030000 Name=t01an1.nc
188:CDF_NetCdf::CloseOpenedFile: Handle=3 NcFileId=00040000 Name=s01an1.nc
189:CDF_OpenExt: Opening t01an2.nc
190:CDF_OpenExt:      Handle=2 NcFileId=00050000
191:CDF_OpenExt: Opening s01an2.nc
192:CDF_OpenExt:      Handle=3 NcFileId=00060000
201:CDF_NetCdf::CloseOpenedFile: Handle=2 NcFileId=00050000 Name=t01an2.nc
202:CDF_NetCdf::CloseOpenedFile: Handle=3 NcFileId=00060000 Name=s01an2.nc
..........
629897:CDF_OpenExt: Opening t02an7.nc
629898:CDF_OpenExt:      Handle=2 NcFileId=fffc0000
629899:CDF_OpenExt: Opening s02an7.nc
629900:CDF_OpenExt:      Handle=3 NcFileId=fffd0000
629916:CDF_NetCdf::CloseOpenedFile: Handle=2 NcFileId=fffc0000
Name=t02an7.nc
629917:CDF_NetCdf::CloseOpenedFile: Handle=3 NcFileId=fffd0000
Name=s02an7.nc
629918:CDF_OpenExt: Opening t02an8.nc
629919:CDF_OpenExt:      Handle=2 NcFileId=fffe0000
629920:CDF_OpenExt: Opening s02an8.nc
629921:CDF_OpenExt:      Handle=3 NcFileId=ffff0000
629937:CDF_NetCdf::CloseOpenedFile: Handle=2 NcFileId=fffe0000
Name=t02an8.nc
629938:CDF_NetCdf::CloseOpenedFile: Handle=3 NcFileId=ffff0000
Name=s02an8.nc
629939:CDF_OpenExt: Opening t03an1.nc
629940:CDF_OpenExt:      Handle=2 NcFileId=00000000
629941:CDF_OpenExt: Opening s03an1.nc
629942:CDF_OpenExt:      Handle=3 NcFileId=00010000
Here is the problem: FileId 0x00010000 returned while it is not closed
(see at the beginning), so we have two opened files with the same id.
And when we try to access the first one (Bathy_1_12.nc) in fact we are
pointing to the last one (s01an1.nc) which fortunately don't have the
same structure and we could saw the problem.

As a solution, maybe the external file handle should be reused (as in
necdf3) or, which may be simpler, the formula "++last_file_id <<
ID_SHIFT" may be evaluated while the returned number is already in use.

Philippe.

PS: Maybe there is another place to send this kind of message, tell me
if so.

-- 
Philippe Poilbarbe
CLS - Space Oceanography Division

• Follow-Ups:
  • Re: [netcdfgroup] Bug in nc4file.c
    • From: Philippe Poilbarbe