NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
NOTICE: This version of the NSF Unidata web site (archive.unidata.ucar.edu) is no longer being updated.
Current content can be found at unidata.ucar.edu.
To learn about what's going on, see About the Archive Site.
Hi Russ, I am puzzled on how to proceed with this. You recommend that I run 4.1.3, but my problem is specifically on Windows with the Microsoft VS compiler. When I look on the netCDF Web site I find that for Windows with VS my choices are this: NOTE: User Viet Eitner has contributed a port of 4.1.1 to Visual Studio, including an F90 port to Intel Fortran. Download source (ftp://ftp.unidata.ucar.edu/pub/netcdf/contrib/win32/netcdf-4.1.1-win32- src.zip) <ftp://ftp.unidata.ucar.edu/pub/netcdf/contrib/win32/netcdf-4.1.1-win32- src.zip> or binary (ftp://ftp.unidata.ucar.edu/pub/netcdf/contrib/win32/netcdf-4.1.1-win32- bin.zip) <ftp://ftp.unidata.ucar.edu/pub/netcdf/contrib/win32/netcdf-4.1.1-win32- bin.zip> versions. This port was done before the code was refactored in 4.1.2. Or this: Two versions of the netCDF DLLs are available, for different Fortran compilers: * NetCDF for Intel and Portland Group Fortran compilers. <ftp://ftp.unidata.ucar.edu/pub/netcdf/contrib/win32/win32_vs_PGI_dll_4. 0.1.zip> * NetCDF for other Fortran compilers. <ftp://ftp.unidata.ucar.edu/pub/netcdf/contrib/win32/win32_vs_f2c_dll_4. 0.1.zip> The former is 4.1.1, and the latter are 4.0.1. What is your suggestion as to how to build with VS on 4.1.3? I do build my application with Cygwin, but I also need to build with VS, because there are many detector drivers that do not build correctly with Cygwin, they require VS. So I need to build my application with both compilers. Note that I am not using the VS development environment, I am using gnumake. But I am invoking the VS compiler and linker. Can anyone answer the question as to whether netCDF 3.6.3 with Windows VS can correctly write classic netCDF files with sizes >2GB, as Linux and Cygwin can? Thanks, Mark -----Original Message----- From: Russ Rew [mailto:russ@xxxxxxxxxxxxxxxx] Sent: Wednesday, June 29, 2011 1:34 PM To: Mark Rivers Cc: netcdfgroup@xxxxxxxxxxxxxxxx Subject: Re: [netcdfgroup] Problem with netCDF "classic" files larger than 2GB on Windows Hi Mark, > I am having a problem writing netCDF "classic" files larger than 2GB > when building with the Microsoft Visual Studio 2008 compiler. The same > code works fine when build on Linux, and on Cygwin. > > A bit of background. This is a project to build generic file writers > for cameras and detectors in the areaDetector package > (http://cars9.uchicago.edu/software/epics/areaDetectorDoc.html) for the > EPICS (http://www.aps.anl.gov/epics/) real-time control system. This is > a large project, with its own build system based on gnumake. I am > building the basic netCDF library from the same source code on all > supported platforms (Linux 32 and 64-bit, Windows 32 and 64-bit with > Microsoft compiler, Windows with Cygwin gcc compilet, vxWorks, Darwin, > Solaris, etc.). Because I have another file writer that handles HDF5, I > am using netCDF 3.6.3, since I only want to netCDF to create "classic" > files. I am using 3.6.3 because it is less complex than 4.x, not > requiring any HDF5 support, etc. I recommend you use the latest netCDF-4 release, 4.1.3. If you don't have and HDF5 library installed, it will be built without support for netCDF-4, but with bug fixes since 3.6.3 was released 3 years ago. For example, these bug fixes were mentioned in the RELEASE_NOTES for 4.1.2: Fixed two large-file bugs with using classic format or 64-bit offset format and accessing multidimensional variables with more than 2**32 values. If the problem you're seeing still occurs in releases since 4.1.2, it may be something new. --Russ > The application is typically streaming uncompressed images, using the > UNLIMITED dimension as the streaming dimension. Thus, each record is > small, only a few MB, and the file size limitations of the classic > format are not a problem. Here is an ncdump of a file header created > with this file writer on Linux: > > corvette:ADApp/op/adl>ncdump -h /home/epics/scratch/netcdf_test_1.nc > netcdf netcdf_test_1 { > dimensions: > numArrays =3D UNLIMITED ; // (4100 currently) > dim0 =3D 1024 ; > dim1 =3D 1024 ; > attrStringSize =3D 256 ; > variables: > int uniqueId(numArrays) ; > double timeStamp(numArrays) ; > byte array_data(numArrays, dim0, dim1) ; > int Attr_ColorMode(numArrays) ; > double Attr_AcquireTime(numArrays) ; > double Attr_RingCurrent(numArrays) ; > char Attr_RingCurrent_EGU(numArrays, attrStringSize) ; > double Attr_ID_Energy(numArrays) ; > char Attr_ID_Energy_EGU(numArrays, attrStringSize) ; > int Attr_ImageCounter(numArrays) ; > int Attr_MaxSizeX(numArrays) ; > int Attr_MaxSizeY(numArrays) ; > char Attr_CameraModel(numArrays, attrStringSize) ; > char Attr_CameraManufacturer(numArrays, attrStringSize) ; > > // global attributes: > :dataType =3D 1 ; > :NDNetCDFFileVersion =3D 3. ; > :numArrayDims =3D 2 ; > :dimSize =3D 1024, 1024 ; > ... > > So the only large array is called "array_data", and in this case it is > [4100, 1024, 1024], where 4100 is the unlimited dimension. Thus, this > file is over 4GB, and it can be written and read with no problems on > Linux and Cygwin. It also works fine when writing files on Windows with > the Microsoft compiler, up to file sizes of 2GB. > > However, when I try to write a file on Windows larger than 2GB using the > program built with the Visual Studio compiler I get the following error: > > Assertion failed: pxp->bf_offset <=3D offset && offset < pxp->bf_offset = > + > (off_t) pxp->bf_extent, file ..\posixio.c, line 325 > > When I look at the code at line 325 in posixio.c, I see that offset and > pdxp->bf_offset are of type off_t. I added a printf() in that code to > print the sizeof(offset) and sizeof(off_t), and it comes up as 4, not 8. > > > But when I look at the config.h file that comes in the win32/NET > directory in netCDF 3.6.3 it has the following: > corvette:areaDetector/ADApp/netCDFSrc>grep -C3 off_t > /usr/local/netcdf/netcdf-3.6.3/win32/NET/config.h > /* #undef HAVE_ST_BLKSIZE */ > > /* Define to `long' if <sys/types.h> doesn't define. */ > /* #undef off_t */ > > /* Define to `unsigned' if <sys/types.h> doesn't define. */ > /* #undef size_t */ > -- > /* The number of bytes in a size_t */ > #define SIZEOF_SIZE_T 4 > > /* The number of bytes in a off_t */ > #define SIZEOF_OFF_T 8 > > /* Define to `int' if system doesn't define. */ > > > So it defines SIZEOF_OFF_T to be 8, not 4. > > I have generated the netCDF config.h file on Linux, but then edited it > to correctly (?) define things on other platforms, like _WIN32 and > vxWorks. > > The compiler flags being used on Windows are illustrated in the > following output when I build: > > cl -c /nologo /D__STDC__=3D0 /D_CRT_SECURE_NO_DEPRECATE > /D_CRT_NONSTDC_NO_DEPRECATE /Ox /GL /W3 /w44355 > /D_WIN32_WINNT=3D0x0503 -D_FILE_OFFSET_BITS=3D64 /MT -DEPICS_DLL_NO = > -I. > -I..\\O.Common -I. -I.. -I..\\..\\..\\include\\os\\WIN32 > -I..\\..\\..\\include -IJ:\\epics\\devel\\asyn-4-17\\include > -IJ:\\epics\\devel\\calc-2-8\\include > -IJ:\\epics\\devel\\busy-1-3\\include > -IJ:\\epics\\devel\\sscan-2-6-6\\include > -IJ:\\epics\\devel\\mca-6-12-4\\include > -IJ:\\epics\\devel\\autosave-4-7\\include\\os\\WIN32 > -IJ:\\epics\\devel\\autosave-4-7\\include > -IJ:\\epics\\devel\\areaDetector-1-7beta1\\include\\os\\WIN32 > -IJ:\\epics\\devel\\areaDetector-1-7beta1\\include > -IH:\\epics\\base-3.14.12.1\\include\\os\\WIN32 > -IH:\\epics\\base-3.14.12.1\\include ..\\var.c > > There is something I don't understand here. > > Has netCDF 3.6.3 been tested to correctly write classic files > 2GB with > the Microsoft compiler? Why am I getting the assert error? > > Thanks very much, > Mark Rivers
netcdfgroup archives: