Re: [netcdfgroup] Problem with netCDF "classic" files larger than 2GB on Windows

To: Mark Rivers <rivers@xxxxxxxxxxxxxxxxx>
Subject: Re: [netcdfgroup] Problem with netCDF "classic" files larger than 2GB on Windows
From: Christoph Gohlke <cgohlke@xxxxxxx>
Date: Thu, 07 Jul 2011 20:10:04 -0700



On 7/7/2011 2:25 PM, Mark Rivers wrote:

Hi Christoph,

Thanks very much, that indeed fixed the problem!  When I first tried
your lines below I got compiler errors, but then I looked at your
version of config.h and saw that you also need to #include io.h and
process.h before the #defines.  Thus my config.h now contains:

/* The size of `off_t', as computed by sizeof. */
#if defined(vxWorks)
   #define SIZEOF_OFF_T 4
#elif defined(_WIN32)
   #include<io.h>
   #include<process.h>
   #define SIZEOF_OFF_T 8
   #define lseek _lseeki64
   #define off_t __int64
   #define _off_t __int64
   #define _OFF_T_DEFINED
#else
   #define SIZEOF_OFF_T 8
#endif

Thanks again for the help!

It's interesting that netCDF 3.6.3 contains a config.h in the win32/
directory, but it does not contain the above fixes, so it does not work
for creating files>2GB.

I also noted that your config.h file contains the following

/* I added the following to this config.h file by hand, after being
abducted by
    aliens last week in Kansas. (All Hail Zorlock, Mighty Destroyer of
Worlds!) */
#include<io.h>
#include<process.h>
#define lseek _lseeki64
#define off_t __int64
#define _off_t __int64
#define _OFF_T_DEFINED
#define snprintf sprintf_s
#define strcasecmp _stricmp

So did the aliens teach you this trick, or where did you learn it?

I can't take credit for that. The config.h file was actually part ofnetcdf-4.0.1 but was removed around netcdf-4.1.1 along with the VisualStudio project files. I just kept using and adjusting those files for myneeds.


Christoph


Cheers,
Mark

-----Original Message-----
From: netcdfgroup-bounces@xxxxxxxxxxxxxxxx
[mailto:netcdfgroup-bounces@xxxxxxxxxxxxxxxx] On Behalf Of Christoph
Gohlke
Sent: Tuesday, July 05, 2011 5:46 PM
To: netcdfgroup@xxxxxxxxxxxxxxxx
Subject: Re: [netcdfgroup] Problem with netCDF "classic" files larger
than 2GB on Windows

Try to compile with 64 bit off_t and lseek. E.g. in config.h

#define SIZEOF_OFF_T 8
#define lseek _lseeki64
#define off_t __int64
#define _off_t __int64
#define _OFF_T_DEFINED

Your test_big_classic.c works for me. The config.h I'm using with msvc9
is attached.

Christoph


On 7/5/2011 3:13 PM, Mark Rivers wrote:

Hi Russ and Ed,

Thanks for your replies.

I have now upgraded from netCDF 3.6.3 to 4.1.3 as you suggested.

I have reproduced the problem in a simple test program, which I have
attached. The test program creates a netCDF file of the following

format

as revealed by ncdump:

corvette:areaDetector/ADApp/netCDFSrc>ncdump -k

/home/epics/scratch/test.nc


classic

corvette:areaDetector/ADApp/netCDFSrc>ncdump -h

/home/epics/scratch/test.nc


netcdf test {

dimensions:

numArrays = UNLIMITED ; // (4100 currently)

YSize = 1024 ;

XSize = 1024 ;

variables:

byte array_data(numArrays, YSize, XSize) ;

}

So it writes a 1024x1024 byte array to the file 4100 times, where 4100
is the unlimited dimension. This tests writing files that are larger
than both the 2GB and 4GB potential limits.

The test program runs fine on Linux and Cygwin.

However, when compiled with the Microsoft Visual Studio 2008 compiler,
it fails with exactly the same error I got in netCDF 3.6.3, except it

is

now in line 342 rather than 325 of posixio.c.

writing record 2047/4100

writing record 2048/4100

Assertion failed: pxp->bf_offset<= offset&&  offset<  pxp->bf_offset

(off_t) pxp->bf_extent, file

..\..\..\ADApp\netCDFSrc\libsrc\posixio.c,

line 342

By way of background on why I need to use the Microsoft compiler:

I run a fairly large NSF-funded user facility at the Advanced Photon
Source at Argonne National Laboratory. We collect data using a wide
variety of x-ray and other detectors under a distributed real-time
control system called EPICS. Some of these detectors can be controlled
from Linux and Cygwin, but many of them have vendor-supplied drivers
that can only be called from code compiled with the Microsoft

compiler.

I also need to build my application to run on the vxWorks and RTEMS
real-time operating systems, which we use to control specialized VME
hardware. These use the gcc compiler, but cross-compiling for the
real-time target on a Linux host.

I am building with gnumake, not the Visual Studio application
development environment on all platforms, including when building with
the Microsoft VC++ compiler on Windows.

I have also attached a copy of the config.h file I am using, which has
been modified from the version built by "configure" on Linux to do
things differently if _WIN32 or vxWorks are defined.

I had to make some minor changes to the 4.1.3 source code in order to
allow it to build with the Microsoft VS 2008 compiler, and to build on
vxWorks. I will send those changes in a separate e-mail.

I have also attached an IDL version of the same program. It should be
completely equivalent to the C program. It works fine on both Linux

and

on Windows. It appears to me that the idl_netcdf.dll file that ITT
provides with IDL on Windows is built with the Microsoft compiler
(judging from dumpbin /imports), but I am not certain of this. If so,

it

indicates that it is possible to write netCDF classic files when
compiling with Visual Studio, and I am probably just doing something

wrong.


Perhaps it is in my flags to the compiler. Here is the output when I
compile posixio.c as an example of the flags I am using:

J:\epics\devel\areaDetector\ADApp\netCDFSrc>make install.win32-x86

make -C O.win32-x86 -f ../Makefile TOP=../../.. T_A=win32-x86 install

make[1]: Entering directory
`J:/epics/devel/areaDetector/ADApp/netCDFSrc/O.win32-x86'

cl -c /nologo /D__STDC__=0 /D_CRT_SECURE_NO_DEPRECATE
/D_CRT_NONSTDC_NO_DEPRECATE /Ox /GL /W3 /w44355 /D_WIN32_WINNT=0x0503
-DHAVE_CONFIG_H /MT -DEPICS_DLL_NO - I. -I..\\O.Common - I. -I..
-I..\\..\\..\\ADApp\\netCDFSrc\\include
-I..\\..\\..\\ADApp\\netCDFSrc\\libsrc
-I..\\..\\..\\ADApp\\netCDFSrc\\libdispatch
-I..\\..\\..\\ADApp\\netCDFSrc\\liblib

-I..\\..\\..\\include\\os\\WIN32

-I..\\..\\..\\include -IJ:\\epics\\devel\\asyn-4-17\\include
-IJ:\\epics\\devel\\calc-2-8\\include
-IJ:\\epics\\devel\\busy-1-3\\include
-IJ:\\epics\\devel\\sscan-2-6-6\\include
-IJ:\\epics\\devel\\mca-6-12-4\\include
-IJ:\\epics\\devel\\autosave-4-7\\include\\os\\WIN32
-IJ:\\epics\\devel\\autosave-4-7\\include
-IJ:\\epics\\devel\\areaDetector-1-7beta1\\include\\os\\WIN32
-IJ:\\epics\\devel\\areaDetector-1-7beta1\\include
-IH:\\epics\\base-3.14.12.1\\include\\os\\WIN32
-IH:\\epics\\base-3.14.12.1\\include
-I..\\..\\..\\ADApp\\netCDFSrc\\include
..\\..\\..\\ADApp\\netCDFSrc\\libsrc\\posixio.c

posixio.c

..\..\..\ADApp\netCDFSrc\libsrc\posixio.c(433) : warning C4018: '<' :
signed/unsigned mismatch

..\..\..\ADApp\netCDFSrc\libsrc\posixio.c(441) : warning C4018: '<=' :
signed/unsigned mismatch

..\..\..\ADApp\netCDFSrc\libsrc\posixio.c(449) : warning C4018: '<=' :
signed/unsigned mismatch

..\..\..\ADApp\netCDFSrc\libsrc\posixio.c(454) : warning C4018: '>' :
signed/unsigned mismatch

Thanks,

Mark

-----Original Message-----
From: Russ Rew [mailto:russ@xxxxxxxxxxxxxxxx]
Sent: Wednesday, June 29, 2011 1:34 PM
To: Mark Rivers
Cc: netcdfgroup@xxxxxxxxxxxxxxxx
Subject: Re: [netcdfgroup] Problem with netCDF "classic" files larger
than 2GB on Windows

Hi Mark,

  I am having a problem writing netCDF "classic" files larger than 2GB

  when building with the Microsoft Visual Studio 2008 compiler. The

same

  code works fine when build on Linux, and on Cygwin.

  A bit of background. This is a project to build generic file writers

  for cameras and detectors in the areaDetector package

  (http://cars9.uchicago.edu/software/epics/areaDetectorDoc.html) for

the

  EPICS (http://www.aps.anl.gov/epics/) real-time control system. This

is

  a large project, with its own build system based on gnumake. I am

  building the basic netCDF library from the same source code on all

  supported platforms (Linux 32 and 64-bit, Windows 32 and 64-bit with

  Microsoft compiler, Windows with Cygwin gcc compilet, vxWorks,

Darwin ,

  Solaris, etc.). Because I have another file writer that handles

HDF5, I

  am using netCDF 3.6.3, since I only want to netCDF to create

"classic"

  files. I am using 3.6.3 because it is less complex than 4.x, not

  requiring any HDF5 support, etc.


I recommend you use the latest netCDF-4 release, 4.1.3. If you don't

have and HDF5 library installed, it will be built without support for

netCDF-4, but with bug fixes since 3.6.3 was released 3 years ago. For

example, these bug fixes were mentioned in the RELEASE_NOTES for

4.1.2:


Fixed two large-file bugs with using classic format or

64-bit offset format and accessing multidimensional

variables with more than 2**32 values.

If the problem you're seeing still occurs in releases since 4.1.2, it

may be something new.

--Russ

  The application is typically streaming uncompressed images, using

the

  UNLIMITED dimension as the streaming dimension. Thus, each record is

  small, only a few MB, and the file size limitations of the classic

  format are not a problem. Here is an ncdump of a file header created

  with this file writer on Linux:

  corvette:ADApp/op/adl>ncdump -h /home/epics/scratch/netcdf_test_1.nc

  netcdf netcdf_test_1 {

  dimensions:

  numArrays =3D UNLIMITED ; // (4100 currently)

  dim0 =3D 1024 ;

  dim1 =3D 1024 ;

  attrStringSize =3D 256 ;

  variables:

  int uniqueId(numArrays) ;

  double timeStamp(numArrays) ;

  byte array_data(numArrays, dim0, dim1) ;

  int Attr_ColorMode(numArrays) ;

  double Attr_AcquireTime(numArrays) ;

  double Attr_RingCurrent(numArrays) ;

  char Attr_RingCurrent_EGU(numArrays, attrStringSize) ;

  double Attr_ID_Energy(numArrays) ;

  char Attr_ID_Energy_EGU(numArrays, attrStringSize) ;

  int Attr_ImageCounter(numArrays) ;

  int Attr_MaxSizeX(numArrays) ;

  int Attr_MaxSizeY(numArrays) ;

  char Attr_CameraModel(numArrays, attrStringSize) ;

  char Attr_CameraManufacturer(numArrays, attrStringSize) ;

  // global attributes:

  :dataType =3D 1 ;

  :NDNetCDFFileVersion =3D 3. ;

  :numArrayDims =3D 2 ;

  :dimSize =3D 1024, 1024 ;

...

  So the only large array is called "array_data", and in this case it

is

  [4100, 1024, 1024], where 4100 is the unlimited dimension. Thus,

this

  file is over 4GB, and it can be written and read with no problems on

  Linux and Cygwin. It also works fine when writing files on Windows

with

  the Microsoft compiler, up to file sizes of 2GB.

  However, when I try to write a file on Windows larger than 2GB using

the

  program built with the Visual Studio compiler I get the following

error:

  Assertion failed: pxp->bf_offset<=3D offset&&  offset<

pxp->bf_offset =

  (off_t) pxp->bf_extent, file ..\posixio.c, line 325

  When I look at the code at line 325 in posixio.c, I see that offset

and

  pdxp->bf_offset are of type off_t. I added a printf() in that code

to

  print the sizeof(offset) and sizeof(off_t), and it comes up as 4,

not 8.

  But when I look at the config.h file that comes in the win32/NET

  directory in netCDF 3.6.3 it has the following:

  corvette:areaDetector/ADApp/netCDFSrc>grep -C3 off_t

  /usr/local/netcdf/netcdf-3.6.3/win32/NET/config.h

  /* #undef HAVE_ST_BLKSIZE */

  /* Define to `long' if<sys/types.h>  doesn't define. */

  /* #undef off_t */

  /* Define to `unsigned' if<sys/types.h>  doesn't define. */

  /* #undef size_t */

--

  /* The number of bytes in a size_t */

  #define SIZEOF_SIZE_T 4

  /* The number of bytes in a off_t */

  #define SIZEOF_OFF_T 8

  /* Define to `int' if system doesn't define. */

  So it defines SIZEOF_OFF_T to be 8, not 4.

  I have generated the netCDF config.h file on Linux, but then edited

it

  to correctly (?) define things on other platforms, like _WIN32 and

  vxWorks.

  The compiler flags being used on Windows are illustrated in the

  following output when I build:

  cl -c /nologo /D__STDC__=3D0 /D_CRT_SECURE_NO_DEPRECATE

  /D_CRT_NONSTDC_NO_DEPRECATE /Ox /GL /W3 /w44355

  /D_WIN32_WINNT=3D0x0503 -D_FILE_OFFSET_BITS=3D64 /MT -DEPICS_DLL_NO

-I.

  -I..\\O.Common - I. -I.. -I..\\..\\..\\include\\os\\WIN32

  -I..\\..\\..\\include -IJ:\\epics\\devel\\asyn-4-17\\include

  -IJ:\\epics\\devel\\calc-2-8\\include

  -IJ:\\epics\\devel\\busy-1-3\\include

  -IJ:\\epics\\devel\\sscan-2-6-6\\include

  -IJ:\\epics\\devel\\mca-6-12-4\\include

  -IJ:\\epics\\devel\\autosave-4-7\\include\\os\\WIN32

  -IJ:\\epics\\devel\\autosave-4-7\\include

  -IJ:\\epics\\devel\\areaDetector-1-7beta1\\include\\os\\WIN32

  -IJ:\\epics\\devel\\areaDetector-1-7beta1\\include

  -IH:\\epics\\base-3.14.12.1\\include\\os\\WIN32

  -IH:\\epics\\base-3.14.12.1\\include ..\\var.c

  There is something I don't understand here.

  Has netCDF 3.6.3 been tested to correctly write classic files>  2GB

with

  the Microsoft compiler? Why am I getting the assert error?

  Thanks very much,

  Mark Rivers




_______________________________________________
netcdfgroup mailing list
netcdfgroup@xxxxxxxxxxxxxxxx
For list information or to unsubscribe,  visit:

http://www.unidata.ucar.edu/mailing_lists/

References:
- [netcdfgroup] Problem with netCDF "classic" files larger than 2GB on Windows
  - From: Mark Rivers
- Re: [netcdfgroup] Problem with netCDF "classic" files larger than 2GB on Windows
  - From: Russ Rew
- Re: [netcdfgroup] Problem with netCDF "classic" files larger than 2GB on Windows
  - From: Mark Rivers
- Re: [netcdfgroup] Problem with netCDF "classic" files larger than 2GB on Windows
  - From: Christoph Gohlke
- Re: [netcdfgroup] Problem with netCDF "classic" files larger than 2GB on Windows
  - From: Mark Rivers

2011 messages navigation, sorted by:
1. Thread
2. Subject
3. Author
4. Date
5. ↑ Table Of Contents
Search the netcdfgroup archives: