[netcdfgroup] Netcdf 4.1.1, C++ interface, swapn4b, invalid write in libnetcdf.so.6.0.0

• To: netcdfgroup@xxxxxxxxxxxxxxxx
• Subject: [netcdfgroup] Netcdf 4.1.1, C++ interface, swapn4b, invalid write in libnetcdf.so.6.0.0
• From: Thomas Taylor <thomas.taylor@xxxxxxxxxxxx>
• Date: Sat, 14 Aug 2010 20:14 +0200
• Followup-to: gmane.comp.lib.netcdf

Hello!

First I would like to start on a positive note, since I very much appreciate 
netcdf and its usage by AMBER, as this certainly is one of the saner file 
formats out there used for MD trajectories.

This is precisely what I use netcdf for in my program, to read AMBER created 
netcdf files. However this does not work as expected.

(1) Recently I changed from 4.0.1 to 4.1.1, where 4.0.1 worked nicely with 
the Offset64Bits flag in the NcFile constructor, now I have to use Classic, 
which appears rather strange to me. Using Offset64Bits with 4.1.1 will 
return false for is_valid() after construction.
(ncks output: Opened file emim_trif_50-1.mdcrd: dimensions = 6, variables = 
7, global atts. = 8, ID = 65536, type = NC_FORMAT_64BIT)

(2) With both 4.0.1 and 4.1.1 reproducible, but seemingly random, crashes 
occur when reading AMBER files. Running a small test program via valgrind 
gives the following:

[ OPEN FILE A
[ READ VALUES "coordinates" from 0,0,0 to 9,0,2
[ OPEN FILE B
[ READ VALUES "coordinates" from 0,0,0 to 9,0,2

[ FIRST ERROR BY valgrind (AFAIK ALWAYS after having read data from each 
file once, so probably occuring when trying to open file A for the second 
time...
Invalid write of size 1
   at 0x7D1AB65: swapn4b (in /usr/lib/libnetcdf.so.6.0.0)
   by 0x7D1E015: ncx_getn_float_float (in /usr/lib/libnetcdf.so.6.0.0)
   by 0x7D2896F: getNCv_float (in /usr/lib/libnetcdf.so.6.0.0)
   by 0x7D2E5E1: nc3_get_vara_float (in /usr/lib/libnetcdf.so.6.0.0)
   by 0x7D09589: nc_get_vara_float (in /usr/lib/libnetcdf.so.6.0.0)
   by 0x4E374B6: NcVar::get(float*, long const*) const (in 
/usr/lib/libnetcdf_c++.so.5.0.0)
   by 0x4267AD: gepetto::system::amber::netcdf::Parser::readTAX(std::string 
const&, long*, long*, float*) (parser.cpp:334)
   by 0x406EB5: calc(std::vector<std::string, std::allocator<std::string> > 
const&) (correlation.cpp:82)
   by 0x407FC9: main (correlation.cpp:179)
 Address 0x955b728 is 0 bytes after a block of size 40 alloc'd
   at 0x4C25509: operator new[](unsigned long) (in 
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
   by 0x406BBE: calc(std::vector<std::string, std::allocator<std::string> > 
const&) (correlation.cpp:49)
   by 0x407FC9: main (correlation.cpp:179)

[ READ LOTS OF MORE VALUES, always cycling through all files
[ SECOND ERROR after arbitrary number of reads/file switches, presumably 
again when trying to open one of the files...
valgrind: m_mallocfree.c:248 (get_bszB_as_is): Assertion 'bszB_lo == 
bszB_hi' failed.
valgrind: Heap block lo/hi size mismatch: lo = 112, hi = 
4717696923473706361.
This is probably caused by your program erroneously writing past the
end of a heap block and corrupting heap metadata.  If you fix any
invalid writes reported by Memcheck, this assertion failure will
probably go away.  Please try that before reporting this as a bug.

   at 0x3802B8B9: ??? (in /usr/lib/valgrind/memcheck-amd64-linux)
   by 0x3802BA88: ??? (in /usr/lib/valgrind/memcheck-amd64-linux)
   by 0x380372D8: ??? (in /usr/lib/valgrind/memcheck-amd64-linux)
   by 0x38001B19: ??? (in /usr/lib/valgrind/memcheck-amd64-linux)
   by 0x38064842: ??? (in /usr/lib/valgrind/memcheck-amd64-linux)
   by 0x38073F84: ??? (in /usr/lib/valgrind/memcheck-amd64-linux)

sched status:
  running_tid=1

Thread 1: status = VgTs_Runnable
   at 0x4C2497E: operator delete[](void*) (in 
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
   by 0x4E362ED: NcVar::~NcVar() (in /usr/lib/libnetcdf_c++.so.5.0.0)
   by 0x4E36318: NcVar::~NcVar() (in /usr/lib/libnetcdf_c++.so.5.0.0)
   by 0x4E3A312: NcFile::close() (in /usr/lib/libnetcdf_c++.so.5.0.0)
   by 0x42D1A6: gepetto::system::amber::netcdf::Parser::close() 
(parser.h:82)
   by 0x42CDF0: gepetto::system::amber::netcdf::Parser::open(std::string 
const&) (parser.h:70)
   by 0x406E74: calc(std::vector<std::string, std::allocator<std::string> > 
const&) (correlation.cpp:80)
   by 0x407FC9: main (correlation.cpp:179)


Please note that I have tried this scheme with different numbers of files, 
and with both 4.0.1 and 4.1.1, always to the same effect.
Please also note that every call to FilePointer = new NcFile(args), is 
preceded by OldFilePointer->close()

Feel free to offer any advice and suggestions, including how to turn this 
into a proper bug report.

Thank you,
Thomas Taylor

• Follow-Ups:
  • Re: [netcdfgroup] Netcdf 4.1.1, C++ interface, swapn4b, invalid write in libnetcdf.so.6.0.0
    • From: Thomas Taylor