Kazuho@Cybozu Labs

Thanks to a user of Q4M, I have found a bug that would likely lead to data corruption on 32bit versions of Q4M.  64bit versions are unaffected.

Q4M by default uses mmap(2) to read from data files.  On 32bit systems, it tries to map max. 1GB per each table into memory using mmap.  When mmap fails to map memory due to low memory, Q4M falls back to file I/O to read the data.

However there was a bug in handling the response from mmap, that led to reading corrupt data from database files when mmap(2) failed after the size of the underlying file was grown / shrunk by Q4M.  And since Q4M writes back the corrupt data into the database file when rows are being consumed, the bug will likely destroy the database files.

I have fixed the bug and have uploaded Q4M 0.9.2, into the prerelease directory at q4m.31tools.com/dist/pre.  Source tarball and prebuilt binaries for MySQL 5.1.42 for 32bit linux are available.

If you are using 32bit versions of Q4M, I highly recommend either to update to 0.9.2 or switch to 64bit versions if possible.

BTW in 0.9.2 release, I also changed the maximum mmap size per table from 1GB to 256MB, to lower the possiblity of low memory.  However this countermeasure might not be sufficient in some cases, i.e. databases having many Q4M tables or if other storage engines also used a lot of memory.  I am considering of just disabling the use of mmap(2) on 32bit systems in future releases.  If you have any comments on this, please let me know.