Andy,

Magic's Btrieve ISAM gateway hasn't been substantially modified since at least v9.4. AFAIK, the last real modification happened sometime in v9, to finally squash an abandoned record lock bug first identified with Magic's implementation of the Btrieve API's GetExtended() operation in v7. (You might recall hearing me chatter from decade to decade about adding "/EXTENDED=N" to the Btrieve DBMS Properties column.) That the Btrieve ISAM Gateway has not been significantly altered since way, WAY back is true, again AFAIK, all the way up to and including the Btrieve ISAM Gateway installed with Magic xpa v4.8.x. With the general release of Magic xpa v4.9, however, MSE are now shipping a brand spanking new ISAM gateway as well, and installed in addition to the legacy Btrieve ISAM Gateway. This new ISAM gateway implements Actian Zen's new Btrieve 2 API. 

Indeed a HUGE step forward, the Btrieve 2 API gateway is a version-one product with some known issues. For one, it appears to be flat-out incompatible with Zen 15's Btrieve 2 API — MSE supports the gateway with v14 only so far, but I expect they'll all get sorted out in ensuing service packs. If you opt to install the Btrieve ISAM Gateway with v4.9, you actually get two gateway DLLs, the original Btrieve gateway, now RE-named MGBtrieveAPI1.dll; and the new gateway which inherited the original gateway's file name MGBtrieve.dll.

The previous Pervasive SQL gateway remains unchanged.

Steve,

After running the program for a week (in 9.4 using Actian Zen 14) to combine the data files into one file, it crashed with File manager fatal error 132 (The file has reached its size limit).
The file has 540,764,583 records in it, which is about a quarter of the total data in need to load. According to Windows 10, the file is 268,416,352 KB (391 GB).

So where do I go from here?
If I were to use the new XPA and the new gateways would it get past this error?

By the way, a similar procedure took me about an hour to load the data to a MongoDB table/collection.
Does anyone know if the ODBC gateway from 9.4 will connect to MongoDB?

Thanks
Andy

Hello all,
I reached out to Bill Bach about this and below is his response.
Please let me know what you think of this.
Thanks
Andy

Bill's response:
While the Actian Zen v13.30 and newer engines support files up to 64TB and billions of records with ease (I have personally tested it to 600GB and over 4.5B rows), your limiting factor in this case is your development environment -- Magic itself. 

The Magic Gateway included with the older versions of Magic simply cannot handle files with more than 256GB of data (which can be less than this when compressed) or 4 billion rows.  (Not even sure if it will handle more than 2B rows.)  The latest release from January (v4.9?) does include a new Magic Gateway available that purports to support this new file format capability -- but the initial release is buggy.  Apparently, if you have any descending keys, the new Gateway chokes on it.  This issue is already known to Magic (for at least 2 months) and at last check (about a week ago), they confirmed that it is not yet fixed.

So, the magic formula (pun intended) is thus:
- Actian PSQL v13.30 or higher is required.  Zen v14/v15 recommended.
- Data file must be converted to the v13 file format.  Note that doing this on v15.10 will result in changes to the file such that it cannot be used on v15.01 or older.  As such, if you need to support older versions, you should avoid v15.10 or newer for now.
- Magic v4.9 or higher is required, and you must use the new Magic Gateway provided in this release.  Again, watch out for descending keys, which can crash it.  We don't know what other bugs may be present in this brand-new release.
- The Limit File Size configuration option inside the database engine must be unchecked.  This limit is there to prevent users NOT YET using the new Gateway from inserting more data than the application can handle.  If you leave this checked, then the engine will still return the 132 when you hit 256GB.  If you uncheck it and are still using the older Gateway, extended calls to the file will start failing, and your application will likely fail in some spectacular fashion.

If you can get to ALL of those, then you can make the file larger.  Alternatively, I could add that if you eschew the direct Magic access and instead use ODBC access ONLY for that file, then this becomes:
- Actian PSQL v13.30 or higher is required.  Zen v14/v15 recommended.
- Data file must be converted to the v13 file format.  Note that doing this on v15.10 will result in changes to the file such that it cannot be used on v15.01 or older.  As such, if you need to support older versions, you should avoid v15.10 or newer for now.
- Magic ODBC Access Only is required -- you cannot use the Magic Gateway to access this file directly. 
- The Limit File Size configuration option inside the database engine must be unchecked.  This limit is there to prevent users NOT YET using the new Gateway from inserting more data than the application can handle.  If you leave this checked, then the engine will still return the 132 when you hit 256GB.  If you uncheck it and are still using the older Gateway, extended calls to the file will start failing, and your application will likely fail in some spectacular fashion.

The Limit File Size configuration option inside the database engine must be unchecked: This setting was added to v15.  With v14, there is no such protection -- if you create a v13 file and then exceed the file size limits of the older BTRV function (and the older Magic Gateway), then your application will simply break without warning.  Actian added the extra switch to v15 after I pointed out this dangerous situation with v14.

The last response I have on the bug report is from Yuval Asheri <yuval_asheri@...> on March 12, who indicated it was still an open issue.