Ability to pack a table while in use
Design is almost complete, implementation starts soon! :)
17 comments
-
Alex
commented
For sure, pack table could work with #TEMP tables at first place. Because in the moment those #TEMP functionality doesn't looks like complete for everyday work. In my view this functionality was clearly under priced.
-
Lev P.
commented
Could the logic be similar to FLOCK() ? Then any of the following could be performed: PACK, ZAP, REINDEX, etc., followed by an equivalent of UNLOCK.
-
billseven
commented
This would be good if it would work on free tables and dbfs as well as adts
-
Rodd Graham
commented
Should R/W ops be allowed to block during pack?
If and only if the blocked op eventually succeeds without an error returned to the client AND the pack is implemented to block for no more than a few seconds at a time.
Since ADS does not currently implement any blocked operation failures at the Sql layer besides long running transactions (we don't do those do we...), most ADS applications are not defensively programmed against Sql failures due to locking/blocking or written to provide user feedback during extraordinary delays. Even if long transactions are used, the code would know which transactions are at risk of failing unlike a PACK block which could occur against the simplest sql operation thereby requiring all operations to be defensive.
-
Lev P.
commented
Also related: allow ZAP in shared mode if there are no locks on any records.
-
Rodd Graham
commented
If the memo file has inverse pointers to the row data (recnos in the memo header), it should be scannable from bottom to top with a simple online move down compaction. No index maintenance required in this phase.
Online row data and memo data compaction should be independent with system procedures to do one or both as desired.Online indexing is more difficult and schema changes a nightmare.
-
Rodd Graham
commented
For packing, consideration might be given to just scanning from both ends of the table and moving rows from the top to the bottom with normal index operations. Assuming that concurrent inserts will use deleted rows first, when the two scans meet, the remainder of the physical table can be truncated in place.
-
Rodd Graham
commented
These (online indexing/packing/structure mods) are not 'dangerous' operations as other SQL implementation support them just fine. However they are made more complicated by the physical (non-paged) nature of the underlying ISAM.
-
tmjac2
commented
I was thinking that you could almost do it with straight SQL script. If you could see the deleted rows, inside a transaction the db could move the data from the last record to the first deleted row. It might mean 1 transaction operation per moved row. It would be slow, but should be fairly safe if a server crash happens. Granted I don't know how this works inside ADS...
-
ma1
commented
With this kind of suggestions (pack a table while in use and add indexes to tables while in use) you may risk table integrity and index integrity. Please DO NOT implement these dangerous suggestions!
-
jeremym1234
commented
It will be easier, but not as safe. If the server crashes during a backup, you don't lose any data, we store the old record data in memory, not the new data, it is already on disk. You are correct that we certainly can use some of the backup technology to address a live pack, however.
-
Clemens Maier, Dontenwill AG
commented
If you implement "live packing" with the same procedures as "online backup" uses, it should be "easy" to do the same with "live indexing" and "live altering" ?
-
jeremym1234
commented
Yeah, something like that would be possible. We would need to also try to minimize the potential for data loss.
-
joachimd
commented
I mean something different...maybe I just didn't make it clear enough:
- select a.*, newfield into newtable from oldtable
+ still allow datamodification on oldtable - but additionally keep track of it!
- disallow modification on oldtable
+ put modifications into newtable
+ rename oldtable to .bak
+ rename newtable to oldtable
- allow modifications of updated "oldtable" -
jeremym1234
commented
We could do that, but it would be prone to data loss if the server went down before in-memory changes were flushed. The backup is the opposite; it keeps OLD data in memory when tables are changed during a backup. The backup reads the old data, and if the server goes down it doesn't matter if you lose old data.
-
joachimd
commented
What about using the same procedure as online backup uses? Create the new file(s) in the background and keep data modifications in memory to apply after the new file(s) was/where created. Then have just a short outage whilst renaming the file(s).
-
Tyrolean
commented
I see this related with the wish to make a new index or reindex while a table is in use!