Really really large zip files

[mwilliams]

Hi,

I'm trying to use vclzip to do really really large backups. At the mopment I am trying to get it to back up approx 550,000 files with a size of 20GB. A few questions:

1 Am I barking up the wrong tree?
2 What is the best way to do this. I started by building a comprehensive list of the files to backup in the vclZip FilesList object. With 550,000 files and their paths this was obviously some list to be sotring in memory. I basically had problems (though I can't specifically remember what now) and so I switched things around so that I now write the files one at a time to a filestream and then when I have built the list, I read them into the fileslist object in blocks (currently) of 10,000 at a time, zip them and continue. The filestream works file and the list gets built but when I come to zipping the application always hangs before it gets to the 1 GB point (although where it actually hangs is random).
3 Am I going about this the right way? Would it be better to build a list of the directories only to FilesList and let vclZip expand them when it needs to. Doing it this way, after it has compressed the directory, will FilesList dispose of the files listed for that directory or do they remain in memory until the whole zip job is completed.

I should say that I don't think the in memory stringslist is a problem as there is plenty of ram to spare on the server machine on which I am trying to run this utility. However, the application ram usage jumps by about 30MB when it has built the list, so I think it would generally be a good idea to reduce this.

Thanks

Mark

[Kevin]

Hi Mark,

I have a couple of suggestions for you but will have to get back to you late tonight or tomorrow with the meat of the possible solution.

Basically I think your problem has to do with memory fragmentation due to the number of files you are backing up. It does not have to do with the total size of the archive you are building, just the sheer number of files. Borland's memory manager has a real problem with this.

If you have another 3rd party memory manager you can use, that may very well fix the problem. If you don't I have a free one that I can send you that may fix the problem too. I also have a small code change that might also do the trick and the two changes together might really do the trick, but I can't say for sure.

I will get back to you with those things later today if possible, otherwise it will be early tomorrow as I am on my way out the door right now and won't be back till late tonight.

Oh, and to answer your question about whether it is better to just put directories/wildcards into the FilesList and let VCLZip expand the list.. Neither is necessarily better since the same stringlist still gets created either way. The memory problem here happens when the storage objects for each file compressed is created. Fragmentation occurs at this point.

Kevin

[Kevin]

Here are two things you can try. You can use one, the other, or both. I believe option two worked for the user that sent me that fix without using option 1.

==============================================================
Solution 1: Here is the url where you can get a zip file containing an alternative memory manager that I have found to help greatly with this. You might want to try it out, it's free and it has been around for quite awhile. I did not write it.

www.vclzip.com/multimm.zip

To use it, unzip both files into your project directory. Then open up
your project source file (not your mainform file, it must be your
project source file) and include MultiMM as the very first in your
USES list, something like:

uses
MultiMM, <-- Add this
Forms,
IncZip in '..\IncZip.pas' {IncompleteZipDlg},
...

This will cause a lot less total memory to be used when working with
archives containing very many files. If you feel more comfortable
using something that is supported I believe there are commercially
available memory managers that should accomplish the same thing.

=================================================================
Solution 2: This one I have not tested myself (though I have added it to the code which will be part of the next version) but it was sent to me by a registered user. Open KPCNTN.PAS and change the memory allocation strategy:

procedure tObjectList.Grow;
Var Delta:tOLSize;
begin
if FCapacity > 8
//then Delta := 16
then Delta := Round(Sqrt(FCapacity)) // IK 06.04.04 important for huge Zips
else if FCapacity > 4
then Delta := 8
else Delta := 4;
SetCapacity(FCapacity+Delta);
end;
================================================================

Hopefully this helps. Let me know if it does. I'd be interested to
know which option or options worked best for you. I'd especially be
curious if option 2 works by itself.

And let me know if you have any questions.

Thanks,

Kevin

[mwilliams]

Thanks for reply Kevin. Will try option 2 tonight and report back on success or not.

[Kevin]

Thanks Mark, I'll be very interested to know.

Kevin

[mwilliams]

Option 2 worked a treat. Did away with filestream and loaded 500,000+ files into the fileslist with an uncompressed size of 22GB - compressed to 17GB.

Excellent stuff. Many thanks.

[Kevin]

Glad to hear that. Thanks for getting back!

Kevin