Very large self-extracting archives?

[sbussinger]

I'm dealing with very large archives (5+ GB). I'd planned on making them self-extracting with a custom stub program, but I'm running into some problems and have a few questions:

1) It appears that self-extracting archives won't work past 2GB (due to Windows limitations). Can anyone confirm that for me? If that's not the case can someone point me to what the limitations are? I realize that FAT32 partitions have their own limitations and I'm not concerned about those.

2) Is it possible to have "blocked" self-extracting archives? Any tricks I should know about?

3) An old forum message mentions that when creating a blocked archive it makes the blocks in the temp directory and copies them to the destination. Is there any problem with setting the "TempPath" property to the same directory as the destination archive file (e.g. filename conflicts)?

Thanks for any help!

[sbussinger]

I'm experimenting with the blocked self-extracting archives and having not luck whatsoever with either MakeSFX() or MakeNewSFX() on a blocked ZIP file. Any thoughts?

Also, I can't seem to open the blocked archives using the 7-Zip GUI. Is this because of a limitation of 7-Zip, a lack of multi-volume ZIP standards or something else?

Thanks!

[sbussinger]

I tried doing a MakeNewSFX() followed by Split() and that doesn't work either.

Incidentally, the Split() routine appears to have a couple of small bugs in it. There are two calls similar to

OnFileNameForSplitPart(self, tmpZipName, CurrentDisk, spLast);

which should probably be something like

DoFileNameForSplitPart(tmpZipName, CurrentDisk, spLast);

so that DefaultFileNameForSplitPart() is used correctly in this routine.

[Kevin]

Sorry, I did not see this till late tonight. I will take a look at this tomorrow and get back.

Kevin

[Kevin]

May 3, 2009 12:51:42 GMT -5 sbussinger said:

I'm dealing with very large archives (5+ GB). I'd planned on making them self-extracting with a custom stub program, but I'm running into some problems and have a few questions:

1) It appears that self-extracting archives won't work past 2GB (due to Windows limitations). Can anyone confirm that for me? If that's not the case can someone point me to what the limitations are? I realize that FAT32 partitions have their own limitations and I'm not concerned about those.

2) Is it possible to have "blocked" self-extracting archives? Any tricks I should know about?

3) An old forum message mentions that when creating a blocked archive it makes the blocks in the temp directory and copies them to the destination. Is there any problem with setting the "TempPath" property to the same directory as the destination archive file (e.g. filename conflicts)?

Thanks for any help!

1) What "custom stub program" are you using? Whether it handles greater than 2gb or not depends on what stub you use. You can create your own stubs with Delphi as long as stubs around 300kb are not too big for you. If you create your own stubs with Delphi and VCLZip, then Zip64 format is used for archives and files greater than 2gb.

2) No, I don't think that is possible. When creating an SFX, the headers all have to be modified, and modifying a split or spanned archive is not possible.

3) No, I'm not sure which forum thread you were reading, but when creating any NEW archive, including blocked archives, the archive is created directly to its destination. The temp directory is used when modifying an archive, such as adding files to it, deleting files from it, etc.

Kevin

[Kevin]

May 3, 2009 15:16:59 GMT -5 sbussinger said:

I'm experimenting with the blocked self-extracting archives and having not luck whatsoever with either MakeSFX() or MakeNewSFX() on a blocked ZIP file. Any thoughts?

Also, I can't seem to open the blocked archives using the 7-Zip GUI. Is this because of a limitation of 7-Zip, a lack of multi-volume ZIP standards or something else?

Thanks!

Unfortunately, Self Extracting archives and split archives don't mix, as explained above.

I'm not sure what 7-zip's format is, VCLZip uses PKZip/WinZip standard for blocked archives.

[sbussinger]

1) What "custom stub program" are you using? Whether it handles greater than 2gb or not depends on what stub you use. You can create your own stubs with Delphi as long as stubs around 300kb are not too big for you. If you create your own stubs with Delphi and VCLZip, then Zip64 format is used for archives and files greater than 2gb.

My stub is a special Delphi program using VCLZip that understands my special needs and is about 1MB in size. It's working great as long as the archive doesn't get too large. It appears that there's a Windows limitation on the maximum size of an EXE that can be run (either 2GB or 4GB depending on where you read about it -- I never found a definative source). I can definitely tell you that a 5GB .EXE file just gets a "that is not a valid application" error message if you try and run it. The data can be extracted, you just can't execute it.

2) No, I don't think that is possible. When creating an SFX, the headers all have to be modified, and modifying a split or spanned archive is not possible.

That is a major bummer! I'll have to rethink how this is going to work for me a bit. I loved the simplicity of self-extracting archives for backup and recovery, but the size issues are killing me.

3) No, I'm not sure which forum thread you were reading, but when creating any NEW archive, including blocked archives, the archive is created directly to its destination. The temp directory is used when modifying an archive, such as adding files to it, deleting files from it, etc

.

The forum message I was referring to was the last message in this thread:
vclzip.proboards.com/index.cgi?board=VCLZipPro&action=display&thread=83
In re-reading it, i see that only one file at a time is compressed in the TEMP directory and not one block at a time as I first read it.

Thanks for the help!