Nested Archives Handling

[moligarch]

First of all, I wanted to thank you for the library that you prepared, it is very practical and easy to use.

Currently, I was working on a project where I needed to handle nested compressed archives in two ways:
First - a method to have the content and the output address of those nested archives, which can almost be said to have been achieved by using Recursion and .extractTo along side Reader that gets buffer instead of path.

Second - I just wanted to use it to get the content list inside those archives without needing their content.

Where does the problem occur? In order to access the contents of nested archives, I must first extract it into the memory and then give that buffer to the Reader again so that I can use methods like .items() and iterate on it. This can cause a lot of memory to be used... for some reason I can't extract to disk and I have to work with memory only. Do you think there is a solution for this?

I was looking for how to extract and display the content list inside the archive by 7z, Does it do the same thing or does it get that information by reading the header file?
If it gets it by reading the header, is there a way I can use that method without having to extract the whole file?

for more information, here is part of code I wrote to get Content of Archive recursively (it's not list only, it's list and buffer together):

Content ArcSnp::ArcSnpImpl::GetContent(const std::string& filePath, size_t depthLimit)
{
	mDepthLimit = depthLimit;
	Content result{};

	bit7z::BitArchiveReader arc{ *mLib , filePath, bit7z::BitFormat::Auto };
	auto arcItems = arc.items();

	for (const auto& item : arcItems) {
		const auto itemPath{ filePath + "->" + item.path() };
		//Check the state of the item of archive for further processes like extraction, detection, etc.
		const auto itemState = CheckItem(item, arc);

		auto tmpBuffer = std::make_unique<bit7z::buffer_t>(item.size());
		if (itemState == ArcSnpState::kExtractable)
		{
			arc.extractTo(*tmpBuffer, item.index());
			//This method takes a buffer and its base path, and recursively extracts its content into a vector of Decompressed structure (Content)
			auto output = ResolveBuffer(itemPath, *tmpBuffer); 
			result.insert(result.end(), output.begin(), output.end());
		}
		else if (itemState != ArcSnpState::kNotExist &&
			itemState != ArcSnpState::kIsDirectory)
		{
			arc.extractTo(*tmpBuffer, item.index());
			result.emplace_back(Decompressed(
				{
				.msBasePath = itemPath,
				.msDepth = mDepth + 1,
				.msState = itemState },
				*tmpBuffer));
		}
	}
	return result;
}

[rikyoz]

Hi!

First of all, I wanted to thank you for the library that you prepared, it is very practical and easy to use.

You're welcome! Thank you for using it!

Where does the problem occur? In order to access the contents of nested archives, I must first extract it into the memory and then give that buffer to the Reader again so that I can use methods like .items() and iterate on it. This can cause a lot of memory to be used... for some reason I can't extract to disk and I have to work with memory only. Do you think there is a solution for this?

Unfortunately, there's no solution using bit7z, at least at the moment.
This feature was requested many times (e.g., #92), as it would definitely help in many use cases like yours.

I was looking for how to extract and display the content list inside the archive by 7z, Does it do the same thing or does it get that information by reading the header file?
If it gets it by reading the header, is there a way I can use that method without having to extract the whole file?

The main problem is that the nested archive is compressed using the outer archive format. So we need to first decompress the outer archive in order to be able to read the nested one using its own compression format.
7-Zip API doesn't make it easy to implement the extraction of nested archives, as any extraction request is basically a blocking/synchronous operation.
However, I have a few ideas on how to implement it in bit7z.
Nested archives handling will be the main new functionality I'm going to add to the next bit7z v4.1.

[moligarch]

Thank you for your explanation!

I have another question: In documentation of your library about modifying Archives (Advance usage) , I didn't find an example or usage structure.
Would you help me to understand how to modify an archives?
In my case I need to open an archive (it's better to open it from buffer), remove some item from it, and then reconstruct the buffer and compress it.

Is there a way to do so (or even similar option)?

[rikyoz]

I have another question: In documentation of your library about modifying Archives (Advance usage) , I didn't find an example or usage structure.

Yeah, I still need to add some parts of the Wiki documentation, sorry!

Would you help me to understand how to modify an archives?
In my case I need to open an archive (it's better to open it from buffer), remove some item from it, and then reconstruct the buffer and compress it.

Since the v4, bit7z provides the class BitArchiveEditor specifically for modifying existing archives.
Unfortunately, BitArchiveEditor only supports opening filesystem archives, so it might not be ideal for your use case.
I hope to remove such limitation in the future.

Anyway, here is an example of how it can be used, in case you might need it:

BitArchiveEditor editor{ lib, "path/to/archive.7z", BitFormat::SevenZip };

// Adding a new item
editor.addFile( fileBuffer, "example.txt" ); // Adding a file named 'example.txt' using a buffer for its content

// Updating an existing item within the (original) archive
editor.updateItem( "path/to/file.txt", fileBuffer2 );

// Deleting an item within the (original) archive
editor.deleteItem( "folder/compressed_file.txt" ); // path of the file to be deleted from the archive

editor.applyChanges(); // Applying the changes.
// or
editor.compressTo( outputBuffer ); // Compress the edited archive to the given buffer.

Note: the actual deletion only happens when you call the applyChanges()/compressTo method; in this way, you can chain multiple edit operations and apply them all together to the archive.