Springy and 7-zip archives

Dragan, posted Jul 2nd 2009 at 7:52PM

A considerable number of existing and potential users asked about the status of support for 7-zip and RAR archives in Springy. I’ve already promised to write a few words about it, but due to some other higher priority issues I haven’t be able to keep that promise so far. So, I’m just about to do it now.

Since a lot of things can be written about support for these two archive types, this story will be divided in two blog posts; the first one (the one you read) will look more closely into the 7-zip support, since things are more certain there, while the second one, which I’ll post in a couple of days, will be dedicated to the RAR support.

When including support for a new archive type in an archiving utility, one can choose one of the following paths:

1. Take a library which deals with specific archive type files, developed by someone else, which has clearly defined API and is preferably already proven and in use on the market. This approach has all the benefits; it’s quick and enables handling archived files in a proper way. Furthermore, if the library is open sourced, it can be easily modified (of course, if licensing permits that) to fulfil specific needs. This is how ZIP, TAR, PAX, CPIO, SIT, GZIP, BZIP2 and UNIX Compress are supported in Springy. More information on underlying libraries, their names and authors (big kudos to them) can be found in Springy About Panel.

2. Take the archive format specification documents (if available), sit behind your desk, take a deep breath and start developing a library yourself. The benefit is clear; you can implement anything you like in the way you like. The drawback is clear as well; it requires a lot of time and effort. And if you’re like me, developing your product not as a primary job (something else still puts bread on my table), but in your spare time just for the passion of developing it (and earning some extra cash, it never hurts), you usually don’t have that much time. Especially if the list of other features users wants you to put into your product is rather long.

3. If neither a library nor specification of the archive format are available, one can try to find a source code of the utility doing the job already. Including the source code unmodified, as for completely separate application, is usually not very effective thing to do, but if it is portable enough and if licensing allows for it, it can be modified and turned into a proper library. From then on, the path is the same as in #1. This is how RAR is supported in Springy. The support is not complete (creation and modification of RAR archives are not possible, but that’s the topic for the next blog post).

4. Finally, one can take already existing command line (CLI) utility handling desired archive type and use it as a processing engine. The application itself would be just a UI wrapper around the CLI utility, receiving users actions and handing them to the engine and present the results of processing done by the engine to the user. I’ll stay a bit more at this particular option for the obvious reason that many archiving utilities for Mac use exactly that approach and since it gives me the nice opportunity to answer question/comment/rant of some users “How come that many others support that archive type, and Springy still doesn’t?”

I have to say that I really dislike concept of an UI front-end application with CLI tool running in the background doing all heavy works. That is the reason Springy doesn’t use any such tool running in the background for archive processing (apart from system built-in hdiutil, which does disk image mounting and ejection). There are many reasons why I don’t like going this path, and here I’ll quote just what authors of Version Control with Subversion book, Ben Collins-Sussman, Brian W. Fitzpatrick, C. Michael Pilato say about that, which sums it very nicely:

Binding Directly: A Word About Correctness
Why should your GUI program bind directly with a libsvn_client instead of acting as a wrapper around a command-line program? Besides simply being more efficient, this can address potential correctness issues as well. A command-line program (like the one supplied with Subversion) that binds to the client library needs to effectively translate feedback and requested data bits from C types to some form of human-readable output. This type of translation can be lossy. That is, the program may not display all of the information harvested from the API, or may combine bits of information for compact representation.

If you wrap such a command-line program with yet another program, the second program has access only to already-interpreted (and as we mentioned, likely incomplete) information, which it must again translate into its representation format. With each layer of wrapping, the integrity of the original data is potentially tainted more and more, much like the result of making a copy of a copy (of a copy …) of a favorite audio or video cassette.

As they explained nicely about correctness issues, Springy talks for itself when it comes to efficiency; Springy, while being much richer with its UI options and views, proves to be faster in archive processing than many of numerous applications available for Mac OS X which serve just as a UI wrapper around CLI tools running in the background. There are so many things that can be done when using proper library or implementing things your own way than running a separate tool in the background. They usually lead to faster operation, lower memory consumption and most often (and most important I would add) in case of archiving utilities, much less disk read/write activity (which contributes to speed as well).

Now with this clear conclusion, let’s go back to the original topic of this post: support for 7-zip in Springy. There isn’t publicly available library that can process 7-zip archives. Implementing it from level zero myself would definitely take much, much longer than anyone would’ve wanted and been ready to wait for. My first attempt was to use LMZA SDK. Unfortunately, the source for this SDK is full of Windows-isms, and completely undocumented and non-commented. It’s not portable at all and it’s very hard to use it as a foundation of a custom made library which would handle 7-zip archives. Then, I tried my luck with 7-zip for Windows source code. The situation with that was even worse. Nearly impossible to change it and compile for some other platform, at least not in a short period of time. Then, I though that the source code of p7zip would be the solution. It’s available for many flavours of UNIX, even for Mac, so it’s portable, right? Unfortunately again, it’s not much better. It’s just the same 7-zip code with a Win32 compatibility layer compiled in alongside. The only really portable 7z code is 7z-C code, which is part of LMZA SDK distribution. But it’s very limited; it supports only extraction/decompression, no support for file attributes, split and solid archives, encrypted files and archives etc. So, I didn’t want to think of it at all.

Of course, I could’ve gone the undesired path as well; take the existing p7zip CLI utility compiled for Mac OS X (its distribution supplies make files to compile it on Mac), embed it into Springy application bundle and use it as an engine for 7z archiving/extraction with Springy as a UI wrapper. But as already concluded above, that isn’t going to happen.

So, the question remains: what is going to happen with support for 7-zip archives in Springy? The good news is: it will come! And it will come very soon! I decided to go path #2 from the above list. I took the source code of the p7zip CLI utility and started modifying it all over the place in order to make a proper library out of it. I started this long time (more than a year) ago and I still haven’t finished due to time (un)availability, other features in a queue, but most significantly, due to the p7zip code itself. It really needed a lot of rework and modifying in order to make it the way I wanted. Please, don’t get me wrong, I’m not saying the code is bad or anything similar. I just want to point out that it’s nowhere near to being easily adapted to make a library out of it. And that’s exactly what I want.

What this all mean for the users? The reading/browsing/extracting part of functionality is already finished and working beautifully. Currently I’m struggling with archiving/compressing part, which should work properly and include numerous options and archiving parameters 7-zip archive supports. I still don’t know when it will be ready, I’m working hard to finish it before the version 1.6 is released, and that should be somewhere in October. If I don’t succeed, the reading/browsing/extracting part will be included in version 1.6 anyway, so at least users will be able to open, browse and extract files from 7-zip archives. Archiving part will follow soon after. Off course, if I manage to finish it before October, the full 7-zip support (including archiving and modifying of archives) will be available in version 1.6.

I hope this post finally clarifies the status of support for 7-zip archives in Springy. I also hope it explains why it still isn’t supported and why it took me so long to make it. In the next blog post I’ll provide similar information (not that detailed though) on support for creating and modifying RAR archives using Springy.

Dragan