New archive file format (was: [omgps] collect feature requests)

[Laszlo KREKACS]

On Thu, Jul 2, 2009 at 10:24 AM, Laszlo
KREKACS<laszlo.krekacs.list at gmail.com> wrote:
> Need a serious benchmark here, if the extra overhead is true or not.

Ok, I have written the python implementation of the file archive maker.
I need to finish (ie. write the unpacking part) of it.

I compiled few benchmarks...

I compressed the whole OSM maps tiles on my laptop (I repeated it 10 times):
lol at buldergep:~/Maps/OSM$ echo -e "\noutput.kiss"; time python
../../Asztal/down/openmoko/paroli/data/kiss/kiss.py >>
../report.txt;mv output.kiss ..; echo -e "\noutput.tar"; time tar -cf
../output.tar .; echo -e "\noutput.zip"; time zip -0 -r output * >>
../report.txt; mv output.zip ..; echo -e "\noutput_comp.zip"; time zip
-r output_comp * >> ../report.txt; mv output_comp.zip ..; rm
../output*; rm ../report.txt

output.kiss

real	0m4.447s
user	0m2.748s
sys	0m1.520s

output.tar

real	0m4.039s
user	0m0.236s
sys	0m1.188s

output.zip

real	0m5.556s
user	0m1.276s
sys	0m2.632s

output_comp.zip

real	0m12.438s
user	0m8.437s
sys	0m2.620s


So the speed is about the same as in .tar file case. And it beats the zip.
File sizes:
-rw-r--r-- 1 lol lol 109M 2009-07-02 19:11 output_comp.zip
-rw-r--r-- 1 lol lol 125M 2009-07-02 19:11 output.kiss
-rw-r--r-- 1 lol lol 156M 2009-07-02 19:11 output.tar
-rw-r--r-- 1 lol lol 113M 2009-07-02 19:11 output.zip

-rw-r--r--  1 lol lol  93M 2009-07-02 19:11 output.kiss.bz2
-rw-r--r--  1 lol lol  94M 2009-07-02 19:11 output.tar.bz2

Total size of invidual files:
lol at buldergep:~/Maps/OSM$ du -hs .
290M	.

Pretty strange, it reserves half the size ....

I think this file format worth the effort.

Best regards,
 Laszlo