Another attempt at speeding up yaouh, with persistent connections

Helge Hafting helge.hafting at hist.no
Mon Mar 2 20:54:53 CET 2009


My last attempt at speeding up yaouh was not only clunky to use, it also
drove the cpu load up to 12-15, which made the phone extremely sluggish
during the update.

This time I took a different approach that keeps the load in the 0.3-0.5
range.

There is no more parallelism than what you find in standard yaouh v. 0.4,
most of the speedup comes from avoiding network latency. Standard yaouh uses
curl to fetch the checksum for one file at a time. My approach have curl
downloading all the files in a directory in one operation. This helps a lot,
because curl will use persistent connections instead of making a new
connection per file. Not having to start a new process for each file helps
too. It looks like openstreetmap can have up to 50 files in a directory.

The same trick is used for md5sum, as md5sum also is capable of processing
several files in one go. This avoid some process startup time as well.

Finally, files that aren't up-to-date are downloaded with curl instead of
wget. This way one invocation can download all stale tiles in a directory in
one go, again utilizing persistent connections. 

So far this program has checked 5100 files in 12 minutes, and downloaded
260. It looks like it will get through all my 50.000 tiles in under 2 hours.
I use wifi, the usb connection may be slower.

If anyone want to try, here is the download link.
http://www.aitel.hist.no/~helgehaf/openmoko/yaouh_new.py

Install yaouh v 0.4 from opkg.org if you don't already have it, so that all
dependencies are in place. My modified script should then work.

Helge Hafting
-- 
View this message in context: http://n2.nabble.com/Another-attempt-at-speeding-up-yaouh%2C-with-persistent-connections-tp2411310p2411310.html
Sent from the Openmoko Community mailing list archive at Nabble.com.





More information about the community mailing list