Command-line Yaouh! clone

[Helge Hafting]

rhn wrote:
> On Thu, 23 Apr 2009 15:45:28 +0200, Helge Hafting <helge.hafting at hist.no>
> wrote:
>> Risto H. Kurppa wrote:
>>> On Thu, Apr 23, 2009 at 4:11 PM, Helge Hafting <helge.hafting at hist.no>
>>> wrote:
>>>>> These here sound good to me to have around (in Yaouh, too)
>>>>> * skipping files modified recently
>>>> How recent would that be?
>>>> I run yaouh to get the latest tiles where I have updated the map. Of
>>>> course, this is often the same tiles as I refreshed a couple of days
>>>> earlier. So I think this has to be optional. Some kind of uses need to
>>>> check fairly new tiles.
> 
> I use the program in exactly the same way. Skipping files is not for sake
> of less transfer, but for less time spent on contacting remote servers
> (that's the same reason why I believe deleting files long unused is
> necessary). As of now, skipping files one day old is completely sufficient
> for me. Unless you do many updates every day and the tile server gets
> updated quicker than in my area.
> The value should be adjustable (including 0) in my opinion.

Sometimes yaouh crash (or the phone crash.) When restarting the 
download, avoiding checking the same stuff again would be nice. Skipping 
the most recent files will do that - another way is to have the app 
record how far it gets and restart from the same place.

> 
>>>>> * forcing download empty tiles
>>>> What does that mean?
> 
> The current version treats the empty files (size 0) left by tangoGPS as
> ones needed to download "by force", that is, not checking for the server
> modification date.
> 
>>> For example download every tile that has 2 (or 3 or four) neighbors
>>> downloaded to 'fill the gaps' in the map.
>>>
> 
> That's an interesting idea... unfortunately, the program would need to be
> rewritten because it uses recursive traversal of dictionaries now (and I
> think Yaouh! does so, too).
> 
Yaouh recurses through the directories, but python lets you control the 
order. You can have the top level first or the bottom level first.


>> Ah, excellent!
>> It'd also be nice to have an option for automatically "zooming out".
>> That is, if I have a bunch of tiles at zoom 15 (downloaded using 
>> tangogps perhaps) then I want the zoom 14 tiles covering the same area, 
>> as well as the zoom 13 tiles, and so on all the way up to the top.
>>
>> Of course, the number of tiles quickly gets lower as one goes up, and 
>> usually hits a tile that is there already after a few levels.
>>
> 
> I think this will be the next thing I implement when I get some free time.
> 
I'm looking forward to that. :-)

>> Another thing that'd be nice to have, is duplicate tile detection.
>> There are a lot of "sea" tiles, "empty land" tiles, and probably some 
>> tiles containing only forest or similiar. This could save lots of space, 
>> but not download time. It'd still be necessary to check if "empty land" 
>> suddenly isn't empty anymore.
> 
> Is it really that good? Thousands of tiles wold be necessary to make any
> real difference, even if the internal storage of a Freerunner is used. 

I have about 60.000 tiles. The blank tile has size 103. I assume that 
any tile with actual detail won't compress so well. There are 28.000 
such tiles, almost half of them.

On ext3, they actually use 4096 byte each, as they have to use whole 
blocks. That's 109MB saved from a total of 316MB.

A reiserfs system with tail packing might get closer to the 2.75MB 
actually needed.


> A  few megabytes saved when a few hundred is laying there isn't worth the
> trouble. Imagine storing modify dates for each sea tile independently when
> they are all symlinked to the same file (if you want to use the skipping
> functionality mentioned above to save time).
> 
"ls -l" shows me different timestamps for a file and for the symlink 
pointing to it. The symlink timestamp seems to be the creation time for 
the link. The file timestamp is updated when the file is updated, no 
matter what link is used to update it.

So symlinks can indeed be used. When you check a symlink and find that 
the file has changed - break the link and download the now unique file. 
(In theory, it might match another existing file, e.g. when empty land 
turns into empty forest.)

When the file isn't changed, just delete the link and recreate it, in 
order to force a new link timestamp.

Helge Hafting