[wikireader] Fully automated generation script

[Ed Kapitein]

Hi Thomas,

Since you are available for suggestions.... :-)
Perhaps it is a good idea to use the iso country codes from [1] to
define a language.
autowiki.sh be would fetch the belgium wikipedia in that case.

Kind regards,
Ed
[1]
http://www.iso.org/iso/country_codes/iso_3166_code_lists/english_country_names_and_code_elements.htm

On Tue, 2010-02-09 at 08:28 +0100, Thomas HOCEDEZ wrote:
> (This message was first send on the wrong tree, sorry)
> 
> Hi wikireaders,
> 
> As I told before, my automated script is ready !And this script is 
> totally "cronable" :!
> It manage every step, from downloading Wikipedia archive in your 
> language to the installation. 
> 
> it is available here : http://freerunner.daily.free.fr/files/autowiki.sh
> Just modify with your ftp parameters (for ftp'ing images) & mail (to be 
> adverted on each step). then launch
> 
> $ autowiki.sh fr
> 
> for french Wikipedia
> 
> $ autowiki be
> 
> for Biello-russian Wikipedia
> 
> 
> Some of you would say that it also could check if the wikireader folder 
> exists & install or upgrade source from the git, well, I let you some 
> work to do ;-)
> 
> 
> I'm still available for any suggestion or anything else.
> 
> The code is following.
> 
> #!/bin/sh
> # Automation script for rendering a Wikipedia image for the FR
> # Written by AstHrO  / openmoko-fr.org / thomas.hocedez at free.fr
> # V 1.0
> # Todo
> #  param1 : language (fr,en,be,lu....)
> 
> # =========================
> # Ftp host configuration :
> 
> ftp_host=""
> ftp_login=""
> ftp_passd=""
> mail=""
> 
> # Folder where the wikireader tools are installed
> 
> wr_folder="/media/stocks/wikireader"
> 
> # =========================
> 
> # Language extension of the WP (fr,en, nl ...)
> if [[ "$1" = "" ]];  then
> lang="fr"
>  else
> lang=$1
> fi
> 
> 
> # You don't have to change following stuff, but you can ...
> cfile="${lang}wiki-latest-pages-articles.xml.bz2"
> ufile="${lang}wiki-latest-pages-articles.xml"
> mail_msg="message.txt"
> sep="=============================================================="
> 
> echo "---" > $mail_msg
> 
> # Going to the working folder :
> cd $wr_folder
> 
> # cleaning old treatments :
> echo $sep
> echo "Step 1 : CLeaning..."
> make clean &>/dev/null
> 
> # downloading interesting WP :
> echo $sep
> echo "Step 2 : Downloading dump ..."
> rm $cfile &>null
> wget http://download.wikimedia.org/${lang}wiki/latest/$cfile 
> ls -l $cfile | mail -s "[WR] Wikipedia dump downloaded"  "$mail" < $mail_msg
> 
> # uncompressing :
> echo $sep
> echo "Step 3 : Uncompressing ..."
> rm $ufile &>/dev/null
> bzip2 -d ./$cfile 2> log.txt &1> /dev/null
> 
> #creating some folders :
> mkdir work &>/dev/null
> mkdir image &>/dev/null
> 
> echo $sep
> echo "Step 4 : Indexing Articles..."
> # Creating index of articles :
> make index XML_FILES="$ufile" DESTDIR=image WORKDIR=work 2> log.txt 1> 
> /dev/null
> 
> # Parsing  :  (30>60 sec / 1000)
> echo $sep
> echo "Step 5 : Parsing Articles ..."
> make parse -j3  XML_FILES="$ufile" DESTDIR=image WORKDIR=work 2>log.txt
> #mail -s "[WR] Parsing of $ufile complete !"  "$mail" < $mail_msg
> 
> # Rendering the file : (60>200 sec / 1000)
> echo $sep
> echo "Step 6 : Rendering ..."
> make render -j3  XML_FILES="$ufile" DESTDIR=image WORKDIR=work 2>log.txt
> #mail -s "[WR] Rendering of $ufile complete !"  "$mail" < $mail_msg
> 
> echo $sep
> echo "Step 7 : Finalizing ..."
> # Combining articles indexes (few seconds)
> make combine -j3 DESTDIR=image WORKDIR=work > $mail_msg
> 
> # Generating a Hash (last few seconds dude !)
> make hash -j3  DESTDIR=image WORKDIR=work >> $mail_msg
> 
> # Going to output folder
> cd image
> echo $sep
> echo "Step 8 : Compressing files..."
> # Compressing  data files
> tar -cvf wr_${lang}_$(date '+%d-%m-%Y').tar.gz pedia*.*
> 
> # a little HASH to be sure ...
> shasum wr_${lang}_$(date '+%d-%m-%Y').tar.gz >  sha_${lang}.txt
> #mail -s "[WR] Hash of your file..."  "$mail" < $mail_msg
> echo $sep
> echo "Step 9 : Let's FTP all this !"
> # Let's FTP all this stuff
> ftp -n << EOF
> open $ftp_host
> user $ftp_login $ftp_passd
> binary
> put wr_${lang}_$(date '+%d-%m-%Y').tar.gz
> EOF
> echo $sep
> echo "Step 10 : Enjoy !"
> #mail -s "[WR] All done, WP image ready to use !"  "$mail"
> 
> # That's it you can now send a mail to your friends.
> 
> 
> 
> _______________________________________________
> Openmoko community mailing list
> community at lists.openmoko.org
> http://lists.openmoko.org/mailman/listinfo/community
> -- 
> Thomas HOCEDEZ
> _______________________________________________
> Openmoko community mailing list
> community at lists.openmoko.org
> http://lists.openmoko.org/mailman/listinfo/community