One interesting bit is downloading the access logs. Initially, I did not even realize that it's possible to download them using the appengine, but I stumbled upon it when (re-)reading the appengine docs (for the n-th time :-) ). It's actually quite convenient, but there was one issue that was giving me a headache:
We've three separate appengine deployments: The production environment, the development staging environment and the beta-testing environment. All three are deployed from the same code by simply updating the app.yaml file. It's normally set to development, only sometimes to beta and almost never to production. Now here is the catch:
When downloading the logs using appcfg.py, you have to specify the path of your app. All it does is reading the app name from app.yaml and then download the corresponding log files. However, in our case we can't really use the source directory, because it points to the development stuff 99% of the time. And accesslogs from the development server are usually quite boring.
To fix this, you can create fake app directories, only containing the app.yaml file, adjusted to the different deployment names. So we have 3 directories "fake_beta", "fake_dev", "fake_prod", each just containing the app.yaml with a different "application: ..." line. Now all we need to do is create a small shellscript that calls
appcfg.py -e $email --passin -n 0 -a request_logs fake_$name tm_$name.log < $password
for each of the deployments, where $name is its name, $email and $password are the corresponding gmail account. This can then be stuffed into a cronjob and it will download all accesslog entries ever produced, and then regularly append new ones, not re-downloading all.
Now you can stick the result into your favourite log analyzer -- or just use python and write your own :-P
-- Volker
