Google App Engine Logs - The Undocumented “—include_all” Flag

It’s no secret that we run a lot of the zLocation platform code on Google App Engine. GAE is great, and for the most part we love it. But, it can definitely get a little frustrating at times. The latest thing that caused a lot of frustration on our side was trying to download the detailed GAE application logs. Well, let’s just say that we wanted to pull the app logs for some offline analysis.  To go into a little more detail, we needed to run SPLUNK on the logs to generate some basic metrics (e.g. API response times).

So, Google’s documentation on downloading the logs is located here http://bit.ly/N3U932, and they pretty much tell you that the following will take care of all your needs:

appcfg.py request_logs myapp/ mylogs.txt

Well, that will get you some of your logs but it will definetly not get you everything that you can see in the Log Viewer when logged into the App Engine Admin Console.

This is what you need and it works really well for us:

appcfg.py --append --num_days=0 --include_all request_logs /path/to/your/app/ /var/log/gae/yourapp.log

Anyway, the line above will get all your log records and append them to a log file if you’ve executed this line before, and if not, it will create a new log file. It actually looks at your existing log (well, if it’s there at all), and don’t worry, it will not get any duplicates. You can run this without the —append flag if you want, but use it if you are automating log downloads.

Again, the key here is the —include_all flag which seems to be undocumented in Google’s official online documentation. This flag will get all the data that you see if you use GAE’s web log viewer. So, you will get fields such as: ms=71 cpu_ms=32 api_cpu_ms=12 cpm_usd=0.000921… 

Just to be fair to the GAE team, if you run “appcfg.py —help” from the terminal (shell or command prompt) you will see that the flag is not fully undocumented, as it will say:

 —include_all         Include everything in log messages.

We hope this saves someone a ton of time. We also hope that this is just an oversight and that the feature will make it into the official GAE documentation soon.

The zLocation Team — http://www.zlocation.com

  1. zlocation posted this
Blog comments powered by Disqus