Export logs to Azure storage
We are using Azure Automation to orchestrate a number of operations including (1) moving some files around in BLOB storage (2) provisioning an HDInsight cluster and (3) kicking off a job on that cluster to process the data files.
The HDInsight job stores its logs in BLOB storage and our Operational Support have requested that we store all logs from our system in the same place.
The Azure Automation output logs are available through the Azure Portal or via Powershell, however that does not meet the requirement to store all logs from our system in the same place. We would love to be able to specify that the Azure Automation output logs automatically get stored in some given location in Azure BLOB Storage. This would enable to consume them however we like (our current preference is to expose them as a Hive table within our HDInsight cluster).
In summary this request is ... give us the option to store the Azur Automation output in BLOB Storage so that we can consume those logs however we wish.
We are currently working on exporting Automation logs to Azure blob storage and to Log Analytics.
We are starting with job status and then adding in job streams. Please post back if there are specific pieces of information that you need to see included in these logs.
Charles M. commented
After such a long time - February 2016 - there is still nothing available... The logs seem to be stored in a DocumentDB which is always reaching thresholds. It's unusable... Can't we have an option to export these logs or store them in a blob as this was requested?
Jamie Thomson commented
For those that are interested I've written a runbook that can do this for you and made it available on the script gallery: https://gallery.technet.microsoft.com/scriptcenter/Archive-Azure-Automation-898a1aa8
Blogged about it here: http://sqlblog.com/blogs/jamie_thomson/archive/2014/11/25/archiving-azure-automation-logs-to-azure-blob-storage.aspx