Download & Process Amazon Cloudfront Logs with Awstats

These days we use Amazon Cloudfront for content delivery. Amazon has made it very easy to deliver files in a Amazon Simple Storage Service (S3) bucket using Amazon Cloudfront distribution. If you are using Cloudfront as Content Delivery Network (CDN) your next task will be monitoring the usage. For this Amazon Cloudfront has a provision to store access logs to a S3 bucket. My hurdle was to process the log files stored by Cloudfront. For sites hosted with apache I use Awstats for reading the logs. So my vote was for awstats. Please follow the steps one by one 😉

1. Need to download the log files stored in the S3 bucket. For this I had to use the a python script done by wpstorm.net but I had to make some modification so that it worked for me. Please follow the blog post if you need any help setting up the required libraries.

get-aws-logs.py

#! /usr/bin/env python
"""Download and delete log files for AWS S3 / CloudFront
 
Usage: python get-aws-logs.py [options]
 
Options:
  -b ..., --bucket=...    AWS Bucket
  -p ..., --prefix=...    AWS Key Prefix
  -a ..., --access=...    AWS Access Key ID
  -s ..., --secret=...    AWS Secret Access Key
  -l ..., --local=...     Local Download Path
  -h, --help              Show this help
  -d                      Show debugging information while parsing
 
Examples:
  get-aws-logs.py -b eqxlogs
  get-aws-logs.py --bucket=eqxlogs
  get-aws-logs.py -p logs/cdn.example.com/
  get-aws-logs.py --prefix=logs/cdn.example.com/
 
This program requires the boto module for Python to be installed.
"""
 
__author__ = "Johan Steen (http://www.artstorm.net/)"
__version__ = "0.5.0"
__date__ = "28 Nov 2010"
 
import boto
import getopt
import sys, os
from boto.s3.key import Key
 
_debug = 0
 
class get_logs:
    """Download log files from the specified bucket and path and then delete them from the bucket.
    Uses: http://boto.s3.amazonaws.com/index.html
    """
    # Set default values
    AWS_BUCKET_NAME = '{AWS_BUCKET_NAME}'
    AWS_KEY_PREFIX = ''
    AWS_ACCESS_KEY_ID = '{AWS_ACCESS_KEY_ID}'
    AWS_SECRET_ACCESS_KEY = '{AWS_SECRET_ACCESS_KEY}'
    LOCAL_PATH = '/tmp'
    # Don't change below here
    s3_conn = None
    bucket = None
    bucket_list = None
 
    def __init__(self):
        s3_conn = None
        bucket_list = None
        bucket = None
 
    def start(self):
        """Connect, get file list, copy and delete the logs"""
        self.s3Connect()
        self.getList()
        self.copyFiles()
 
    def s3Connect(self):
        """Creates a S3 Connection Object"""
        self.s3_conn = boto.connect_s3(self.AWS_ACCESS_KEY_ID, self.AWS_SECRET_ACCESS_KEY)
 
    def getList(self):
        """Connects to the bucket and then gets a list of all keys available with the chosen prefix"""
        self.bucket = self.s3_conn.get_bucket(self.AWS_BUCKET_NAME)
        self.bucket_list = self.bucket.list(self.AWS_KEY_PREFIX)
 
    def copyFiles(self):
        """Creates a local folder if not already exists and then download all keys and deletes them from the bucket"""
        # Using makedirs as it's recursive
        if not os.path.exists(self.LOCAL_PATH):
            os.makedirs(self.LOCAL_PATH)
        for key_list in self.bucket_list:
            key = str(key_list.key)
            # Get the log filename (L[-1] can be used to access the last item in a list).
            filename = key.split('/')[-1]
            # check if file exists locally, if not: download it
            if not os.path.exists(self.LOCAL_PATH+filename):
                key_list.get_contents_to_filename(self.LOCAL_PATH+filename)
                print "Downloaded				"+filename
            # check so file is downloaded, if so: delete from bucket
            if os.path.exists(self.LOCAL_PATH+filename):
                key_list.copy(self.bucket,'archive/'+key_list.key)
                print "Moved					"+filename
                key_list.delete()
                print "Deleted					"+filename
 
def usage():
    print __doc__
 
def main(argv):
    try:
        opts, args = getopt.getopt(argv, "hb:p:l:a:s:d", ["help", "bucket=", "prefix=", "local=", "access=", "secret="])
    except getopt.GetoptError:
        usage()
        sys.exit(2)
    logs = get_logs()
    for opt, arg in opts:
        if opt in ("-h", "--help"):
            usage()
            sys.exit()
        elif opt == '-d':
            global _debug
            _debug = 1
        elif opt in ("-b", "--bucket"):
            logs.AWS_BUCKET_NAME = arg
        elif opt in ("-p", "--prefix"):
            logs.AWS_KEY_PREFIX = arg
        elif opt in ("-a", "--access"):
            logs.AWS_ACCESS_KEY_ID = arg
        elif opt in ("-s", "--secret"):
            logs.AWS_SECRET_ACCESS_KEY = arg
        elif opt in ("-l", "--local"):
            logs.LOCAL_PATH = arg
    logs.start()
 
if __name__ == "__main__":
    main(sys.argv[1:])

Note: The above script will download the s3 logs to specified folder. Please make sure you put your Amazon access keys.

2. Now we have bash script which will uses the above python script to download the log files and combine all of them into a single log file and then it will be analyzed by awstats.

Warning: Please read through the script files and make necessary changes needed.
Note: You should have awstats installed on your system. The bellow script uses awstats.
Note: You can download the script files at the end of this blog post where awstats configuration with custom setup for cloudfront log format is also provided.

get-aws-logs.sh

#!/bin/bash
# Initial, cron script to download and merge AWS logs
# 29/11 - 2010, Johan Steen
 
# 1. Setup variables
date=`date +%Y-%m-%d`
static_folder="/tmp/log_static_$date/"

mkdir -pv $static_folder 
python /var/www/scripts/get-aws-logs.py --prefix=logs/www.imthi.com --local=$static_folder
gunzip --quiet  ${static_folder}*
 
/usr/local/awstats/tools/logresolvemerge.pl ${static_folder}* | sed -r -e 's/([0-9]{4}-[0-9]{2}-[0-9]{2})\t([0-9]{2}:[0-9]{2}:[0-9]{2})/\1 \2/g'  >> /var/www/logs/www.imthi.com.log
 
rm -vrf $static_folder
/usr/local/awstats/wwwroot/cgi-bin/awstats.pl -config=imthi -update

I would suggest you to test run the above scripts on a staging / testing environment before moving to a production. Again please change the scripts with your domain details and Amazon access keys.

Download the scripts to download and process Amazon Cloudfront Logs with Awstats.

Have a nice journey exploring the cloud 😉