After Googling for few hour and with trial and error method I got the correct Apache log format so that the real remote IP is logged.

I created a custom log format with a name “combined_new” and used it in all the virtual host configured in Apache. But you can use the format however you want.

#
# The following directives define some format nicknames for use with
# a CustomLog directive (see below).
#
LogFormat "\"%{X-Forwarded-For}i\" %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined_new
#....

#...
#
# START_HOST example.com
<VirtualHost *:80>
	ServerName example.com
	DocumentRoot "/var/www/example.com/html"
	<Directory "/var/www/example.com/html">
		Options Includes FollowSymLinks
		AllowOverride All
		Order allow,deny
		Allow from all
	</Directory>
	CustomLog /var/www/logs/example.com/access_log combined_new
	ErrorLog /var/www/logs/example.com/error_log
</VirtualHost>
# END_HOST example.com

Before you make any changes to httpd.conf please go through Apache log module.

I hope this small tip helps

Note: Always backup your original httpd.conf before you make any changes.

1. Need to download the log files stored in the S3 bucket. For this I had to use the a python script done by wpstorm.net but I had to make some modification so that it worked for me. Please follow the blog post if you need any help setting up the required libraries.

get-aws-logs.py

#! /usr/bin/env python
"""Download and delete log files for AWS S3 / CloudFront

Usage: python get-aws-logs.py [options]

Options:
  -b ..., --bucket=...    AWS Bucket
  -p ..., --prefix=...    AWS Key Prefix
  -a ..., --access=...    AWS Access Key ID
  -s ..., --secret=...    AWS Secret Access Key
  -l ..., --local=...     Local Download Path
  -h, --help              Show this help
  -d                      Show debugging information while parsing

Examples:
  get-aws-logs.py -b eqxlogs
  get-aws-logs.py --bucket=eqxlogs
  get-aws-logs.py -p logs/cdn.example.com/
  get-aws-logs.py --prefix=logs/cdn.example.com/

This program requires the boto module for Python to be installed.
"""

__author__ = "Johan Steen (http://www.artstorm.net/)"
__version__ = "0.5.0"
__date__ = "28 Nov 2010"

import boto
import getopt
import sys, os
from boto.s3.key import Key

_debug = 0

class get_logs:
    """Download log files from the specified bucket and path and then delete them from the bucket.
    Uses: http://boto.s3.amazonaws.com/index.html
    """
    # Set default values
    AWS_BUCKET_NAME = '{AWS_BUCKET_NAME}'
    AWS_KEY_PREFIX = ''
    AWS_ACCESS_KEY_ID = '{AWS_ACCESS_KEY_ID}'
    AWS_SECRET_ACCESS_KEY = '{AWS_SECRET_ACCESS_KEY}'
    LOCAL_PATH = '/tmp'
    # Don't change below here
    s3_conn = None
    bucket = None
    bucket_list = None

    def __init__(self):
        s3_conn = None
        bucket_list = None
        bucket = None

    def start(self):
        """Connect, get file list, copy and delete the logs"""
        self.s3Connect()
        self.getList()
        self.copyFiles()

    def s3Connect(self):
        """Creates a S3 Connection Object"""
        self.s3_conn = boto.connect_s3(self.AWS_ACCESS_KEY_ID, self.AWS_SECRET_ACCESS_KEY)

    def getList(self):
        """Connects to the bucket and then gets a list of all keys available with the chosen prefix"""
        self.bucket = self.s3_conn.get_bucket(self.AWS_BUCKET_NAME)
        self.bucket_list = self.bucket.list(self.AWS_KEY_PREFIX)

    def copyFiles(self):
        """Creates a local folder if not already exists and then download all keys and deletes them from the bucket"""
        # Using makedirs as it's recursive
        if not os.path.exists(self.LOCAL_PATH):
            os.makedirs(self.LOCAL_PATH)
        for key_list in self.bucket_list:
            key = str(key_list.key)
            # Get the log filename (L[-1] can be used to access the last item in a list).
            filename = key.split('/')[-1]
            # check if file exists locally, if not: download it
            if not os.path.exists(self.LOCAL_PATH+filename):
                key_list.get_contents_to_filename(self.LOCAL_PATH+filename)
                print "Downloaded				"+filename
            # check so file is downloaded, if so: delete from bucket
            if os.path.exists(self.LOCAL_PATH+filename):
                key_list.copy(self.bucket,'archive/'+key_list.key)
                print "Moved					"+filename
                key_list.delete()
                print "Deleted					"+filename

def usage():
    print __doc__

def main(argv):
    try:
        opts, args = getopt.getopt(argv, "hb:p:l:a:s:d", ["help", "bucket=", "prefix=", "local=", "access=", "secret="])
    except getopt.GetoptError:
        usage()
        sys.exit(2)
    logs = get_logs()
    for opt, arg in opts:
        if opt in ("-h", "--help"):
            usage()
            sys.exit()
        elif opt == '-d':
            global _debug
            _debug = 1
        elif opt in ("-b", "--bucket"):
            logs.AWS_BUCKET_NAME = arg
        elif opt in ("-p", "--prefix"):
            logs.AWS_KEY_PREFIX = arg
        elif opt in ("-a", "--access"):
            logs.AWS_ACCESS_KEY_ID = arg
        elif opt in ("-s", "--secret"):
            logs.AWS_SECRET_ACCESS_KEY = arg
        elif opt in ("-l", "--local"):
            logs.LOCAL_PATH = arg
    logs.start()

if __name__ == "__main__":
    main(sys.argv[1:])

Note: The above script will download the s3 logs to specified folder. Please make sure you put your Amazon access keys.

2. Now we have bash script which will uses the above python script to download the log files and combine all of them into a single log file and then it will be analyzed by awstats.

Warning: Please read through the script files and make necessary changes needed.
Note: You should have awstats installed on your system. The bellow script uses awstats.
Note: You can download the script files at the end of this blog post where awstats configuration with custom setup for cloudfront log format is also provided.

get-aws-logs.sh

#!/bin/bash
# Initial, cron script to download and merge AWS logs
# 29/11 - 2010, Johan Steen

# 1. Setup variables
date=`date +%Y-%m-%d`
static_folder="/tmp/log_static_$date/"

mkdir -pv $static_folder
python /var/www/scripts/get-aws-logs.py --prefix=logs/www.imthi.com --local=$static_folder
gunzip --quiet  ${static_folder}*

/usr/local/awstats/tools/logresolvemerge.pl ${static_folder}* | sed -r -e 's/([0-9]{4}-[0-9]{2}-[0-9]{2})\t([0-9]{2}:[0-9]{2}:[0-9]{2})/\1 \2/g'  >> /var/www/logs/www.imthi.com.log

rm -vrf $static_folder
/usr/local/awstats/wwwroot/cgi-bin/awstats.pl -config=imthi -update

I would suggest you to test run the above scripts on a staging / testing environment before moving to a production. Again please change the scripts with your domain details and Amazon access keys.

Download the scripts to download and process Amazon Cloudfront Logs with Awstats.

Have a nice journey exploring the cloud

They must have liked Alexander picture very much LOL !!!

Next step is to install secure shell ssh.

#sudo apt-get install ssh

Ubuntu comes with remote desktop manager. I tested the same and it would not work when there is no monitor connected to the system. I tried different methods and nothing worked out for me. So I decided to ditch the built in remote desktop and installed vncserver.

#sudo apt-get install vnc4server

After it is installed you should be logged as normal user and not root.

#vncserver :1 -geometry 1024x768 -depth 16

Once you issue the above command it will prompt for password. This password will be used for connecting to the server. Once it is complete we can change the setting for the server. Before we do that we have to kill the server.

#vncserver -kill :1

The configuration is kept in the file /home/userxx/.vnc/xstartup I edited this file so that I can start the server with gnome. My file looks exactly like below.

#!/bin/sh

# Uncomment the following two lines for normal desktop:
unset SESSION_MANAGER
# exec /etc/X11/xinit/xinitrc

gnome-session &

[ -x /etc/vnc/xstartup ] && exec /etc/vnc/xstartup
[ -r $HOME/.Xresources ] && xrdb $HOME/.Xresources
xsetroot -solid grey
vncconfig -iconic &
# xterm -geometry 80x24+10+10 -ls -title "$VNCDESKTOP Desktop" &
twm &

Now everything is done. All you have to do is to restart the system which is good for the setting to work properly. One you have restarted please start the vnc server.

#vncserver

I used chicken of the vnc to connect from mac.

After connecting you get gnome session which you can keep it alive even if the chicken vnc is closed.

Now all set and I don’t want my server to start gnome session during boot. To avoid that I have to disable gdm server.

That’s all folks have a nice day

Update: Amy Rinehart from remotedesktopmac.com has done a great comparison chart for remote desktop softwares.

Step 1

You need to have Xcode installed on your mac. In my case I had Xcode 3.1.2 installed. You can install from your Leopard DVD or you can install iPhone SDK.

Step 2

Download and install the latest Mysql database server.

Step 3

Install Fink. You can download the version 0.9.0 which I used for compiling from here. Once you have installed open terminal and issue the following commands.

# sudo fink selfupdate
# sudo fink update-all

The above two commands will update the needed files for fink library to work properly. Once this is done you can install needed libraries for php by issuing the following command

# sudo fink install libjpeg libtiff libpng3 libxml t1lib5 libmcrypt libmhash mysql15-dev curl wget

Step 4

Install FrontBase if it is needed for you. It is installer file you can download it from here. If you don’t want to use this database system you can remove the option ‘–with-fbsql’ from the php configure command.

Step 5

Install Freetds if you want use MS-SQL in PHP5.

# sudo su
# curl -O http://ibiblio.org/pub/Linux/ALPHA/freetds/stable/freetds-stable.tgz
# tar xvzf freetds-stable.tgz
# cd freetds-0.82/
# ./configure && make && sudo make install

Incase if you feel this is not needed for your installation you can remove the option ‘–with-mssql’ and ‘–enable-dbase’ from the php configure command.

Step 6

Install Imap Client libraries files. You need this lib to be installed if you wish to compile imap. If you don’t want you can remove the option ‘–with-imap’ and ‘–with-imap-ssl’. Further reading on this topic visit here.

# sudo su
# curl -O ftp://ftp.cac.washington.edu/imap/c-client.tar.Z
# tar xvzf c-client.tar.Z
# cd imap-2007e/
# make oxp
# cd c-client/
# cp *.h /usr/local/include
# cp *.c /usr/local/lib
# cp c-client.a /usr/local/lib/libc-client.a

Step 7

Install gettext. This is a must for lot of open source projects and for localization.

# sudo su
# curl -O ftp://ftp.gnu.org/gnu/gettext/gettext-0.17.tar.gz
# tar xvzf gettext-0.17.tar.gz
# cd gettext-0.17
# cd gettext-tools/
# ./configure
# make
# make install

Step 8

Install freetype. You need if you wish to load free type fonts in GD.

#sudo su
# wget http://mirrors.zerg.biz/nongnu/freetype/freetype-2.3.8.tar.gz
# tar xvzf freetype-2.3.8.tar.gz
# cd freetype-2.3.8
# ./configure
# make
# make install

Step 9

I think we are pretty there and now it is time to compile the big boss the PHP itself.

# sudo su
# wget http://www.php.net/get/php-5.2.8.tar.gz/from/in.php.net/mirror
# tar xvzf php-5.2.8.tar.gz
# cd php-5.2.8
# ./configure --prefix=/usr/local/php5.2.8 --with-apxs2 --with-config-file-scan-dir=/usr/local/php5.2.8/php.d --with-openssl=/usr --with-zlib=/usr --with-gd --enable-gd-native-ttf --with-jpeg-dir=/sw --with-jpeg-dir=/sw --with-png-dir=/sw --with-zlib-dir=/usr --with-ldap --with-xmlrpc --with-iconv-dir=/usr --with-snmp=/usr --enable-exif --enable-soap --enable-ftp --enable-sockets --enable-mbstring --enable-calendar --enable-bcmath --with-bz2=/usr --with-curl --with-kerberos=/usr --with-libxml-dir=/sw --with-xsl --with-t1lib=/sw --with-mcrypt=/sw --with-mhash=/sw --with-mysql=/sw --with-mysqli --with-pdo-mysql=/sw --enable-zip --with-imap --with-imap-ssl --with-gettext --with-curl=/sw --with-pear --with-regex --with-freetype-dir=/usr/local --enable-gd-native-ttf --with-iconv=shared,/opt/local --enable-dbase --enable-sqlite-utf8 --with-fbsql
# make
# make install

If you have completed till the above step without any error means that you have successfully installed PHP5. If you restart your apache server from system settings the server might not work for you. Since it might conflict a bit due to various architecture problems in apache and php build. To over come this issue you need to re-compile apache. You can do it by issuing the following commands. Further reading on this issue.

Step 10

Recompile Apache 2.

# sudo su
# wget http://apache.siamwebhosting.com/httpd/httpd-2.2.11.tar.gz
# tar xvzf httpd-2.2.11.tar.gz
# cd httpd-2.2.11/
# ./configure --enable-layout=Darwin --enable-mods-shared=all
# make
# make install

Once everything is done you should be able to run PHP. Have fun compiling your own native PHP on your mac .!!!

Downloaded the i686 DVD version and started the installation. In VMware OS list there was no fedora so I had to select red hat Linux as option. The installation went very smooth and no issues or error. I felt the installater was little heavy when compared to Ubuntu installation.

Fedora booted up well and detected most of the hardware of VMware. But It was not as smooth as Ubuntu was on VMware.

I heard there is lot of improvement and enhancements. But Ubuntu is very far ahead when it comes to Desktop Linux. I prefer sticking with Ubuntu than Fedora sulphur

I found the ip of the iPhone using the command ifconfig. Latter I was able to connect to the iPhone using ssh from my laptop . I was even able to ftp files to my iphone over SSH.

I Installed apache, php edited httpd.conf and I was able to run php scripts from my iPhone. It has all the basic modules that is needed to write a simple powerful application with sqlite. Taken a screenshot too

Wow this gadget is really amazing

I am such a fan of fedora and wanted to test fedora so checked out the website it had new release fedora 8 werewolf. It had live cd so downloaded it and tested on the laptop. To my surprise even fedora live cd detected everything. So reinstalled the laptop operating system with fedora 8.

I installed all the need software using yum as usual . Then I installed the Quake 3 and wanted burn test my laptop for performance. I opened couple of movies, music player and quake running auto demo on 800×600. The processor usage was around 90 to 95%. Everything worked amazing fast and in another workspace I was able to do normal installation and browsing. I ran the test for more than 3 hours and the laptop temperature was so normal. No overheating and no issues. I was totally amazed by the laptop performance over Linux.

I am really happy to see such a stable fedora core which is now capable of detected all hardware. Lenovo laptop performance was amazingly superb

Other than that I used the following

• Pentium 4 3.2 HT processor
• 80 GB Sata hard disk
• 1 GB RAM
• Realtek gigabyte Ethernet card
• DVD drive (mainly used for installing purpose).

Now the system is ready to rock and roll. The next big step was to install some flavor of linux and test the firewall. I was thinking of fedora latter my friend Lijeesh showed me some new variant of linux which are there only for this purpose routing and firewall solution. Some distribution which are done for routing purpose are

• IPCOP an open source distribution.
• Smoothwall has both both open source and commercial solution
• Clarkconnect is commercial solution.

I installed and tested in the same above mentioned order. And I found some issues with each installation. Let me explain the hurdles I faced in each.

IPCOP – The installation was smooth. I had some issues in setting up my on board Ethernet card. Basically it was marvell chipset. I tried installing sk98lin drivers for it and it wont work. Then the firewall not supporting NAT.

Smoothwall – The installation was smooth. It even detected the on board Ethernet and it worked. It had really a good web interface with funky ajax effects. Like live bandwidth monitor, live messenger chat viewer etc. But when it comes to routing features the was very much limited . One such problem was it didn’t NAT routing.

Clarkconnet – It is a commercial solution. It is based on red hat distribution. I had some issues with my onboard ethernet card. It was detecting it but the drivers was not working. I had to issue command “modprobe sk98lin” and everything worked well. It had all the routing features please features. I tested it well and it was stable. Currently I am using this for my routing solution.

It took me 4 days to figure out the best options for routing solution based on Linux. I am not sure that the above conclusion are 100% correct. May be I have screwed it up somewhere. But it was fun experimenting

It is quite powerful and give total control on incoming and outgoing mails. To solve my problem I need to play only with outgoing mails. There is a file named /etc/antivirus.exim is the file we need to play with. I edited the file and added the below lines to make it work for me.

if first_delivery
and ( ("$h_from:" contains "test1@imthi.com")
or ("$h_from:" contains "test2@imthi.com")
or ("$h_from:" contains "test3@imthi.com")
)
then
unseen deliver "tracker@imthi.com"
endif

The above block checks if the outgoing mail is on first delivery and who is the sender. Based on the sender I am sending a bcc copy to another mailbox. You can play more by studying the exim filter commands more