Building a CD Bootable Firewall with FreeBSD

By Chatchawan Wongsiriprasert
Some of the information in this page come from Etienne de Bruin (eT) Building a CD Bootable Firewall.

1. Network Topology

Before start building firewall, You need to know your network topology of my network and some special machine on the network.
	Intranet<------->Firewall<------>Internet

    FileServer<-----------|
I need to store the outbound traffic statistic , to the server. All Firewall log message via syslogd must go to FileServer. The firewall configuration is store on FileServer which must be loaded to FireWall when the machine start. This allow me to reconfig FireWall without rewrite new CD (FireWall has only a CD-ROM -- no writeable storage on FireWall).

2. FreeBSD distribution

I build my Firewall on FreeBSD 4.7p10 version because all other of FreeBSD in the company use this version.
It should be any problem with FreeBSD 4.8. I don't have enought bravery to try 5.0.
After install and build FreeBSD in normal way, just copy all the files you need to a new location with tar (assume that you put user data to /home as me). This is my method.
	cd /home
	mkdir fwCD
	cd fwCD
	mkdir images
	cd images
	tar -X ./tar-exclude -c -p -v -f - / | tar xpf -
And this is content of tar-exclude
	/tmp
	/home
	/usr/obj
Now, you can use the images folder as a images for the boot able CD.
We need to modify some files/folder for our Firewall CD.The modification are
  1. Modify images/etc/rc.conf as show in 4.
  2. Create images/cdboot
  3. Create images/cdboot/rc.cdboot as , also, show in 4.
  4. Create images/usr/local/etc/rc.d0/000.smbfs.sh, don't forget to make it user executable (chmod u+x 000.smbfs.sh).
  5. Create images/stat
  6. Put FileServer user/password to images/etc/nsmb.conf. Don't forget file mode to 0600.
  7. Don't forget to modify image/etc/fstab. It should be only one line
    		proc	/proc	procfs	rw	0	0
    	

3. Kernel Configuration

You need no special option for the CD-Bootable version, except that you must have
	options CD9660
	options CD9660_ROOT
All other options are the same as standard FreeBSD firewall.This is my kernel configuration.

4. Special directory and configuration files

There are 3 directories on FreeBSD that need to be writeable. Which are A
	/var
	/tmp
	/etc
They need to be mounted to mfs (memory file system). These file systems must be mounted as soon as posible. To archive this, you need to set /etc/rc.conf.
This is my /etc/rc.conf:
ifconfig_xl0="DHCP"
diskless_mount="/cdboot/rc.cdboot"
sshd_enable="YES"
inetd_enable="NO"
sendmail_enable="NO"
In the /etc/rc.conf, I config my primary interface which connect to the Intranet, with DHCP, therefore, I can change the IP address of the Firewall later.
The diskless_mount line, allow me to setup read-write directory very early by /etc/rc ( see your /etc/rc )
I enable sshd in the rc.conf to allow me to assess to this machine, if something went wrong. inetd and sendmail(received mode) is not need to this machine. The /cdboot/rc.cdboot is a small script that create mfs for /var, /tmp and /etc (/tmp and /var are created by standard FreeBSD rc.diskless2 script).
	/sbin/mount_mfs swap /mnt
	cd /mnt
	/bin/cp -R -p /etc /mnt
	/sbin/mount_mfs swap /etc
	cd /
	/bin/cp -R -p /mnt/etc /
	umount /mnt	

	sh /etc/rc.diskless2
First, just copy /etc off CD-ROM, remount /etc , and copy tempory stored files to /etc.
Then, use /etc/rc.diskless2 to create /var and /tmp.

5. Setting firewall -- bootstrap

At this point, the machine should work as normal FreeBSD. I put the script 000.smbfs.sh to /usr/local/etc/rc.d to make sure that this script is the last to run.
#!/bin/sh
# $Id: index.html,v 1.11 2006/04/26 08:51:59 cws Exp $

# start
if [ "x$1" = "xstart" ]; then

m=`/sbin/mount | /usr/bin/grep "/stat"`
if [ "x$m" = "x" ] 
then
/sbin/mount_smbfs //FIREWALL@SERVER/FIREWALL /stat
fi
if [ -f /stat/rc ]
then
A

IFCONFIG=/sbin/ifconfig
AWK=/usr/bin/awk
GREP=/usr/bin/grep
if1=`$IFCONFIG -l -u | $AWK -F ' ' '{print $1}'`
ip=`$IFCONFIG ${if1} | $GREP -w inet | $AWK -F ' ' '{print $2}'`
/bin/sh /stat/rc ${ip}
fi

fi
First, the script try to mount /stat to FIREWALL SMB shared on file server(SERVER) using account FIREWALL, the password is stored in /etc/nsmb.conf -- to /stat.
Then, if there is /stat/rc, the script try to run that file and give the IP address of primary network interface as a parameter.

The /stat/rc just check the parameter and call the appropiate scrip base on the given IP -- (this file must be place in root of FIREWALL share)
#!/bin/sh
#
#$Id: index.html,v 1.11 2006/04/26 08:51:59 cws Exp $
#
PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin
base_folder=`/usr/bin/dirname $0`
script_name=`echo $base_folder/rc.d/rc.$1`
if [ -f $script_name ];
then
        exec $script_name
fi

if [ "x$1" = "x" ];
then
echo usage : $0 caller_ip
else
echo Unknown script name [$script_name]
fi

6. The Firewall configuration script

You can put any command you want to /stat/rc.d/rc.x.x.x.x. This include fixing error in CD-ROM image, install cron tab , run additional startup script and add firewall rules This is my sample:
#!/bin/sh
#
#$Id: index.html,v 1.11 2006/04/26 08:51:59 cws Exp $
#
PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin
base_path=`dirname $0`

#Fix errors in CD-ROM images
chmod og+rw /dev/null

#Kill unused daemon that start in CD-ROM etc/rc.conf
killall -9 inetd
killall -9 sendmail
killall -9 dhclient

#Sendmail for outbound mail only.
sendmail -q30m &

hostname firewall.net0.intranet

#Config network interface and routing
route delete default 2>/dev/null
ifconfig xl1 inet 192.168.1.1 netmask 255.255.255.0
route add default 192.168.1.254

#Make /etc read/write
chown -R root:wheel /etc
cd /etc/mail
make

#Copy program need for ipaudit
mkdir /var/ipaudit
cp ${base_path}/../scripts/ipaudit /var/ipaudit
cp ${base_path}/../scripts/cappkg.pl /var/ipaudit
cp ${base_path}/../scripts/synclock.sh /var/ipaudit

#Home for root
mkdir /var/root
mkdir /var/admin
chown admin:admin /var/admin

#Start package capture stat
/var/ipaudit/cappkg.pl &           
/var/ipaudit/synclock.sh &

#Install cron
crontab -u root ${base_path}/../config/10.0.0.200/crontab	
cp ${base_path}/../config/192.168.0.1/periodic.conf /etc

#--------------------
# Firewall configuration 
# Prevent any host to use this host as a switch.
#--------------------
#Enable gateways and routing
sysctl net.inet.ip.forwarding=1 >/dev/null
routed -s &

ipfw=/sbin/ipfw
sysctl net.inet.ip.fw.dyn_max=8192
${ipfw} -q flush
#Not need with current rule set because we have keep-stat
#Rule 
#${ipfw} add check-state

#Limit bandwidth
#${ipfw} add pipe 1 ip from ${x.x.x.x} to any out
#${ipfw} add pipe 2 ip from ${x.x.x.x} to any in
#${ipfw} pipe 1 config bw 64Kbit/s queue 10Kbytes
#${ipfw} pipe 2 config bw 64Kbit/s queue 10Kbytes

#Traffic to firewall
${ipfw} add allow ip from me to any
${ipfw} add allow ip from any to me

#If you have multiple external lines and want to divert some traffic
#${ipfw} add forward a.b.c.d ip from x.x.x.x to not ${x.x.x.x/24}

#Open firewall -- trade security with convenient 
${ipfw} add 65535 allow ip from any to any

7. Burn CD

After put everything you want to images folder. This is a time to burn the CD

7.1 Contructintg boot.flp

You need FreeBSD boot.flp which can be found on your FreeBSD CD or any FreeBSD FTP site
(for example ftp://ftp1.freebsd.org/pub/FreeBSD/releases/i386/4.7-RELEASE/floppies/boot.flp).
Modify the boot.flp as you need with the following steps:
	vnconfig -s labels -c vn0 boot.flp
	mount /dev/vn0 /mnt

	#Copy new kernel
	gzip -c images/kernel > /mnt/kernel.gz
To make standard kernel boot from CD-ROM instead of first hard disk , modify /mnt/boot/loader.rc to
	set vfs.root.mountfrom="cd9660:acd0a"
	load /kernel
	echo \007\007
	autoboot 3
When finished with boot.flp ,
	umount /mnt
	vnconfig -u vn0
	cp boot.flp images/boot.flp

7.2 Making an ISO images

You need ports/sysutils/mkisofs to do this. After install the port,
	mkisofs -l -R -L -o fwCD.iso -b boot.flp -c boot.catalog images
Then burn the CD.
	burncd -f /dev/acd0a -s 8 data fwCD.iso fixate
Now you got them

8. Miscellaneous

8.1 synclock.sh

I use synclock.sh script to syncronize firewall clock every 2 hours. I prefer ntpdate to ntpd becase it run only when I want.

8.2 Package capture

I run package capture on Firewall to collect traffic statistic. The capture program come from ipaudit project. But I make some modification to suit my need. This is the modified version.

Attension: Due to the change in ipaudit code (mostly hash.c/hash.h). My modified version of ipaudit.c can not complie with the hash.c/hash.h from ipaudit.sourceforge.net. Until I have a time to fix my ipaudit code you must use the binary version that compiled on FreeBSD 4.8 from here. If you want to run it on 5.x , you must install "ports/misc/compat4x" or at least libpcap.so.2 and libc.so.4.

Package capture also need a perl scrip to run. This is the cappkg.pl.

8.3 crontab

I need cron to run synclock every 2 hours and cappkg.pl every hours.
# DO NOT EDIT THIS FILE - edit the master and reinstall.
# (/tmp/crontab.nkFFKn2GAm installed on Thu Apr 10 10:02:12 2003)
# (Cron version -- $FreeBSD: src/usr.sbin/cron/crontab/crontab.c,v 1.12.2.4 2001/06/16 03:18:37 peter Exp $)
20 0-23/2  * * * /var/ipaudit/synclock.sh
0  *	      * * * /var/ipaudit/cappkg.pl

8.4 periodic.conf

Some of the default periodic.conf is not make sense on CD-ROM (such as update locate and whatis database). This is my version of periodic.conf for Firewall
#!/bin/sh
#$Id: index.html,v 1.11 2006/04/26 08:51:59 cws Exp $
#
# What files override these defaults ?
# See /etc/default/periodic.conf for options
# Daily options

# Weekly options

# 310.locate
weekly_locate_enable="NO"				# Update locate weekly

# 320.whatis
weekly_whatis_enable="NO"				# Update whatis weekly

8.5 Server configuration

With my configuration , I need to set
  1. /usr/local/etc/dhcp.conf ( I use isc-dhcpd ). Add these lines
    	host firewall {
    	  hardware ethernet XX:XX:XX:XX:XX:XX;
    	  fixed-address firewall.net0.intranet;
    	  option routers X.X.X.X;
    	}
    	
    Don't forget the line routers -- set it to any IP that is not firewall.net0.intranet.
  2. Add new SMB share for store configuration and statistic for firewall.

9. Stat Viewer

How to view the statistic collected from the firewall?
First, I try IPAudit Web, but ,Unfortunately, I can not make it work with my server.
Next, I turn to net/ntop port. I can make it work but ntop show current data only. I also need to view archived data.
So, I write the firewall statistic viewer myself and call it fwstat.

9.1 System Requirement

fwstat is written in PHP. So, all it need is apache+php and if you want to see network loading graph, your php must support gd2.

9.2 How to install

  1. Download the source code. Click here to download.
  2. Unpack the source codes,and make them accessible from web browser.
    In my case, I store them in /home/firewall/public_html which are home page of the user that I use for smb share.
  3. Modify config.php to match your network configuration
    The configuration you may need to change are
        $STAT_CFG['RAWSTAT_FOLDER'] = '/home/firewall/traffic/raw';
        $STAT_CFG['COOKEDSTAT_FOLDER']='/home/firewall/traffic/cooked';
        $STAT_CFG['CURRENTSTAT_FOLDER']='/home/firewall/traffic/raw/current';
        $STAT_CFG['CRONLOG_FILE'] = '/home/firewall/cronlog.txt';
        $STAT_CFG['STAT_COOKER'] = '/home/firewall/public_html/tools/cookstat.php';
    	
    Change them to the folder that you store the statistic collected from firewall. See cappkg.pl source code for more detail.
        $STAT_CFG['LOCALNET'] = '10\.\d{1,3}\.\d{1,3}\.\d{1,3}';
        $STAT_CFG['LOADGRAPH'] = array( array('TITLE'=>'ALL','NET'=>'255.255.255.255'),
                                 array('TITLE'=>'NET0','NET'=>'10.0.0.255'),
                                 array('TITLE'=>'NET1','NET'=>'10.0.1.255'));
    	
    Change them to match your local network configuration. The LOCALNET string is perl pattern that match the IP range of your local network. The LOADGRAPH is an array to indicate number of graphs to be shown.
        $STAT_CFG['MAX_TRAFFIC'] = 128;
    	
    Change the value to match maximum bandwidth of your network. The unit of this value is Kbit per second.
  4. Add cron job to cook the statictic. Some of the statistic need not to be cooked but some such as yearly and monthly data need some pre-processing to increase execution time. Add the following cron job to your cron tab
        2 * * * * /home/firewall/public_html/tools/cron_run.php
        10 0 * * * /home/firewall/public_html/tools/cook_remotehost.php
    	
    Note: I run the scripts at xx:02 and 0:10 becase I want the cappkg.pl a time to copy the data to file server.
  5. Protect the statistic page from your user. Read apache authenticate for more detail.
  6. If something wrong, fix them (you have the source code :) and please mail the fix to me at cws@miraclenet.co.th. I never test the code at any other network except the small network of my company.

9.3 Screen shot

Network loading graph of last 60 minutes.
Daily Network loading graph.
Network loading detail.
Local Host Monthly Network loading statistic.
This may be the most effective stat that this program was produced. After I show the manager this page, The network load reduce dramatically. Especially the traffic to movie trailer sites.

The document theme copy from FreeBSD Handbook
For questions about this documentation, e-mail <cws@miraclenet.co.th>.
Last update : $Id: index.html,v 1.11 2006/04/26 08:51:59 cws Exp $