Squid Proxy Server Mac Address based filtering
Setting up ACL's based on MAC address
Open squid.conf:
# vi /etc/squid/squid.conf
Local acl, section and append ACL as follows:
acl macf1 arp mac-address
acl macf2 arp 00:11:22:33:44:55
http_access allow macf1
http_access allow macf2
http_access deny all
Save and close the file. Restart squid server:
# /etc/init.d/squid restart
Thanks to cyberciti.biz
Transparent proxy with Squid
Server Configuration
- Step #1 : Squid configuration so that it will act as a transparent proxy
- Step #2 : Iptables configuration
- a) Configure system as router
- b) Forward all http requests to 3128 (DNAT)
- Step #3: Run scripts and start squid service
First, Squid server installed (use up2date squid) and configured by adding following directives to file:
# vi /etc/squid/squid.conf
Modify or add following squid directives:
httpd_accel_host virtual
httpd_accel_port 80
httpd_accel_with_proxy on
httpd_accel_uses_host_header on
acl lan src 192.168.1.1 192.168.2.0/24
http_access allow localhost
http_access allow lan
Where,
- httpd_accel_host virtual: Squid as an httpd accelerator
- httpd_accel_port 80: 80 is port you want to act as a proxy
- httpd_accel_with_proxy on: Squid act as both a local httpd accelerator and as a proxy.
- httpd_accel_uses_host_header on: Header is turned on which is the hostname from the URL.
- acl lan src 192.168.1.1 192.168.2.0/24: Access control list, only allow LAN computers to use squid
- http_access allow localhost: Squid access to LAN and localhost ACL only
- http_access allow lan: -- same as above --
Here is the complete listing of squid.conf for your reference (grep will remove all comments and sed will remove all empty lines, thanks to David Klein for quick hint ):
# grep -v "^#" /etc/squid/squid.conf | sed -e '/^$/d'
OR, try out sed (thanks to kotnik for small sed trick)
# cat /etc/squid/squid.conf | sed '/ *#/d; /^ *$/d'
Output:
hierarchy_stoplist cgi-bin ?
acl QUERY urlpath_regex cgi-bin \?
no_cache deny QUERY
hosts_file /etc/hosts
refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern . 0 20% 4320
acl all src 0.0.0.0/0.0.0.0
acl manager proto cache_object
acl localhost src 127.0.0.1/255.255.255.255
acl to_localhost dst 127.0.0.0/8
acl purge method PURGE
acl CONNECT method CONNECT
cache_mem 1024 MB
http_access allow manager localhost
http_access deny manager
http_access allow purge localhost
http_access deny purge
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
acl lan src 192.168.1.1 192.168.2.0/24
http_access allow localhost
http_access allow lan
http_access deny all
http_reply_access allow all
icp_access allow all
visible_hostname myclient.hostname.com
httpd_accel_host virtual
httpd_accel_port 80
httpd_accel_with_proxy on
httpd_accel_uses_host_header on
coredump_dir /var/spool/squid
Iptables configuration
Next, I had added following rules to forward all http requests (coming to port 80) to the Squid server port 3128 :
iptables -t nat -A PREROUTING -i eth1 -p tcp --dport 80 -j DNAT --to 192.168.1.1:3128
iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 80 -j REDIRECT --to-port 3128
Here is complete shell script. Script first configure Linux system as router and forwards all http request to port 3128 (Download the fw.proxy shell script):
#!/bin/sh
# squid server IP
SQUID_SERVER="192.168.1.1"
# Interface connected to Internet
INTERNET="eth0"
# Interface connected to LAN
LAN_IN="eth1"
# Squid port
SQUID_PORT="3128"
# DO NOT MODIFY BELOW
# Clean old firewall
iptables -F
iptables -X
iptables -t nat -F
iptables -t nat -X
iptables -t mangle -F
iptables -t mangle -X
# Load IPTABLES modules for NAT and IP conntrack support
modprobe ip_conntrack
modprobe ip_conntrack_ftp
# For win xp ftp client
#modprobe ip_nat_ftp
echo 1 > /proc/sys/net/ipv4/ip_forward
# Setting default filter policy
iptables -P INPUT DROP
iptables -P OUTPUT ACCEPT
# Unlimited access to loop back
iptables -A INPUT -i lo -j ACCEPT
iptables -A OUTPUT -o lo -j ACCEPT
# Allow UDP, DNS and Passive FTP
iptables -A INPUT -i $INTERNET -m state --state ESTABLISHED,RELATED -j ACCEPT
# set this system as a router for Rest of LAN
iptables --table nat --append POSTROUTING --out-interface $INTERNET -j MASQUERADE
iptables --append FORWARD --in-interface $LAN_IN -j ACCEPT
# unlimited access to LAN
iptables -A INPUT -i $LAN_IN -j ACCEPT
iptables -A OUTPUT -o $LAN_IN -j ACCEPT
# DNAT port 80 request comming from LAN systems to squid 3128 ($SQUID_PORT) aka transparent proxy
iptables -t nat -A PREROUTING -i $LAN_IN -p tcp --dport 80 -j DNAT --to $SQUID_SERVER:$SQUID_PORT
# if it is same system
iptables -t nat -A PREROUTING -i $INTERNET -p tcp --dport 80 -j REDIRECT --to-port $SQUID_PORT
# DROP everything and Log it
iptables -A INPUT -j LOG
iptables -A INPUT -j DROP
Save shell script. Execute script so that system will act as a router and forward the ports:
# chmod +x /etc/fw.proxy
# /etc/fw.proxy
# service iptables save
# chkconfig iptables on
Start or Restart the squid:
# /etc/init.d/squid restart
# chkconfig squid on
Desktop / Client computer configuration
Point all desktop clients to your eth1 IP address (192.168.2.1) as Router/Gateway (use DHCP to distribute this information). You do not have to setup up individual browsers to work with proxies.
How do I test my squid proxy is working correctly?
See access log file /var/log/squid/access.log:
# tail -f /var/log/squid/access.log
Above command will monitor all incoming request and log them to /var/log/squid/access_log file. Now if somebody accessing a website through browser, squid will log information.
Problems and solutions
(a) Windows XP FTP Client
All Desktop client FTP session request ended with an error:
Illegal PORT command.
I had loaded the ip_nat_ftp kernel module. Just type the following command press Enter and voila!
# modprobe ip_nat_ftp
Please note that modprobe command is already added to a shell script (above).
(b) Port 443 redirection
I had block out all connection request from our router settings except for our proxy (192.168.1.1) server. So all ports including 443 (https/ssl) request denied. You cannot redirect port 443, from debian mailing list, "Long answer: SSL is specifically designed to prevent "man in the middle" attacks, and setting up squid in such a way would be the same as such a "man in the middle" attack. You might be able to successfully achive this, but not without breaking the encryption and certification that is the point behind SSL".
Therefore, I had quickly reopen port 443 (router firewall) for all my LAN computers and problem was solved.
(c) Squid Proxy authentication in a transparent mode
You cannot use Squid authentication with a transparently intercepting proxy.
Backing Up and Restoring Your MySQL Database
Do you need to change your web host or switch your database server? This is probably the only time when you really think of backing up your MySQL data. If you've got a website with a database or your custom database running for your applications, it is imperative that you make regular backups of the database. In this article, I will outline two easy ways of backing up and restoring databases in MySQL.
The easiest way to backup your database would be to telnet to the your database server machine and use the mysqldump command to dump your whole database to a backup file. If you do not have telnet or shell access to your server, don't worry about it; I shall outline a method of doing so using the PHPMyAdmin web interface, which you can setup on any web server which executes PHP scripts.
Playing with mysqldump
If you have either a shell or telnet access to your database server, you can backup the database using mysqldump. By default, the output of the command will dump the contents of the database in SQL statements to your console. This output can then be piped or redirected to any location you want. If you plan to backup your database, you can pipe the output to a sql file, which will contain the SQL statements to recreate and populate the database tables when you wish to restore your database. There are more adventurous ways to use the output of mysqldump.
A Simple Database Backup:
You can use mysqldump to create a simple backup of your database using the following syntax.
mysqldump -u [username] -p [password] [databasename] > [backupfile.sql]
o [username] - this is your database username
o [password] - this is the password for your database
o [databasename] - the name of your database
o [backupfile.sql] - the file to which the backup should be written.
The resultant dump file will contain all the SQL statements needed to create the table and populate the table in a new database server. To backup your database 'Customers' with the username 'sadmin' and password 'pass21' to a file custback.sql, you would issue the command:
mysqldump -u sadmin -p pass21 Customers > custback.sql
You can also ask mysqldump to add a drop table command before every create command by using the option --add-drop-table. This option is useful if you would like to create a backup file which can rewrite an existing database without having to delete the older database manually first.
mysqldump --add-drop-table -u sadmin -p pass21 Customers > custback.sql
Backing up only specified tables
If you'd like restrict the backup to only certain tables of your database, you can also specify the tables you want to backup. Let's say that you want to backup only customer_master & customer_details from the Customers database, you do that by issuing
mysqldump --add-drop-table -u sadmin -p pass21 Customers customer_master customer_details> custback.sql
So the syntax for the command to issue is:
mysqldump -u [username] -p [password] [databasename] [table1 table2 ....]
o [tables] - This is a list of tables to backup. Each table is separated by a space.
Backing Up and Restoring Your MySQL Database - What about Multiple Databases?
If you are a database administrator who has to look after multiple databases, you'll need to back up more than one database at a time. Here's how you can backup multiple databases in one shot.
If you want to specify the databases to backup, you can use the --databases parameter followed by the list of databases you would like to backup. Each database name has to be separated by at least one space when you type in the command. So if you have to backup 3 databases, let say Customers, Orders and Comments, you can issue the following command to back them up. Make sure the username you specify has permissions to access the databases you would like to backup.
mysqldump -u root -p pass21 --databases Customers Orders Comments > multibackup.sql
This is okay if you have a small set of databases you want to backup. Now how about backing up all the databases in the server? That's an easy one, just use the --all-databases parameter to backup all the databases in the server in one step.
mysqldump --all-databases> alldatabases.sql
Backing up only the Database Structure
Most developers need to backup only the database structure to while they are developing their applications. You can backup only the database structure by telling mysqldump not to back up the data. You can do this by using the --no-data parameter when you call mysqldump.
mysqldump --no-data --databases Customers Orders Comments > structurebackup.sql
Compressing your Backup file on the Fly
Backups of databases take up a lot of space. You can compress the output of mysqldump to save valuable space while you're backing up your databases. Since mysqldump sends its output to the console, we can pipe the output through gzip or bzip2 and send the compressed dump to the backup file. Here's how you would do that with bzip2 and gzip respectively.
mysqldump --all-databases | bzip2 -c >databasebackup.sql.bz2
mysqldump --all-databases | gzip >databasebackup.sql.gz
A Shell Script for Automating Backups?
You can automate the backup process by making a small shell script which will create a daily backup file. How do you get cron to back up your database without overwriting the older backup? You can use a tiny shell script to add the date to your backup file. An example of a shell script you could use is shown below.
#!/bin/sh
date=`date -I`
mysqldump --all-databases | gzip > /var/backup/backup-$date.sql.gz
Now that you've got backups of your database, let's learn how to restore your backup in case your database goes down. Here's how you can restore your backed up database using the mysql command.
Restore using mysql
If you have to re-build your database from scratch, you can easily restore the mysqldump file by using the mysql command. This method is usually used to recreate or rebuild the database from scratch.
Here's how you would restore your custback.sql file to the Customers database.
mysql -u sadmin -p pass21 Customers < custback.sql
Easy isn't it ? Here's the general format you would follow:
mysql -u [username] -p [password] [database_to_restore] < [backupfile]
Now how about those zipped files? You can restore your zipped backup files by first uncompressing its contents and then sending it to mysql.
gunzip < custback.sql.sql.gz | mysql -u sadmin -p pass21 Customers
You can also combine two or more backup files to restore at the same time, using the cat command. Here's how you can do that.
cat backup1.sql backup.sql | mysql -u sadmin -p pass21
Moving Data Directly Between Databases
How would you like to replicate your present database to a new location? When you are shifting web hosts or database servers, you can directly copy data to the new database without having to create a database backup on your machine and restoring the same on the new server. mysql allows you to connect to a remote database server to run sql commands. Using this feature, we can pipe the output from mysqldump and ask mysql to connect to the remote database server to populate the new database. Let's say we want to recreate the Customers database on a new database server located at 202.32.12.32, we can run the following set of commands to replicate the present database at the new server.
mysqldump -u sadmin -p pass21 Customers | mysql --host=202.32.12.32 -C Customers
How to install Xcache module for Apache
XCache is a fast, stable PHP opcode cacher that has been tested and is now running on production servers under high load. It is tested (on linux) and supported on all of the latest PHP cvs branches such as PHP_4_3 PHP_4_4 PHP_5_0 PHP_5_1 PHP_5_2 HEAD(6.x). ThreadSafe/Windows is also supported. It overcomes a lot of problems that has been with other competing opcachers such as being able to be used with new PHP versions. See Introduction for more information.
You don't have to check the following list yourself, the configure script will do for you, unless you have problem with configure/make.
Check version with cli
$ php-cgi -v
PHP 4.4.3-dev (cgi-fcgi) (built: Mar 10 2006 18:46:02)
Copyright (c) 1997-2006 The PHP Group
Zend Engine v1.3.0, Copyright (c) 1998-2004 Zend Technologies
or setup a file with:
Check version with phpinfo
<?php
phpinfo();
?>
and request it from your browser.
* Get the php works with webserver without XCache first!
* common build tools: c compiler, make, libtool (required by php build env)
* php building env installed. if you've install php yourself, make sure you have do "make install". for some os distro, there is "php-devel" package. check it out with:
Check phpize
$ which phpize
/usr/local/bin/phpize
(or)
/usr/bin/phpize
you output may be vary from this, depending on your installtion of php. if it's not found, you should find it yourself
* m4
* indent (optional)
Building:
~ $ cd ~/src/xcache
~/src/xcache $ ls
(you XCache source is here)
~/src/xcache $ phpize
(generating configure .... everytime you upgrade php, or update to a new XCache, you have to run phpize again)
(it is suggested to build outside of the source directory, so make an build directory first and enter it)
~/src/xcache $ mkdir ../xcache-build
~/src/xcache $ cd ../xcache-build
~/src/xcache-build $ ../xcache/configure --help
......
--enable-xcache Include XCACHE support.
--enable-xcache-optimizer XCACHE: (N/A)
--enable-xcache-coverager XCACHE: Enable code coverage dumper
--enable-xcache-assembler XCACHE: (N/A)
--enable-xcache-disassembler XCACHE: Enable opcode to php variable dumper
--enable-xcache-encoder XCACHE: (N/A)
--enable-xcache-decoder XCACHE: (N/A)
--enable-xcache-test XCACHE: Enable self test - FOR DEVELOPERS ONLY!!
......
(run configure with options you selected now)
~/src/xcache-build $ ../xcache/configure --enable-xcache --enable-xcache-coverager
(many output here, if you have problem, read the error message twice)
(and search inside config.log, and check Pre-requirement in this page above)
~/src/xcache-build $ make
(many output here again, check if it success or error out.)
NOTE: It's always better not to enable unnecessary modules for production server unless you're not the maintainer of the server. Play with it locally.
WARNING: If you're using something like /opt/php/bin/phpize which isn't the 1st one found in $PATH, remember to configure --with-php-config=/opt/php/bin/php-config, exactly the same directory as phpiz.
Special path
~/src/xcache-build $ PATH="/opt/php/bin/:$PATH" ../xcache/configure \
--with-php-config=/opt/php/bin/php-config \
--enable-xcache \
--enable-xcache-coverager
(line is broken up for readability)
~/src/xcache-build $ make
Installing
~/src/xcache-build $ su
Password:
(input your root password here. whenever u see a red # in code listing in this wiki, it means you need to be root to do that)
~/src/xcache-build # make install
(many output here, and you can see where the XCache extension is installed into, remember the extension path)
You have to modify php.ini to make XCache enable in your php!
How to install mod_security for Apache
What is mod_security or modsecurity?
ModSecurity is an open source intrusion detection and prevention engine for web applications. It operates embedded into the web server, acting as a powerful umbrella - shielding applications from attacks. ModSecurity supports Apache web server.
Short explanation of how to embedd:
Installation
ModSecurity installation consists of the following steps:
1. ModSecurity 2.x works with Apache 2.0.x or better.
2. Make sure you have mod_unique_id installed.
mod_unique_id is packaged with Apache httpd.
3. Install the latest version of libxml2, if it isn't already installed on the server.
http://xmlsoft.org/downloads.html
4. Optionally install the latest version of Lua in the 5.1.x branch, if it isn't already installed on the
server and you will be using the new Lua engine.
http://www.lua.org/download.html
Note that ModSecurity requires the dynamic libraries. These are not built by default in the
source distribution, so the binary distribution is recommended.
5. Stop Apache httpd
6. Unpack the ModSecurity archive
7. Building differs for UNIX (or UNIX-like) operating systems and Windows.
• UNIX
a. Run the configure script to generate a Makefile. Typically no options are needed.
./configure
Options are available for more customization (use ./configure --help for a full
list), but typically you will only need to specify the location of the apxs command in-
stalled by Apache httpd with the --with-apxs option.
./configure --with-apxs=/path/to/httpd-2.x.y/bin/apxs
b. Compile with: make
c. Optionally test with: make test
NOTE: This is step is still a bit experimental. If you have problems, please send the
full output and error from the build to the support list. Most common issues are related
to not finding the required headers and/or libraries.
d. Optionally build the ModSecurity Log Collector with: make mlogc
e. Optionally install mlogc: Review the INSTALL file included in the
apache2/mlogc-src directory in the distribution.
f. Install the ModSecurity module with: make install
• Windows (MS VC++ )
a. Edit Makefile.win to configure the Apache base and library paths.
b. Compile with: nmake -f Makefile.win
c. Install the ModSecurity module with: nmake -f Makefile.win install
d. Copy the libxml2.dll and lua5.1.dll to the Apache bin directory. Alternat-
ively you can follow the step below for using LoadFile to load these libraries.
8. Edit the main Apache httpd config file (usually httpd.conf)
On UNIX (and Windows if you did not copy the DLLs as stated above) you must load libxml2
and lua5.1 before ModSecurity with something like this:
LoadFile /usr/lib/libxml2.so
LoadFile /usr/lib/liblua5.1.so
Load the ModSecurity module with:
LoadModule security2_module modules/mod_security2.so
9. Configure ModSecurity
10. Start Apache httpd
11. You should now have ModSecurity 2.x up and running.
Note
If you have compiled Apache yourself you might experience problems compiling ModSecurity
against PCRE. This is because Apache bundles PCRE but this library is also typically provided
by the operating system. I would expect most (all) vendor-packaged Apache distributions to be
configured to use an external PCRE library (so this should not be a problem).
You want to avoid Apache using the bundled PCRE library and ModSecurity linking against the
one provided by the operating system. The easiest way to do this is to compile Apache against the
PCRE library provided by the operating system (or you can compile it against the latest PCRE
version you downloaded from the main PCRE distribution site). You can do this at configure time
using the --with-pcre switch. If you are not in a position to recompile Apache, then, to com-
pile ModSecurity successfully, you'd still need to have access to the bundled PCRE headers (they
are available only in the Apache source code) and change the include path for ModSecurity (as
you did in step 7 above) to point to them (via the --with-pcre ModSecurity configure op-
ion).
Do note that if your Apache is using an external PCRE library you can compile ModSecurity with
WITH_PCRE_STUDY defined,which would possibly give you a slight performance edge in regu-
ar expression processing.
Latest release here!
QOS and IP Accounting with BGP under linux
At NSP we’ve go a fibre connection into the building, and a 10MBit feed from our ISP, and over that we’re allowed 10MBit of national and 3 Mbit PIR of international traffic. Note that this adds up to more than 10Mbit in total! This can cause annoying problems, like someone doing a lot of national or APE traffic at 10MBit, and closing out real international traffic. For a long time I’ve wanted to separate this out, but have not had the time to look into it
This week I finally organised a BGP from my ISP, and had a look at what my options were. I’d seen the Route-based QOS mini-HOWTO a while back, and it looked like it would work ok, but had a few problems. There’s no current way it to apply tc or iptables rules selectively based on a routing decision, or even on a route table. You can match on a route realm, however. The mini-HOWTO suggests copying your BGP routes into a separate table and into a realm at the same time, and then using tc and iptable’s realm matching code.
A quick aside: route realms are best described as a collection of routes. The decision as to which realm a route is placed is made by the local administrator, and each realm can contain routes from a mix of origins. Realms are used to allow administrators to perform bulk operations on large groups of routes in an easy manner. From the iproute command reference:
The main application of realms is the TC route classifier [7], where they are used to help assign packets to traffic classes, to account, police and schedule them according to this classification.
After a bit of digging, I found a link to a patch for quagga to provide route realms support. It’s even still maintained! After a bit of battling with autotools[1], and a bit of battling with linux capabilities[2], I had it up and running.
The route realms patch page covered off the BGP configuration I needed, and now I have a set of iptables counters for national, international and total traffic (for completeness). The only bit it doesn’t cover off is graphing, but we already have a set of perl scripts which pull information from interface totals or iptables FWMARK counters, so I modified that to pull from these counters as well, and set up RRD graphs. I was previously graphing interface totals out the external nic, and it’s interesting to note that the iptables “total” traffic, while adding up to the sum of national and international, does not correspond to the interface totals.
It’s worth pointing out that, as seen in iproute command reference, the rtacct tool will grab realm counts for you without needing iptables, so if you just want to something to graph things quickly, rtacct might do the job:
Realm BytesTo PktsTo BytesFrom PktsFrom
BPSTo PPSTo BPSFrom PPSFrom
unknown 5949K 57188 15839K 61776
0 0 0 0
national 15839K 61776 5949K 57188
0 0 0 0
rtacct has a naive limit of 256 realms however, where as the actual implementation supports a 16 bit number, so if you have a large number of realms, or you autoclassify your inbound BGP into realms based on the AS number, you will have to use iptables only
I’m currently only accounting for traffic using this mechanism, but I can also do QOS on it - tc will match directly on realm tags, and any iptables based match systems you may have can be adapted to match on a realm as well.
[1] The realms patch touched configure.ac, which then required the autotools chain to rebuild everything, but it needed a very particular combination of autoconf and automake. Because it took me an hour or so to get this right, I’ll record it here:
aclocal-1.7
autoheader
autoconf
autoconf2.50
libtoolize -c
automake-1.7 –gnu –add-missing –copy
./configure –enable-realms –enable-user=quagga –enable-group=quagga –enable-vty-group=quaggavty –enable-vtysh –localstatedir=/var/run/quagga –enable-configfile-mask=0640 –enable-logfile-mask=0640
autoheader and autoconf above are version 2.13. I have no idea why I had to run autoconf2.13 then autoconf2.50, but it seems that this actually worked.
[2] I initially tried building against quagga-0.98.6, because the quaggarealms patch site implied this was the “stable” verson, but it seems that quagga drops priviledges too soon. This works out fine if you have “capabilities” support in your kernel, which mine didn’t. They’ve changed this behaviour in 0.99.5, and incidentally this is the version in debian etch.
Traffic classification using BGP (a quagga+realms approach)
Realms patch - Quagga 0.98.6
Stable: quagga-0.98.6-realms.diff
Development: quagga-0.99.5-realms.diff
Updated versions (>0.99.5) - http://linux.mantech.ro/quagga+realm_en.html
This patch enables Linux route realms support in quagga 0.98.6
By Arcady Stepanov’s patch for zebra 0.93b http://win.mol.ru/penguin/zebra-hacks/, adapted it to quagga 0.98.4 interface and added some useful features.
The following commands are supported:
* Route-map
o
bgpd(config-route-map)# set realm
<1-255> Realm id for Linux FIB routes
WORD Realm name for Linux FIB routes
origin-as Use route origin AS as realm id
peer-as Use route peer AS as realm id
o
bgpd(config-route-map)# no set realm
<0-255> Realm value
WORD Realm name
origin-as Origin AS - realm
peer-as Peer AS - realm
<cr>
* Neighbor
o
bgpd(config-router)# neighbor x.x.x.x realm
<0-255> default realm id
WORD default realm name
origin-as Set default realm to received route origin AS
peer-as Set default realm to peer AS
o
bgpd(config-router)# no neighbor x.x.x.x realm
<0-255> default realm id
WORD default realm name
origin-as Set default realm to received route origin AS
peer-as Set default realm to peer AS
<cr>
Note:
’set realm origin-as’ was added with inter-AS traffic accounting in mind. For now, this is possible only with the iptables realm match which can match on the full 16bit realm value. The current realm accounting code in the kernel (rtacct - /proc/net/rt_acct) supports only 256 values for realms, and displays incorrect statistics.
Bugs/suggestions should go to: vcalinusATgemenii.ro
Brief usage guide…
0. kernel support (if you want to classify traffic into htb classes using tc)
CONFIG_NET_CLS_ROUTE=y
1. /eetc/iproute2/rt_realms
Assign meaningful names to realm numbers...
user@router:/# cat /eetc/iproute2/rt_realms
10 localnet
20 metro-isp
22 metro-other
30 international
2. compile/install quagga
Stable Quagga 0.98.6
quagga 0.98.6 - official release
+
quagga 0.98.6 realms patch
Big thanks to Alin Nastac for updating the patch to 0.98.6!
Patch for development Quagga 0.99.5
quagga-0.99.5-realms.diff
Older patches
quagga-0.98.5-realms.diff
quagga-0.98.4-realms.diff
quagga-0.98.3-realms.diff
Remember to use ./configure --enable-realms
3. BGP CONFIGURATION
a possible bgp setup:
(if you hold the full routing table - replace defgw with a match on the desired community)
AS-regexp match is also possible
neighbor xxx.xxx.xxx.xxx remote-as XXXXX
neighbor xxx.xxx.xxx.xxx soft-reconfiguration inbound
neighbor xxx.xxx.xxx.xxx route-map isp_in in
ip prefix-list defgw seq 5 permit 0.0.0.0/0
ip community-list standard metro-isp permit XXXXX:comm1
ip community-list standard metro-other permit XXXXX:comm2
route-map isp_in permit 10
match ip address prefix-list defgw
set realm 30
!
route-map isp_in permit 20
match community metro-isp
set realm 20
!
route-map isp_in permit 30
match community metro-other
set realm 22
!
route-map isp_in permit 40
3.1 'ip route sh' will show kernel routes - they should have the realms specified in the route-map
something like....
62.217.192.0/18 via 193.19.192.65 dev eth1 proto zebra equalize realm 20
82.137.0.0/18 via 172.16.100.1 dev eth2 proto zebra equalize realm 22
84.243.64.0/18 via 172.16.100.1 dev eth2 proto zebra equalize realm 20
82.208.128.0/18 via 193.19.192.65 dev eth1 proto zebra equalize realm 22
4. iptables
Can be used in FORWARD or POSTROUTING (remember that realms are valid only after the forwarding decision)
Download: match default route, community 1, and community 2 sets
-A FORWARD -i eth3 -m realm --realm 0x1e0000/0xffff0000 -j sometarget...
-A FORWARD -i eth3 -m realm --realm 0x140000/0xffff0000 -j sometarget...
-A FORWARD -i eth3 -m realm --realm 0x160000/0xffff0000 -j sometarget...
Upload: match default route, community 1, and community 2 sets
-A FORWARD -o eth3 -m realm --realm 0x1e/0xffff -j sometarget...
-A FORWARD -o eth3 -m realm --realm 0x14/0xffff -j sometarget...
-A FORWARD -o eth3 -m realm --realm 0x16/0xffff -j sometarget...
(realms 30,20 and 22 are specified in hexadecimal)
5. tc
Excerpt from LARTC
# ip route add 192.168.2.0/24 dev eth2 realm 2
# tc filter add dev eth1 parent 1:0 protocol ip prio 100
route from 2 classid 1:2
Here the filter specifies that packets from the subnetwork 192.168.2.0 (realm 2) will match class id 1:2.
You can also find useful QoS stuff at: http://kernel.umbrella.ro/net/
6. what are realms after all?
Realms are 16bit integer values used to group routes into sets, according to
some defined policy. Each route in the set will have the same realm.
Each packet routed will have a 32bit integer value specifying a source and a destination realm. (they may be 0 - or unknown)
On the leftmost 16bits you will find the source realm, on the rightmost 16bits the destination realm.
More info: http://www.policyrouting.org/iproute2.doc.html#ss9.9
Quagga realms Tutorial
Quagga is an open source routing software based on the Zebra router, for which development was stopped in 2003. It supports the main standardised routing protocols such as RIP, OSPF or BGP and can be installed on any Linux system with a 2.4 or higher kernel.
Quagga is composed of several daemons, one per routing protocol and another one called Zebra acting as the kernel routing manager. Each daemon has its own configuration file and terminal interface which can be accessed by telnet.
The vtysh tool is provided to configure the Quagga router from the localhost, in a unique interface.
See the Quagga tutorial for instructions on configuring the router.
Quagga works independently from the operational system (OS) over which it is installed. This is not the case for the open source Vyatta router or the commercial routers where the OS and the routing engine are built together.
With Vyatta routers, you can access the operational system (a modified version of the Debian Linux distribution) but even this is not necessary to configure the router.
With commercial routers like Cisco or Nortel, you can access only the router interface.
It must be emphasised that Quagga owns only routing capabililies and functionalities associated with it, such as access lists or route maps. It does not provide "non-routing" functionalities such as DHCP server, NTP server/client or ssh access but it is often possible to enable them on the operating system supporting your Quagga router.
See the Routers' comparison to get an overview about the functionalities available with Cisco, Vyatty or Quagga routers.
The Quagga Command line Interface is very similar to the Cisco IOS software allows it to be configured very easily for those who are familiar with Cisco.
And finally, little story: the word 'quagga' comes from an extinct animal which was a subspecies of the plains zebra.
1. QUAGGA DAEMONS:
You have to activate the Quagga daemons matching the routing protocols you want to set on your router.
zebra:
bgpd:
ospfd:
ospf6d:
ripd:
ripngd: Interface declaration and static routing
BGP routing protocol
OSPF routing protocol
OSPF IPv6 routing protocol
RIP v2 routing protocol
RIP Ipv6 routing protocol
In the example below, the zebra and IPv4 OSPF daemon have been activated.
#vim /etc/quagga/daemons
zebra=yes
bgpd=no
ospfd=yes
ospf6d=no
ripd=no
ripngd=no
Restart the Quagga service:
#/etc/init.d/quagga restart
Stopping Quagga daemons (prio:0): ospfd zebra (bgpd) (ripd) (ripngd) (ospf6d) (isisd).
Removing all routes made by zebra.
Nothing to flush.
Loading capability module if not yet done.
Starting Quagga daemons (prio:10): zebra ospfd.
You can check the Quagga daemons status:
#ps -ef | grep quagga
UID PID PPID C STIME TTY TIME CMD
quagga 4632 1 0 22:25 ? 00:00:00 /usr/lib/quagga/ospfd --daemon
quagga 4636 1 0 22:25 ? 00:00:00 /usr/lib/quagga/zebra --daemon
If a Quagga daemon doesn't stop properly, you can kill it manually and start the quagga service:
#kill -9 "UID number"
#/etc/init.d/quagga start
2. CONFIGURATION FILES: (/etc/quagga/*.conf files)
You must create a configuration file (even if it is empty) each time you activate a Quagga daemon.
Each daemon is associated with a specific file name:
zebra:
bgpd:
ospfd:
ospf6d:
ripd:
ripngd: zebra.conf
bgpd.conf
ospfd.conf
ospf6d.conf
ripd.conf
ripngd.conf
To create the config files, copy the sample config files as follows:
In our example, as we activated the zebra and ospfd daemons; we need to create the zebra.conf and ospfd.conf files.
#cp /usr/share/doc/quagga/examples/zebra.conf.sample /etc/quagga/zebra.conf
#cp /usr/share/doc/quagga/examples/ospfd.conf.sample /etc/quagga/ospfd.conf
Another way to do it is to create two empty files called /etc/quagga/ospfd.conf and /etc/quagga/zebra.conf. But in this case you cannot telnet a daemon, you need to configure the telnet permissions with vtsh (see below).
Finally, give user and group ownership to respectively quagga and quaggavty to the files inside the /etc/quagga directory:
#chown quagga.quaggavty /etc/quagga/*.conf
#chmod 640 /etc/quagga/*.conf
Restart the Quagga service:
#/etc/init.d/quagga restart
Top of the page
3. DEBIAN.CONF FILE
By default, the Quagga daemons are listening only to the loopback interface 127.0.0.1. It means that you can telnet a daemon only on its loopback address 127.0.0.1 and thus cannot access it remotely.
If you want to telnet a Quagga daemon remotely you can, in the /etc/quagga/debian.conf file. Either indicate one or several IP addresses or remove the -A option meaning that you can telnet a daemon on any of its IP addresses.
Here are two examples:
The ospfd daemon is listening to the 127.0.0.1 and 192.168.1.104 IP addresses.
ospfd_options=" --daemon -A 127.0.0.1 192.168.1.104"
The zebra daemon is listening to all the Linux interfaces IP addresses. We recommend using this setting. If you want to filter who can access to your router, configure access lists on the router software.
zebra_options=" --daemon "
Here is the recommended debian.conf file:
#vim /etc/quagga/debian.conf
# If this option is set the /etc/init.d/quagga script automatically loads
# the config via "vtysh -b" when the servers are started.
# Check /etc/pam.d/quagga if you intend to use "vtysh"!
#
vtysh_enable=yes
zebra_options=" --daemon -A "
bgpd_options=" --daemon -A "
ospfd_options=" --daemon -A "
ospf6d_options="--daemon -A "
ripd_options=" --daemon -A "
ripngd_options="--daemon -A "
isisd_options=" --daemon -A "
The "vtysh_enable=yes" setting is required to access the Quaggga router via vtysh. (see vtysh section).
Restart the Quagga service
#/etc/init.d/quagga restart
Top of the page
4. VTYSH
As indicated in the Quagga introduction, you can access the daemons by telnetting their port number because each daemon has its own configuration file and terminal interface.
zebra:
ripd:
ripng:
ospfd:
bgpd:
ospf6d: 2601
2602
2603
2604
2605
2606
By instance, to access the ospfd daemon:
#telnet localhost 2604
As it's not very practical to configure your router by telnetting its daemons separately, vtysh has been created to configure everything in one single interface.
To use vtysh, you must first create its configuration file as follows:
#cp /usr/share/doc/quagga/examples/vtysh.conf.sample /etc/quagga/vtysh.conf
/etc/quagga/vtysh.conf
!
! Sample
!
! service integrated-vtysh-config
hostname quagga-router
username root nopassword
!
Apply correct permissions and restart Quagga:
#chown quagga.quaggavty /etc/quagga/*.conf
#chmod 640 /etc/quagga/*.conf
#/etc/init.d/quagga restart
In the example above the "service integrated-vtysh-config" setting has been disabled (recommended). In this case, when you save the config under vtysh, it will be stored in separate files depending on the protocols you activated.
Below, an example where the Quagga configuration is saved under vtysh. (The zebra and ospfd daemons have been enabled.)
#vtysh
quagga-router#write
Configuration saved to /etc/quagga/zebra.conf
Configuration saved to /etc/quagga/ospfd.conf
If you activate "service integrated-vtysh-config", the configuration under vtysh will be saved in one file called Quagga.conf in the /etc/quagga/ directory.
With this setting, when you access a daemon via telnet, the daemon will look first to the Quagga.conf file before looking for its own file. This means that, when you telnet a device, there can be a difference between what you see after the "show run" command and the content of the associated file, for example zebra.conf.
#vtysh
quagga-router#write
Configuration saved to /etc/quagga/Quagga.conf
It is recommended to disable "service integrated-vtysh-config" because if this setting is enabled and in case of a syntax error in the Quagga.conf file, this can lead to all your daemons being unable to start up. This will not be case when "service integrated-vtysh-config" is disabled because the configurations are stored in separate files.
Check that the default "vtysh_enable=yes" setting are configured in your /etc/quagga/debian.conf file. You can read the previous paragraph about the debian.conf file to get more information.
Then it's useful to add the "VTYSH_PAGER=more" setting in your /etc/environment file, otherwise you will see an unfriendly "(END)" blinking in the left-down corner of the screen each time your enter a command and will need to press the "q" key to continue.
#echo VTYSH_PAGER=more > /etc/environnement
Log off and log on to enable the environment setting. You can now access the Quagga router with the vtysh command:
#vtysh
Hello, this is Quagga (version 0.99.6).
Copyright 1996-2005 Kunihiro Ishiguro, et al.
quagga-router#
If you want to run a Quagga command from the Linux shell:
#vtysh -c "command"
For instance, vtysh -c "show ip route" will display the Quagga routing table.
You can use ping and traceroute to perform connectivity checks from the vtysh prompt. Of course, these two programs need to be installed on the Linux machine. Ping is generally installed by default but traceroute often not.
To install traceroute:
#apt-get install traceroute
Top of the page
5. IP FORWARDING:
IP forwarding is required to transfer packets between the network interfaces of a Linux system.
See a picture of the Linux kernel routing.
#echo "1" > /proc/sys/net/ipv4/ip_forward
The command above will add the "1" value inside the /proc/sys/net/ipv4/ip_forward file and thus activate the IP forwarding.
If you want to keep the IP forwarding after a Linux reboot:
#echo "net.ipv4.ip_forward = 1" >> /etc/sysctl.conf
It is possible to check the ip_forwarding status under the Quagga router:
#show ip forwarding
IP forwarding is on
In this case the IP forwarding is activated.
Top of the page.
6. SPEED/DUPLEX:
It is not possible to set the duplex and speed settings on the Quagga plateform. You have to configure them at the Linux level.
Use the interface configuration tutorial for assistance.
Here is a comparison of some characteristics you can find on four routers: the first two are the best open source routers available, Quagga and Vyatta and the remaining two are commercial devices and best-sellers from Cisco: a standard router from the 26xx family and a Layer Three Switch 3750.
| RFC | QUAGGA 0.99.6 |
VYATTA VC 3.0 |
CISCO 2651 |
CISCO 3750 L3 |
|
| Software type | IOS-Like | JunOS-Like | Cisco IOS | Cisco IOS | |
| Opensource | YES | YES | NO | NO | |
| Installation | YES | YES | NO | NO | |
| Static Routing | - | YES | YES | YES | YES |
| RIPv2 | 2453 | YES | YES | YES | YES |
| OSPFv2 | 2328 | YES | YES | YES | YES |
| BGPv4 | 1771/4271 | YES | YES | YES | YES |
| NAT | 2766 | NO/Linux | YES | YES | NO |
| VRRP | 3768 | NO/Linux | YES | YES | NO |
| Access lists | YES | YES | YES | YES | |
| Route maps | YES | YES | YES | YES | |
| VPN IPSec | 4301 | NO/Linux (1) | YES | YES | YES |
| VPN SSL | NO/Linux (2) | NO/Linux (2) | NO | NO | |
| FTP client | 959 | NO/Linux | YES | YES | YES |
| TFTP client | 1350 | NO | YES | YES | YES |
| Telnet server | 854 | YES (3) | YES | YES | YES |
| ssh server | 4251 | NO/Linux | YES | YES/NO | YES/NO |
| HTTP server | NO | YES | YES | YES | |
| DHCP server | 2131 | NO/Linux | YES | YES | YES |
| DHCP relay | NO/Linux | YES | YES | YES | |
| NTP server | 1305 | NO/Linux | NO | YES/NO | NO |
| NTP client | 1305 | NO/Linux | YES | YES | YES |
| SNMP | 3412/1157 | YES (4) | YES | YES | YES |
| Ping | YES (5) | YES | YES | YES | |
| traceroute | YES (5) | YES | YES | YES |
Legend:
| YES: YES/NO: BUGGED: NO/Linux: (1) (2) (3) (4) |
The protocol is supported. Available on specific Cisco IOSs only. There is a major bug on the protocol. Not supported on the Quagga router but can be enabled at the Linux level. Openswan can be used as an implementation of IPsec for Linux. OpenVPN can be used to build SSL VPNs on Linux. Each routing protocol daemon and the zebra kernel routing manager have to be telnetted separately. Quagga must be compiled with the SNMP functionality. Ping or traceroute must be installed at the Linux level to be used under Quagga. |
Htb-Tools Tutorial – Traffic shaping
HTB-tools was started by Ionut Spirlea ionut.spirlea[at]rdsnet.ro, Manuel Subredu diablo[at]infoiasi.ro, Vasile Laurentiu Stanimir stanimir[at]cr.nivis.com. Over the time, they couldn't dedicate more resources to this project.
Needing a tool to manage the bandwidth for a wireless network, HTB-tools came across. As it was unmaintained, the author of this site decided to take over the project. Meanwhile, other great people contributed to this project, please see the Credits list. Thanks, guys! You were great!
HTB-tools is a software suite with several tools that help simplify the difficult process of bandwidth allocation, for both upload and download traffic: generate and check configuration files, real time traffic overview for each separate client.
Features:
* bandwidth limitation using public IP addresses, using the two configuration files for upload and download
* bandwidth limitation using private IP addresses (SNAT), using a single configuration file
* match mark
* match mark in u32
* metropolitan/external limitation
* menu based management software for configuration and administration of HTB-tools (starting with version 0.3.0)
How To use HTB-tools
Hierarchical Token Bucket ( HTB ) successfully replaces Class Based Queueing ( CBQ ) due to the precise and easy to understand way it operates. The difference from CBQ is that the bandwidth is allocated to one (or more) classes, and when the class's allocated bandwidth is exceeded, it can (temporarily) borrow unused bandwidth from another class. Moreover, unlike CBQ, you can allocate several clients to one class. Using HTB-tools all classes and all clients can be defined in a configuration file.
To better understand how it works, let's suppose that we have a bandwidth of 256kbps shared by 4 clients (with routable IP addresses), each client having a guaranteed minimum of 48kbps and a guaranteed maximum of 64kbps upload/download.
After HTB-tools is installed, you can proceed to editing the configuration files.
Assuming that eth0 is the interface to your provider and eth1 the interface to your clients, on the eth0 interface you can only limit the upload (only if you use public IP addresses) and on eth1 only the download, because you can only control the debit of the packages that leave the router ( the information that enters the router is imposed by the external environment ).
For upload limitations/guarantees you have to edit the file /etc/htb/eth0-qos.cfg and for download limitations/guarantees you have to edit the file /etc/htb/eth1-qos.cfg
Configuration files for HTB-tools
Remarks:
- the format of the configuration files resembles the format of bind's configuration files;
- the bandwidth users are divided into classes;
- these classes can not share the bandwidth among themselves;
- the members of a class (clients) can share the same bandwidth according to the parameters specified in the configuration file;
- a class may contain one or more clients;
- a special class is the default class, that defines the bandwidth allocated to those which are not included in any class;
- the transfer rate is specified in kbit;
- the lines having # as the first character are considered comments.
- you can NOT specify the fields src and dst for a class; these fields are only defined for clients.
The class syntax:
bandwidth 192; - the guaranteed minimum for the class; represents the minimum total sum guaranteed for the clients of the class;
limit 256; - the maximum transfer rate for the class;
burst 2; - the maximum number of kbits sent at once by the class;
!!! NOTE:
If you don't set the burst value properly, the limits might not work
correctly. Eg: for the limit 10000kbit you must use a burst of 12 kbit.
THIS IS AVAILABLE ONLY IF YOU USE PUBLIC IPs !
burst 0; - ONLY for HTB-tools beta4 - if burst is set to 0 then HTB-tools will calculate the burst value automatically. This is valid for clients side too.
priority 1; - priority of the class; there are 8 priority levels: from 0 to 7. The packets are served in ascending order of priority. For example, if the priority is 0 then the packets will first be served to this class, on the other hand if the priority is 4 then the packets will first be sent to the class having priority 0 and only then to the class having priority 4;
que sfq; - we can specify the qdisc for a class, if not specified the default is pfifo limit 5.
(or esfq;)
For more details about esfq, please see [link]
As specified above, we have a bandwidth of 256kbps that we want to allocate to 4 clients, both for upload and for download. For this, we first define the class:
class class_1 {
bandwidth 192;
limit 256;
burst 2;
priority 1;
que sfq;
The clients syntax:
bandwidth 48; - guaranteed minimum for the client;
limit 64; - maximum transfer rate for the client;
burst 2; - maximum number of kbits sent to the client at once;
mark 20; - if only mark is specified, without dst/src, then fw will be used; if the source/destination is specified, then u32 will be used with the possibility to mark (match_in_u32);
dst or src can be used together in the configuration file only for clients; if you wish to limit the upload then you must use src, and if you wish to limit the download then you must use dst; in our example above we have used limitation/allocation for download and the configuration file will be eth1-qos.cfg;
priority 1; - client priority;
- in the case of a single IP, the representation looks like this: 192.168.1.10/32;
- in the case of a class, the representation looks like this: 192.168.1.0/24;
- if you wish to apply the limitation to a particular port for a single IP:
192.168.1.10/32 80; - limits the traffic to port 80 (http)
- if you wish to apply the limitation to a particular port for an IP class:
192.168.1.0/24 21; - limits the traffic to port 21 (ftp)
The next step is to define the 4 clients. For each client a minimum of 48kbit and a maximum of 64kbit will be allocated.
client client1 {
bandwidth 48;
limit 64;
burst 2;
priority 1;
mark 20;
dst {
192.168.100.4/32;
};
};
client client2{
bandwidth 48;
limit 64;
burst 2;
priority 1;
mark 20;
dst {
192.168.100.5/32;
};
};
client client3 {
bandwidth 48;
limit 64;
burst 2;
priority 1;
mark 20;
dst {
192.168.100.8/32;
};
};
client clien4 {
bandwidth 48;
limit 64;
burst 2;
priority 1;
mark 20;
dst {
192.168.100.10/32;
};
};
class default { bandwidth 8; };
};
};
Only for HTB-tools beta4
Another new feature is the "upload" function, which would assist in managing the
upload, if you use SNAT (non-routeable IP addresses). This can be done using the same configuration file that you would use to manage the download.
WARNING: this type of shaping (ingress) drops the packets that are over the
limit, which generates additional traffic when entering the interface. For this reason it is
recommended to have the shaping machine in the same LAN with the shaped machines.
The "upload" option uses dst but don't work with mark.
Configuration examples (client side):
client client {
bandwidth 350;
limit 1200;
burst 0; # new
priority 1;
upload 300; #( in kbits) <- new
dst {
192.168.100.5/32;
};
};
client client {
bandwidth 350;
limit 1200;
burst 0; # new
priority 1;
upload 300; #( in kbits) <- new
dst {
192.168.100.30/32;
};
};
UPLOAD option available from 0.3.0-beta4 works only with dst.
You can specify mark or src but this options will NOT
affect outgoing trafic.
The next step is checking the configuration file using the q_checkcfg command:
q_checkcfg /etc/htb/eth1-qos.cfg
Default bandwidth: 8
Class class_1, CIR: 192, MIR: 256
** 4 clients, CIR2: 192, MIR2: 256
1 classes; CIR / MIR = 192 / 256; CIR2 / MIR2 = 192 / 256
- the configuratin files are syntactically correct.
The rc.htb init script
------------------------
With this script you can start/stop/monitor/generate/restart the limitations/guarantees for upload/download or for both together.
To start at boot the bandwidth policies for upload and download you must add to /etc/rc.d/rc.local the line:
/etc/rc.d/rc.htb start
If you wish to limit/guarantee the bandwidth only for download then you must use the command:(this example is works on Slackware linux)
/etc/rc.d/rc.htb start_eth1
If you wish to limit/guarantee the bandwidth only for upload then you must use the command:
/etc/rc.d/rc.htb start_eth0
If you wish to generate the traffic rules script you can use:
for eth0
[center]/etc/rc.d/rc.htb gen_eth0[/center]
for eth1
[center]/etc/rc.d/rc.htb gen_eth1[/center]
q_show allows you to see in real-time the traffic and bandwidth usage for each client (download). In order to see realtime trafic run:
/etc/rc.d/rc.htb show_eth1
...and the result is:
class_1
224.80
2
192
256
client_1
62.25
1
48
64
client_2
51.05
1
48
64
client_3
48.25
1
48
64
client_4
63.25
1
48
64
_default_
0
0
0
0
If you like to pass some options to q_show , please see the q_show(8) man page.
rc.htb complete options and usage
----------------------------------------
/etc/rc.d/rc.htb start | stop | restart |
start_eth0 | stop_eth0 | restart_eth0 |
start_eth1 | stop_eth1 | restart_eth1 |
start_eth2 | stop_eth2 | restart_eth2 |
show_eth0 | show_eth1 | show_eth2 |
gen_eth0 | gen_eth1 |
Web q_show
---------------
Web q_show is a tool that displays in a webpage the traffic status and allocated bandwidth for each class/client according to the configuration file.
A cron job collects the traffic data in a .log file, namely q_show.log. From here on it is the job of .php script (q_show.php) that parses the file and displays the content in a webpage.
Configuration
----------------
Before starting anything, you must have php installed and a working
web server (apache). Let's assume that you have domain.ro and the
default directory for the webpages is /var/www/htdocs/. The default
directory specified at the install time (with "make install_web") will
have a folder called webhtb with q_show.php script.
Before adding the line to crontab, set:
- the time interval (*/1 every minute or */5 every 5 minutes etc) at which to generate the traffic logs in the file;
- the correct path to the configuration file
- the interface, ethx where x = 0, 1, 2 ... etc, the interface you want to monitor;
- ethx-qos.cfg the configuration file for the monitored interface;
- the path to the webhtb directory;
Add to crontab (crontab -e):
*/1 * * * * /sbin/q_show -i eth1 -f /etc/htb/eth1-qos.cfg -1 > /var/www/htdocs/webhtb/q_show.log
You can see the traffic at the address [link] .
Web HTB-tools configuration files generator
---------------------------------------------------
The web q_show install will get you whtbcfg in the same directory as webhtb (i.e.: /var/www/htdocs/whtbcfg). You will need a functional httpd (apache) with php support to be able to use it.
php.ini settings
------------------
Please set register_globals to ON in php.ini and disable error_reporting like
this:
register_globals = on<br>#error_reporting = E_ALL & ~E_NOTICE
After modifying php.ini, please restart the httpd server and point your browser to [link] to be able to generate configuration files.
Possible configurations in HTB-tools beta 4
--------------------------------------------------
Please see the file cfg/possible_configs
IPTables HowTo
1. Introduction
CentOS has an extremely powerful firewall built in, commonly referred to as iptables, but more accurately is iptables/netfilter. Iptables is the userspace module, the bit that you, the user, interact with at the command line to enter firewall rules into predefined tables. Netfilter is a kernel module, built into the kernel, that actually does the filtering. There are many GUI front ends for iptables that allow users to add or define rules based on a point and click user interface, but these often lack the flexibility of using the command line interface and limit the users understanding of what's really happening. We're going to learn the command line interface of iptables.
Before we can really get to grips with iptables, we need to have at least a basic understanding of the way it works. Iptables uses the concept of IP addresses, protocols (tcp, udp, icmp) and ports. We don't need to be experts in these to get started (as we can look up any of the information we need), but it helps to have a general understanding.
Iptables places rules into predefined chains (INPUT, OUTPUT and FORWARD) that are checked against any network traffic (IP packets) relevant to those chains and a decision is made about what to do with each packet based upon the outcome of those rules, i.e. accepting or dropping the packet. These actions are referred to as targets, of which the two most common predefined targets are DROP to drop a packet or ACCEPT to accept a packet.
Chains
These are 3 predefined chains in the filter table to which we can add rules for processing IP packets passing through those chains. These chains are:
- INPUT - All packets destined for the host computer.
- OUTPUT - All packets originating from the host computer.
- FORWARD - All packets neither destined for nor originating from the host computer, but passing through (routed by) the host computer. This chain is used if you are using your computer as a router.
For the most part, we are going to be dealing with the INPUT chain to filter packets entering our machine - that is, keeping the bad guys out.
Rules are added in a list to each chain. A packet is checked against each rule in turn, starting at the top, and if it matches that rule, then an action is taken such as accepting (ACCEPT) or dropping (DROP) the packet. Once a rule has been matched and an action taken, then the packet is processed according to the outcome of that rule and isn't processed by further rules in the chain. If a packet passes down through all the rules in the chain and reaches the bottom without being matched against any rule, then the default action for that chain is taken. This is referred to as the default policy and may be set to either ACCEPT or DROP the packet.
The concept of default policies within chains raises two fundamental possibilities that we must first consider before we decide how we are going to organize our firewall.
1. We can set a default policy to DROP all packets and then add rules to specifically allow (ACCEPT) packets that may be from trusted IP addresses, or for certain ports on which we have services running such as bittorrent, FTP server, Web Server, Samba file server etc.
or alternatively,
2. We can set a default policy to ACCEPT all packets and then add rules to specifically block (DROP) packets that may be from specific nuisance IP addresses or ranges, or for certain ports on which we have private services or no services running.
Generally, option 1 above is used for the INPUT chain where we want to control what is allowed to access our machine and option 2 would be used for the OUTPUT chain where we generally trust the traffic that is leaving (originating from) our machine.
2. Getting Started
Working with iptables from the command line requires root privileges, so you will need to become root for most things we will be doing.
|
|
IMPORTANT: We will be turning off iptables and resetting your firewall rules, so if you are reliant on your Linux firewall as your primary line of defense you should be aware of this. |
|
Iptables should be installed by default on all CentOS 3.x, 4.x and 5.x installations. You can check to see if iptables is installed on your system by:
$ rpm -q iptables iptables-1.3.5-1.2.1
And to see if iptables is actually running, we can check that the iptables modules are loaded and use the -L switch to inspect the currently loaded rules:
# lsmod | grep ip_tables ip_tables 29288 1 iptable_filter x_tables 29192 6 ip6t_REJECT,ip6_tables,ipt_REJECT,xt_state,xt_tcpudp,ip_tables
# iptables -L Chain INPUT (policy ACCEPT) target prot opt source destination RH-Firewall-1-INPUT all -- anywhere anywhere Chain FORWARD (policy ACCEPT) target prot opt source destination RH-Firewall-1-INPUT all -- anywhere anywhere Chain OUTPUT (policy ACCEPT) target prot opt source destination Chain RH-Firewall-1-INPUT (2 references) target prot opt source destination ACCEPT all -- anywhere anywhere ACCEPT icmp -- anywhere anywhere icmp any ACCEPT esp -- anywhere anywhere ACCEPT ah -- anywhere anywhere ACCEPT udp -- anywhere 224.0.0.251 udp dpt:mdns ACCEPT udp -- anywhere anywhere udp dpt:ipp ACCEPT tcp -- anywhere anywhere tcp dpt:ipp ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:ssh REJECT all -- anywhere anywhere reject-with icmp-host-prohibited
Above we see the default set of rules in addition to access to the SSH service.
If iptables is not running, you can enable it by running:
# system-config-securitylevel
3. Writing a Simple Rule Set
|
|
IMPORTANT: At this point we are going to clear the default rule set. If you are connecting remotely to a server via SSH for this tutorial then there is a very real possibility that you could lock yourself out of your machine. You must set the default input policy to accept before flushing the current rules, and then add a rule at the start to explicitly allow yourself access to prevent against locking yourself out. |
|
We will use an example based approach to examine the various iptables commands. In this first example, we will create a very simple set of rules to set up a Stateful Packet Inspection (SPI) firewall that will allow all outgoing connections but block all unwanted incoming connections:
# iptables -P INPUT ACCEPT # iptables -F # iptables -A INPUT -p tcp --dport 22 -j ACCEPT # iptables -P INPUT DROP # iptables -P FORWARD DROP # iptables -P OUTPUT ACCEPT # iptables -A INPUT -i lo -j ACCEPT # iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT # iptables -L -v
which should give the following output:
Chain INPUT (policy DROP 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 ACCEPT all -- lo any anywhere anywhere
0 0 ACCEPT all -- any any anywhere anywhere state RELATED,ESTABLISHED
Chain FORWARD (policy DROP 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Now lets look at each of the 8 commands above in turn and understand exactly what we've just done:
-
iptables -P INPUT ACCEPT If connecting remotely we must first temporarily set the default policy on the INPUT chain to ACCEPT otherwise once we flush the current rules we will be locked out of our server.
-
iptables -F We used the -F switch to flush all existing rules so we start with a clean state from which to add new rules.
-
iptables -A INPUT -p tcp --dport 22 -j ACCEPT Here we add a rule allowing SSH connections over tcp port 22. This is to prevent accidental lockouts when working on remote systems over an SSH connection. We will explain this rule in more detail later.
-
iptables -P INPUT DROP The -P switch sets the default policy on the specified chain. So now we can set the default policy on the INPUT chain to DROP. This means that if an incoming packet does not match one of the following rules it will be dropped. If we were connecting remotely via SSH and had not added the rule above, we would have just locked ourself out of the system at this point.
-
iptables -P FORWARD DROP Similarly, here we've set the default policy on the FORWARD chain to DROP as we're not using our computer as a router so there should not be any packets passing through our computer.
-
iptables -P OUTPUT ACCEPT and finally, we've set the default policy on the OUTPUT chain to ACCEPT as we want to allow all outgoing traffic (as we trust our users).
-
iptables -A INPUT -i lo -j ACCEPT Now it's time to start adding some rules. We use the -A switch to append (or add) a rule to a specific chain, the INPUT chain in this instance. Then we use the -i switch (for interface) to specify packets matching or destined for the lo (localhost, 127.0.0.1) interface and finally -j (jump) to the target action for packets matching the rule - in this case ACCEPT. So this rule will allow all incoming packets destined for the localhost interface to be accepted. This is generally required as many software applications expect to be able to communicate with the localhost adaptor.
-
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT This is the rule that does most of the work, and again we are adding (-A) it to the INPUT chain. Here we're using the -m switch to load a module (state). The state module is able to examine the state of a packet and determine if it is NEW, ESTABLISHED or RELATED. NEW refers to incoming packets that are new incoming connections that weren't initiated by the host system. ESTABLISHED and RELATED refers to incoming packets that are part of an already established connection or related to and already established connection.
-
iptables -L -v Finally, we can list (-L) the rules we've just added to check they've been loaded correctly.
Finally, the last thing we need to do is save our rules so that next time we reboot our computer our rules are automatically reloaded:
# /sbin/service iptables save
This executes the iptables init script, which runs /sbin/iptables-save and writes the current iptables configuration to /etc/sysconfig/iptables. Upon reboot, the iptables init script reapplies the rules saved in /etc/sysconfig/iptables by using the /sbin/iptables-restore command.
Obviously typing all these commands at the shell can become tedious, so by far the easiest way to work with iptables is to create a simple script to do it all for you. The above commands may be entered into your favourite text editor and saved as myfirewall, for example:
#!/bin/bash # # iptables example configuration script # # Flush all current rules from iptables # iptables -F # # Allow SSH connections on tcp port 22 # This is essential when working on remote servers via SSH to prevent locking yourself out of the system # iptables -A INPUT -p tcp --dport 22 -j ACCEPT # # Set default policies for INPUT, FORWARD and OUTPUT chains # iptables -P INPUT DROP iptables -P FORWARD DROP iptables -P OUTPUT ACCEPT # # Set access for localhost # iptables -A INPUT -i lo -j ACCEPT # # Accept packets belonging to established and related connections # iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT # # Save settings # /sbin/service iptables save # # List rules # iptables -L -v
Note: We can also comment our script to remind us what were doing.
now make the script executable:
# chmod +x myfirewall
We can now simply edit our script and run it from the shell with the following command:
# ./myfirewall
4. Interfaces
In our previous example, we saw how we could accept all packets incoming on a particular interface, in this case the localhost interface:
iptables -A INPUT -i lo -j ACCEPT
Suppose we have 2 separate interfaces, eth0 which is our internal LAN connection and ppp0 dialup modem (or maybe eth1 for a nic) which is our external internet connection. We may want to allow all incoming packets on our internal LAN but still filter incoming packets on our external internet connection. We could do this as follows:
iptables -A INPUT -i lo -j ACCEPT iptables -A INPUT -i eth0 -j ACCEPT
But be very careful - if we were to allow all packets for our external internet interface (for example, ppp0 dialup modem):
iptables -A INPUT -i ppp0 -j ACCEPT
we would have effectively just disabled our firewall!
5. IP Addresses
Opening up a whole interface to incoming packets may not be restrictive enough and you may want more control as to what to allow and what to reject. Lets suppose we have a small network of computers that use the 192.168.0.x private subnet. We can open up our firewall to incoming packets from a single trusted IP address (for example, 192.168.0.4):
# Accept packets from trusted IP addresses iptables -A INPUT -s 192.168.0.4 -j ACCEPT # change the IP address as appropriate
Breaking this command down, we first append (-A) a rule to the INPUT chain for the source (-s) IP address 192.168.0.4 to ACCEPT all packets (also note how we can use the # symbol to add comments inline to document our script with anything after the # being ignored and treated as a comment).
Obviously if we want to allow incoming packets from a range of IP addresses, we could simply add a rule for each trusted IP address and that would work fine. But if we have a lot of them, it may be easier to add a range of IP addresses in one go. To do this, we can use a netmask or standard slash notation to specify a range of IP address. For example, if we wanted to open our firewall to all incoming packets from the complete 192.168.0.x (where x=1 to 254) range, we could use either of the following methods:
# Accept packets from trusted IP addresses iptables -A INPUT -s 192.168.0.0/24 -j ACCEPT # using standard slash notation iptables -A INPUT -s 192.168.0.0/255.255.255.0 -j ACCEPT # using a subnet mask
Finally, as well as filtering against a single IP address, we can also match against the MAC address for the given device. To do this, we need to load a module (the mac module) that allows filtering against mac addresses. Earlier we saw another example of using modules to extend the functionality of iptables when we used the state module to match for ESTABLISHED and RELATED packets. Here we use the mac module to check the mac address of the source of the packet in addition to it's IP address:
# Accept packets from trusted IP addresses iptables -A INPUT -s 192.168.0.4 -m mac --mac-source 00:50:8D:FD:E6:32 -j ACCEPT
First we use -m mac to load the mac module and then we use --mac-source to specify the mac address of the source IP address (192.168.0.4). You will need to find out the mac address of each ethernet device you wish to filter against. Running ifconfig (or iwconfig for wireless devices) as root will provide you with the mac address.
This may be useful for preventing spoofing of the source IP address as it will allow any packets that genuinely originate from 192.168.0.4 (having the mac address 00:50:8D:FD:E6:32) but will block any packets that are spoofed to have come from that address. Note, mac address filtering won't work across the internet but it certainly works fine on a LAN.
6. Ports and Protocols
Above we have seen how we can add rules to our firewall to filter against packets matching a particular interface or a source IP address. This allows full access through our firewall to certain trusted sources (host PCs). Now we'll look at how we can filter against protocols and ports to further refine what incoming packets we allow and what we block.
Before we can begin, we need to know what protocol and port number a given service uses. For a simple example, lets look at bittorrent. Bittorrent uses the tcp protocol on port 6881, so we would need to allow all tcp packets on destination port (the port on which they arrive at our machine) 6881:
# Accept tcp packets on destination port 6881 (bittorrent) iptables -A INPUT -p tcp --dport 6881 -j ACCEPT
Here we append (-A) a rule to the INPUT chain for packets matching the tcp protocol (-p tcp) and entering our machine on destination port 6881 (--dport 6881).
Note: In order to use matches such as destination or source ports (--dport or --sport), you must first specify the protocol (tcp, udp, icmp, all).
We can also extend the above to include a port range, for example, allowing all tcp packets on the range 6881 to 6890:
# Accept tcp packets on destination ports 6881-6890 iptables -A INPUT -p tcp --dport 6881:6890 -j ACCEPT
7. Putting It All Together
Now we've seen the basics, we can start combining these rules.
A popular UNIX/Linux service is the secure shell (SSH) service allowing remote logins. By default SSH uses port 22 and again uses the tcp protocol. So if we want to allow remote logins, we would need to allow tcp connections on port 22:
# Accept tcp packets on destination port 22 (SSH) iptables -A INPUT -p tcp --dport 22 -j ACCEPT
This will open up port 22 (SSH) to all incoming tcp connections which poses a potential security threat as hackers could try brute force cracking on accounts with weak passwords. However, if we know the IP addresses of trusted remote machines that will be used to log on using SSH, we can limit access to only these source IP addresses. For example, if we just wanted to open up SSH access on our private lan (192.168.0.x), we can limit access to just this source IP address range:
# Accept tcp packets on destination port 22 (SSH) from private LAN iptables -A INPUT -p tcp -s 192.168.0.0/24 --dport 22 -j ACCEPT
Using source IP filtering allows us to securely open up SSH access on port 22 to only trusted IP addresses. For example, we could use this method to allow remote logins between work and home machines. To all other IP addresses, the port (and service) would appear closed as if the service were disabled so hackers using port scanning methods are likely to pass us by.