Friday, February 22, 2013

Prevent DOS with iptables


Prevent DOS with iptables

After a recent conversation on the Ubuntu Forums I wanted to post an example of using iptables.
Of course there are several types of DOS attacks , in this post I will demonstrating the use if iptables to limit the traffic on port 80.
The goal is to keep your web server “responsive” to legitimate traffic, but to throttle back on excessive (potential DOS) traffic.
In this demonstration iptables is configured :
  1. The default policy is ACCEPT (to prevent lockout in the event of flushing the rules with iptables -F).
  2. “Legitimate” traffic is then allowed. In this example I am allowing traffic only on port 80.
  3. All other traffic is then blocked at the end of the INPUT chain (the final rule in the INPUT chain is to DROP all traffic).
The rules I will demonstrate are as follows:

First rule : Limit NEW traffic on port 80

sudo iptables -A INPUT -p tcp --dport 80 -m state --state NEW -m limit --limit 50/minute --limit-burst 200 -j ACCEPT
Lets break that rule down into intelligible chunks.
-p tcp --dport 80 => Specifies traffic on port 80 (Normally Apache, but as you can see here I am using nginx).
-m state NEW => This rule applies to NEW connections.
-m limit --limit 50/minute --limit-burst 200 -j ACCEPT =>This is the essence of preventing DOS.
  • “--limit-burst” is a bit confusing, but in a nutshell 200 new connections (packets really) are allowed before the limit of 50 NEW connections (packets) per minute is applied.
For a more technical review of this rule, see this netfilet page. Scroll down to a bit to the “limit” section.

Second rule – Limit established traffic

This rule applies to RELATED and ESTABLISHED all traffic on all ports, but is very liberal (and thus should not affect traffic on port 22 or DNS).
If you understood the above rule, you should understand this one as well.
sudo iptables -A INPUT -m state --state RELATED,ESTABLISHED -m limit --limit 50/second --limit-burst 50 -j ACCEPT
In summary, 50 ESTABLISHED (and/or RELATED) connections (packets really) are allowed before the limit of 50 ESTABLISHED (and/or RELATED) connections (packets) per second is applied.
Do not let that rule fool you, although it seems very open, it does put some limits on your connections.
Test it for yourself, try using the first rule with and without the second rule.

Full set of rules

After the above commands, here is the complete set of rules I am testing:
iptables-save
# Generated by iptables-save v1.4.4 on --
*nat
:PREROUTING ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
COMMIT
# Completed on --
# Generated by iptables-save --
*mangle
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
COMMIT
# Completed on --
# Generated by iptables-save v1.4.4 on --
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -i lo -j ACCEPT
-A INPUT -m state --state RELATED,ESTABLISHED -m limit --limit 50/sec \
--limit-burst 50 -j ACCEPT
-A INPUT -p icmp -m limit --limit 1/sec -j ACCEPT
-A INPUT -p tcp -m tcp --dport 80 -m state --state NEW -m limit --limit 50/min \
--limit-burst 200 -j ACCEPT
-A INPUT -j LOG
-A INPUT -j DROP
-A FORWARD -j DROP
-A OUTPUT -o lo -j ACCEPT
COMMIT
# Completed on --
This rule set is for demonstration only and is NOT a complete set of rules for a web server. Do no use this rule set unmodified on a production server.

Testing the rule set

Human interaction

Open Firefox, point it to your web page. The web page should load nice and fast.
Hit F5 repetitively, load the page as fast as you can. Your web site should remain nice and responsive.
So far, so good, we want our site to remain responsive.

Simulated DOS

Actual DOS attacks are many times faster then humans, here I will use ab.
See this link or the Apache documentation for information of ab.

Baseline, without the above 2 rules

ab -n 100 -c 10 http://bodhi's_test_server.com/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking bodhi's_test_server.com (be patient).....done


Server Software:        nginx
Server Hostname:       bodhi's_test_server.com
Server Port:            80

Document Path:          /
Document Length:        59786 bytes

Concurrency Level:      10
Time taken for tests:   13.174 seconds
Complete requests:      100
Failed requests:        0
Write errors:           0
Total transferred:      6002700 bytes
HTML transferred:       5978600 bytes
Requests per second:    7.59 [#/sec] (mean)
Time per request:       1317.369 [ms] (mean)
Time per request:       131.737 [ms] (mean, across all concurrent requests)
Transfer rate:          444.98 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:      122  129   2.2    128     134
Processing:  1151 1182  19.1   1177    1260
Waiting:      125  132   8.2    128     170
Total:       1280 1310  19.3   1305    1390

Percentage of the requests served within a certain time (ms)
  50%   1305
  66%   1313
  75%   1316
  80%   1321
  90%   1328
  95%   1354
  98%   1386
  99%   1390
 100%   1390 (longest request)
Notice:
Requests per second: 7.59 [#/sec] .
Total time for requests: 13 seconds .
(Data) Transfer rate: 444.98 [Kbytes/sec] .

With the above rules

First attempt:
ab -n 100 -c 10 http://bodhi's_test_server.com/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking bodhi's_test_server.com (be patient)...
apr_poll: The timeout specified has expired (70007)
Total of 99 requests completed
Oh no ! timed out, LOL
Second attempt (I reduced the number of requests to 90):
ab -n 90 -c 10 http://bodhi's_test_server.com/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking bodhi's_test_server.com (be patient).....done


Server Software:        nginx
Server Hostname:       bodhi's_test_server.com
Server Port:            80

Document Path:          /
Document Length:        59786 bytes

Concurrency Level:      10
Time taken for tests:   69.684 seconds
Complete requests:      90
Failed requests:        0
Write errors:           0
Total transferred:      5402430 bytes
HTML transferred:       5380740 bytes
Requests per second:    1.29 [#/sec] (mean)
Time per request:       7742.658 [ms] (mean)
Time per request:       774.266 [ms] (mean, across all concurrent requests)
Transfer rate:          75.71 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:      123  128   4.3    127     155
Processing:  1036 6269 10081.4   1921   51059
Waiting:      125 1240 5908.7    128   49656
Total:       1159 6396 10081.1   2044   51186

Percentage of the requests served within a certain time (ms)
  50%   2044
  66%   2981
  75%   5478
  80%   7047
  90%  20358
  95%  27356
  98%  48218
  99%  51186
 100%  51186 (longest request)
Notice :
Requests per second: 1.29 [#/sec] (mean)
Total time for requests: 69 seconds.
(Data) Transfer rate: 75.71 [Kbytes/sec] [Kbytes/sec].

For those unfamiliar with ab, that is a “minor” DOS

For comparison, here is what ab can do to the server (iptables was flushed [disabled]):
ab -n 1000 -c 100 http://bodhi's_test_server.com/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking bodhi's_test_server.com (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests


Server Software:        nginx
Server Hostname:       bodhi's_test_server.com
Server Port:               80

Document Path:          /
Document Length:        58708 bytes

Concurrency Level:      100
Time taken for tests:   59.324 seconds
Complete requests:      1000
Failed requests:        945
   (Connect: 0, Receive: 0, Length: 945, Exceptions: 0)
Write errors:           0
Total transferred:      59190450 bytes
HTML transferred:       58945935 bytes
Requests per second:    16.86 [#/sec] (mean)
Time per request:       5932.368 [ms] (mean)
Time per request:       59.324 [ms] (mean, across all concurrent requests)
Transfer rate:          974.37 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:      127  908 817.9    788    8016
Processing:   735 4779 1805.2   4368   15707
Waiting:      128  981 827.2    811   12143
Total:       1058 5687 1880.8   5269   17450

Percentage of the requests served within a certain time (ms)
  50%   5269
  66%   5899
  75%   6340
  80%   6863
  90%   8078
  95%   9001
  98%  10937
  99%  11730
 100%  17450 (longest request)

Notice :
Requests per second: 16.86 [#/sec]
Total time for requests: 69 seconds.
(Data) Transfer rate: 974.37 [Kbytes/sec] .
As you can see, the server has no problem dishing out 974.37 [Kbytes/sec] !!!

Closing remarks

Hopefully you now understand this “simple” example limiting a DOS on port 80.
With these rules your web site remains responsive to human interaction in firefox. Go ahead, hit F5 (refresh the page) as fast as you can, see if you can get your web page to slow down =) .
The difference is that as with a DOS attack, ab is hitting the server faster then you can with F5 , so your site is responsive to “normal” activity, but blocks DOS.
Obviously this is but one example and there are several types of DOS attacks. The goal is to demonstrate the use of iptables using a few “simple” rules.
You task is to take this knowledge and apply it to you own server.

Howto: Performance Benchmarks a Webserver


Howto: Performance Benchmarks a Webserver

by  on JUNE 9, 2006 · 34 COMMENTS· LAST UPDATED NOVEMBER 13, 2008
You can benchmark Apache, IIS and other web server with apache benchmarking tool called ab. Recently I was asked to performance benchmarks for different web servers.
It is true that benchmarking a web server is not an easy task. From how to benchmark a webserver:
First, benchmarking a web server is not an easy thing. To benchmark a web server the time it will take to give a page is not important: you don't care if a user can have his page in 0.1 ms or in 0.05 ms as nobody can have such delays on the Internet.
What is important is the average time it will take when you have a maximum number of users on your site simultaneously. Another important thing is how much more time it will take when there are 2 times more users: a server that take 2 times more for 2 times more users is better than another that take 4 times more for the same amount of users."
Here are few tips to carry out procedure along with an example:

Apache Benchmark Procedures

  • You need to use same hardware configuration and kernel (OS) for all tests
  • You need to use same network configuration. For example, use 100Mbps port for all tests
  • First record server load using top or uptime command
  • Take at least 3-5 readings and use the best result
  • After each test reboot the server and carry out test on next configuration (web server)
  • Again record server load using top or uptime command
  • Carry on test using static html/php files and dynamic pages
  • It also important to carry out test using the Non-KeepAlive and KeepAlive (the Keep-Alive extension to provide long-lived HTTP sessions, which allow multiple requests to be sent over the same TCP connection) features
  • Also don't forget to carry out test using fast-cgi and/or perl tests

Webserver Benchmark Examples:

Let us see how to benchmark a Apache 2.2 and lighttpd 1.4.xx web server.

Static Non-KeepAlive test for Apache web server

i) Note down server load using uptime command
$ uptime
ii) Create a static (small) html page as follows (snkpage.html) (assuming that server IP is 202.54.200.1) in /var/www/html (or use your own webroot):
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<html>
<head>
<title>Webserver test</title>
</head>
<body>
This is a webserver test page.
</body>
</html>
 
Login to Linux/bsd desktop computer and type following command:
$ ab -n 1000 -c 5 http://202.54.200.1/snkpage.html
Where,
  • -n 1000: ab will send 1000 number of requests to server 202.54.200.1 in order to perform for the benchmarking session
  • -c 5 : 5 is concurrency number i.e. ab will send 5 number of multiple requests to perform at a time to server 202.54.200.1
For example if you want to send 10 request, type following command:
$ ab -n 10 -c 2 http://www.somewhere.com/
Output:
This is ApacheBench, Version 2.0.41-dev <$Revision: 1.141 $> apache-2.0
Copyright (c) 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright (c) 1998-2002 The Apache Software Foundation, http://www.apache.org/
Benchmarking www.cyberciti.biz (be patient).....done
Server Software:
Server Hostname:        www.somewhere.com
Server Port:            80
Document Path:          /
Document Length:        16289 bytes
Concurrency Level:      1
Time taken for tests:   16.885975 seconds
Complete requests:      10
Failed requests:        0
Write errors:           0
Total transferred:      166570 bytes
HTML transferred:       162890 bytes
Requests per second:    0.59 [#/sec] (mean)
Time per request:       1688.597 [ms] (mean)
Time per request:       1688.597 [ms] (mean, across all concurrent requests)
Transfer rate:          9.59 [Kbytes/sec] received
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:      353  375  16.1    386     391
Processing:  1240 1312  52.1   1339    1369
Waiting:      449  472  16.2    476     499
Total:       1593 1687  67.7   1730    1756
Percentage of the requests served within a certain time (ms)
  50%   1730
  66%   1733
  75%   1741
  80%   1753
  90%   1756
  95%   1756
  98%   1756
  99%   1756
 100%   1756 (longest request)
Repeat above command 3-5 times and save the best reading.

Static Non-KeepAlive test for lighttpd web server

First, reboot the server:
# reboot
Stop Apache web server. Now configure lighttpd and copy /var/www/html/snkpage.html to lighttpd webroot and run the command (from other linux/bsd system):
$ ab -n 1000 -c 5 http://202.54.200.1/snkpage.html
c) Plot graph using Spreadsheet or gnuplot.

How do I carry out Web server Static KeepAlive test?

Use -k option that enables the HTTP KeepAlive feature using ab test tool. For example:
$ ab -k -n 1000 -c 5 http://202.54.200.1/snkpage.html
Use the above procedure to create php, fast-cgi and dynmic pages to benchmarking the web server.
Please note that 1000 request is a small number you need to send bigger (i.e. the hits you want to test) requests, for example following command will send 50000 requests :
$ ab -k -n 50000 -c 2 http://202.54.200.1/snkpage.html

How do I save result as a Comma separated value?

Use -e option that allows to write a comma separated value (CSV) file which contains for each percentage (from 1% to 100%) the time (in milliseconds) it took to serve that percentage of the requests:
$ ab -k -n 50000 -c 2 -e apache2r1.cvs http://202.54.200.1/snkpage.html

How do I import result into excel or gnuplot programs so that I can create graphs?

Use above command or -g option as follows:
$ ab -k -n 50000 -c 2 -g apache2r3.txt http://202.54.200.1/snkpage.html
Put following files in your webroot (/var/www/html or /var/www/cgi-bin) directory. Use ab command.

Sample test.php file

#!/usr/bin/perl
$command=`perl -v`;
$title = "Perl Version";
 
print "Content-type: text/html\n\n";
print "<html><head><title>$title</title></head>\n<body>\n\n";
 
print "<h1>$title</h1>\n";
print $command;
 
print "\n\n</body></html>";
 
Run ab command as follows:
$ ab -n 3000 -c 5 http://202.54.200.1/cgi-bin/test.pl

Sample psql.php (php+mysql) file

<html>
<head><title>Php+MySQL</title></head>
<body>
<?php
   $link = mysql_connect("localhost", "USERNAME", "PASSWORD");
   mysql_select_db("DATABASE");
 
   $query = "SELECT * FROM TABLENAME";
   $result = mysql_query($query);
 
   while ($line = mysql_fetch_array($result))
   {
      foreach ($line as $value)
       {
         print "$value\n";
      }
   }
 
    mysql_close($link);
?>
</body>
</html>
 
Run ab command as follows:
$ ab -n 1000 -c 5 http://202.54.200.1/psql.phpReference Link
http://www.cyberciti.biz/tips/howto-performance-benchmarks-a-web-server.html

Thursday, February 14, 2013

CentOS Linux install and configure NTP to synchronize the system clock


Q. How do I install and configure NTP under CentOS Linux 5 server to synchronize the system clock?
A. You can easily install NTP (Network Time Protocol, a means of transmitting time signals over a computer network) using yum command under Redhat or CentOS/Fedora core Linux.

Procedure

Login as the root user
Type the following command to install ntp
# yum install ntp
Turn on service
# chkconfig ntpd on
Synchronize the system clock with 0.pool.ntp.org server:
# ntpdate pool.ntp.org
Start the NTP:
# /etc/init.d/ntpd start

Install lamp with 1 command in Ubuntu 12.04, 12.10 QuantalQuetzal & LinuxMint13


This tutorial was has been tested on Ubuntu 10.04, 10.10, 11.04, 11.10, 12.04 LTS Precise Pangolin. Also tested in LinuxMint13 and works fine.
Open terminal and Type the command :install it   first  with


sudo apt-get install tasksel

Now to install LAMP, type the taskel command in terminal :

sudo  tasksel

And select LAMP Server


During the installation  you  will be  asked  to insert the  mysql root  password
Now check if php is working :

$sudo vi /var/www/info.php
and add
<?php
phpinfo();
?>
save and exit
restart apache2 ,
#sudo /etc/init.d/apache2 restart

sudo  apt-get  install  phpmyadmin
To login  to phpmyadmin, open browser and type :
http://ip/phpmyadmin   or http://localhost/phpmyadmin




Monday, February 11, 2013

Setup MySQL Replication in 11 Easy Steps


Well, Fasten Your Seat-belts
  1. Install a MySQL instance that will serve as a Master
  2. Install a MySQL instance that will serve as a Slave
  3. Configure the Master my.cnf file (located at /etc/ at CentOS) with the server id and the log file name: 
    1. [mysqld]
    2. server­-id = 1
    3. log­-bin   = master­-bin.log
  4. Configure the Slave my.cnf with the server id, reply logs and key configuration databases;
    1. server­-id = 2 
    2. relay­-log-­index = slave-­relay-­bin.index
    3. relay­-log = slave­-relay­-bin
    4. replicate-wild-ignore-table=mysql.%
    5. replicate-wild-ignore-table=information_schema.%
    6. replicate-wild-ignore-table=performance_schema.%
  5. Restart the MySQL daemons on both servers to apply the my.cnf changes.
  6. If both servers were just installed there is no need to sync their data files (since they  should have the same data files). O/w you should either stop the slave and master and copy the data files using SCP or perform just perform a mysqldump. Notice! Before copying the files, get the Master location describe below.
  7. Get the master location, so we can sync the slave to it:
    1. master> FLUSH TABLES WITH READ LOCK;
    2. master> SHOW MASTER STATUS;
    3. +-------------------+----------+--------------+------------------+
    4. | File              | Position | Binlog_Do_DB | Binlog_Ignore_DB |
    5. +-------------------+----------+--------------+------------------+
    6. | master-bin.000013 |  1233453 |              |                  |
    7. +-------------------+----------+--------------+------------------+
    8. master> UNLOCK TABLES;
  8. Provide the slave with replication permissions:
    1. master> GRANT REPLICATION SLAVE ON *.* to `repl`@`%` IDENTIFIED BY 'slavepass';
    2. master> FLUSH PRIVILEGES;
  9. Setup the slave using the chosen user/password and the master location we found before:
    1. slave> CHANGE MASTER TO
    2.     ->     MASTER_HOST='10.17.16.253',
    3.     ->     MASTER_PORT=3306,
    4.     ->     MASTER_USER='repl',
    5.     ->     MASTER_PASSWORD='slavepass',
    6.     ->     MASTER_LOG_FILE='master-bin.000013',
    7.     ->     MASTER_LOG_POS=1233453;
  10. Now, start the slave and verify it's running: 
    1. slave> start slave;show slave status\G
  11. If everything is Okay, verify the replication really works by:
    1. Creating a table and inserting values to it in the master:
      1. master> CREATE DATABASE a;
      2. master> CREATE TABLE a.b (c int);
      3. master> INSERT INTO a.b (c) VALUES (1);
    2. Verifying that SLAVE/MASTER> SELECT * FROM a.b; return the same values in the master and slave;
    3. Dropping the database in the master: master> DROP DATABASE IF EXISTS a;

Thursday, February 7, 2013

Creating a MySQL slave from a running master


 Creating a MySQL slave from a running master

First, start by reading the online documentation. The MySQL documentation is well written and accurate. It goes through all of the steps in a good level of detail except for one. Unfortunately, that one step is critical.
To start MySQL replication, you need your new slave server to have an exact snapshot of the master at a point in time. MySQL recommends shutting down the master to take a backup. That sounds nice, but I can't afford several hours of downtime. The naive approach is just to use mysqldump to export a copy of the master and load it on the slave server. This works if you only have one database. With multiple database, you'll end up with inconsistent data. Mysqldump will dump data from each database on the server in a different transaction. That means that your export will have data from a different point in time for each database.
Instead of dumping all of the data at once, I dump my databases one at a time. Once the first dump completes, I then dump the next database. All databases are dumped using


mysqldump -u root -e -q --single-transaction --master-data database_name

Once I have dumped all of the databases on the server, I load the slave by importing the first database . 
I configure replication to only process replication events for this database by setting
replicate-wild-do-table= database_name.%
Next, I find out what the master log position is for my next import.NIt's in the first 25 lines of the MySQL 
dmp. Once I have that, I start the slave and tell mysql to replicate until until this position. You can
do this by passing the until argument to start slave. For example:

START SLAVE UNTIL MASTER_LOG_FILE='bin.000029', MASTER_LOG_POS=651322976;

Now, the database will be in the same state as when I dumped the next database. I then load that export. Once the load is done, I shutdown mysql and add a record for the newly import database to the replication setup. It will now look like: replicate-wild-do-table= database_name.% replicate-wild-do-table= another_db.% I then start mysql and immediately stop the slave. I can the repeat as necessary, running replication until the same time as the previous export, exporting the data then adding the new database to replication. This process allows me to easily build a new slave server from an existing MySQL master with no downtime.
Reference


How to re-sync the Mysql DB if Master and slave have different database incase of Mysql replication


Mysql Replication Broken issue

Mysql Server1 is running as MASTER.
Mysql Server2 is running as SLAVE.
Now DB replication is happening from MASTER to SLAVE.
Server2 is removed from network and re-connet it back after 1 day. After this there is mismatch in database in master and slave.
How to re-sync the DB again as after restoring DB taken from Master to Slave also doesn't solve the problem ?

This is the full step-by-step procedure to resync a master-slave replication from scratch:
At the master:
RESET MASTER;
FLUSH TABLES WITH READ LOCK;
SHOW MASTER STATUS;
 
And copy the values of the result of the last command somewhere.
Wihtout closing the connection to the client (because it would release the read lock) issue the command to get a dump of the master:
mysqldump -uroot -p --all-database > /a/path/mysqldump.sql Now you can release the lock, even if the dump hasn't end. To do it perform the following command in the mysql client:
UNLOCK TABLES; Now copy the dump file to the slave using scp or your preferred tool.
 
At the slave:
Open a connection to mysql and type:
STOP SLAVE; Load master's data dump with this console command:
mysql -uroot -p < mysqldump.sql Sync slave and master logs:
RESET SLAVE; CHANGE MASTER TO MASTER_LOG_FILE='mysql-bin.000001', MASTER_LOG_POS=98; Where the values of the above fields are the ones you copied before.
Finally type
START SLAVE; And to check that everything is working again, if you type
SHOW SLAVE STATUS; you should see:
Slave_IO_Running: Yes Slave_SQL_Running: Yes

Reference Link
http://stackoverflow.com/questions/2366018/how-to-re-sync-the-mysql-db-if-master-and-slave-have-different-database-incase-o