Mitigating a DDOS attack – can you really do it?
Today I would like to smash your confidence about the security of your organization. You spent hundreds of dollars or euros on your security devices and you think you are safe. I believe there are many organizations like yours that think the same thing. Once they become victims, they realize they are not as safe as they had thought.
Let’s talk about DOS/DDOS attacks. I think most people have some knowledge about them. Today we should not talk only about DOS attacks because the risk of being seriously impacted is very low. Most network devices can easily mitigate such attacks because they come from one source IP address. The situation is different when we receive illegitimate traffic from many IP addresses. There are two possibilities here: we can be flooded by traffic from a real, existing source IP address or addresses; or there is one real source IP, but the packets contain random source IPs, which means you are not able to block them easily.
We can distinguish the following types of DOS/DDOS attacks:
- DOS – one attacker who initiates an attack from one machine (one IP address)
- DDOS – one attacker who randomly changes the source IP address
- DDOS – traffic sent from ‘zombies’, i.e. infected machines managed by one central host.
I would like to analyze these three types of attacks in order to mitigate them. I prepared a virtual lab with the following elements:
- ISP – Cisco router – IOS 15.2
- Perimeter Firewall – ASA – 8.4
10.0.0.10 <---------- A T T A C K ------------------------------|
/-----\ ----- /----\ -----------
|WEB SRV|-------| ASA1 |----------| ISP |--[INTERNET ]------| ATTACKER |
\-----/ ----- \----/ -----------
Fig.1 Network Diagram
From the ‘Attacker’ machine I will generate traffic to the ‘Web server’, located in the DMZ. This scenario is the most common for small and medium-sized businesses (SMBs) segment. Usually these businesses manage their own firewall and have one ISP. IPS/IDS appliances are not too popular with SMBs .
Lab #1 – DOS attack
I started the first simulation with a DOS attack on a server (10.0.0.10). I set the firewall to permit all TCP traffic from any source to the destination 10.0.0.10 on the port 80.
First I checked the current configuration and utilization on all devices:
ISP#sh processes cpu sorted
CPU utilization for five seconds: 0%/0%; one minute: 1%; five minutes: 7%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
239 17132 254464 67 0.23% 0.26% 0.30% 0 ISG MIB jobs Man
112 9740 127455 76 0.15% 0.14% 0.15% 0 IPAM Manager
2 2660 874 3043 0.07% 0.04% 0.05% 0 Load Meter
59 6960 1304 5337 0.07% 0.06% 0.07% 0 HC Counter Timer
268 5224 877 5956 0.07% 0.07% 0.07% 0 Compute load avg
174 3548 42094 84 0.07% 0.06% 0.07% 0 RBSCP Background
82 7964 127454 62 0.07% 0.15% 0.15% 0 IP ARP Retry Age
8 0 2 0 0.00% 0.00% 0.00% 0 Timers
7 96 29 3310 0.00% 0.00% 0.00% 0 DiscardQ Backgro
10 26904 1459 18440 0.00% 0.08% 0.15% 0 ARP Input
6 11828 950 12450 0.00% 0.00% 0.00% 0 Pool Manager
--More--
Fig.2 CPU usage – ISP router
ISP#sh run int gig0/0
Building configuration...
Current configuration : 165 bytes
!
interface GigabitEthernet0/0
ip address 192.168.157.10 255.255.255.0
ip access-group OUTSIDE in
media-type gbic
speed 1000
duplex full
negotiation auto
end
!
ISP#sh run | b access-list
ip access-list extended OUTSIDE
permit tcp any host 10.0.0.10 eq www log
!
Fig.3 Interface settings and ACL on ISP
PER-FW# sh cpu usage
CPU utilization for 5 seconds = 0%; 1 minute: 0%; 5 minutes: 0%
PER-FW#
Fig.4 CPU usage – Perimeter Firewall
PER-FW# sh access-list
access-list cached ACL log flows: total 0, denied 0 (deny-flow-max 4096)
alert-interval 300
access-list OUT; 1 elements; name hash: 0xcd7d0798
access-list OUT line 1 extended permit tcp any host 10.0.0.10 eq www (hitcnt=1) 0x5f077c5e
PER-FW#
Fig.5 Number of connection passed by the firewall
Let’s start the DOS attack using the ‘t50’ tool (part of the KALI distribution).
root@kali:~# t50 10.0.0.10 --flood --syn -s 192.168.157.200 --dport 80
entering in flood mode...
hit CTRL+C to break.
T50 5.4.1-rc1 successfully launched on Aug 30th 2014 22:41:39
After one minute I found almost 80,000 SYN packets:
ISP#sh access-lists
Extended IP access list OUTSIDE
10 permit tcp any host 10.0.0.10 eq www log (79287 matches)
ISP#
As we see, CPU usage is close to 100%:
ISP#sh processes cpu
CPU utilization for five seconds: 96%/100%; one minute: 32%; five minutes: 30%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
1 0 2 0 0.00% 0.00% 0.00% 0 Chunk Manager
2 5220 1525 3422 0.07% 0.04% 0.05% 0 Load Meter
3 83540 5068 16483 3.73% 0.78% 0.58% 0 Exec
4 0 1 0 0.00% 0.00% 0.00% 0 RO Notify Timers
5 11812 1307 9037 0.00% 0.15% 0.12% 0 Check heaps
6 57772 4431 13038 2.30% 1.02% 0.99% 0 Pool Manager
7 212 67 3164 0.00% 0.00% 0.00% 0 DiscardQ Backgro
8 0 2 0 0.00% 0.00% 0.00% 0 Timers
9 344 397 866 0.00% 0.00% 0.00% 0 WATCH_AFS
10 46644 4333 10764 0.07% 0.21% 0.18% 0 ARP Input
11 4880 7878 619 0.07% 0.00% 0.00% 0 ARP Background
--More--
Here are examples of connections initiated from the ‘Attacker’ seen on the ISP’s router:
ISP#sh logg
...
*Aug 31 00:20:46.451: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 192.168.157.200(9599) -> 10.0.0.10(80), 1 packet
*Aug 31 00:20:46.451: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 192.168.157.200(2627) -> 10.0.0.10(80), 1 packet
*Aug 31 00:20:46.451: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 192.168.157.200(8561) -> 10.0.0.10(80), 1 packet
*Aug 31 00:20:46.451: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 192.168.157.200(1329) -> 10.0.0.10(80), 1 packet
*Aug 31 00:21:17.215: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 192.168.157.200(18443) -> 10.0.0.10(80), 1 packet
*Aug 31 00:21:18.227: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 192.168.157.200(13679) -> 10.0.0.10(80), 1 packet
*Aug 31 00:21:19.247: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 192.168.157.200(9995) -> 10.0.0.10(80), 1 packet
*Aug 31 00:21:20.275: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 192.168.157.200(5977) -> 10.0.0.10(80), 1 packet
*Aug 31 00:21:21.311: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 192.168.157.200(3745) -> 10.0.0.10(80), 1 packet
*Aug 31 00:21:22.343: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 192.168.157.200(15022) -> 10.0.0.10(80), 1 packet
From the Perimeter Firewall’s point of view, the attack was not so heavy. The CPU usage was around 1-2%:
PER-FW# sh access-list
access-list cached ACL log flows: total 1529, denied 0 (deny-flow-max 4096)
alert-interval 300
access-list OUT; 1 elements; name hash: 0xcd7d0798
access-list OUT line 1 extended permit tcp any host 10.0.0.10 eq www log informational interval 300 (hitcnt=879) 0x5f077c5e
PER-FW# sh cpu usage
CPU utilization for 5 seconds = 1%; 1 minute: 2%; 5 minutes: 1%
PER-FW#
The reason for the low impact might be the number of packets which hit the PER-FW. Let’s check the counters on the ISP:
ISP#sh int gig0/0 stat
GigabitEthernet0/0
Switching path Pkts In Chars In Pkts Out Chars Out
Processor 81789 4906579 17 1658
Route cache 0 0 0 0
Total 81789 4906579 17 1658
ISP#sh int gig1/0 stat
GigabitEthernet1/0
Switching path Pkts In Chars In Pkts Out Chars Out
Processor 0 0 8962 538358
Route cache 349 20242 0 0
Total 349 20242 8962 538358
As you can see, the number of packets sent out from the ISP router is 10 times less than it received. This could be caused by a lack of resources to process all packets.
Here we have sample connections initiated by the ‘Attacker’ as seen on the PER-FW:
%ASA-6-302013: Built inbound TCP connection 12328 for outside:192.168.157.200/13402 (192.168.157.200/13402) to inside:10.0.0.10/80 (10.0.0.10/80)
%ASA-6-106100: access-list OUT permitted tcp outside/192.168.157.200(23561) -> inside/10.0.0.10(80) hit-cnt 1 first hit [0x5f077c5e, 0x0]
%ASA-6-302013: Built inbound TCP connection 12329 for outside:192.168.157.200/23561 (192.168.157.200/23561) to inside:10.0.0.10/80 (10.0.0.10/80)
%ASA-6-106100: access-list OUT permitted tcp outside/192.168.157.200(29917) -> inside/10.0.0.10(80) hit-cnt 1 first hit [0x5f077c5e, 0x0]
%ASA-6-302013: Built inbound TCP connection 12330 for outside:192.168.157.200/29917 (192.168.157.200/29917) to inside:10.0.0.10/80 (10.0.0.10/80)
%ASA-6-106100: access-list OUT permitted tcp outside/192.168.157.200(17876) -> inside/10.0.0.10(80) hit-cnt 1 first hit [0x5f077c5e, 0x0]
%ASA-6-302013: Built inbound TCP connection 12331 for outside:192.168.157.200/17876 (192.168.157.200/17876) to inside:10.0.0.10/80 (10.0.0.10/80)
%ASA-6-106100: access-list OUT permitted tcp outside/192.168.157.200(14328) -> inside/10.0.0.10(80) hit-cnt 1 first hit [0x5f077c5e, 0x0]
%ASA-6-302013: Built inbound TCP connection 12332 for outside:192.168.157.200/14328 (192.168.157.200/14328) to inside:10.0.0.10/80 (10.0.0.10/80)
%ASA-6-106100: access-list OUT permitted tcp outside/192.168.157.200(5033) -> inside/10.0.0.10(80) hit-cnt 1 first hit [0x5f077c5e, 0x0]
%ASA-6-302013: Built inbound TCP connection 12333 for outside:192.168.157.200/5033 (192.168.157.200/5033) to inside:10.0.0.10/80 (10.0.0.10/80)
<--- More --->
In this case we can easily locate the source of our problems. Let’s see how we can do this.
The first problem you may notice is the ‘log’ parameter in the access-list. This is a very CPU intensive parameter and it shouldn’t be used in your production configuration:
conf t
ip access-list extended OUTSIDE
no permit tcp any host 10.0.0.10 eq www log
permit tcp any host 10.0.0.10 eq www
end
Here I’ve initiatedan attack once again to check the results:
ISP#sh processes cpu sorted
CPU utilization for five seconds: 31%/100%; one minute: 61%; five minutes: 33%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
145 27216 12685 2145 1.03% 0.27% 0.10% 0 CEF: IPv4 proces
239 38360 494717 77 0.71% 0.74% 0.44% 0 ISG MIB jobs Man
112 20068 251110 79 0.31% 0.28% 0.20% 0 IPAM Manager
82 15004 251100 59 0.31% 0.26% 0.19% 0 IP ARP Retry Age
10 52028 5623 9252 0.31% 0.58% 0.27% 0 ARP Input
2 6136 1793 3422 0.07% 0.06% 0.06% 0 Load Meter
27 4340 8667 500 0.07% 0.02% 0.00% 0 IPC Periodic Tim
174 9048 84571 106 0.07% 0.12% 0.10% 0 RBSCP Background
270 66340 8950 7412 0.07% 0.67% 0.42% 0 Per-Second Jobs
9 360 475 757 0.00% 0.00% 0.00% 0 WATCH_AFS
7 220 76 2894 0.00% 0.00% 0.00% 0 DiscardQ Backgro
--More--
CPU usage has dropped from 96% to 31%!
ISP#sh int gig0/0 stat
GigabitEthernet0/0
Switching path Pkts In Chars In Pkts Out Chars Out
Processor 193 11307 30 2774
Route cache 390697 23441820 0 0
Total 390890 23453127 30 2774
ISP#sh int gig1/0 stat
GigabitEthernet1/0
Switching path Pkts In Chars In Pkts Out Chars Out
Processor 0 0 20 2174
Route cache 413 23954 6971 376434
Total 413 23954 6991 378608
ISP#
Let’s check the CPU usage of the PER-FW:
PER-FW# sh cpu usage
CPU utilization for 5 seconds = 10%; 1 minute: 5%; 5 minutes: 2%
PER-FW# sh access-list
access-list cached ACL log flows: total 5011, denied 0 (deny-flow-max 4096)
alert-interval 300
access-list OUT; 1 elements; name hash: 0xcd7d0798
access-list OUT line 1 extended permit tcp any host 10.0.0.10 eq www log informational interval 300 (hitcnt=5123) 0x5f077c5e
PER-FW#
In this case we have to isolate the traffic from the PRE-FW and the DMZ. The easiest way is to add an access-list (ACL) which blocks the traffic based on its source IP address:
!
interface GigabitEthernet0/0
ip address 192.168.157.10 255.255.255.0
ip access-group ATTACK-ACL in
media-type gbic
speed 1000
duplex full
negotiation auto
!
!
ip access-list extended ATTACK-ACL
deny tcp host 192.168.157.200 host 10.0.0.10 eq www
!
The negative effect of the ACL is higher CPU usage, but as you see not one packet hit the PER-FW:
ISP#sh processes cpu sorted
CPU utilization for five seconds: 89%/100%; one minute: 77%; five minutes: 35%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
83 1339100 99226 13495 37.87% 31.74% 11.97% 0 IP Input
3 127132 9806 12964 1.12% 0.70% 1.22% 0 Exec
145 38500 18525 2078 0.71% 0.18% 0.10% 0 CEF: IPv4 proces
112 29524 366360 80 0.47% 0.19% 0.17% 0 IPAM Manager
239 60840 722891 84 0.39% 0.37% 0.35% 0 ISG MIB jobs Man
2 9348 2610 3581 0.31% 0.10% 0.06% 0 Load Meter
44 3724 12985 286 0.23% 0.02% 0.00% 0 GraphIt
59 19504 3887 5017 0.07% 0.09% 0.08% 0 HC Counter Timer
98 8872 50501 175 0.07% 0.07% 0.02% 0 SSS Feature Time
82 23952 366347 65 0.07% 0.13% 0.15% 0 IP ARP Retry Age
270 88148 12990 6785 0.07% 0.49% 0.24% 0 Per-Second Jobs
ISP#
PER-FW# sh cpu usage
CPU utilization for 5 seconds = 1%; 1 minute: 1%; 5 minutes: 1%
PER-FW# sh access-list
access-list cached ACL log flows: total 0, denied 0 (deny-flow-max 4096)
alert-interval 300
access-list OUT; 1 elements; name hash: 0xcd7d0798
access-list OUT line 1 extended permit tcp any host 10.0.0.10 eq www log informational interval 300 (hitcnt=0) 0x5f077c5e
PER-FW#
The next option is to use Policy Based Routing (PBR) for traffic coming from the ‘Attacker’:
!
access-list 1 permit 192.168.157.200
!
route-map ATTACK permit 10
match ip address 1
set interface Null0
!
interface gig0/0
ip policy route-map ATTACK
!
ISP#sh route-map ATTACK
route-map ATTACK, permit, sequence 10
Match clauses:
ip address (access-lists): 1
Set clauses:
interface Null0
Policy routing matches: 598255 packets, 35895300 bytes
ISP#
It should be processed by the hardware, so it will not impact the CPU on the ISP’s router.
Lab #2 – DDOS attack
Due to technical limitations I will only test a scenario in which an attack comes from one physical machine, but the packets contain spoofed source IP addresses. I removed all security features applied in lab #1, so now only http traffic is allowed from any source to the destination 10.0.0.10.
I started this attack using random IP sources :
root@kali:~# t50 10.0.0.10 --flood --syn --dport 80
entering in flood mode...
hit CTRL+C to break.
T50 5.4.1-rc1 successfully launched on Aug 30th 2014 22:41:39
As expected CPU usage on the ISP’s router is very high:
ISP#sh processes cpu sorted
CPU utilization for five seconds: 94%/100%; one minute: 31%; five minutes: 14%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
83 1474212 106143 13888 26.77% 9.08% 3.87% 0 IP Input
270 93876 16754 5603 2.39% 0.49% 0.20% 0 Per-Second Jobs
6 91056 7058 12901 1.83% 0.96% 0.42% 0 Pool Manager
3 145300 11877 12233 0.95% 0.51% 0.53% 0 Exec
174 15604 160192 97 0.39% 0.09% 0.08% 0 RBSCP Background
112 33524 479984 69 0.39% 0.18% 0.16% 0 IPAM Manager
239 70372 949141 74 0.39% 0.36% 0.32% 0 ISG MIB jobs Man
50 27368 6837 4002 0.39% 0.10% 0.06% 0 Net Background
82 27472 479970 57 0.15% 0.13% 0.13% 0 IP ARP Retry Age
11 8368 17366 481 0.15% 0.02% 0.00% 0 ARP Background
5 24248 2830 8568 0.07% 0.12% 0.12% 0 Check heaps
The number of hits is very similar to test #1:
ISP#sh access-lists OUTSIDE
Extended IP access list OUTSIDE
10 permit tcp any host 10.0.0.10 eq www log (72422 matches)
ISP#
ISP#sh int gig0/0 stats
GigabitEthernet0/0
Switching path Pkts In Chars In Pkts Out Chars Out
Processor 72550 4357936 36 4074
Route cache 0 0 0 0
Total 72550 4357936 36 4074
ISP#sh int gig1/0 stats
GigabitEthernet1/0
Switching path Pkts In Chars In Pkts Out Chars Out
Processor 0 0 8852 532396
Route cache 0 0 0 0
Total 0 0 8852 532396
ISP#
As you see in the following log, the connections came from different source IPs:
ISP#sh logging
Syslog logging: enabled (0 messages dropped, 1549 messages rate-limited, 0 flushes, 0 overruns, xml disabled, filtering disabled)
…
*Aug 31 02:52:06.902: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 220.199.63.3(155) -> 10.0.0.10(80), 1 packet
*Aug 31 02:52:06.902: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 164.55.130.104(28412) -> 10.0.0.10(80), 1 packet
*Aug 31 02:52:06.902: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 83.17.127.33(2912) -> 10.0.0.10(80), 1 packet
*Aug 31 02:52:06.902: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 80.202.164.15(7333) -> 10.0.0.10(80), 1 packet
*Aug 31 02:52:06.902: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 35.163.122.116(14173) -> 10.0.0.10(80), 1 packet
*Aug 31 02:52:06.902: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 160.76.89.101(23452) -> 10.0.0.10(80), 1 packet
*Aug 31 02:54:33.038: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 150.234.98.10(30593) -> 10.0.0.10(80), 1 packet
*Aug 31 02:54:34.082: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 50.7.245.9(8186) -> 10.0.0.10(80), 1 packet
*Aug 31 02:54:35.098: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 32.64.62.34(8789) -> 10.0.0.10(80), 1 packet
*Aug 31 02:54:36.114: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 197.138.244.20(31482) -> 10.0.0.10(80), 1 packet
*Aug 31 02:54:37.126: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 77.36.118.8(24865) -> 10.0.0.10(80), 1 packet
*Aug 31 02:54:38.174: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 103.95.142.100(11182) -> 10.0.0.10(80), 1 packet
*Aug 31 02:54:39.194: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 64.230.212.54(24549) -> 10.0.0.10(80), 1 packet
*Aug 31 02:54:40.218: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 210.25.64.40(23580) -> 10.0.0.10(80), 1 packet
*Aug 31 02:54:41.218: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 130.11.182.87(3643) -> 10.0.0.10(80), 1 packet
*Aug 31 02:54:42.230: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 181.182.64.60(14103) -> 10.0.0.10(80), 1 packet
*Aug 31 02:54:43.242: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 92.8.226.119(24074) -> 10.0.0.10(80), 1 packet
*Aug 31 02:54:44.250: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 34.72.28.48(22367) -> 10.0.0.10(80), 1 packet
*Aug 31 02:54:44.566: %SEC-6-IPACCESSLOGRL: access-list logging rate-limited or missed 13666 packets
*Aug 31 02:54:45.254: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 25.9.251.30(23508) -> 10.0.0.10(80), 1 packet
*Aug 31 02:54:46.258: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 201.27.106.117(26452) -> 10.0.0.10(80), 1 packet
*Aug 31 02:54:47.278: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 190.52.74.50(6904) -> 10.0.0.10(80), 1 packet
*Aug 31 02:54:48.286: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 169.147.17.89(12970) -> 10.0.0.10(80), 1 packet
*Aug 31 02:54:49.294: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 196.174.184.49(24059) -> 10.0.0.10(80), 1 packet
*Aug 31 02:54:50.326: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 118.237.206.110(1664) -> 10.0.0.10(80), 1 packet
*Aug 31 02:54:51.326: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 132.72.239.123(22164) -> 10.0.0.10(80), 1 packet
*Aug 31 02:54:52.358: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 26.238.113.114(12075) -> 10.0.0.10(80), 1 packet
*Aug 31 02:54:53.366: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 200.78.129.114(27771) -> 10.0.0.10(80), 1 packet
*Aug 31 02:54:54.374: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 44.69.75.92(25733) -> 10.0.0.10(80), 1 packet
*Aug 31 02:54:55.402: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 185.39.35.54(9501) -> 10.0.0.10(80), 1 packet
*Aug 31 02:54:56.406: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 149.238.124.68(689) -> 10.0.0.10(80), 1 packet
*Aug 31 02:54:57.458: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 215.40.122.24(14267) -> 10.0.0.10(80), 1 packet
*Aug 31 02:54:58.534: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 141.225.192.103(24388) -> 10.0.0.10(80), 1 packet
*Aug 31 02:54:59.542: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 65.67.18.17(7899) -> 10.0.0.10(80), 1 packet
*Aug 31 02:55:00.542: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 57.135.236.19(3134) -> 10.0.0.10(80), 1 packet
*Aug 31 02:55:01.610: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 240.55.235.108(23735) -> 10.0.0.10(80), 1 packet
*Aug 31 02:55:02.726: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 52.202.105.109(23049) -> 10.0.0.10(80), 1 packet
*Aug 31 02:55:03.738: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 226.5.29.67(12367) -> 10.0.0.10(80), 1 packet
*Aug 31 02:55:04.766: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 1.160.121.91(10350) -> 10.0.0.10(80), 1 packet
*Aug 31 02:55:05.786: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 254.144.152.15(28777) -> 10.0.0.10(80), 1 packet
*Aug 31 02:55:06.802: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 90.84.122.90(31084) -> 10.0.0.10(80), 1 packet
*Aug 31 02:55:07.806: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 67.119.134.11(24975) -> 10.0.0.10(80), 1 packet
*Aug 31 02:55:08.830: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 48.195.167.60(31799) -> 10.0.0.10(80), 1 packet
*Aug 31 02:55:09.834: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 117.232.51.94(29094) -> 10.0.0.10(80), 1 packet
*Aug 31 02:55:10.854: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 83.219.21.118(24424) -> 10.0.0.10(80), 1 packet
--More—
The CPU on the PER-FW is not very high:
PER-FW# sh cpu usage
CPU utilization for 5 seconds = 1%; 1 minute: 1%; 5 minutes: 1%
PER-FW# sh access-list
access-list cached ACL log flows: total 732, denied 0 (deny-flow-max 4096)
alert-interval 300
access-list OUT; 1 elements; name hash: 0xcd7d0798
access-list OUT line 1 extended permit tcp any host 10.0.0.10 eq www log informational interval 300 (hitcnt=732) 0x5f077c5e
PER-FW#
PER-FW# sh int gig0 stats
Interface GigabitEthernet0 "outside", is up, line protocol is up
Hardware is Linux Ethernet Dev, BW 100 Mbps, DLY 100 usec
(Full-duplex), (100 Mbps)
Input flow control is unsupported, output flow control is unsupported
MAC address 00ab.cd92.5200, MTU 1500
IP address 192.168.158.11, subnet mask 255.255.255.0
7814 packets input, 443038 bytes, 0 no buffer
Received 0 broadcasts, 0 runts, 0 giants
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
0 pause input, 0 resume input
0 L2 decode drops
8 packets output, 534 bytes, 0 underruns
0 pause output, 0 resume output
0 output errors, 0 collisions, 0 interface resets
0 late collisions, 0 deferred
0 input reset drops, 0 output reset drops
input queue (blocks free curr/low): hardware (0/0)
output queue (blocks free curr/low): hardware (0/0)
Traffic Statistics for "outside":
918 packets input, 38467 bytes
0 packets output, 0 bytes
92 packets dropped
1 minute input rate 0 pkts/sec, 16 bytes/sec
1 minute output rate 0 pkts/sec, 0 bytes/sec
1 minute drop rate, 0 pkts/sec
5 minute input rate 2 pkts/sec, 106 bytes/sec
5 minute output rate 0 pkts/sec, 0 bytes/sec
5 minute drop rate, 0 pkts/sec
PER-FW# sh int gig1 stats
Interface GigabitEthernet1 "inside", is up, line protocol is up
Hardware is Linux Ethernet Dev, BW 100 Mbps, DLY 100 usec
(Full-duplex), (100 Mbps)
Input flow control is unsupported, output flow control is unsupported
MAC address 00ab.cd92.5201, MTU 1500
IP address 10.0.0.1, subnet mask 255.255.255.0
1329 packets input, 88071 bytes, 0 no buffer
Received 0 broadcasts, 0 runts, 0 giants
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
0 pause input, 0 resume input
0 L2 decode drops
7139 packets output, 385491 bytes, 0 underruns
0 pause output, 0 resume output
0 output errors, 0 collisions, 0 interface resets
0 late collisions, 0 deferred
0 input reset drops, 0 output reset drops
input queue (blocks free curr/low): hardware (0/0)
output queue (blocks free curr/low): hardware (0/0)
Traffic Statistics for "inside":
393 packets input, 19227 bytes
830 packets output, 33200 bytes
393 packets dropped
1 minute input rate 0 pkts/sec, 11 bytes/sec
1 minute output rate 0 pkts/sec, 0 bytes/sec
1 minute drop rate, 0 pkts/sec
5 minute input rate 1 pkts/sec, 46 bytes/sec
5 minute output rate 2 pkts/sec, 97 bytes/sec
5 minute drop rate, 1 pkts/sec
PER-FW#
As you can see below, the half-open connections are kept only for 30 seconds:
PER-FW# sh conn
388 in use, 2542 most used
TCP outside 132.16.242.95:4437 inside 10.0.0.10:80, idle 0:00:00, bytes 0, flags SaAB
TCP outside 45.180.196.105:7374 inside 10.0.0.10:80, idle 0:00:00, bytes 0, flags SaAB
…
TCP outside 139.45.38.60:29808 inside 10.0.0.10:80, idle 0:00:00, bytes 0, flags SaAB
TCP outside 251.45.52.118:9456 inside 10.0.0.10:80, idle 0:00:00, bytes 0, flags SaAB
TCP outside 120.180.46.9:32555 inside 10.0.0.10:80, idle 0:00:02, bytes 0, flags SaAB
TCP outside 190.97.134.67:15763 inside 10.0.0.10:80, idle 0:00:02, bytes 0, flags SaAB
…
TCP outside 93.216.144.56:12253 inside 10.0.0.10:80, idle 0:00:03, bytes 0, flags SaAB
TCP outside 79.73.193.9:28466 inside 10.0.0.10:80, idle 0:00:03, bytes 0, flags SaAB
TCP outside 115.141.225.36:2679 inside 10.0.0.10:80, idle 0:00:03, bytes 0, flags SaAB
TCP outside 202.235.0.52:18265 inside 10.0.0.10:80, idle 0:00:03, bytes 0, flags SaAB
TCP outside 46.100.213.111:3357 inside 10.0.0.10:80, idle 0:00:03, bytes 0, flags SaAB
…
TCP outside 52.54.191.123:30871 inside 10.0.0.10:80, idle 0:00:13, bytes 0, flags SaAB
TCP outside 171.140.183.72:13459 inside 10.0.0.10:80, idle 0:00:13, bytes 0, flags SaAB
TCP outside 146.14.23.38:10416 inside 10.0.0.10:80, idle 0:00:13, bytes 0, flags SaAB
TCP outside 29.105.29.26:25413 inside 10.0.0.10:80, idle 0:00:13, bytes 0, flags SaAB
TCP outside 134.132.159.121:1246 inside 10.0.0.10:80, idle 0:00:13, bytes 0, flags SaAB
….
TCP outside 207.26.176.50:18211 inside 10.0.0.10:80, idle 0:00:20, bytes 0, flags SaAB
TCP outside 46.13.236.42:20774 inside 10.0.0.10:80, idle 0:00:20, bytes 0, flags SaAB
TCP outside 184.58.42.97:28843 inside 10.0.0.10:80, idle 0:00:20, bytes 0, flags SaAB
…
TCP outside 16.185.54.105:12733 inside 10.0.0.10:80, idle 0:00:27, bytes 0, flags SaAB
TCP outside 157.244.239.122:11022 inside 10.0.0.10:80, idle 0:00:27, bytes 0, flags SaAB
TCP outside 130.80.98.111:2086 inside 10.0.0.10:80, idle 0:00:27, bytes 0, flags SaAB
TCP outside 102.155.65.57:9322 inside 10.0.0.10:80, idle 0:00:29, bytes 0, flags SaAB
TCP outside 79.180.252.112:20785 inside 10.0.0.10:80, idle 0:00:29, bytes 0, flags SaAB
TCP outside 112.112.154.108:18489 inside 10.0.0.10:80, idle 0:00:29, bytes 0, flags SaAB
TCP outside 191.31.218.24:2074 inside 10.0.0.10:80, idle 0:00:29, bytes 0, flags SaAB
TCP outside 213.125.40.95:19220 inside 10.0.0.10:80, idle 0:00:29, bytes 0, flags SaAB
TCP outside 65.75.170.103:8192 inside 10.0.0.10:80, idle 0:00:29, bytes 0, flags SaAB
PER-FW#
After 30 seconds they are removed:
PER-FW# sh logg
Syslog logging: enabled
Facility: 20
Timestamp logging: disabled
Standby logging: disabled
Debug-trace logging: disabled
Console logging: disabled
Monitor logging: disabled
Buffer logging: level informational, 22071 messages logged
Trap logging: disabled
Permit-hostdown logging: disabled
History logging: disabled
Device ID: disabled
Mail logging: disabled
ASDM logging: disabled
nside:10.0.0.10/80 duration 0:00:30 bytes 0 SYN Timeout
%ASA-6-302014: Teardown TCP connection 3419 for outside:61.84.231.56/5802 to inside:10.0.0.10/80 duration 0:00:30 bytes 0 SYN Timeout
…
%ASA-6-302014: Teardown TCP connection 3503 for outside:92.159.9.91/21461 to inside:10.0.0.10/80 duration 0:00:30 bytes 0 SYN Timeout
%ASA-6-302014: Teardown TCP connection 3504 for outside:175.111.170.38/10465 to inside:10.0.0.10/80 duration 0:00:30 bytes 0 SYN Timeout
%ASA-6-302014: Teardown TCP connection 3505 for outside:58.116.52.96/8622 to inside:10.0.0.10/80 duration 0:00:30 bytes 0 SYN
…
%ASA-6-302014: Teardown TCP connection 3522 for outside:95.189.67.74/17573 to inside:10.0.0.10/80 duration 0:00:30 bytes 0 SYN Timeout
%ASA-6-302014: Teardown TCP connection 3523 for outside:150.0.190.70/2709 to inside:10.0.0.10/80 duration 0:00:30 bytes 0 SYN Timeout
%ASA-6-302014: Teardown TCP connection 3524 for outside:211.185.70.62/32324 to inside:10.0.0.10/80 duration 0:00:30 bytes 0 SYN Timeout
%ASA-6-302014: Teardown TCP connection 3525 for outside:204.68.234.108/26687 to inside:10.0.0.10/80 duration 0:00:30 bytes 0 SYN
Below you can find the CPU usage of the WEB-SRV:
WEB-SRV#sh processes cpu sorted
CPU utilization for five seconds: 12%/2%; one minute: 4%; five minutes: 2%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
83 227296 34712 6548 4.79% 1.78% 0.81% 0 IP Input
3 11328 1288 8795 3.43% 0.48% 0.36% 0 Exec
239 21852 907335 24 0.47% 0.32% 0.29% 0 ISG MIB jobs Man
150 20708 11147 1857 0.31% 0.18% 0.05% 0 TCP Timer
270 7356 14662 501 0.15% 0.03% 0.00% 0 Per-Second Jobs
82 8544 453167 18 0.15% 0.15% 0.13% 0 IP ARP Retry Age
112 10512 453168 23 0.15% 0.16% 0.15% 0 IPAM Manager
59 11256 4395 2561 0.07% 0.08% 0.07% 0 HC Counter Timer
173 184 1463 125 0.07% 0.00% 0.00% 0 QoS stats proces
174 3136 145816 21 0.07% 0.07% 0.07% 0 RBSCP Background
268 11416 2934 3890 0.07% 0.08% 0.07% 0 Compute load avg
WEB-SRV#
WEB-SRV#sh int gig0/0
GigabitEthernet0/0 is up, line protocol is up
Hardware is i82543 (Livengood), address is ca02.10e8.0008 (bia ca02.10e8.0008)
Internet address is 10.0.0.10/24
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full Duplex, 1000Mbps, link type is auto, media type is SX
output flow-control is XON, input flow-control is XON
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:08:13, output 00:00:00, output hang never
Last clearing of "show interface" counters 00:11:26
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
830 packets input, 44820 bytes, 0 no buffer
Received 0 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
435 packets output, 30199 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
WEB-SRV#sh int gig0/0 stats
GigabitEthernet0/0
Switching path Pkts In Chars In Pkts Out Chars Out
Processor 830 44820 436 30259
Route cache 0 0 0 0
Total 830 44820 436 30259
WEB-SRV#
Let’s think a while about how we can mitigate the attack or how we can minimize its impact on our network. As you can see, we can’t apply an access list with the source of the attacker’s IP address because in this case every connection comes from a different IP address. We can’t use policy base routing (PBR), because we can’t identify the ‘bad’ traffic. We can deny all connections to our web server, but it is not a good solution the server needs to be available to our clients. This is a list of some possible solutions:
- Limit the number of embryonic connections – you have to know your ‘normal’ traffic: how many connection you see on average and what is the maximum number of these connections. Based on this information you can apply connection limits. To do this, you can use one of the following features:
a) Context-Based Access Control (CBAC):
!CBAC TCP/UDP Intercept Feature
R3:
ip inspect max-incomplete low 500
ip inspect max-incomplete high 700
ip inspect one-minute low 90
ip inspect one-minute high 100
ip inspect tcp synwait-time 15
ip inspect tcp max-incomplete host 50 block-time 5
!
ip inspect name PROTECTDOS tcp
ip inspect name PROTECTDOS udp
ip inspect name PROTECTDOS icmp
!
ip access-list extended ATTACK
deny ip any any log
!
interface FastEthernet 0/0
ip inspect PROTECTDOS out
ip access-group ATTACK in
b) Zone Base Policy Firewall (ZBPF)
parameter-map type inspect ATTACK_CONTROL
audit-trail on
alert off
max-incomplete low 200
max-incomplete high 300
udp idle-time 50
dns-timeout 5
tcp idle-time 60
tcp synwait-time 5
zone security outside
zone security dmz
interface GigabitEthernet0/1
zone-member security dmz
!
interface GigabitEthernet0/0
zone-member security outside
class-map type inspect match-any DMZ-protocols
match protocol http
policy-map type inspect outside_to_dmz
class type inspect DMZ-protocols
inspect ATTACK_CONTROL
class class-default
drop
zone-pair security OUT>DMZ source outside destination dmz
service-policy type inspect outside_to_dmz
c) ASA:
class-map ATTACK
match any
policy-map ATTACK
class ATTACK
set connection conn-max 500 embryonic-conn-max 100
set connection timeout idle 2:0:0 embryonic 0:40:0 half-closed 0:20:0 dcd
service-policy CONNS interface outside
- Keep the timeout of half open connections as low as possible, it will delete all not-completed TCP sessions faster.
- Contact your ISP and ask them what kind of protections he can guarantee during a DDOS attack. Every ISP offers different services but for some of them you may have to pay more. This information should be included in your “Business Recovery Plan”. Don’t wait to do this until you are under attack, as it will be too late.
- If you need your WEB Server available for your customers, you should prepare a backup plan. It can be a backup server located in a Data Center, for example Amazon (Elastic Compute Cloud EC2). You pay only for what you use, and if you need the backup service only occasionally, it should not ruin your budget.
- If your server architecture is very complex, or due to regulations you can’t keep your data out of your company/country, you can consider using companies like Prolexis (now Akamai), but it is rather for the enterprise segment. They have many Data Centers around the world and your content can be easily moved around. They also work closely with ISPs to track down attackers.
- The next option for the medium to large segment is to use the following appliances that can mitigate DOS/DDOS attacks: DDOS Protector (Checkpoint), BIG-IP ASM and/or BIG-IP LTM (F5). You can also buy security services from AT&T or IBM, but as I said earlier, this is rather for the enterprise segment.
The number of DOS/DDOS attacks will grow as it is the easiest way to stop someone’s business. The motivations may be different, but the effect is the same: your service is not available. You should prepare your business for these kinds of attacks. Maybe you will never use this plan, but it’s definitely better to be prepared. You can’t think ‘I have a firewall, so I’m safe!’ It isn’t true, even Cisco openly says: “Although firewalls play a critical role in any organization’s security solution, they are not purpose-built DDoS prevention devices.”
3
Kudos
3
Kudos