Posted by anantagati on May 6, 2010 at 2:12am
Our site is sometimes getting DOS attack, on my previous LAMP installation I could install mod_evasive and it stopped all those attacks.
I installed mod_evasive on Mercury, but pages to attacking IP address are still served. It behaves like mod_evasive is completely ignored.
Anybody could put server down by using:
ab -k -n 1000 -c 50 http://domain/
I changed maximal number of servers in Apache config, so previous command doesn't crash server now, but it is still taking resources by generating pages.
How are you protecting your server against such attacks?
Comments
Are you sure you can 'put it down'?
The reason why your mod_evasive configurations didn't do anything, is because Varnish listens on port 80 on Mercury, not Apache. Varnish is a libevent-based reverse proxy server, which means that all the nasty tuning you had to do to Apache to keep it from choking itself are no longer needed. Are you postive the site is going down when running that test, or are you just assuming it is? If you're able to bring down a Mercury site with just 50 concurrent threads on AB, then something isn't right in your configuration. Start another discussion if that's the case so that the right people can help you out.
On my OOTB Mercury install (512MB RAM VPS), I can throw ab -k -n 100000 -c 500 at it and still have all requests served up in 20ms or less with no impact to the site.
What about ab -k -n 100000 -c
What about ab -k -n 100000 -c 500 -C DRUPAL_UID=1 http://example.com/?
You can very easily fight
You can very easily fight against both kinds of DoS attempts (both anonymous and "logged in" -C) by using Nginx on the front with enabled
limit_zone- see my example: http://github.com/omega8cc/nginx-for-drupal/blob/master/nginx.conf - lines 38 and 75.Furthermore, we are using custom script to monitor Nginx access.log and it almost immediately blocks all those attempts using standard system firewall (iptables).
HTH ~ Grace
you concerns are unfounded
Here are the results of an ab attack on an OOTB production Mercury (small EC2).
You won't touch your CPU until you exceed 150 concurrent.
This ab test does max out the CPU but still serves it up just fine.
I think you concerns are unfounded.
Thanks Chapter Three!!!!
~# ab -k -n 100000 -c 500 http://.../
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking ... (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Completed 100000 requests
Finished 100000 requests
Server Software: Apache/2.2.11
Server Hostname: ...
Server Port: 80
Document Path: /
Document Length: 371 bytes
Concurrency Level: 500
Time taken for tests: 20.862 seconds
Complete requests: 100000
Failed requests: 2
(Connect: 0, Receive: 0, Length: 2, Exceptions: 0)
Write errors: 0
Non-2xx responses: 100000
Keep-Alive requests: 99998
Total transferred: 81800000 bytes
HTML transferred: 37100000 bytes
Requests per second: 4793.34 [#/sec] (mean)
Time per request: 104.311 [ms] (mean)
Time per request: 0.209 [ms] (mean, across all concurrent requests)
Transfer rate: 3829.05 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 2.3 0 89
Processing: 3 104 99.9 80 3793
Waiting: 0 104 99.9 80 3793
Total: 3 104 99.9 80 3793
Percentage of the requests served within a certain time (ms)
50% 80
66% 100
75% 116
80% 127
90% 177
95% 294
98% 345
99% 397
100% 3793 (longest request)
Regards,
Michael Hofmockel
Open Source || Open Access || Open Mind
Not so good Document Length:
Not so good
Document Length: 371 bytes...
Non-2xx responses: 100000
Note that ab is not a
Note that ab is not a particularly good test ... it doesn't simulate a real web browser's behavior at all. It's not bad for a very basic requests/sec test, but note that it does not pull the CSS, JS, images, etc. that make up a significant proportion (often the majority) of a visitor's traffic. I'd recommend pointing a FunkLoad or JMeter test at it if you want to get a good idea how many real users the system may be able to support. Both of those tools simulate a full browser page load - they parse the HTML and request the associated CSS, images and JS along with the HTML. I use FunkLoad and have found it to be extremely helpful in identifying performance bottlenecks.
Thank you for your inputs.
Thank you for your inputs. Seems like Varnish is not active as I can see with 'top' that it always tries to use Apache instance.
I just tried it on fresh Mercury install on Linode and Varnish works good. But when I transferred live site to Mercury it just use Apache and not Varnish.
Seems like Varnish is not
I wouldn't recommend using 'top' to see if Varnish is doing anything. Varnish uses remarkably little CPU even when under heavy load, so it's not likely to show on 'top'. I would do a couple of things to see if Varnish is really doing anything:
sub vcl_deliver {# return (deliver);
#add cache hit data
if (obj.hits > 0) {
#if hit add hit count
set resp.http.X-Cache = "HIT";
set resp.http.X-Cache-Hits = obj.hits;
} else {
set resp.http.X-Cache = "MISS";
}
}
I am using 'top' and it shows
I am using 'top' and it shows many instances of Apache.
When using 'ab' these items get increased in 'varnishstat': Client requests received, Cache hits for pass, Backend connections success.
0+07:02:52 www
Hitrate ratio: 10 13 13
Hitrate avg: 0.0052 0.0055 0.0055
2095 0.00 0.08 Client connections accepted
8396 14.99 0.33 Client requests received
93 0.00 0.00 Cache hits
6092 14.99 0.24 Cache hits for pass
1224 0.00 0.05 Cache misses
8303 14.99 0.33 Backend connections success
849 0.00 0.03 Backend connections reuses
2190 0.00 0.09 Backend connections recycles
144 . . N struct srcaddr
1 . . N active struct srcaddr
53 . . N struct sess_mem
51 . . N struct sess
135 . . N struct object
210 . . N struct objecthead
229 . . N struct smf
5 . . N small free smf
2 . . N large free smf
48 . . N struct vbe_conn
51 . . N struct bereq
51 . . N worker threads
125 0.00 0.00 N worker threads created
0 -1.00 0.00 N queued work requests
1274 0.00 0.05 N overflowed work requests
1 . . N backends
1138 . . N expired objects
93 . . N LRU moved objects
8315 13.99 0.33 Objects sent with write
2095 0.00 0.08 Total Sessions
8347 13.99 0.33 Total Requests
1 0.00 0.00 Total pipe
7029 13.99 0.28 Total pass
8253 13.99 0.33 Total fetch
4653621 8349.82 183.42 Total header bytes
172333839 345181.69 6792.28 Total body bytes
1003 0.00 0.04 Session Closed
7352 13.99 0.29 Session herd
689590 1131.89 27.18 SHM records
49638 44.96 1.96 SHM writes
61 0.00 0.00 SHM MTX contention
16537 29.97 0.65 allocator requests
222 . . outstanding allocations
1892352 . . bytes allocated
132325376 . . bytes free
8302 14.99 0.33 Backend requests made
1 0.00 0.00 N vcl total
1 0.00 0.00 N vcl available
1 . . N total active purges
1 0.00 0.00 N new purges added
After few times reloading homepage in Firefox as anonymous user there is always X-Cache: MISS
.
Cookies
Cookies are almost certainly the issue.
In addition to session cookies (e.g. from not running Pressflow), many common JS widgets for analytics or other purposes set cookies in the user agent. The default VCL that comes with Mercury is aware of Google Analytics, but if you have others in play, you will need to adjust your VCL to get any benefits from Varnish.
https://pantheon.io | http://www.chapterthree.com | https://www.outlandishjosh.com
Cookies?
If your live site is setting cookies, you need to inform Varnish how to deal with it in order for it to cache properly. Please start a new discussion with a subject like "Varnish not caching anonymous requests on my site". People more knowledgeable than I can tell you exactly what to do.
Problem seems Masquerade
Problem seems Masquerade module and modules to restrict access. Now it is served by Varnish.
But if somebody will use this command:
ab -k -n 1000000 -c 500 http://domain/pageit will be still serve through Varnish and will use bandwidth and some CPU. Is there some way how to set maximum requests from one IP, so after xxx number of requests for IP will Varnish ignore it for some time?