Yesterday, an internal staff member was running two XML sitemap scans on our site and we were seeing loads on the Linux web head servers in excess of 12-15. We are running a Varnish/Pound proxy in front of the drupal web head with the MySQL DB on a 3rd server. We asked them to stop and the load disappeared.
The question I have is that since anyone could conceivable do this from the outside what is the best practice to help keep our server more resilient to a scan of every page on the site from the same IP? Is there a way to throttle the connections so that the load is more manageable?