{"id":1313,"date":"2010-10-21T22:43:33","date_gmt":"2010-10-21T15:43:33","guid":{"rendered":"http:\/\/www.jfdesignnet.com\/?p=1313"},"modified":"2010-10-21T22:43:33","modified_gmt":"2010-10-21T15:43:33","slug":"applying-anti-bandwidth-sucker-measure-into-apache","status":"publish","type":"post","link":"https:\/\/www.jfdesignnet.com\/?p=1313","title":{"rendered":"applying anti-bandwidth-sucker measure into apache"},"content":{"rendered":"

\"\"<\/a><\/p>\n

This time I have to deal with another form of leech other than woman I know of … he he he. Some user hogging my server bandwidth for several time, and this abuse couldn’t continue. Here is some tricks that can denying the bandwidth sucker user by detecting their browser user agent. Fortunately, apache has a built-in directive called BrowserMatch, and its sister BrowserMatchNoCase. These allow you to block clients based on their UserAgent string, and they will work without mod_rewrite module. Yeahh … I know … the UserAgent string itself can be spoofed, but hell yeah … just like in real life … she can be very deceiving also with cosmetic and tears, the most lethal weapon from women \ud83d\ude42 … but, back to topics, this will take care of the vast majority of your would-be bandwidth hogs.<\/p>\n

Looking for web spiders and site suckers :<\/strong>
\nHere’s a shortcut for identifying all user agents in your logs.
\n{code}cat access.log | awk -F “”” {‘print $6’} | sort | uniq | grep -v Mozilla{\/code}<\/p>\n

Translation:<\/span><\/strong> Run the logfile through a filter, treating double-quotes as field separators, pick out the 6th field (which is the UserAgent field), sort the resulting list, toss out all duplicates, and don’t show me anything containing “Mozilla”. NOTE that many programs include “Mozilla” in their useragent strings; also, some download accelerators operate as plugins to the regular browser and then append their useragent specifics to the browser’s string. So, if you want to be more thorough than this, leave off everything after “uniq” and you’ll see it all — including stuff like :
\nMozilla\/5.0 (Windows; U; Windows NT 5.1; vi; rv:1.9.2.10) Gecko\/20100914 Firefox\/3.6.10 ;ShopperReports
\nMozilla\/5.0 (Windows; U; Windows NT 5.2; en-US) AppleWebKit\/534.3 (KHTML, like Gecko) Chrome\/6.0.472.63 Safari\/534.3
\nMozilla\/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.13) Gecko\/20100914 Ant.com Toolbar 2.0.1 Firefox\/3.5.13<\/em><\/p>\n

which may or may not be legit, but are definitely unusual.<\/p>\n

This will take awhile for large logs. Leave off everything after “uniq” to see all<\/strong> of the UserAgents; I filter out Mozilla since it is what appears for most normal browsers, including IE.<\/p>\n

Also, this assumes you are using “combined” log format. If you don’t get the results you expect (see below), then try replacing the “$6” with another number — your log format may order the fields differently.<\/p>\n

Sample results:<\/span><\/strong>
\n{code}cat \/var\/log\/httpd\/access.log | awk -F “”” {‘print $6’} | sort | uniq | grep -v Mozilla{\/code}<\/p>\n

Feedfetcher-Google; (+http:\/\/www.google.com\/feedfetcher.html; feed-id=17527023618099803589)
\nFeedfetcher-Google; (+http:\/\/www.google.com\/feedfetcher.html; feed-id=7166567229941680435)
\nFeedjit Favicon Crawler 1.0
\nFirefox (SBUA)
\nFlashGet
\nGetGo Download Manager 4.0 (www.getgosoft.com)Pragma: no-cache
\nGooglebot-Image\/1.0
\nGooglebot\/2.1 (+http:\/\/www.google.com\/bot.html)
\nAxel 1.0b (Linux)
\nBaiduImagespider+(+http:\/\/www.baidu.jp\/spider\/)
\nBaiduspider+(+http:\/\/www.baidu.com\/search\/spider.htm)
\nBaiduspider+(+http:\/\/www.baidu.jp\/spider\/)
\nBlackBerry8520\/4.6.1.314 Profile\/MIDP-2.0 Configuration\/CLDC-1.1 VendorID\/123
\nBlackBerry8520\/4.6.1.314 Profile\/MIDP-2.0 Configuration\/CLDC-1.1 VendorID\/611
\nBlackBerry8520\/4.6.1.314 Profile\/MIDP-2.0 Configuration\/CLDC-1.1 VendorID\/613
\nIRLbot\/1.0 (+http:\/\/irl.cs.tamu.edu\/crawler)
\nIltrovatore-Setaccio\/1.2 (It-bot; http:\/\/www.iltrovatore.it\/bot.html; info@iltrovatore.it)
\nJava1.3.1
\nLWP::Simple\/5.64
\nLeechGet 2004 (www.leechget.net)
\nLinks (2.1pre15; Linux 2.6.7-hardened-r16 i686; 80×40)
\nLynx\/2.8.4rel.1 libwww-FM\/2.14
\nMediapartners-Google\/2.1
\nMonica\/1.4
\nOpera\/7.50 (X11; Linux i386; U) [en]
\nRealDownload\/4.0.0.40
\nSIE-M55\/10 UP.Browser\/6.1.0.5.c.6 (GUI) MMP\/1.0 (Google WAP Proxy\/1.0)
\nSafariBookmarkChecker\/1.26 (+http:\/\/www.coriolis.ch\/)
\nSiteBar\/3.2.6
\nOpera\/9.80 (S60; SymbOS; Opera Mobi\/499; U; en) Presto\/2.4.18 Version\/10.00
\nOpera\/9.80 (Windows Mobile; Opera Mini\/5.1.21594\/20.2497; U; en) Presto\/2.5.25
\nWget\/1.10.2 (Red Hat modified)
\nWget\/1.11.4
\nWget\/1.12 (linux-gnu)
\nMWG Atom Life Profile\/MIDP-2.0 Configuration\/CLDC-1.1 UNTRUSTED\/1.0
\nMxAgent
\nNetScape\/0.02 [fu] (MAC OS; X; SK)
\nNokia5300\/2.0 (05.51) Profile\/MIDP-2.0 Configuration\/CLDC-1.1
\nNokia6070\/2.0 (03.20) Profile\/MIDP-2.0 Configuration\/CLDC-1.1
\nNokia6085\/2.0 (03.71) Profile\/MIDP-2.0 Configuration\/CLDC-1.1
\nNokia6235\/1.0 (S190V0200.nep) UP.Browser\/6.2.3.2 MMP\/2.0
\niCab\/2.9.8 (Macintosh; U; 68K)
\nSonyEricssonG502\/R1FA Browser\/NetFront\/3.4 Profile\/MIDP-2.1 Configuration\/CLDC-1.1
\nSonyEricssonK510i\/R4EA Browser\/NetFront\/3.3 Profile\/MIDP-2.0 Configuration\/CLDC-1.1
\nSonyEricssonW380i\/R10CA Browser\/NetFront\/3.3 Profile\/MIDP-2.0 Configuration\/CLDC-1.1
\nSonyEricssonX2\/R3AA Browser\/Mozilla\/4.0 (compatible; MSIE 6.0; Windows CE; IEMobile 8.12; MSIEMobile 6.5) Profile\/MIDP-2.1 Configuration\/CLDC-1.1
\nSosospider+(+http:\/\/help.soso.com\/webspider.htm)
\nStackRambler\/2.0 (MSIE incompatible)
\nmsnbot\/1.0 (+http:\/\/search.msn.com\/msnbot.htm)
\nWindows-Media-Player\/10.00.00.3993
\nWindows-Media-Player\/11.0.6000.6352
\nWordPress\/3.0.1; http:\/\/trik-fb.co.cc
\nWordPress\/3.0; http:\/\/www.ahlidesain.com
\nXunlei@Http Download
\nYahooMobile\/1.0 (Resource; Server; 1.0.0)
\nYeti\/1.0 (NHN Corp.; http:\/\/help.naver.com\/robots\/)<\/em><\/p>\n

When you see something odd or suspicious (like “LeechGet”, gee I wonder what that does), Google around for it. If the name is too general, add “user agent” or “spider” or “search engine” to your query.<\/p>\n

Now, it’s time to applying those knowledge :<\/p>\n

1. To apache2.conf, add the following<\/p>\n

{code}BrowserMatchNoCase ^NameOfBadProgram1 nameofenv
\nBrowserMatchNoCase ^NameOfBadProgram2 nameofenv
\nBrowserMatchNoCase ^NameOfBadProgram3 nameofenv{\/code}<\/p>\n

Use the same “nameofenv” value for all of the agents you want to block. I added this section to some preexisting BrowserMatch directives that had to do with forcing HTTP responses to certain browser versions. Here’s what mine looks like right now, I will be adding to it as my logs reveal new twits:<\/p>\n

{code}# anti-bandwidth-sucker measures by jfdesign added
\nBrowserMatchNoCase ^wget suckers
\nBrowserMatchNoCase ^SiteSucker suckers
\nBrowserMatchNoCase ^iGetter suckers
\nBrowserMatchNoCase ^larbin suckers
\nBrowserMatchNoCase ^LeechGet suckers
\nBrowserMatchNoCase ^RealDownload suckers
\nBrowserMatchNoCase ^Teleport suckers
\nBrowserMatchNoCase ^Webwhacker suckers
\nBrowserMatchNoCase ^WebDevil suckers
\nBrowserMatchNoCase ^Webzip suckers
\nBrowserMatchNoCase ^Attache suckers
\nBrowserMatchNoCase ^SiteSnagger suckers
\nBrowserMatchNoCase ^WX_mail suckers
\nBrowserMatchNoCase ^EmailCollector suckers
\nBrowserMatchNoCase ^WhoWhere suckers
\nBrowserMatchNoCase ^Roverbot suckers
\nBrowserMatchNoCase ^ActiveAgent suckers
\nBrowserMatchNoCase ^EmailSiphon suckers{\/code}<\/p>\n

2. Now, inside the Directory blocks for each directory you want to apply these to, put<\/p>\n

{code}deny from env=suckers{\/code}<\/p>\n

You could also add this to individual directories’ .htaccess files, but I have not tried this yet. I simply added it at the end of my main Directory block:<\/p>\n

{code}Options Indexes FollowSymLinks
\nAllowOverride AuthConfig
\ndeny from env=suckers{\/code}<\/p>\n

You can create other environments for things like known email-harvesting bots, known-evil web spiders, etc. and do more creative things based on which type of malicious visitors they are. I am content to just refuse connections from all of them.<\/p>\n

Don’t forget to restart apache (service apache2 restart) after applying any of these changes.<\/p>\n","protected":false},"excerpt":{"rendered":"

This time I have to deal with another form of leech other than woman I know of … he he he. Some user hogging my server bandwidth for several time, and this abuse couldn’t continue. Here is some tricks that can denying the bandwidth sucker user by detecting their browser user agent. Fortunately, apache has […]<\/p>\n","protected":false},"author":1,"featured_media":1318,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[16,9],"tags":[34,36],"_links":{"self":[{"href":"https:\/\/www.jfdesignnet.com\/index.php?rest_route=\/wp\/v2\/posts\/1313"}],"collection":[{"href":"https:\/\/www.jfdesignnet.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.jfdesignnet.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.jfdesignnet.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.jfdesignnet.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1313"}],"version-history":[{"count":0,"href":"https:\/\/www.jfdesignnet.com\/index.php?rest_route=\/wp\/v2\/posts\/1313\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.jfdesignnet.com\/index.php?rest_route=\/wp\/v2\/media\/1318"}],"wp:attachment":[{"href":"https:\/\/www.jfdesignnet.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1313"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.jfdesignnet.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1313"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.jfdesignnet.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1313"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}