Иногда боты настолько нагружают сервер и создают ненужную активность, что проще вообще им запретить посещать сайт. Мы составили черный список ботов, которые смогли найти в сети.
Ниже код помогает установить блокировку на ботов через файл .htaccess
RewriteEngine On # Блокировка по User-Agent RewriteCond %{HTTP_USER_AGENT} ^(360Spider|80legs|Abonti|AhrefsBot|AiHitBot|Alexibot|AlphaBot|AcoonBot|AddThis.com|adidxbot|ADmantX|Advanced\ Email|AIBOT|Aipbot|Antivirus|Apexoo|archive.org_bot|Arora|Astute|Attach|autoemailspider|BackDoorBot|Backlink-Ceck|BackStreet|BackWeb|Badass|Baiduspider|BeetleBot|BillyBob|BingPreview|Bitacle|BlackWidow|BlockNote.Net|BlogsharesSpider|BlowFish|BoardReader|Bolt|BotALot|BotRightHere|Bot\ mailto:craftbot@yahoo.com|Bubing|Buddy|BuiltBotTough|Bullseye|BunnySlippers|BuzzBot|CASIE|CCBot|Cegbfeieh|CheeseBot|CherryPicker|ChinaClaw|Chlooe|Claritybot|Cliqzbot|Cloud\ mapping|Cogentbot|Collector|Copier|CopyRightCheck|Copyscape|Cosmos|Craftbot|Crawling|Crescent|CrunchBot|Curious|Custo|Cyborg|DBLBot|DIIbot|DISCo|DittoSpyder|Dotbot|Download\ Demon|Drip|DSearch|Dumbot|EasyDL|eCatch|ecxi|EirGrabber|EmailCollector|EmailSiphon|EmailWolf|EroCrawler|Exabot|Express\ WebPictures|ExtractorPro|EyeNetIE|Ezooms|FDM\ 1|Fetch|FEZhead|FileHound|Findxbot|Flaming\ AttackBot|FlashGet|Foobot|Freeuploader|FrontPage|FyberSpider|GalaxyBot|Genieo|GetRight|GetWeb!|Gigabot|Go-Ahead-Got-It|Go!Zilla|GoZilla|GrabNet|Grafula|GrapeshotCrawler|GridBot|GT::WWW|Haansoft|Harvest|Havij|Hloader|HMView|httplib|HTTrack|Humanlinks|HybridBot|IDBot|IDG/IT|IlseBot|Image\ Stripper|Image\ Sucker|Indy\ Library|InfoNaviRobot|Insta\ crawl|Intelliseek|InterGET|Internet\ Ninja|InternetSeer\.com|Iria|IRLbot|Jakarta|JennyBot|JetCar|JikeSpider|JOC\ Web\ Spider|Jooblebot|kanagawa|Kapere|Kenjin.Spider|Keyword.Density|KlikBot|Koepa|Larbin|LeechFTP|LexiBot|Lftp|LibWeb|Libwhisker|LieBaoFast|Lightspeedsystems|Likse|LinkextractorPro|LinkpadBot|LinkScan|LinkWalker|LinqiaMetadataDownloaderBot|LinqiaRSSBot|LinqiaScrapeBot|Lipperhey|Litemage_walker|Lmspider|LNSpiderguy|Lwp-trivial|Magnet|Mag-Net|Majestic12|Majestic-SEO|MarkMonitor|Mass\ Downloader|Mata.Hari|MauiBot|Mb2345Browser|MeanPathbot|Mediatoolkitbot|MegaIndex.ru|Metauri|MicroMessenger|Microsoft.URL|MIIxpc|Mister\ PiX|MJ12bot|MojeekBot|Morfeus\ Fucking\ Scanner|Mozlila|Mr.4x3|MSFrontPage|MSIECrawler|MSRBOT|Mydictionarybot|Nbot|NearSite|NetAnts|NetLyzer|NetMechanic|NetSpider|NetZIP|NextGenSearchBot|NICErsPRO|Niki-bot|NimbleCrawler|Ninja|Nutch|Octopus|Offline\ Explorer|Offline\ Navigator|Openfind|OpenLinkProfiler|Openvas|OrangeBot|Owlin|PageGrabber|Pandalytics|Panscient|Papa\ Foto|pavuk|pcBrowser|PECL::HTTP|PeoplePal|Photon|PHPCrawl|Picscout|Picsearch|PictureFinder|Piepmatz|Pinterest|Pi-Monster|Pockey|ProPowerBot|ProWebWalker|psbot|QueryN.Metasearch|Quick-Crawler|RankActiveBot|RankFlex|Rankivabot|RankurBot|RealDownload|Reaper|RebelMouse|Recorder|RedesScrapy|ReGet|RepoMonkey|Ripper|RocketCrawler|Rogerbot|RSSingBot|RyzeCrawler|SafeSearch|SBIder|ScanAlert|Scrapy|Screaming|ScreenerBot|ScreenerBot|SearchmetricsBot|Search\ Snoop|SemrushBot|SentiBot|SeoSiteCheckup|SeznamBot|ShellBot|Shodan|Siphon|SiteCheckerBotCrawler|SiteExplorer|SiteLockSpider|SiteSnagger|SlySearch|SmartDownload|SMTBot|Snake|Snapbot|Snoopy|Sogou|Sosospider|SpaceBison|SpankBot|spanner|Spbot|SpiderBot|Spyder|Sqlmap|Sqworm|Steeler|Stripper|Sucker|SuperBot|SuperHTTP|Surfbot|SurveyBot|Suzuran|Swiftbot|Szukacz|Takeout|Teleport\ Pro|Telesoft|Telesphoreo|TheNomad|TightTwatBot|Titan|Toata\ dragostea\ mea\ pentru\ diavola|Toweyabot|TurnitinBot|TurnitinBot|TwengaBot|Twiceler|Typhoeus|UnisterBot|Upflow|URI::Fetch|URLy.Warning|Vacuum|Vagabondo|VB\ Project|VCI|VoidEYE|VoilaBot|WBSearchBot|Web\ Downloader|WebAuto|WebBandit|WebCopier|WebEMailExtrac|WebEnhancer|WebFetch|WebGo\ IS|WebImageCollector|WebLeacher|WebmasterWorldForumBot|WebReaper|WebSauger|Website\ eXtractor|Website\ Quester|WebStripper|WebWhacker|WebZIP|WeSEE|Wget|Whack|WhosTalking|Widow|WinHTTP|WiseGuys\ Robot|Wotbox|WPScan|WWWOFFLE|WWW-Collector-E|Xaldon\ WebSpider|XoviBot|Y!J|Yahoo!\ Slurp|YaK|YisouSpider|YoudaoBot|Zade|Zermelo|Zeus|ZumBot|ZyBorg) [NC] RewriteRule ^.* - [F,L]