Skip to content

Trim UA block list for false positives #189

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 15 commits into
base: master
Choose a base branch
from
Open

Trim UA block list for false positives #189

Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
ccd9484
Remove overly greedy entry "Not", which blocks every phone or tablet …
Feb 5, 2023
9539ad2
Remove "PyCurl" entry, which is just a Python wrapper for cURL, and n…
Feb 5, 2023
dfd2b7a
Remove LWP entries - this is just Perl's normal request library
Feb 5, 2023
c79f42d
Remove overly greedy entry "Leap", which blocks every app with "Mobil…
Feb 5, 2023
1ee8f92
Remove overly greedy entry "Ninja", which blocks every request from M…
Feb 5, 2023
954ea71
Remove overly greedy entry "oBot", which matches (among any other typ…
Feb 5, 2023
d55d6b2
Remove overly greedy entry WEBDAV, which can (and in many cases will)…
Feb 5, 2023
a53f776
Remove "Bullseye" entry. This is the codename for Debian 11 (the curr…
Feb 5, 2023
f783035
Remove "Siteimprove" - this is a paid service for assessing your site…
Feb 5, 2023
89810ab
Remove "archive.org_bot" - this is Archive.org, a non-profit working …
Feb 5, 2023
e8b82cf
Remove "Netcraft" entry. Netcraft is an online intelligence company f…
Feb 5, 2023
2b25b9b
Remove "Shodan" entry. Shodan is an extremely powerful tool for vulne…
Feb 5, 2023
03d3b6b
Remove various Chinese entries said to be legit - actual search engin…
Feb 5, 2023
af68f5a
Remove "Exabot" entry, this is said to be a good and respectful bot -…
Feb 5, 2023
8922398
Remove "Mojeek" entry, this is said to be a good and respectful bot -…
Feb 5, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 0 additions & 22 deletions _generator_lists/bad-user-agents.list
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
01h4x.com
360Spider
404checker
404enemy
80legs
Expand Down Expand Up @@ -62,7 +61,6 @@ Buck
Buddy
BuiltBotTough
BuiltWith
Bullseye
BunnySlippers
BuzzSumo
CATExplorador
Expand Down Expand Up @@ -135,7 +133,6 @@ Ecxi
EirGrabber
EroCrawler
Evil
Exabot
Express\ WebPictures
ExtLinksBot
Extractor
Expand Down Expand Up @@ -181,7 +178,6 @@ HTMLparser
HTTP::Lite
HTTrack
Haansoft
HaosouSpider
Harvest
Havij
Heritrix
Expand Down Expand Up @@ -225,10 +221,8 @@ Keyword\ Density
Kinza
Kozmosbot
LNSpiderguy
LWP::Simple
Lanshanbot
Larbin
Leap
LeechFTP
LeechGet
LexiBot
Expand Down Expand Up @@ -278,13 +272,11 @@ Meanpathbot
Mediatoolkitbot
MegaIndex.ru
Metauri
MicroMessenger
Microsoft\ Data\ Access
Microsoft\ URL\ Control
Minefield
Mister\ PiX
Moblie Safari
Mojeek
Mojolicious
MolokaiBot
Morfeus\ Fucking\ Scanner
Expand All @@ -306,7 +298,6 @@ NetMechanic
NetSpider
NetZIP
Net\ Vampire
Netcraft
Nettrack
Netvibes
NextGenSearchBot
Expand All @@ -315,9 +306,7 @@ Niki-bot
Nikto
NimbleCrawler
Nimbostratus
Ninja
Nmap
Not
Nuclei
Nutch
Octopus
Expand Down Expand Up @@ -362,7 +351,6 @@ Psbot
Pu_iN
Pump
PxBroker
PyCurl
QueryN\ Metasearch
Quick-Crawler
RSSingBot
Expand Down Expand Up @@ -410,7 +398,6 @@ SentiBot
SeoSiteCheckup
SeobilityBot
Seomoz
Shodan
Siphon
SiteCheckerBotCrawler
SiteExplorer
Expand All @@ -419,7 +406,6 @@ SiteSnagger
SiteSucker
Site\ Sucker
Sitebeam
Siteimprove
Sitevigil
SlySearch
SmartDownload
Expand All @@ -428,7 +414,6 @@ Snapbot
Snoopy
SocialRankIOBot
Sociscraper
Sogou\ web\ spider
Sosospider
Sottopop
SpaceBison
Expand Down Expand Up @@ -494,7 +479,6 @@ Voil
Voltron
WASALive-Bot
WBSearchBot
WEBDAV
WISENutbot
WPScan
WWW-Collector-E
Expand Down Expand Up @@ -563,7 +547,6 @@ ZoominfoBot
ZumBot
ZyBorg
adscanner
archive.org_bot
arquivo-web-crawler
arquivo.pt
autoemailspider
Expand All @@ -574,7 +557,6 @@ clark-crawler
coccocbot
cognitiveseo
com.plumanalytics
crawl.sogou.com
crawler.feedback
crawler4j
dataforseo.com
Expand All @@ -591,14 +573,11 @@ ips-agent
isitwp.com
iubenda-radar
linkdexbot
lwp-request
lwp-trivial
magpie-crawler
meanpathbot
mediawords
muhstik-scan
netEstate\ NE\ Crawler
oBot
page\ scorer
pcBrowser
plumanalytics
Expand All @@ -617,7 +596,6 @@ serpstatbot
sexsearcher
sitechecker.pro
siteripz
sogouspider
sp_auditbot
spyfu
sysscan
Expand Down