لیست لینک • Bots und Crawler

Name des Bots __________ User Agents

007ac9 Crawler __________ 007ac9 Crawler
1001FirmsBot __________ 1001FirmsBot
200PleaseBot __________ 200PleaseBot
Aboundex __________ Aboundex
AddThis.com __________ AddThis.com
ADmantX __________ admantx
Admedia [Bot] __________ AdMedia bot
AdnormCrawler __________ AdnormCrawler
AdsTxtCrawler __________ AdsTxtCrawler
Adstxtlab.com [Crawler] __________ adstxtlab.com Crawler
Advbot __________ Advbot
Affectv Robot __________ Affectv Robot
Affiliatewindow __________ affiliatewindow.com
AlphaBot __________ AlphaBot
AmazonAdBot __________ AmazonAdBot
Anonymous Coward __________ anonymous coward
Applebot __________ Applebot
AppleWebKit __________ AppleWebKit
Archive-tv __________ archive-tv.com
Archive.org_bot __________ Archive.org_bot
BDCbot __________ BDCbot
Beta [Bot] __________ betaBot
Bidswitchbot __________ bidswitchbot
Bitly [Bot] __________ bitlybot
Bot.AraTurka __________ Bot.AraTurka.com
BPImageWalker __________ BPImageWalker
BUbiNG __________ BUbiNG
Bulk-retrieval [Bot] __________ bulk-retrieval bot
Buzzbot __________ Buzzbot
Ca-crawler __________ ca-crawler
Calypso V __________ Calypso v
Centro Ads.txt [Crawler] __________ Centro Ads.txt Crawler
CheckMarkNetwork __________ CheckMarkNetwork
Cipa [Crawler] __________ CipaCrawler
Clarity [Bot] __________ Claritybot
Clickagy Intelligence [Bot] __________ Clickagy Intelligence Bot
Cliqzbot __________ Cliqzbot
CMS [Crawler] __________ CMS Crawler
Coccocbot Web __________ coccocbot-web
Codewisebot __________ Codewisebot
CoinCorner [Bot] __________ CoinCornerBot
Compatible [Crawler] __________ crawler-complaints
Dataprovider __________ Dataprovider
Contentdetection __________ contentDetection
ContextAd [Bot] __________ ContextAd Bot
Contxbot __________ contxbot
Crawler4j __________ crawler4j
Crazywebcrawler __________ CRAZYWEBCRAWLER
CrystalSemantics [Bot] __________ CrystalSemanticsBot
Datagnionbot __________ datagnionbot
DeuSu __________ DeuSu
Digincore Crawler [Bot] __________ Digincore crawler bot
DnyzBot __________ DnyzBot
DomainCheck.io __________ DomainCheck.io
DomainCrawler __________ DomainCrawler
DomainSigmaCrawler __________ DomainSigmaCrawler
DotBot __________ DotBot
DuckDuckGo [Bot] __________ DuckDuckGo-Favicons-Bot
Elefent __________ elefent
Epicbot __________ epicbot
EuripBot __________ EuripBot
Everyonedomainsbot __________ Everyonedomainsbot
ExtLinksBot __________ ExtLinksBot
Facebot __________ Facebot
Fastbot.de [Crawler] __________ fastbot.de crawler
Fetchbot __________ Fetchbot
Finbot __________ finbot
Findxbot __________ Findxbot
Fr-crawler __________ fr-crawler
Garlik [Crawler] __________ GarlikCrawler
GDPR [Bot] __________ GDPR-Bot
Genieo __________ Genieo
GetIntent [Crawler] __________ GetIntent Crawler
GigablastOpenSource __________ GigablastOpenSource
Gimme60 [Bot] __________ Gimme60bot
Gluten Free [Crawler] __________ Gluten Free Crawler
Google-Adwords-Instant __________ Google-Adwords-Instant
Gowikibot __________ Gowikibot
Grapeshot [Crawler] __________ GrapeshotCrawler
Hivemind __________ Hivemind
HRCrawler __________ HRCrawler
HyperCrawl __________ HyperCrawl
HyScore __________ HyScore
Ia_archiver __________ ia_archiver
IABTechLab Ads.txt [Crawler] __________ IABTechLab Ads.txt Crawler
IAS crawler __________ IAS crawler
IBusiness Shopcrawler __________ iBusiness Shopcrawler
DoCoMo __________ DoCoMo
Hybrid [Bot] __________ HybridBot
Implisense [Bot] __________ ImplisenseBot
Indeed [Bot] __________ IndeedBot
Insites [Bot] __________ Insitesbot
It2media-domain-crawler __________ it2media-domain-crawler
Izsearch.com __________ iZSearch.com
James [BOT] __________ James BOT
Jooble [Bot] __________ Jooblebot
Keybot Translation-Search-Machine __________ Keybot Translation-Search-Machine
Komodia [Bot] __________ KomodiaBot
Kraken __________ Kraken/Nutch-2.2.1
Laserlike [Bot] __________ Laserlikebot
Lb-spider __________ lb-spider
LCC __________ LCC ( http://corpora.informatik.uni-leipzig.d ... r_faq.html)
LightspeedSystemsCrawler __________ LightspeedSystemsCrawler
Linkis [Bot] __________ LinkisBot
Linkpad [Bot] __________ LinkpadBot
LinksCrawler 0.1beta __________ LinksCrawler 0.1beta
LinqiaCrawler [Bot] __________ LinqiaCrawlerBot
Livelap [Bot] __________ LivelapBot
LSSRocket [Crawler] __________ LSSRocketCrawler
Magpie [Crawler] __________ magpie-crawler
Maluuba [Crawler] __________ maluuba-crawler
Maui [Bot] __________ MauiBot
MaxPoint [Crawler] __________ MaxPointCrawler
MBCrawler __________ MBCrawler
mbr-Web-Crawler __________ mbr-Web-Crawler
Meanpathbot __________ meanpathbot
Galaxy Nexus Build __________ Galaxy Nexus Build
Mediatoolkitbot __________ Mediatoolkitbot
MegaIndex.ru __________ MegaIndex.ru
Mediapartners-Google __________ Mediapartners-Google
Meta_Bot __________ Meta_Bot
MetaJobBot __________ MetaJobBot
MetaURI API __________ MetaURI API
SearchmetricsBot __________ SearchmetricsBot
Metricsbot __________ Metricsbot
Mindupbot __________ mindUpBot
MixrankBot __________ MixrankBot
My Nutch [Spider] __________ My Nutch Spider
Trident __________ Trident/
MSIE [Crawler] __________ MSIECrawler
Mojeek [Bot] __________ MojeekBot
Moatbot __________ moatbot
MnoGoSearch __________ MnoGoSearch
MLIB_NetWork [Bot] __________ MLIB_NetWorkBot
NerdyBot __________ NerdyBot
Naaraa __________ Naaraa/
Honeso [Spider] __________ Honeso Spider
Toshiba Digital __________ Toshiba Digital Solution TDSL
Nutch-1.4 __________ nutch-1.4/Nutch-1.4
NewShareCounts __________ NewShareCounts.com/
Woo [Bot] __________ woobot/
Apercite __________ Apercite; http://www.apercite.fr
ODP link checker __________ pmoz.info ODP link checker
OFM Search __________ OFM search_robot .*google.*
OpenWebSpider __________ OpenWebSpider
OptimizationCrawler __________ OptimizationCrawler
Outclicks [Bot] __________ OutclicksBot/
Orliac [Bot] __________ E. Orliac/
PagesInventory __________ PagesInventory
PaperLiBot __________ PaperLiBot
PathDefender __________ PathDefender
Pinterest __________ Pinterestbot
HUAWEI VNS-L31 __________ HUAWEI VNS-L31 Build/
Python-urllib __________ Python-urllib/
PycURL __________ PycURL/
PulsePoint-Ads __________ PulsePoint-Ads.txt-Crawler
PubMatic Crawler __________ PubMatic Crawler Bot
Publiclibraryarchive __________ publiclibraryarchive.org
Psbot-page __________ psbot-page
PPBot __________ PPBot/
Plukkie __________ Plukkie/
Plista Datafication __________ Plista Datafication
Pipl [Bot] __________ PiplBot ( http://www.pipl.com/bot/)
Qwantify __________ Qwantify/ https://www.qwant.com/
RSSingBot __________ RSSingBot https://www.rssing.com
Rogerbot __________ rogerbot/
RobotsChecker __________ RobotsChecker/
ResearchBot __________ ResearchBot/
Redditbot __________ redditbot/
R6_FeedFetcher __________ R6_FeedFetcher
R6_CommentReader __________ R6_CommentReader
SafeDNSBot __________ SafeDNSBot
SafeSearch microdata __________ SafeSearch microdata crawler
Screaming Frog SEO [Spider] __________ Screaming Frog SEO Spider/
Pr-cy.ru Screenshot Bot __________ pr-cy.ru Screenshot Bot
SemrushBot-SA __________ SemrushBot-SA/
Seedingup __________ seedingup/status_crawler/
Seebot __________ seebot/
Seobility [Bot] __________ SeobilityBot
SEOkicks __________ SEOkicks; https://www.seokicks.de
SEOkicks-Robot __________ SEOkicks-Robot
Seznam [Bot] __________ SeznamBot/
Showyou [Bot] __________ ShowyouBot showyou.com
Sky nutch crawler __________ sky nutch crawler
Slackbot __________ Slackbot-LinkExpanding
SMTBot __________ SMTBot/
SocialRankIO [Bot] __________ SocialRankIOBot; http://socialrank.io
Sogou Pic [Spider] __________ Sogou Pic Spider/
Speedy Spider __________ Speedy Spider https://www.entireweb.com
Spiderbot __________ Spiderbot/
SurveyBot __________ SurveyBot/
SurdotlyBot __________ SurdotlyBot/
Streamline3 [Bot] __________ Streamline3Bot/
Stq_bot __________ stq_bot
SpuhexBot __________ SpuhexBot/
Spinn3r __________ aggregator:Spinn3r (Spinn3r 3.1); http://spinn3r.com/robot
Twitter [Bot] __________ Twitterbot/
TweetmemeBot __________ TweetmemeBot/
Tracemyfile __________ tracemyfile/
TouTrix crawler __________ TouTrix crawler/
TelegramBot __________ TelegramBot
TagVisit __________ TagVisit/
Uptimebot.org __________ Uptimebot.org
Uptimebot __________ Uptimebot/1.0; http://www.uptime.com/uptimebot
UMBot-FC __________ uMBot-FC/
Uipbot __________ uipbot/
UCrawler __________ uCrawler/
VoilaBot __________ VoilaBot
Vebidoobot __________ vebidoobot/
WeSEE __________ WeSEE:Ads/PageBot
WebTarantula.com Crawler __________ WebTarantula.com Crawler
HubSpot Webcrawler __________ HubSpot Webcrawler
VelenPublicWebCrawler __________ VelenPublicWebCrawler
WBSearchBot __________ WBSearchBot
WASALive-Bot __________ WASALive-Bot
WhatsApp __________ WhatsApp/
HeadlessChrome __________ HeadlessChrome
WordPress __________ WordPress/
XoviOnpageCrawler __________ XoviOnpageCrawler
XoviBot __________ XoviBot
Wotbox __________ Wotbox/
Yahoo Ad monitoring __________ Yahoo Ad monitoring
Yahoo! __________ Yahoo! Slurp
YaK __________ YaK/1.0; http://linkfluence.com/
YandexSearch __________ YandexSearch
YandexMobileBot __________ YandexMobileBot
YandexFavicons __________ YandexFavicons
YisouSpider __________ YisouSpider/