User-agent: TurnitinBot User-agent: x28-job-bot User-agent: VelenPublicWebCrawler User-agent: Omgilitbot User-agent: Google-Extended User-agent: ChatGPT-User User-agent: GPTBot User-agent: magpie-crawler User-agent: SEOkicks User-agent: DataForSeoBot User-agent: adsbot User-agent: CCBot User-agent: barkrowler User-agent: PetalBot User-agent: Buck User-agent: FemtosearchBot User-agent: Gigabot User-agent: MauiBot User-agent: BLEXBot User-agent: 007ac9 User-agent: MJ12bot User-agent: AhrefsBot User-agent: SeznamBot User-agent: daumoa User-agent: SputnikBot User-agent: WBSearchBot User-agent: Yeti User-agent: sogou spider User-agent: NaverBot User-agent: sistrix User-agent: spbot User-agent: wiederfreibot/1.0 User-agent: dotbot User-agent: XoviBot User-agent: Linguee User-agent: seoscanners.net User-agent: BUbiNG User-agent: serpstatbot Disallow: / User-agent: * Crawl-delay: 240 Disallow: /wiki Disallow: /emacs.pl Disallow: /cgi-bin/emacs Disallow: /cgi-bin/wiki?action= Disallow: /cgi-bin/wiki?search= Disallow: /emacs?action= Disallow: /emacs/?action= Disallow: /emacs-de?action= Disallow: /emacs-en?action= Disallow: /emacs-es?action= Disallow: /emacs-fr?action= Disallow: /emacs-gr?action= Disallow: /emacs-ja?action= Disallow: /emacs-jp?action= Disallow: /emacs-ko?action= Disallow: /emacs-pl?action= Disallow: /emacs-pt?action= Disallow: /emacs-ru?action= Disallow: /emacs-se?action= Disallow: /emacs/download/ Disallow: /cgi-bin/latin-1.pl Disallow: /cgi-bin/wiki.pl?action= Disallow: /cgi-bin/wiki.pl?search= Disallow: /cgi-bin/wiki?action= Disallow: /cgi-bin/wiki?search= Disallow: /cgi-bin/wiki/download/ Disallow: /cgit # Block all known AI crawlers and assistants # from using content for training AI models. # Source: https://robotstxt.com/ai User-Agent: GPTBot User-Agent: ClaudeBot User-Agent: Claude-Web User-Agent: CCBot User-Agent: Googlebot-Extended User-Agent: Applebot-Extended User-Agent: Facebookbot User-Agent: Meta-ExternalAgent User-Agent: Meta-ExternalFetcher User-Agent: diffbot User-Agent: PerplexityBot User-Agent: Omgili User-Agent: Omgilibot User-Agent: webzio-extended User-Agent: ImagesiftBot User-Agent: Bytespider User-Agent: Amazonbot User-Agent: Youbot User-Agent: SemrushBot-OCOB User-Agent: Petalbot User-Agent: VelenPublicWebCrawler User-Agent: TurnitinBot User-Agent: Timpibot User-Agent: OAI-SearchBot User-Agent: ICC-Crawler User-Agent: AI2Bot User-Agent: AI2Bot-Dolma User-Agent: DataForSeoBot User-Agent: AwarioBot User-Agent: AwarioSmartBot User-Agent: AwarioRssBot User-Agent: Google-CloudVertexBot User-Agent: PanguBot User-Agent: Kangaroo Bot User-Agent: Sentibot User-Agent: img2dataset User-Agent: Meltwater User-Agent: Seekr User-Agent: peer39_crawler User-Agent: cohere-ai User-Agent: cohere-training-data-crawler User-Agent: DuckAssistBot User-Agent: Scrapy Disallow: / DisallowAITraining: / # Block any non-specified AI crawlers (e.g., new # or unknown bots) from using content for training # AI models. This directive is still experimental # and may not be supported by all AI crawlers. User-Agent: * DisallowAITraining: /