# www.khd-blog.net/.../robots.txt K.-H. Dittberner - 19.OKT.2010 # V01.21 khd:KHD___robots.txt P 475/xx - 06.NOV.2013 # # Anweisungen fuer WWW-Suchroboter von Suchmaschinen im Internet. # # ============================================================================== # Basiert auf: # KHD-research___robots.txt K.-H. Dittberner - 19.OKT.2010 # V02.28 P 475/13 # ============================================================================== # # # # [xx/xx/13-khd] => V 01.xx: # - Pkt.0: xxx. # - xxx. # # ============================================================================== # 1. Roboter, die ausgeschlossen (oder gebremst) werden: # ============================================================================== #User-agent: * #Disallow: / # -------------------------- User-agent: Accoona-AI-Agent Disallow: / User-agent: Mata Hari Disallow: / User-agent: Slurp China Disallow: / User-agent: Slurp #Disallow: / Crawl-delay: 5 #Werte von 1 -- 10 sind moeglich. #User-agent: Microsoft URL Control #Disallow: / User-agent: IWAgent/ 1.0 - www.brandprotect.com Disallow: / User-agent: LinkWalker Disallow: / User-agent: libwww-perl Disallow: / User-agent: BPImageWalker/2.0 (www.bdbrandprotect.com) Disallow: / User-agent: Toplistbot Disallow: / User-agent: MSR-ISRCCrawler Disallow: / User-agent: WebAlta Crawler/1.3.34 Disallow: / User-agent: WebAlta Crawler/1.3.34 (http://www.webalta.net/ru/about_webmaster.html) (Windows; U; Windows NT 5.1; ru-RU) Disallow: / User-agent: WebAlta Crawler/2.0 Disallow: / User-agent: WebAlta Crawler/2.0 (http://www.webalta.net/ru/about_webmaster.html) (Windows; U; Windows NT 5.1; ru-RU) Disallow: / User-agent: Morfeus Fucking Scanner Disallow: / #User-agent: msnbot/1.1 #Disallow: / #User-agent: msnbot #Disallow: / User-agent: Twiceler Disallow: / #User-agent: Yandex #Disallow: / User-agent: Purebot Disallow: / User-agent: Speedy Spider Disallow: / User-agent: Yeti Disallow: / User-agent: Zscho.de Crawler Disallow: / User-agent: Sosospider Disallow: / User-agent: yacybot Disallow: / #User-agent: Yandex #Disallow: / User-agent: Online24-Bot Disallow: / User-agent: Sogou web spider Disallow: / User-agent: Baiduspider Disallow: / User-agent: spbot Disallow: / User-agent: DotBot Disallow: / User-agent: MyEngines-Bot Disallow: / User-agent: iCcrawler Disallow: / User-agent: Zango Disallow: / User-agent: StackRambler Disallow: / User-agent: Ruky-Roboter Disallow: / User-agent: NaverBot Disallow: / User-agent: psbot Disallow: / User-agent: CCBot Disallow: / User-agent: Linguee Bot Disallow: / # ============================================================================== # 2. Welche Roboter duerfen zugreifen? Alle anderen! # ============================================================================== User-agent: * # ============================================================================== # 3. Wo ist die Site-map? # ============================================================================== Sitemap: http://khd-blog.net/Site-map.html # ============================================================================== # 4. Gesperrte Zugriffe von Verzeichnissen: (Ob's was nutzt?) # ============================================================================== Disallow: /Ads/ Disallow: /Archiv/Icons/ Disallow: /Archiv/Pict/ Disallow: /Audio/ Disallow: /Bln/Icons/ Disallow: /Bln/Pict/ Disallow: /Bln/Pict/Hist/ Disallow: /BSE/ Disallow: /BSE/Icons/ Disallow: /BSE/Images/ Disallow: /BSE/Docs/ Disallow: /cgi-bin/ Disallow: /Food/Icons/ Disallow: /Food/Pict/ Disallow: /Food/Tab/ Disallow: /Fotos/ Disallow: /Heimat/B/Bln/Icons/ Disallow: /Heimat/B/Bln/Pict/ Disallow: /Heimat/W/ Disallow: /Heimat/W/Ads/ Disallow: /Heimat/W/Audio/ Disallow: /Heimat/W/CD-ROM/ Disallow: /Heimat/W/CD-ROM/Icons/ Disallow: /Heimat/W/CD-ROM/PC-Fonts/ Disallow: /Heimat/W/Forum/Icons/ Disallow: /Heimat/W/Forum/Pict/ Disallow: /Heimat/W/Icons/ Disallow: /Heimat/W/Images/ Disallow: /Heimat/W/Images/Deko/ Disallow: /Heimat/W/Pict/ Disallow: /Heimat/W/Pict/Alb/ Disallow: /Heimat/W/Pict/BHf/ Disallow: /Heimat/W/Pict/Bred/ Disallow: /Heimat/W/Pict/Fbl/ Disallow: /Heimat/W/Pict/Misc/ Disallow: /Heimat/W/Pict/Psn/ Disallow: /Heimat/W/Plan/ Disallow: /Heimat/W/Rezepte/ Disallow: /Heimat/W/Video/ Disallow: /Icons/ Disallow: /Images/ Disallow: /Images/Deko/ Disallow: /Images/Wp/ Disallow: /Internet/Icons/ Disallow: /Kultur/Icons/ Disallow: /Kultur/Pict/ Disallow: /Misc/Icons/ Disallow: /My_Lisa/Icons/ Disallow: /My_Lisa/Logos/ Disallow: /My_Lisa/Pict/ Disallow: /My_Progs/ Disallow: /Politik/Icons/ Disallow: /Politik/Pict/ Disallow: /Tech/Icons/ Disallow: /Tech/Images/ Disallow: /ThS/Icons/ Disallow: /ThS/Pict/ Disallow: /Video/ Disallow: /Welcome/Icons/ Disallow: /Welcome/Images/ Disallow: /WWW_Media/Icons/ Disallow: /WWW_Media/Images/ Disallow: /ThS/ Disallow: /t-off/ Disallow: /t-off/Icons/ Disallow: /t-off/Images/ Disallow: /t-off/Docs/ # ============================================================================== # 5. Gesperrte Zugriffe von Einzel-Files: # ============================================================================== Disallow: /BSE/Docs/Doener-Hersteller.pdf Disallow: /BSE/Docs/index.html Disallow: /default.html Disallow: /No_Homepage.html Disallow: /Recherche.html Disallow: /ThS/Docs/UITP_Stadtentw_Li-Sued.jpg Disallow: /ThS/Docs/LaSchGeb_Died_Grossb.jpg Disallow: /ThS/Docs/VIVICO_Golfplan_2010.jpg Disallow: /t-off/Docs/No_go_Telcos.html Disallow: /t-off/Docs/Freenet.de_Blamage_2006.html #Disallow: /xxx # --------------- Disallow: /Heimat/W/Docs/_Wbg_Adressbuch_1924.pdf Disallow: /Heimat/W/Docs/Awd_Adressbuch_1924.txt Disallow: /Heimat/W/Docs/Dsn_Adressbuch_1924.txt Disallow: /Heimat/W/Docs/Fbg_Adressbuch_1924.txt Disallow: /Heimat/W/Docs/Nwd_Adressbuch_1924.txt Disallow: /Heimat/W/Docs/Wbg_Adressbuch_1924.txt Disallow: /Heimat/W/Docs/Awd_Adressbuch_Umg_1924.txt Disallow: /Heimat/W/Docs/Dsn_Adressbuch_Umg_1924.txt Disallow: /Heimat/W/Docs/Fbg_Adressbuch_Umg_1924.txt Disallow: /Heimat/W/Docs/Nwd_Adressbuch_Umg_1924.txt Disallow: /Heimat/W/Docs/Wbg_Adressbuch_Umg_1925.txt # --------------- # --------------- # # # # ==============================================================================