Search Options

Results per page
Sort
Preferred Languages
Advance

Results 1 - 3 of 3 for GoogleBot (0.04 sec)

  2. github.com/codelibs/fess-crawler

    fess-crawler/src/test/java/org/codelibs/fess/crawler/entity/RobotsTxtTest.java 

            // Test GoogleBot
        assertFalse(robotsTxt.allows("/admin/secret.html", "GoogleBot"));
        assertTrue(robotsTxt.allows("/admin/public/info.html", "GoogleBot"));
        assertTrue(robotsTxt.allows("/public/", "GoogleBot"));
        assertEquals(0, robotsTxt.getCrawlDelay("GoogleBot"));

        // Test BingBot
    Registered: Sat Dec 20 11:21:39 UTC 2025
    - Last Modified: Thu Nov 13 13:29:22 UTC 2025
    - 14.4K bytes
    - Viewed (0)
  3. github.com/codelibs/fess-crawler

    fess-crawler/src/test/resources/org/codelibs/fess/crawler/helper/robots.txt 

    User-agent: FessCrawler
Disallow:           # allows all 

User-agent: BruteBot
Disallow: /
Allow: /foo/bar/
Crawl-delay: 1314000

# welcome!
User-agent: Googlebot
Crawl-delay: 1

User-agent: *
Disallow: /private/
Disallow: /help        # disallows /help.html, /help/index.html, etc.
Allow: /help/faq.html
Crawl-delay: 3

User-agent: Crawler
Disallow: /aaa

User-agent: Crawler/1.0
Disallow: /bbb
    Registered: Sat Dec 20 11:21:39 UTC 2025
    - Last Modified: Sun Oct 11 02:16:55 UTC 2015
    - 566 bytes
    - Viewed (0)
  4. github.com/codelibs/fess-crawler

    fess-crawler/src/test/java/org/codelibs/fess/crawler/helper/RobotsTxtHelperTest.java 

                assertTrue(robotsTxt.allows("/foo/bar/index.html", userAgent));
            assertEquals(1314000, robotsTxt.getCrawlDelay(userAgent));
        }

        for (String userAgent : new String[] { "GOOGLEBOT", "GoogleBot", "googlebot" }) {
            assertTrue(robotsTxt.allows("/aaa", userAgent));
            assertTrue(robotsTxt.allows("/private/", userAgent));
            assertTrue(robotsTxt.allows("/private/index.html", userAgent));
    Registered: Sat Dec 20 11:21:39 UTC 2025
    - Last Modified: Mon Nov 24 03:59:47 UTC 2025
    - 20.6K bytes
    - Viewed (0)
Back to top