Search Options

Results per page
Sort
Preferred Languages
Advance

Results 1 - 2 of 2 for googlebot (0.04 sec)

  2. github.com/codelibs/fess-crawler

    fess-crawler/src/test/java/org/codelibs/fess/crawler/helper/RobotsTxtHelperTest.java 

                assertTrue(robotsTxt.allows("/foo/bar/index.html", userAgent));
            assertEquals(1314000, robotsTxt.getCrawlDelay(userAgent));
        }

        for (String userAgent : new String[] { "GOOGLEBOT", "GoogleBot", "googlebot" }) {
            assertTrue(robotsTxt.allows("/aaa", userAgent));
            assertTrue(robotsTxt.allows("/private/", userAgent));
            assertTrue(robotsTxt.allows("/private/index.html", userAgent));
    Registered: Sat Dec 20 11:21:39 UTC 2025
    - Last Modified: Mon Nov 24 03:59:47 UTC 2025
    - 20.6K bytes
    - Viewed (0)
  3. github.com/codelibs/fess

    src/main/java/org/codelibs/fess/crawler/transformer/FessXpathTransformer.java 

                    final String comment = childNode.getNodeValue().trim();
                if (comment.startsWith("googleoff:")) {
                    flag.setValue(false);
                } else if (comment.startsWith("googleon:")) {
                    flag.setValue(true);
                }
            }

            if (!flag.getValue() && childNode.getNodeType() == Node.TEXT_NODE) {
                if (removedNodeList == null) {
    Registered: Sat Dec 20 09:19:18 UTC 2025
    - Last Modified: Fri Dec 12 13:58:40 UTC 2025
    - 54.6K bytes
    - Viewed (0)
Back to top