Search Options

Results per page
Sort
Preferred Languages
Advance

Results 1 - 2 of 2 for SpecialCharBot (2.97 sec)

  1. fess-crawler/src/test/java/org/codelibs/fess/crawler/helper/RobotsTxtHelperTest.java

            assertFalse(robotsTxt.allows("/test/", longBotName));
    
            // Test SpecialCharBot - special characters in paths
            assertFalse(robotsTxt.allows("/path with spaces/", "SpecialCharBot"));
            assertFalse(robotsTxt.allows("/path%20encoded/", "SpecialCharBot"));
            assertFalse(robotsTxt.allows("/path?query=value", "SpecialCharBot"));
    
    Registered: Sat Dec 20 11:21:39 UTC 2025
    - Last Modified: Mon Nov 24 03:59:47 UTC 2025
    - 20.6K bytes
    - Viewed (0)
  2. fess-crawler/src/test/resources/org/codelibs/fess/crawler/helper/robots_malformed.txt

    User-agent: VeryLongBotNameThatExceedsNormalLengthAndShouldStillBeProcessedCorrectlyWithoutAnyIssuesEvenThoughItIsExtremelyLongAndUnusual
    Disallow: /test/
    
    # Case 9: Special characters in paths
    User-agent: SpecialCharBot
    Disallow: /path with spaces/
    Disallow: /path%20encoded/
    Disallow: /path?query=value
    Disallow: /path#fragment
    Allow: /unicode/日本語/
    
    # Case 10: Multiple User-agents in sequence
    User-agent: Bot1
    Registered: Sat Dec 20 11:21:39 UTC 2025
    - Last Modified: Fri Nov 14 12:52:01 UTC 2025
    - 2.6K bytes
    - Viewed (0)
Back to top