Search Options

Results per page
Sort
Preferred Languages
Advance

Results 1 - 1 of 1 for SpecialCharBot (3.13 sec)

  1. fess-crawler/src/test/resources/org/codelibs/fess/crawler/helper/robots_malformed.txt

    User-agent: VeryLongBotNameThatExceedsNormalLengthAndShouldStillBeProcessedCorrectlyWithoutAnyIssuesEvenThoughItIsExtremelyLongAndUnusual
    Disallow: /test/
    
    # Case 9: Special characters in paths
    User-agent: SpecialCharBot
    Disallow: /path with spaces/
    Disallow: /path%20encoded/
    Disallow: /path?query=value
    Disallow: /path#fragment
    Allow: /unicode/日本語/
    
    # Case 10: Multiple User-agents in sequence
    User-agent: Bot1
    Registered: Sat Dec 20 11:21:39 UTC 2025
    - Last Modified: Fri Nov 14 12:52:01 UTC 2025
    - 2.6K bytes
    - Viewed (0)
Back to top