Search Options

Results per page
Sort
Preferred Languages
Advance

Results 1 - 1 of 1 for MixedCaseBot (2.32 sec)

  2. github.com/codelibs/fess-crawler

    fess-crawler/src/test/resources/org/codelibs/fess/crawler/helper/robots_malformed.txt 

    Allow: /path:with:colons

# Case 4: Extra whitespace
User-agent:    ExtraSpaceBot
Disallow:     /spaced/
   Allow:   /also-spaced/

# Case 5: Mixed case directives (should still work)
UsEr-AgEnT: MixedCaseBot
DiSaLlOw: /test1/
AlLoW: /test2/
CrAwL-dElAy: 2
SiTeMaP: http://example.com/sitemap.xml

# Case 6: Comments in various positions
User-agent: CommentBot # inline comment
Disallow: /path1/ # another comment
    Registered: Sat Dec 20 11:21:39 UTC 2025
    - Last Modified: Fri Nov 14 12:52:01 UTC 2025
    - 2.6K bytes
    - Viewed (0)
Back to top