Search Options

Results per page
Sort
Preferred Languages
Advance

Results 1 - 1 of 1 for MultiColonBot (1.04 sec)

  1. fess-crawler/src/test/resources/org/codelibs/fess/crawler/helper/robots_malformed.txt

    Allow: /public/
    Crawl-delay: invalid-number
    Crawl-delay: -10
    Crawl-delay: 5.5
    Crawl-delay:
    Allow:    # empty value
    
    # Case 3: Multiple colons in directive
    User-agent: MultiColonBot
    Disallow: http://example.com:8080/path
    Allow: /path:with:colons
    
    # Case 4: Extra whitespace
    User-agent:    ExtraSpaceBot
    Disallow:     /spaced/
       Allow:   /also-spaced/
    
    Registered: Sat Dec 20 11:21:39 UTC 2025
    - Last Modified: Fri Nov 14 12:52:01 UTC 2025
    - 2.6K bytes
    - Viewed (0)
Back to top