Search Options

Results per page
Sort
Preferred Languages
Advance

Results 1 - 1 of 1 for GoodBot (0.17 sec)

  2. github.com/codelibs/fess-crawler

    fess-crawler/src/test/resources/org/codelibs/fess/crawler/helper/robots_malformed.txt 

    # Case 1: Directives before any User-agent (should be ignored)
Disallow: /orphaned1/
Allow: /orphaned2/

# Case 2: Valid user-agent with various malformed directives
User-agent: GoodBot
Disallow: /admin/
InvalidDirective: some-value
unknown-field: test
Disallow /missing-colon
Disallow:    # empty value (should be treated as allowing all)
Allow: /public/
Crawl-delay: invalid-number
    Registered: Sat Dec 20 11:21:39 UTC 2025
    - Last Modified: Fri Nov 14 12:52:01 UTC 2025
    - 2.6K bytes
    - Viewed (0)
Back to top