Search Options

Results per page
Sort
Preferred Languages
Advance

Results 1 - 1 of 1 for Bot2 (0.01 sec)

  2. github.com/codelibs/fess-crawler

    fess-crawler/src/test/resources/org/codelibs/fess/crawler/helper/robots_malformed.txt 

    Disallow: /path with spaces/
Disallow: /path%20encoded/
Disallow: /path?query=value
Disallow: /path#fragment
Allow: /unicode/日本語/

# Case 10: Multiple User-agents in sequence
User-agent: Bot1
User-agent: Bot2
User-agent: Bot3
Disallow: /shared/

# Case 11: Sitemap with various formats
Sitemap: http://example.com/sitemap.xml
sitemap: http://example.com/sitemap2.xml
SITEMAP: http://example.com/sitemap3.xml
    Registered: Sat Dec 20 11:21:39 UTC 2025
    - Last Modified: Fri Nov 14 12:52:01 UTC 2025
    - 2.6K bytes
    - Viewed (0)
Back to top