- Sort Score
- Result 10 results
- Languages All
Results 1 - 1 of 1 for ExtraSpaceBot (0.09 sec)
-
fess-crawler/src/test/resources/org/codelibs/fess/crawler/helper/robots_malformed.txt
Crawl-delay: Allow: # empty value # Case 3: Multiple colons in directive User-agent: MultiColonBot Disallow: http://example.com:8080/path Allow: /path:with:colons # Case 4: Extra whitespace User-agent: ExtraSpaceBot Disallow: /spaced/ Allow: /also-spaced/ # Case 5: Mixed case directives (should still work) UsEr-AgEnT: MixedCaseBot DiSaLlOw: /test1/ AlLoW: /test2/ CrAwL-dElAy: 2
Registered: Sat Dec 20 11:21:39 UTC 2025 - Last Modified: Fri Nov 14 12:52:01 UTC 2025 - 2.6K bytes - Viewed (0)