- Sort Score
- Result 10 results
- Languages All
Results 1 - 2 of 2 for SpecialCharBot (4.64 sec)
-
fess-crawler/src/test/java/org/codelibs/fess/crawler/helper/RobotsTxtHelperTest.java
assertFalse(robotsTxt.allows("/test/", longBotName)); // Test SpecialCharBot - special characters in paths assertFalse(robotsTxt.allows("/path with spaces/", "SpecialCharBot")); assertFalse(robotsTxt.allows("/path%20encoded/", "SpecialCharBot")); assertFalse(robotsTxt.allows("/path?query=value", "SpecialCharBot"));
Registered: Sat Dec 20 11:21:39 UTC 2025 - Last Modified: Mon Nov 24 03:59:47 UTC 2025 - 20.6K bytes - Viewed (0) -
fess-crawler/src/test/resources/org/codelibs/fess/crawler/helper/robots_malformed.txt
User-agent: VeryLongBotNameThatExceedsNormalLengthAndShouldStillBeProcessedCorrectlyWithoutAnyIssuesEvenThoughItIsExtremelyLongAndUnusual Disallow: /test/ # Case 9: Special characters in paths User-agent: SpecialCharBot Disallow: /path with spaces/ Disallow: /path%20encoded/ Disallow: /path?query=value Disallow: /path#fragment Allow: /unicode/日本語/ # Case 10: Multiple User-agents in sequence User-agent: Bot1
Registered: Sat Dec 20 11:21:39 UTC 2025 - Last Modified: Fri Nov 14 12:52:01 UTC 2025 - 2.6K bytes - Viewed (0)