- Sort Score
- Result 10 results
- Languages All
Results 1 - 1 of 1 for NumericBot (0.04 sec)
-
fess-crawler/src/test/java/org/codelibs/fess/crawler/helper/RobotsTxtHelperTest.java
String[] sitemaps = robotsTxt.getSitemaps(); assertTrue(sitemaps.length >= 3); // At least the valid ones should be parsed // Test NumericBot - various crawl-delay formats // Should handle edge cases gracefully assertTrue(robotsTxt.getCrawlDelay("NumericBot") >= 0); // Test TabBot - tab characters should be treated as whitespace assertFalse(robotsTxt.allows("/tab1/", "TabBot"));
Registered: Sat Dec 20 11:21:39 UTC 2025 - Last Modified: Mon Nov 24 03:59:47 UTC 2025 - 20.6K bytes - Viewed (0)