Search Options

Results per page
Sort
Preferred Languages
Advance

Results 1 - 1 of 1 for NumericBot (0.04 sec)

  1. fess-crawler/src/test/java/org/codelibs/fess/crawler/helper/RobotsTxtHelperTest.java

            String[] sitemaps = robotsTxt.getSitemaps();
            assertTrue(sitemaps.length >= 3); // At least the valid ones should be parsed
    
            // Test NumericBot - various crawl-delay formats
            // Should handle edge cases gracefully
            assertTrue(robotsTxt.getCrawlDelay("NumericBot") >= 0);
    
            // Test TabBot - tab characters should be treated as whitespace
            assertFalse(robotsTxt.allows("/tab1/", "TabBot"));
    Registered: Sat Dec 20 11:21:39 UTC 2025
    - Last Modified: Mon Nov 24 03:59:47 UTC 2025
    - 20.6K bytes
    - Viewed (0)
Back to top