* @param to the end time in HH:MM format
     * @param days comma-separated list of days (1=Sunday, 7=Saturday)
     * @param delay the delay in milliseconds to apply during this interval
     */
    public void addIntervalRule(final String from, final String to, final String days, final long delay) {
        ruleList.add(new IntervalRule(from, to, days, delay));
    }

    /**

    }

    /**
     * Gets the crawl delay value for the specified user agent from robots.txt.
     * The crawl delay specifies the time (in seconds) to wait between successive requests.
     *
     * @param userAgent The user agent string to match against robots.txt directives
     * @return The crawl delay value in seconds. Returns 0 if no matching directive is found
     *         or no crawl delay is specified for the matching directive.
     */

    /**
     * Container for managing crawler components.
     */
    @Resource
    protected CrawlerContainer crawlerContainer;

    /**
     * Controller for managing crawling intervals and delays.
     */
    @Resource
    protected IntervalController intervalController;

    /**
     * Factory for creating crawler clients.
     */
    @Resource

     *
     * @return the registration timeout
     */
    long getWitnessRegistrationTimeout();

    /**
     * Gets the witness reconnect delay in milliseconds.
     *
     * @return the reconnect delay
     */
    long getWitnessReconnectDelay();

    /**
     * Gets whether witness service discovery is enabled.
     *
     * @return true if discovery is enabled
     */

        // First access - should not delay
        final long start1 = System.nanoTime();
        controller.delayBeforeProcessing();
        final long elapsed1 = (System.nanoTime() - start1) / 1000000;
        assertTrue("First access should not delay", elapsed1 < 50);

        // Second access to same host - should delay
        q.setUrl("http://example.com/page2");

    }

    /**
     * Test multiple delay calls
     */
    public void test_multipleDelayCalls() {
        final TestIntervalController controller = new TestIntervalController();

        controller.delay(IntervalController.PRE_PROCESSING);
        controller.delay(IntervalController.POST_PROCESSING);
        controller.delay(IntervalController.NO_URL_IN_QUEUE);
        controller.delay(IntervalController.WAIT_NEW_URL);

                    // Check for potential overflow before multiplication
                    long delay = (retryDelayMs > Long.MAX_VALUE / multiplier) ? MAX_RETRY_DELAY
                            : Math.min(MAX_RETRY_DELAY, retryDelayMs * multiplier);
                    Thread.sleep(delay);
                }

                // Attempt to reset the connection
                connection.reset();

            if (!exponentialBackoff) {
                return initialDelayMs;
            }

            long delay = initialDelayMs;
            for (int i = 1; i < attemptNumber; i++) {
                delay = (long) (delay * backoffMultiplier);
            }

            return Math.min(delay, maxDelayMs);
        }

        public int getMaxAttempts() {
            return maxAttempts;
        }

```

### CrawlerThread (`CrawlerThread.java`)

Worker thread for crawling.

**Flow**: Poll URL → Validate → Get client → Delay → Check last-modified → Execute → Process → Extract children → Queue children → Delay

### CrawlerClientFactory

Pattern-based client selection using `LinkedHashMap<Pattern, CrawlerClient>`.

**Standard Patterns**:
```java

        } finally {
            CloseableUtil.closeQuietly(in);
        }

        // Should not throw exception for invalid crawl-delay
        assertNotNull(robotsTxt);
        // Invalid crawl-delay should be ignored (default 0)
        assertEquals(0, robotsTxt.getCrawlDelay("TestBot"));
        // Other directives should still work
        assertFalse(robotsTxt.allows("/test/", "TestBot"));
    }