* information to search engines and may not correlate exactly to how often
     * they crawl the page. Valid values are:
     * <ul>
     * <li>always</li>
     * <li>hourly</li>
     * <li>daily</li>
     * <li>weekly</li>
     * <li>monthly</li>
     * <li>yearly</li>
     * <li>never</li>
     * </ul>

            } else {
                final String[] values = path.replaceAll("/+", "/").replaceFirst("/$", "").split("/");
                if (values.length == 1) {
                    parent = "/";
                    name = null;
                } else if (values.length == 2) {
                    parent = "/";
                    name = values[1];
                } else {

    transformer.setMaxContentSize(10 * 1024 * 1024); // 10MB limit
});
```

## Troubleshooting

### Common Issues

#### Connection Timeouts
```java
// Increase timeout values
client.setConnectionTimeout(60000);  // 60 seconds
client.setSocketTimeout(120000);     // 120 seconds
```

#### Memory Issues
```java
// Reduce concurrent threads and batch sizes

        assertEquals(100, crawlerContext.getRobotsTxtUrlSet().size());
        assertNotNull(crawlerContext.removeSitemaps());
    }

    /**
     * Test boundary values
     */
    public void test_boundaryValues() {
        // Test with maximum values
        crawlerContext.setActiveThreadCount(Integer.MAX_VALUE);
        assertEquals(Integer.MAX_VALUE, crawlerContext.getActiveThreadCount().intValue());

                    isGroupRecordStarted = true;
                    value = getValue(DISALLOW_RECORD, line);
                    if (value != null) {
                        if (!currentDirectiveList.isEmpty() && value.length() > 0) {
                            for (final Directive directive : currentDirectiveList) {
                                directive.addDisallow(value);
                            }
                        }

        for (final Map.Entry<String, Pattern> entry : regexMap.entrySet()) {
            String value = "";
            final Object obj = map.get(entry.getKey());
            if (obj != null) {
                value = obj.toString();
            }
            final Matcher matcher = entry.getValue().matcher(value);
            if (allRequired) {
                if (!matcher.matches()) {
                    return false;

        // Default constructor
    }

    @Override
    /**
     * Sets the initialization parameters for the underlying CrawlerClient.
     *
     * @param params a map of parameter names and values to be set
     */
    public void setInitParameterMap(final Map<String, Object> params) {
        client.setInitParameterMap(params);
    }

    /**
     * Executes the request with retry logic.

        return directive.allows(path);
    }

    /**
     * Gets the crawl delay value for the specified user agent from robots.txt.
     * The crawl delay specifies the time (in seconds) to wait between successive requests.
     *
     * @param userAgent The user agent string to match against robots.txt directives
     * @return The crawl delay value in seconds. Returns 0 if no matching directive is found

        final CountDownLatch startLatch = new CountDownLatch(1);
        final CountDownLatch endLatch = new CountDownLatch(threadCount);
        final AtomicInteger errorCount = new AtomicInteger(0);

        // Set initial values
        testRule.setRuleId("concurrentRule");
        TestResponseProcessor processor = new TestResponseProcessor();
        testRule.setResponseProcessor(processor);

        for (int i = 0; i < threadCount; i++) {

        assertEquals(75L, merged.getDocFreq()); // Sum of frequencies
        assertEquals(1.5f, merged.getUserBoost(), 0.001f); // Takes item2's boost

        // Check merged arrays contain both items' values
        assertTrue(merged.getTags().length >= 2);
        assertTrue(merged.getRoles().length >= 2);
        assertTrue(merged.getLanguages().length >= 2);
        assertEquals(2, merged.getKinds().length);
    }