```

3. **Add test with sample file** in `src/test/resources/`

### Configuring URL Filtering

```java
// Include patterns (must match)
crawler.urlFilter.addInclude("https://example.com/.*");

// Exclude patterns (must not match)
crawler.urlFilter.addExclude(".*\\.(css|js|png|jpg)$");
```

### Setting Crawl Limits

```java

        try {
            new ContentCache((byte[]) null);
            fail("Expected IllegalArgumentException");
        } catch (IllegalArgumentException e) {
            assertTrue(e.getMessage().contains("data must not be null"));
        }
    }

    @Test
    public void testMemoryBasedCacheDefensiveCopy() throws IOException {
        byte[] data = "Hello, World!".getBytes();
        byte[] originalData = data.clone();

     * Maximum length is 10000 characters.
     */
    @Size(max = 10000)
    public String parameterName;

    /**
     * The expiration date and time for the access token.
     * Must be in ISO 8601 format: YYYY-MM-DDTHH:MM:SS
     */
    @Pattern(regexp = "[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]T[0-9][0-9]:[0-9][0-9]:[0-9][0-9]")
    public String expires;

    /**

 * be invoked directly; please see {@link MultisetTestSuiteBuilder}.
 *
 * @author Chris Povirk
 */
@GwtCompatible
@Ignore("test runners must not instantiate and run this directly, only via suites we build")
// @Ignore affects the Android test runner, which respects JUnit 4 annotations on JUnit 3 tests.
@SuppressWarnings("JUnit4ClassUsedInJUnit3")

 * directly; please see {@link com.google.common.collect.testing.ListTestSuiteBuilder}.
 *
 * @author Chris Povirk
 */
@GwtCompatible
@Ignore("test runners must not instantiate and run this directly, only via suites we build")
// @Ignore affects the Android test runner, which respects JUnit 4 annotations on JUnit 3 tests.
@SuppressWarnings("JUnit4ClassUsedInJUnit3")

     * Creates a new SitemapUrl instance.
     */
    public SitemapUrl() {
        super();
    }

    /**
     * URL of the page. This URL must begin with the protocol (such as http) and
     * end with a trailing slash, if your web server requires it. This value
     * must be less than 2,048 characters.
     */
    private String loc;

    /**
     * The date of last modification of the file. This date should be in W3C

import org.jspecify.annotations.Nullable;

/**
 * Basic implementation of {@code Multiset<E>} backed by an instance of {@code Map<E, Count>}.
 *
 * <p>For serialization to work, the subclass must specify explicit {@code readObject} and {@code
 * writeObject} methods.
 *
 * @author Kevin Bourrillion
 */
@GwtCompatible
abstract class AbstractMapBasedMultiset<E extends @Nullable Object> extends AbstractMultiset<E>

import java.util.Iterator;
import org.junit.Ignore;

/**
 * Tester for {@code BiMap.remove}.
 *
 * @author Louis Wasserman
 */
@GwtCompatible
@Ignore("test runners must not instantiate and run this directly, only via suites we build")
// @Ignore affects the Android test runner, which respects JUnit 4 annotations on JUnit 3 tests.
@SuppressWarnings("JUnit4ClassUsedInJUnit3")

import org.junit.Ignore;

/**
 * Common parent class for {@link ListIndexOfTester} and {@link ListLastIndexOfTester}.
 *
 * @author Chris Povirk
 */
@GwtCompatible
@Ignore("test runners must not instantiate and run this directly, only via suites we build")
// @Ignore affects the Android test runner, which respects JUnit 4 annotations on JUnit 3 tests.
@SuppressWarnings("JUnit4ClassUsedInJUnit3")

 * directly; please see {@link com.google.common.collect.testing.CollectionTestSuiteBuilder}.
 *
 * @author Kevin Bourrillion
 * @author Chris Povirk
 */
@GwtCompatible
@Ignore("test runners must not instantiate and run this directly, only via suites we build")
// @Ignore affects the Android test runner, which respects JUnit 4 annotations on JUnit 3 tests.
@SuppressWarnings("JUnit4ClassUsedInJUnit3")