- **StorageClient**: Cloud storage integration

#### Content Processing Pipeline
- **Extractors**: Content extraction from various formats
- **Transformers**: Data transformation and enrichment
- **Filters**: URL filtering with regex patterns
- **Rules**: Content processing rules and validation

## Building and Testing

### Build Commands

```bash
# Build all modules
mvn clean install

        assertFalse(urlFilter.match("https://other.com/page"));

        // Clear the filter
        urlFilter.clear();

        // After clear, all URLs should match (no filters applied)
        assertTrue(urlFilter.match("https://example.com/page"));
        assertTrue(urlFilter.match("https://other.com/page"));
        assertTrue(urlFilter.match("https://any.com/image.jpg"));
    }

 *   <li>Extracting text from metadata if the main content extraction fails</li>
 *   <li>Reading content as plain text if all other methods fail</li>
 *   <li>Applying post-extraction filters</li>
 *   <li>Handling Tika exceptions, including zip bomb exceptions</li>
 * </ul>
 *
 * <p>
 * The class also supports configuration options such as:
 * </p>
 * <ul>
 *   <li>Output encoding</li>

 *   <li>Cleanup: Deletes the crawled data and clears the URL filter.</li>
 * </ol>
 *
 * <p>The crawler can be configured with various parameters, such as the number of threads,
 * the maximum depth of crawling, and URL filters.
 *
 * <p>Example usage:
 * <pre>
 *   Crawler crawler = new Crawler();
 *   crawler.addUrl("http://example.com/");
 *   crawler.execute();
 *   crawler.close();
 * </pre>
 */