### URL Filtering

```java
// Include patterns
crawler.urlFilter.addInclude("https://example.com/.*");
crawler.urlFilter.addInclude(".*\\.pdf$");

// Exclude patterns  
crawler.urlFilter.addExclude(".*\\.js$");
crawler.urlFilter.addExclude(".*login.*");
```

## Supported Protocols and Formats

### Protocols
- **HTTP/HTTPS**: Full web crawling support with cookies, authentication, redirects

        urlFilter.addExclude(".*\\.(css|js)$");
        urlFilter.addExclude(".*\\/admin\\/.*");
        urlFilter.addExclude(".*#.*");

        assertTrue(urlFilter.match("https://example.com/page.html"));
        assertFalse(urlFilter.match("https://example.com/style.css"));
        assertFalse(urlFilter.match("https://example.com/script.js"));
        assertFalse(urlFilter.match("https://example.com/admin/login"));

    }

    public void test_getTika_js() {
        final InputStream in = ResourceUtil.getResourceAsStream("extractor/program/test.js");
        final Map<String, String> params = new HashMap<String, String>();
        params.put("Content-Type", "text/plain");
        params.put("resourceName", "test.js");
        final ExtractData extractData = tikaExtractor.getText(in, params);
        final String content = extractData.getContent();