/** Property name for redirects enabled setting */
    public static final String REDIRECTS_ENABLED = "redirectsEnabled";

    /** Property name for cookies setting */
    public static final String COOKIES_PROPERTY = "cookies";

    /** Property name for authentication scheme providers setting */
    public static final String AUTH_SCHEME_PROVIDERS_PROPERTY = "authSchemeProviders";

crawler.urlFilter.addExclude(".*\\.js$");
crawler.urlFilter.addExclude(".*login.*");
```

## Supported Protocols and Formats

### Protocols
- **HTTP/HTTPS**: Full web crawling support with cookies, authentication, redirects
- **File System**: Local and network file system access
- **FTP**: FTP server crawling with authentication
- **SMB/CIFS**: Windows network shares

     * @param sessionId The session ID.
     */
    @Override
    public void saveSession(final String sessionId) {
        // TODO use cache
    }

    /**
     * Checks if a URL has been visited by looking in both the queue and access results.
     *
     * @param urlQueue The URL queue entry to check.
     * @return true if the URL has been visited, false otherwise.
     */
    @Override