}

    /**
     * Returns the include filtering pattern.
     * @return The include filtering pattern.
     */
    public String getIncludeFilteringPattern() {
        return includeFilteringPattern;
    }

    /**
     * Sets the include filtering pattern.
     * @param includeFilteringPattern The include filtering pattern.
     */

- **Extensible Architecture**: Plugin system for custom extractors, transformers, and clients
- **Rate Limiting**: Politeness policies and interval controllers
- **URL Filtering**: Regex-based inclusion/exclusion patterns
- **Data Persistence**: Multiple backend options including OpenSearch integration

## Technology Stack

- **Java**: 21+ (requires Java 21 or higher)

import org.codelibs.fess.crawler.service.UrlFilterService;

import jakarta.annotation.Resource;

/**
 * Implementation of the {@link UrlFilterService} interface.
 * This class provides methods for managing URL filtering rules,
 * including adding include and exclude URL patterns, deleting patterns,
 * and retrieving lists of compiled URL patterns. It utilizes a
 * {@link MemoryDataHelper} to store and manage the URL patterns in memory.
 *

    /** Map of session IDs to include URL patterns for filtering URLs. */
    protected volatile Map<String, List<Pattern>> includeUrlPatternMap = new HashMap<>();

    /** Map of session IDs to exclude URL patterns for filtering URLs. */
    protected volatile Map<String, List<Pattern>> excludeUrlPatternMap = new HashMap<>();

    /**

 *
 * <p>Key functionalities include:</p>
 * <ul>
 *   <li>Setting the target index for the search.</li>
 *   <li>Limiting the number of results (size).</li>
 *   <li>Filtering by tags, roles, fields, and languages.</li>
 *   <li>Excluding specific words from the results.</li>
 *   <li>Building the OpenSearch query and rescorer for the popular words search.</li>

 * </p>
 *
 * <p>
 * The reader supports limiting the number of documents read based on a percentage of the total documents
 * or a fixed number. It also allows filtering documents based on their size, using the {@code limitOfDocumentSize}
 * parameter.
 * </p>
 *
 * <p>
 * The reader uses a queue to buffer documents read from Elasticsearch, and it retries failed requests

import jakarta.annotation.Resource;

/**
 * The Crawler class is the main class for web crawling. It manages the crawling process,
 * including adding URLs to the queue, filtering URLs, managing crawler threads,
 * and handling the overall crawling lifecycle.
 *
 * <p>It implements the Runnable interface to be executed in a separate thread,