- Connection pooling with limits
- Background temp file deletion via `FileUtil.deleteInBackground()`

**Fault Tolerance**:
- `FaultTolerantClient` wrapper (retry, circuit breaker)
- Graceful degradation (e.g., robots.txt parsing continues on errors)

---

## Key Components

### Crawler (`Crawler.java`)

Main orchestrator for crawling operations.

**Key Methods**:
```java

suggest.update.contents.limit.num=10000
# Maximum document size for suggestion update.
suggest.update.contents.limit.doc.size=50000
# Scroll size for suggestion source reader.
suggest.source.reader.scroll.size=1
# Cache size for popular word suggestion.
suggest.popular.word.cache.size=1000
# Cache expiration (seconds) for popular word suggestion.
suggest.popular.word.cache.expire=60

     * comment: Tie breaker value for dismax queries.
     * @return The value of found property. (NotNull: if not found, exception but basically no way)
     */
    String getQueryDismaxTieBreaker();

    /**
     * Get the value for the key 'query.dismax.tie_breaker' as {@link java.math.BigDecimal}. <br>
     * The value is, e.g. 0.1 <br>
     * comment: Tie breaker value for dismax queries.

        if (readers.hasNext()) {
            final ImageReader reader = readers.next();
            try {
                reader.setInput(input);
                final ImageReadParam param = reader.getDefaultReadParam();
                final int width = reader.getWidth(0);
                final int height = reader.getHeight(0);
                if (width <= 0 || height <= 0) {

            throw new SuggestIndexException("Failed to index from query_string.", e);
        }
    }

    /**
     * Indexes documents from a query log reader asynchronously.
     * @param queryLogReader The query log reader.
     * @param docPerReq The number of documents to process per request.
     * @param requestInterval The interval between requests.

        String[] headers = null;

        try (BufferedReader reader = new BufferedReader(new InputStreamReader(in, charset))) {
            String line;
            Character detectedDelimiter = delimiter;

            // Read header row if needed
            if (hasHeader) {
                while ((line = reader.readLine()) != null) {
                    if (StringUtil.isBlank(line)) {

 * String indexName = "your_index_name";
 *
 * ESSourceReader reader = new ESSourceReader(client, settings, indexName);
 * reader.setScrollSize(1000); // Set the scroll size
 * reader.setLimitOfDocumentSize(1024 * 1024); // Limit document size to 1MB
 * reader.setQuery(QueryBuilders.termQuery("field", "value")); // Set a query
 *
 * Map<String, Object> document;
 * while ((document = reader.read()) != null) {
 *     // Process the document

    }

    /**
     * Imports elevate words from a CSV file.
     * Expected CSV format: SuggestWord, Reading, Permissions, Labels, Boost
     *
     * @param reader the Reader containing CSV data to import
     */
    public void importCsv(final Reader reader) {
        final PermissionHelper permissionHelper = ComponentUtil.getPermissionHelper();
        final CsvConfig cfg = new CsvConfig(',', '"', '"');

            break;
        case LABEL:
            processLabelRequest(request, response, chain);
            break;
        case POPULARWORD:
            processPopularWordRequest(request, response, chain);
            break;
        case FAVORITE:
            processFavoriteRequest(request, response, chain);
            break;
        case FAVORITES:

        addDocument(indexName, client, num);

        ESSourceReader reader = new ESSourceReader(client, settings, indexName);
        reader.setScrollSize(1);
        reader.setLimitDocNumPercentage("1%");
        int count = 0;
        Set<String> valueSet = Collections.synchronizedSet(new HashSet<>());
        Map<String, Object> source;
        while ((source = reader.read()) != null) {