- **DataService**: Access result persistence, iteration
- Implementations: `UrlQueueServiceImpl`, `DataServiceImpl` (in-memory), `OpenSearchDataService` (persistent)

### Processing Pipeline

```
CrawlerThread → Client → ResponseProcessor → Transformer → Extractor → ExtractData
                                                                            ↓

        if (logger.isDebugEnabled()) {
            logger.debug("Total documents added: {}", documentSize.get());
        }

    }

    /**
     * Processes a document through the ingest pipeline.
     * Applies all available ingesters to transform the document data.
     *
     * @param paramMap the data store parameters
     * @param dataMap the document data to process

- **FileSystemClient**: File system access
- **FtpClient**: FTP protocol support
- **SmbClient**: SMB/CIFS network shares
- **StorageClient**: Cloud storage integration

#### Content Processing Pipeline
- **Extractors**: Content extraction from various formats
- **Transformers**: Data transformation and enrichment
- **Filters**: URL filtering with regex patterns
- **Rules**: Content processing rules and validation

        } catch (final IOException e) {
            return StringUtil.EMPTY; // empty
        }
    }

    /**
     * Processes a crawling request for a specific URL.
     * Executes the full crawling pipeline including client execution, rule processing,
     * transformation, and data extraction.
     *
     * @param crawlingConfig the crawling configuration to use
     * @param crawlingInfoId the crawling session ID

doing that.

### REST Endpoint Conventions

Elasticsearch typically uses singular nouns rather than plurals in URLs.
For example:

    /_ingest/pipeline
    /_ingest/pipeline/{id}

but not:

    /_ingest/pipelines
    /_ingest/pipelines/{id}

You may find counterexamples, but new endpoints should use the singular
form.

### Java Language Formatting Guidelines

            logger.warn("Failed to get data: url={}", accessResult.getUrl(), e);
        }
        return null;
    }

    /**
     * Processes a document through the ingest pipeline if an ingest factory is available.
     * Applies all configured ingesters to transform and enrich the document data.
     *
     * @param accessResult the access result containing document metadata

 * ClosingFuture#finishToFuture()} or {@link
 * ClosingFuture#finishToValueAndCloser(ValueAndCloserConsumer, Executor)} paths to complete a
 * {@link ClosingFuture} pipeline.
 */
@NullUnmarked
@GwtIncompatible
@J2ktIncompatible
public abstract class AbstractClosingFutureTest extends TestCase {
  // TODO(dpb): Use Expect once that supports JUnit 3, or we can use JUnit 4.

 * ClosingFuture#finishToFuture()} or {@link
 * ClosingFuture#finishToValueAndCloser(ValueAndCloserConsumer, Executor)} paths to complete a
 * {@link ClosingFuture} pipeline.
 */
@NullUnmarked
@GwtIncompatible
@J2ktIncompatible
public abstract class AbstractClosingFutureTest extends TestCase {
  // TODO(dpb): Use Expect once that supports JUnit 3, or we can use JUnit 4.

import org.elasticsearch.search.aggregations.MultiBucketConsumerService;
import org.elasticsearch.search.aggregations.bucket.terms.StringTerms;
import org.elasticsearch.search.aggregations.pipeline.PipelineAggregator;
import org.elasticsearch.search.builder.SearchSourceBuilder;
import org.elasticsearch.search.query.QuerySearchResult;
import org.openjdk.jmh.annotations.Benchmark;

### Major Features and Improvements

* `tf.data`
    * Adds `autotune.min_parallelism` to `tf.data.Options` to enable faster input pipeline warm up.
* `tf.lite`