import org.codelibs.fess.util.ComponentUtil;

/**
 * Abstract base class for document ingesters that process and transform documents
 * before they are indexed. Ingesters can be used to modify document content,
 * extract additional metadata, or perform other transformations during the
 * indexing process.
 *
 * Ingesters are processed in priority order, with lower numbers having higher priority.
 */
public abstract class Ingester {

{* ../../docs_src/request_form_models/tutorial001_an_py39.py hl[9:11,15] *}

**FastAPI** will **extract** the data for **each field** from the **form data** in the request and give you the Pydantic model you defined.

## Check the Docs { #check-the-docs }

You can verify it in the docs UI at `/docs`:

			// specified in the Forwarded header. The first element will always be
			// the 'for=' capture, which we ignore. In the case of multiple IP
			// addresses (for=8.8.8.8, 8.8.4.4, 172.16.1.20 is valid) we only
			// extract the first, which should be the client IP.
			if match := forRegex.FindStringSubmatch(fwd); len(match) > 1 {
				// IPv6 addresses in Forwarded headers are quoted-strings. We strip
				// these quotes.

Declare the **cookie** parameters that you need in a **Pydantic model**, and then declare the parameter as `Cookie`:

{* ../../docs_src/cookie_param_models/tutorial001_an_py310.py hl[9:12,16] *}

**FastAPI** will **extract** the data for **each field** from the **cookies** received in the request and give you the Pydantic model you defined.

## Check the Docs { #check-the-docs }

You can see the defined cookies in the docs UI at `/docs`:

   * typically useful for {@code return} statements. That leaves the code with two options: Either
   * add the suppression to the whole method (which turns off checking for a large section of code),
   * or extract a variable, and put the suppression on that. However, a local variable typically
   * doesn't work: Because nullness analyses typically infer the nullness of local variables,

   * typically useful for {@code return} statements. That leaves the code with two options: Either
   * add the suppression to the whole method (which turns off checking for a large section of code),
   * or extract a variable, and put the suppression on that. However, a local variable typically
   * doesn't work: Because nullness analyses typically infer the nullness of local variables,

```

### CrawlerThread (`CrawlerThread.java`)

Worker thread for crawling.

**Flow**: Poll URL → Validate → Get client → Delay → Check last-modified → Execute → Process → Extract children → Queue children → Delay

### CrawlerClientFactory

Pattern-based client selection using `LinkedHashMap<Pattern, CrawlerClient>`.

**Standard Patterns**:
```java
"^https?://.*"     → httpClient

     */
    @Override
    public ExtractData getText(final InputStream in, final Map<String, String> params) {
        if (in == null) {
            throw new CrawlerSystemException("JODConverter input stream is null. Cannot extract text from null input.");
        }

        final String resourceName = params == null ? null : params.get(ExtractData.RESOURCE_NAME_KEY);

        String extension;
        String filePrefix;

     * Gets the appropriate extractor for the given response data.
     * Selects an extractor based on the MIME type or falls back to the Tika extractor.
     *
     * @param responseData the response data containing the document to extract
     * @return the extractor instance for processing the document
     * @throws FessSystemException if no suitable extractor can be found
     */
    @Override

        return "META-INF/maven/extension.xml";
    }

    /**
     * Extracts the extension descriptor (if any) from the specified JAR file.
     *
     * @param extensionJar The JAR file or directory to extract the descriptor from, must not be {@code null}.
     * @return The extracted descriptor or {@code null} if no descriptor was found.
     * @throws IOException If the descriptor is present but could not be parsed.
     */