# Run tests for specific module
mvn test -pl fess-crawler
```

## Examples

### Web Crawling with Custom Rules

```java
// Create crawler with custom configuration
StandardCrawlerContainer container = new StandardCrawlerContainer();

// Configure HTTP client with custom settings
container.singleton("httpClient", HcHttpClient.class, client -> {
    client.setUserAgent("MyBot/1.0");

///

## Custom Headers { #custom-headers }

Keep in mind that custom proprietary headers can be added [using the `X-` prefix](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers).

## Install custom exception handlers { #install-custom-exception-handlers }

You can add custom exception handlers with [the same exception utilities from Starlette](https://www.starlette.dev/exceptions/).

Let's say you have a custom exception `UnicornException` that you (or a library you use) might `raise`.

And you want to handle this exception globally with FastAPI.

You could add a custom exception handler with `@app.exception_handler()`:

        "type": "prometheus",
        "uid": "${DS_PROMETHEUS}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisBorderShow": false,
            "axisCenteredZero": false,
            "axisColorMode": "text",
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,

///

## Returning a custom `Response` { #returning-a-custom-response }

The example above shows all the parts you need, but it's not very useful yet, as you could have just returned the `item` directly, and **FastAPI** would put it in a `JSONResponse` for you, converting it to a `dict`, etc. All that by default.

Now, let's see how you could use that to return a custom response.

            Doc(
                """
                The default response class to be used.

                Read more in the
                [FastAPI docs for Custom Response - HTML, Stream, File, others](https://fastapi.tiangolo.com/advanced/custom-response/#default-response-class).

                **Example**

                ```python
                from fastapi import FastAPI

### Custom Generate Unique ID Function { #custom-generate-unique-id-function }

FastAPI uses a **unique ID** for each *path operation*, which is used for the **operation ID** and also for the names of any needed custom models, for requests or responses.

You can customize that function. It takes an `APIRoute` and outputs a string.

            type[APIRoute],
            Doc(
                """
                Custom route (*path operation*) class to be used by this router.

                Read more about it in the
                [FastAPI docs for Custom Request and APIRoute class](https://fastapi.tiangolo.com/how-to/custom-request-and-route/#custom-apiroute-class-in-a-router).
                """
            ),
        ] = APIRoute,

            }
        };

        assertEquals("Custom protocolType should be WEB", ProtocolType.WEB, customAnnotation.protocolType());
        assertEquals("Custom message should match", "Test message", customAnnotation.message());
        assertEquals("Custom groups should have one element", 1, customAnnotation.groups().length);
        assertEquals("Custom payload should have one element", 1, customAnnotation.payload().length);

    private String id;
    /** The CSS class name to match */
    private String css;
    /** The custom attribute name to match */
    private String attrName;
    /** The custom attribute value to match */
    private String attrValue;

    /**
     * Creates a new PrunedTag instance with the specified tag name.
     *
     * @param tag the HTML tag name to match (e.g., "div", "span", "p")
     */