*/
    public static WebApiManagerFactory getWebApiManagerFactory() {
        return getComponent(WEB_API_MANAGER_FACTORY);
    }

    /**
     * Gets the user agent helper component.
     * @return The user agent helper.
     */
    public static UserAgentHelper getUserAgentHelper() {
        return getComponent(USER_AGENT_HELPER);
    }

    /**
     * Gets the data store factory component.

extractorFactory.addExtractor("text/html", htmlExtractor, 2);  // Weight 2
extractorFactory.addExtractor("text/html", tikaExtractor, 1);  // Fallback
```

### Helpers

**RobotsTxtHelper**: RFC 9309 parsing, user-agent matching, crawl-delay, sitemaps
**SitemapsHelper**: Sitemap XML parsing, index handling
**MimeTypeHelper**: MIME detection via Tika
**EncodingHelper**: Charset detection with BOM
**UrlConvertHelper**: URL normalization

        .method("CONNECT", null)
        .header("Host", route.address.url.toHostHeader(includeDefaultPort = true))
        .header("Proxy-Connection", "Keep-Alive") // For HTTP/1.0 proxies like Squid.
        .header("User-Agent", USER_AGENT)
        .build()

    val fakeAuthChallengeResponse =
      Response
        .Builder()
        .request(proxyConnectRequest)
        .protocol(Protocol.HTTP_1_1)

Using OkHttp is easy. Its request/response API is designed with fluent builders and immutability. It
supports both synchronous blocking calls and async calls with callbacks.

A well behaved user agent
-------------------------

OkHttp follows modern HTTP specifications such as

* HTTP Semantics - [RFC 9110](https://datatracker.ietf.org/doc/html/rfc9110)

labels.regularName=Regular Name
labels.replacement=Replacement
labels.sessionId=Session ID
labels.sortOrder=Sort Order
labels.updatedBy=Updated By
labels.updatedTime=Updated Time
labels.urls=URLs
labels.userAgent=User Agent
labels.username=Username
labels.value=Value
labels.versionNo=Version No
labels.cronExpression=Schedule
labels.dayForCleanup=Delete documents older than specified days
labels.crawlingThreadCount=Concurrent Crawling Count

/// info | `@décorateur` Info

Cette syntaxe `@something` en Python est appelée un "décorateur".

Vous la mettez au dessus d'une fonction. Comme un joli chapeau décoratif (j'imagine que ce terme vient de là 🤷🏻‍♂).

Un "décorateur" prend la fonction en dessous et en fait quelque chose.

Dans notre cas, ce décorateur dit à **FastAPI** que la fonction en dessous correspond au **chemin** `/` avec l'**opération** `get`.

   * by the application:
   *
   * * It may be transformed by the user's interceptors. For example, an application interceptor
   *   may add headers like `User-Agent`.
   * * It may be the request generated in response to an HTTP redirect or authentication
   *   challenge. In this case the request URL may be different than the initial request URL.
   *

      Header("retry-after", ""),
      Header("server", ""),
      Header("set-cookie", ""),
      Header("strict-transport-security", ""),
      Header("transfer-encoding", ""),
      Header("user-agent", ""),
      Header("vary", ""),
      Header("via", ""),
      Header("www-authenticate", ""),
    )

  val NAME_TO_FIRST_INDEX = nameToFirstIndex()

- Kubernetes 1.12 includes a large number of metadata agent improvements, including expanding the metadata agent's access to all API groups and removing metadata agent config maps in favor of command line flags. It also includes improvements to the logging agent, such as multiple fixes and adjustments.

            if (key.startsWith(CRAWLER_PARAM_PREFIX)) {
                factoryParamMap.put(key.substring(CRAWLER_PARAM_PREFIX.length()), entry.getValue());
            }
        }

        // user agent
        final String userAgent = paramMap.get(CRAWLER_USERAGENT);
        if (StringUtil.isNotBlank(userAgent)) {
            factoryParamMap.put(HcHttpClient.USER_AGENT_PROPERTY, userAgent);
        }