extractorFactory.addExtractor("text/html", htmlExtractor, 2);  // Weight 2
extractorFactory.addExtractor("text/html", tikaExtractor, 1);  // Fallback
```

### Helpers

**RobotsTxtHelper**: RFC 9309 parsing, user-agent matching, crawl-delay, sitemaps
**SitemapsHelper**: Sitemap XML parsing, index handling
**MimeTypeHelper**: MIME detection via Tika
**EncodingHelper**: Charset detection with BOM
**UrlConvertHelper**: URL normalization

        .method("CONNECT", null)
        .header("Host", route.address.url.toHostHeader(includeDefaultPort = true))
        .header("Proxy-Connection", "Keep-Alive") // For HTTP/1.0 proxies like Squid.
        .header("User-Agent", USER_AGENT)
        .build()

    val fakeAuthChallengeResponse =
      Response
        .Builder()
        .request(proxyConnectRequest)
        .protocol(Protocol.HTTP_1_1)

Using OkHttp is easy. Its request/response API is designed with fluent builders and immutability. It
supports both synchronous blocking calls and async calls with callbacks.

A well behaved user agent
-------------------------

OkHttp follows modern HTTP specifications such as

* HTTP Semantics - [RFC 9110](https://datatracker.ietf.org/doc/html/rfc9110)

labels.regularName=Regular Name
labels.replacement=Replacement
labels.sessionId=Session ID
labels.sortOrder=Sort Order
labels.updatedBy=Updated By
labels.updatedTime=Updated Time
labels.urls=URLs
labels.userAgent=User Agent
labels.username=Username
labels.value=Value
labels.versionNo=Version No
labels.cronExpression=Schedule
labels.dayForCleanup=Delete documents older than specified days
labels.crawlingThreadCount=Concurrent Crawling Count

   * by the application:
   *
   * * It may be transformed by the user's interceptors. For example, an application interceptor
   *   may add headers like `User-Agent`.
   * * It may be the request generated in response to an HTTP redirect or authentication
   *   challenge. In this case the request URL may be different than the initial request URL.
   *

      Header("retry-after", ""),
      Header("server", ""),
      Header("set-cookie", ""),
      Header("strict-transport-security", ""),
      Header("transfer-encoding", ""),
      Header("user-agent", ""),
      Header("vary", ""),
      Header("via", ""),
      Header("www-authenticate", ""),
    )

  val NAME_TO_FIRST_INDEX = nameToFirstIndex()

            if (key.startsWith(CRAWLER_PARAM_PREFIX)) {
                factoryParamMap.put(key.substring(CRAWLER_PARAM_PREFIX.length()), entry.getValue());
            }
        }

        // user agent
        final String userAgent = paramMap.get(CRAWLER_USERAGENT);
        if (StringUtil.isNotBlank(userAgent)) {
            factoryParamMap.put(HcHttpClient.USER_AGENT_PROPERTY, userAgent);
        }

### Other notable changes

* Metadata Agent Improvements ([#66485](https://github.com/kubernetes/kubernetes/pull/66485), [@bmoyles0117](https://github.com/bmoyles0117))
    * Bump metadata agent version to 0.2-0.0.21-1.
    * Expand the metadata agent's access to all API groups.
    * Remove metadata agent config maps in favor of command line flags.

// Excerpts from @lsegal - https://github.com/aws/aws-sdk-js/issues/659#issuecomment-120477258
//
//	User-Agent:
//
//	    This is ignored from signing because signing this causes problems with generating pre-signed URLs
//	    (that are executed by other agents) or when customers pass requests through proxies, which may
//	    modify the user-agent.
//
//	Authorization:
//
//	    Is skipped for obvious reasons

                    appendJson("languages", entity.getLanguages(), buf).append(',');
                    appendJson("roles", entity.getRoles(), buf).append(',');
                    appendJson("user-agent", entity.getUserAgent(), buf).append(',');
                    appendJson("client-ip", entity.getClientIp(), buf).append(',');
                    appendJson("access-type", entity.getAccessType(), buf).append(',');