*/
    public void setUserAgent(final String userAgent) {
        this.userAgent = userAgent;
    }

    /**
     * Sets the credentials provider.
     * @param credentialsProvider The credentials provider.
     */
    public void setCredentialsProvider(final CredentialsProvider credentialsProvider) {
        this.credentialsProvider = credentialsProvider;
    }

    /**

      origin of the Work and reproducing the content of the NOTICE file.

   7. Disclaimer of Warranty. Unless required by applicable law or
      agreed to in writing, Licensor provides the Work (and each
      Contributor provides its Contributions) on an "AS IS" BASIS,
      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
      implied, including, without limitation, any warranties or conditions

import org.apache.logging.log4j.Logger;
import org.codelibs.core.lang.StringUtil;

/**
 * Utility class for text normalization and processing.
 *
 * This class provides methods to normalize text by reading characters from a provided Reader
 * and processing them according to specific rules. The main functionality is encapsulated
 * within the nested {@link TextNormalizeContext} class.
 *

 * using the Apache Tika library. It extends {@link PasswordBasedExtractor} to handle password-protected files.
 * </p>
 *
 * <p>
 * This class provides methods to extract text from an input stream, handling different scenarios such as:
 * </p>
 * <ul>
 *   <li>Normalizing text content</li>
 *   <li>Handling resource names and content types</li>

 * in a crawler application. This container supports both singleton and prototype component
 * instantiation patterns.
 *
 * <p>The container provides mechanisms for:
 * <ul>
 *   <li>Registering and retrieving components by name</li>
 *   <li>Managing singleton instances with lifecycle hooks</li>
 *   <li>Creating prototype instances on demand</li>

/**
 * {@link HtmlXpathExtractor} is an implementation of the {@link org.codelibs.fess.crawler.extractor.Extractor} interface.
 * It uses XPath expressions to extract text content from HTML documents.
 * <p>
 * This class provides methods to configure the XPath expressions, parser features, and properties.
 * It also includes caching mechanism for XPathAPI instances to improve performance.
 * </p>
 * <p>

import org.codelibs.core.stream.StreamUtil;
import org.codelibs.fess.crawler.Constants;

/**
 * The FormScheme class implements the AuthScheme interface to provide
 * form-based authentication for HTTP clients. It handles the process of
 * obtaining a token and logging in using the provided credentials.
 *
 * <p>This class supports both GET and POST methods for token and login
 * requests. It also allows for the replacement of placeholders in URLs and

 * It uses XPath expressions to extract data from the XML and stores it in a ResultData object.
 * </p>
 *
 * <p>
 * This class provides several configuration options to customize the XML parsing process, such as:
 * </p>
 * <ul>
 *   <li>Namespace awareness</li>
 *   <li>Coalescing</li>
 *   <li>Entity expansion</li>

import org.xml.sax.helpers.DefaultHandler;

/**
 * Helper class for parsing and validating sitemaps.
 * It supports XML sitemaps, XML sitemap indexes, and text sitemaps,
 * and can handle GZIP compressed sitemaps.
 * The class provides methods to check if an input stream is a valid sitemap,
 * and to parse an input stream into a {@link SitemapSet} object.
 * It uses SAX parser for XML sitemaps and XML sitemap indexes,

import org.codelibs.fess.crawler.helper.LogHelper;
import org.codelibs.fess.crawler.log.LogType;

/**
 * Implementation of the {@link LogHelper} interface.
 * This class provides methods for logging various events during the crawling process.
 * It uses Log4j2 for logging.
 *
 * <p>
 * The class contains methods for logging different types of events, such as:
 * </p>
 * <ul>