* It extends the RuntimeException class and provides constructors for creating exceptions with a message, a cause, or both.
 * It also provides a protected constructor that allows to specify whether or not suppression is enabled or stack trace is writable.
 */
public class CrawlerSystemException extends RuntimeException {

 * Implementations of this interface should provide the logic for reading
 * documents and converting them into a key-value structure.
 *
 * <p>This interface extends {@link java.io.Closeable}, so implementations
 * should also handle resource cleanup when the {@link #close()} method is called.</p>
 */
public interface DocumentReader extends Closeable {
    /**
     * Reads a document and returns its contents as a map.
     *

 * It is used to interrupt a running thread when a timeout occurs.
 * The class provides methods to interrupt the thread and stop the timeout target.
 *
 * <p>
 * The class uses a logger to log debug messages.
 * It also uses an AtomicBoolean to track whether the thread is running.
 * </p>
 *
 * <p>
 * The expired method is called when a timeout occurs.
 * It interrupts the running thread up to a maximum number of times.

 * These RequestData objects are added to a set of child URLs, which are then
 * passed to a {@link ChildUrlsException} to be processed by the crawler.
 * </p>
 *
 * <p>
 * The class also handles potential {@link IOException}s that may occur during
 * the parsing of the response body.
 * </p>
 */
public class SitemapsResponseProcessor implements ResponseProcessor {

import org.codelibs.core.lang.StringUtil;

/**
 * FtpAuthentication class holds the authentication information for FTP connections.
 * It includes server address, port number, username, and password.
 * It also provides a method to check if the authentication matches a given FTP path.
 *
 * @author shinsuke
 */
public class FtpAuthentication {
    private static final Logger logger = LogManager.getLogger(FtpAuthentication.class);

 * It also provides a method to retrieve the extracted data as a String from an AccessResultData object.
 *
 * <p>
 * The class handles character encoding issues by attempting to use the specified charset.
 * If the specified charset is invalid, it falls back to UTF-8.
 * </p>
 *
 * <p>

 * and registration with the ExtractorFactory.
 *
 * <p>
 * This class handles the retrieval of essential crawler components like
 * {@link MimeTypeHelper} and {@link ExtractorFactory} from the
 * {@link CrawlerContainer}. It also provides a convenient method for
 * registering the extractor with the {@link ExtractorFactory}.
 * </p>
 *
 * <p>
 * Subclasses should implement the actual extraction logic in their own

 *
 * <p>The class is thread-safe due to the use of {@code synchronized} keyword on methods that
 * modify the internal data structures. The internal maps are also declared as {@code volatile}
 * to ensure visibility of changes across threads.
 *
 * <p>The class also provides methods to manage URL patterns for inclusion and exclusion, which
 * are stored as {@code Pattern} objects.
 */
public class MemoryDataHelper {

 * It uses XPath expressions to extract text content from HTML documents.
 * <p>
 * This class provides methods to configure the XPath expressions, parser features, and properties.
 * It also includes caching mechanism for XPathAPI instances to improve performance.
 * </p>
 * <p>
 * The extracted text is obtained from the nodes selected by the {@code targetNodePath} XPath expression.

 * and storing the result. It also handles "Not Modified" responses by creating an empty
 * result and storing it. Unsuccessful responses are logged for debugging purposes.
 * </p>
 *
 * <p>
 * The class uses {@link CrawlerContainer} to access components like {@link AccessResult}
 * and {@link UrlQueue}. It also uses {@link CrawlingParameterUtil} to access services