* including adding URLs to the queue, filtering URLs, managing crawler threads,
 * and handling the overall crawling lifecycle.
 *
 * <p>It implements the Runnable interface to be executed in a separate thread,
 * and the AutoCloseable interface to ensure resources are properly released after use.
 *
 * <p>The crawler uses several services and components, such as UrlQueueService, DataService,

      this License, without any additional terms or conditions.
      Notwithstanding the above, nothing herein shall supersede or modify
      the terms of any separate license agreement you may have executed
      with Licensor regarding such Contributions.

   6. Trademarks. This License does not grant permission to use the trade

 * processing the response, and extracting child URLs.
 *
 * <p>
 * This class implements the {@link Runnable} interface, allowing it to be executed in a separate thread.
 * It uses various services and components, such as {@link UrlQueueService}, {@link DataService},
 * {@link CrawlerContainer}, {@link LogHelper}, {@link CrawlerClientFactory}, and {@link CrawlerContext},

     * The date of last modification of the file. This date should be in W3C
     * Datetime format. This format allows you to omit the time portion, if
     * desired, and use YYYY-MM-DD.
     *
     * Note that this tag is separate from the If-Modified-Since (304) header
     * the server can return, and search engines may use the information from
     * both sources differently.
     */
    private String lastmod;

    /**

      this License, without any additional terms or conditions.
      Notwithstanding the above, nothing herein shall supersede or modify
      the terms of any separate license agreement you may have executed
      with Licensor regarding such Contributions.

   6. Trademarks. This License does not grant permission to use the trade

/**
 * PdfExtractor extracts text content from PDF files using Apache PDFBox.
 * It supports password-protected PDFs and can extract embedded documents and annotations.
 *
 * <p>The extractor runs text extraction in a separate thread with a configurable timeout
 * to prevent hanging on problematic PDF files. It also extracts metadata from the PDF
 * document and includes it in the extraction result.
 *
 * <p>Features:
 * <ul>

    <!-- see https://www.loc.gov/preservation/digital/formats/fdd/fdd000488.shtml -->
    <root-XML localName="ONIXMessage" namespaceURI="http://ns.editeur.org/onix/3.0/reference"/>
    <!-- do we want a separate mime for this? -->
    <root-XML localName="ONIXmessage" namespaceURI="http://ns.editeur.org/onix/3.0/short"/>
    <root-XML localName="ONIXMessage"/>
    <sub-class-of type="application/xml"/>
  </mime-type>