* second invocation, so implementors have made various choices, including:
 *
 * <ul>
 *   <li>returning the same iterator again
 *   <li>throwing an exception of some kind
 *   <li>or the usual, <i>robust</i> behavior, which all known {@link Collection} implementations
 *       have, of returning a new, independent iterator
 * </ul>
 *

 * <ol>
 *   <li>Initialization: Sets up the crawler context and initializes the URL filter.</li>
 *   <li>Thread Creation: Creates a group of crawler threads to perform the actual crawling.</li>
 *   <li>Execution: Starts the crawler threads and waits for them to complete.</li>
 *   <li>Cleanup: Deletes the crawled data and clears the URL filter.</li>
 * </ol>
 *

 *   <li>Pass two lambda expressions or method references to the {@link #from from} factory method.
 *   <li>Extend this class and implement its {@link #doForward} and {@link #doBackward} methods.
 * </ul>
 *
 * <p>Using a converter:
 *
 * <ul>
 *   <li>Convert one instance in the "forward" direction using {@code converter.convert(a)}.

 * actions taken by the java garbage collection system:
 *
 * <ul>
 *   <li>invoking the {@code finalize} methods of unreachable objects
 *   <li>clearing weak references to unreachable referents
 *   <li>enqueuing weak references to unreachable referents in their reference queue
 * </ul>
 *

   *
   * <ol>
   *   <li>calls {@link ExecutorService#shutdown()}, disabling acceptance of new submitted tasks.
   *   <li>awaits executor service termination for half of the specified timeout.
   *   <li>if the timeout expires, it calls {@link ExecutorService#shutdownNow()}, cancelling
   *       pending tasks and interrupting running tasks.

     * <li>The parent uses aggregation and refers to the child via the modules section</li>
     * <li>The module path to the child is considered to
     * point at the POM rather than its base directory if the path ends with ".xml" (ignoring case)</li>
     * <li>The name of the child's base directory matches the artifact id of the child.</li>
     * </ul>

     * <ul>
     * <li>{@code *} (since 2.0.5)= everything,</li>
     * <li>{@code external:*}  (since 2.0.9)= everything not on the localhost and not file based,</li>
     * <li>{@code external:http:*} (since 3.8.0)= any repository not on the localhost using HTTP,</li>
     * <li>{@code repo,repo1}  (since 2.0.9)= {@code repo} or {@code repo1},</li>

   *
   * <ol>
   *   <li>If {@code index >= end}, {@link IndexOutOfBoundsException} is thrown.
   *   <li><b>If the character at the specified index is not a surrogate, it is returned.</b>
   *   <li>If the first character was a high surrogate value, then an attempt is made to read the
   *       next character.
   *       <ol>

 *
 * <p>Key features:</p>
 * <ul>
 *   <li>Supports multiple user-agent directives with pattern matching</li>
 *   <li>Handles Allow and Disallow rules for path-based access control</li>
 *   <li>Manages crawl delay settings per user agent</li>
 *   <li>Stores sitemap URLs listed in robots.txt</li>
 * </ul>
 *
 * <p>The class uses case-insensitive pattern matching for user agents and supports

 * Benchmarks (as of December 2011) show that:
 *
 * <ul>
 *   <li>for an unnested {@code lock()} and {@code unlock()}, a cycle detecting lock takes 38ns as
 *       opposed to the 24ns taken by a plain lock.
 *   <li>for nested locking, the cost increases with the depth of the nesting:
 *       <ul>
 *         <li>2 levels: average of 64ns per lock()/unlock()
 *         <li>3 levels: average of 77ns per lock()/unlock()