}
        return null;
    }

    /**
     * Adds a directive to the robots.txt rules.
     * The user-agent pattern in the directive is converted to a regular expression pattern,
     * where '*' is replaced with '.*' for pattern matching, and stored case-insensitively.
     *
     * @param directive The directive to add to the robots.txt rules
     */
    public void addDirective(final Directive directive) {

 *   <li>Getting content</li>
 *   <li>Handling redirects</li>
 *   <li>Processing responses</li>
 *   <li>Handling exceptions during crawling and child URL processing</li>
 *   <li>Handling cases where no URL is in the queue</li>
 *   <li>Handling cases where no response processor or rule is found</li>
 *   <li>Handling system errors</li>
 * </ul>
 *
 * <p>
 * Each logging method checks the log level before logging the message.

 * </ul>
 *
 * <p>Components can be registered in two ways:
 * <ul>
 *   <li>As singletons, where one instance is shared throughout the container's lifecycle</li>
 *   <li>As prototypes, where a new instance is created each time the component is requested</li>
 * </ul>
 *
 * <p>The container supports component initialization and destruction through consumer functions,

    /**
     * Output encoding.
     */
    protected String outputEncoding = Constants.UTF_8;

    /**
     * If true, read a content as a text when an extraction fails.
     */
    protected boolean readAsTextIfFailed = false;

    /**
     * Max compression ratio.
     */
    protected long maxCompressionRatio = 100;

    /**

        CrawlerSystemException exception = new CrawlerSystemException(cause);

        assertNotNull(exception);
        assertSame(cause, exception.getCause());
        // When constructed with cause only, message should be cause's toString()
        assertEquals("java.lang.IllegalArgumentException: Root cause", exception.getMessage());
    }

    /**
     * Test constructor with null cause

 * </p>
 *
 * <p>
 * The conversion process involves iterating through the input string, identifying Katakana characters,
 * and replacing them with their corresponding Alphabet representations based on the internal mapping.
 * When a Katakana character has multiple Alphabet representations, the converter generates multiple
 * possible readings.
 * </p>
 *
 * <p>

    /**
     * Cache for session queues.
     */
    protected Map<String, QueueHolder> sessionCache = new ConcurrentHashMap<>();

    /**
     * The number of URLs to fetch when polling.
     */
    protected int pollingFetchSize = 1000;

    /**
     * The maximum size of the crawling queue.
     */
    protected int maxCrawlingQueueSize = 100;

    /**

      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
      (except as stated in this section) patent license to make, have made,
      use, offer to sell, sell, import, and otherwise transfer the Work,
      where such license applies only to those patent claims licensable
      by such Contributor that are necessarily infringed by their
      Contribution(s) alone or by combination of their Contribution(s)

      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
      (except as stated in this section) patent license to make, have made,
      use, offer to sell, sell, import, and otherwise transfer the Work,
      where such license applies only to those patent claims licensable
      by such Contributor that are necessarily infringed by their
      Contribution(s) alone or by combination of their Contribution(s)

        dataService.delete(sessionId);
        urlFilter.clear();
    }

    /**
     * Closes the crawler, releasing resources.
     * This method is called automatically when the crawler is used in a try-with-resources statement.
     */
    @Override
    public void close() {
        clientFactory.close();
    }

    /**
     * Adds an include filter for URLs.