}
        return null;
    }

    /**
     * Adds a directive to the robots.txt rules.
     * The user-agent pattern in the directive is converted to a regular expression pattern,
     * where '*' is replaced with '.*' for pattern matching, and stored case-insensitively.
     *
     * @param directive The directive to add to the robots.txt rules
     */
    public void addDirective(final Directive directive) {

 * and when waiting for new URLs.
 * The delays are configurable via constructor parameters.
 *
 */
public class DefaultIntervalController extends AbstractIntervalController {

    /** Delay in milliseconds after processing a URL */
    protected long delayMillisAfterProcessing = 0L;

    /** Delay in milliseconds when no URL is in the queue */

    /**
     * The name of the component to be retrieved from the CrawlerContainer.
     */
    protected String componentName;

    /**
     * The listener that is called when a pooled object is destroyed.
     */
    protected OnDestroyListener<T> onDestroyListener;

    /**
     * Creates a new object instance from the crawler container.

 *   It is similar to a Promise in JavaScript.
 *   It allows you to register callbacks that will be executed when the computation is complete,
 *   either successfully (resolve) or with an error (reject).
 * </p>
 *
 * <p>
 *   The Deferred class has a Promise inner class that allows you to register callbacks
 *   to be executed when the computation is complete.
 *   The Promise class has then and error methods that allow you to register callbacks

 *   <li>Getting content</li>
 *   <li>Handling redirects</li>
 *   <li>Processing responses</li>
 *   <li>Handling exceptions during crawling and child URL processing</li>
 *   <li>Handling cases where no URL is in the queue</li>
 *   <li>Handling cases where no response processor or rule is found</li>
 *   <li>Handling system errors</li>
 * </ul>
 *
 * <p>
 * Each logging method checks the log level before logging the message.

 * </ul>
 *
 * <p>Components can be registered in two ways:
 * <ul>
 *   <li>As singletons, where one instance is shared throughout the container's lifecycle</li>
 *   <li>As prototypes, where a new instance is created each time the component is requested</li>
 * </ul>
 *
 * <p>The container supports component initialization and destruction through consumer functions,

        this.listener = listener;
    }

    /**
     * Interface for listening to request lifecycle events.
     */
    public interface RequestListener {

        /**
         * Called when a request starts.
         *
         * @param client the fault-tolerant client
         * @param request the request data
         */
        void onRequestStart(FaultTolerantClient client, RequestData request);

 * either express or implied. See the License for the specific language
 * governing permissions and limitations under the License.
 */
package org.codelibs.fess.crawler.exception;

/**
 * This exception is thrown when an error occurs during OpenSearch access.
 *
 * @author shinsuke
 *
 */
public class OpenSearchAccessException extends CrawlerSystemException {

    private static final long serialVersionUID = 1L;

    /**

 * governing permissions and limitations under the License.
 */
package org.codelibs.fess.crawler.exception;

/**
 * MaxLengthExceededException is thrown when the content length exceeds the maximum allowed length.
 * It extends CrawlingAccessException, indicating a crawling-related access issue.
 *
 */
public class MaxLengthExceededException extends CrawlingAccessException {

 * governing permissions and limitations under the License.
 */
package org.codelibs.fess.crawler.exception;

/**
 * UnsupportedExtractException is thrown when the content extraction is not supported.
 * It extends ExtractException and indicates that the requested extraction operation cannot be performed.
 *
 */
public class UnsupportedExtractException extends ExtractException {