*   <li>While waiting for new URLs to be added to the queue ({@link #delayForWaitingNewUrl()})</li>
 * </ul>
 *
 * <p>
 * Subclasses are responsible for implementing the abstract methods to define the actual delay
 * mechanism for each of these stages.
 * </p>
 *
 * <p>
 * The class also provides a mechanism to ignore exceptions that may occur during the delay process.

 * governing permissions and limitations under the License.
 */
package org.codelibs.fess.crawler.interval;

/**
 * The {@code IntervalController} interface defines methods for controlling
 * the delay intervals in a web crawler. It includes constants representing
 * different types of processing states and a method to introduce a delay
 * based on the type of processing.
 * <p>

 * either express or implied. See the License for the specific language
 * governing permissions and limitations under the License.
 */
package org.codelibs.fess.crawler.log;

/**
 * LogType defines the different types of log messages that can be generated during the crawling process.
 * Each enum constant represents a specific event or state in the crawler's execution.
 */
public enum LogType {

 * represents a valid sitemap. It uses a SitemapsHelper to validate the response body as an InputStream.
 * The rule checks if the URL matches the defined regex pattern and then validates the content as a sitemap.
 * If any exception occurs during the sitemap validation, it logs the error and returns false.
 *
 */
public class SitemapsRule extends RegexRule {
    /**

 * and provides getter and setter methods for common properties such as
 * {@code ruleId} and {@code responseProcessor}.
 * </p>
 *
 * <p>
 * Subclasses should extend this class and implement the abstract methods
 * defined in the {@link Rule} interface to provide specific rule logic.
 * </p>
 *
 */
public abstract class AbstractRule implements Rule {

    private static final long serialVersionUID = 1L;

    /** The rule ID. */

 * The class supports caching of include and exclude patterns for scenarios where a session ID is not available.
 * It also provides methods to initialize the filter with a session ID, clear the filter,
 * match a URL against the defined patterns, and process a URL to add include or exclude patterns based on predefined filtering patterns.
 *
 */
/**
 * This class is an implementation of a URL filter.
 */
public class UrlFilterImpl implements UrlFilter {

     * @param settings the suggest settings to apply
     * @param index the name of the index from which documents will be deleted
     * @param queryBuilder the query that defines which documents to delete
     * @return a result object containing information about the delete operation
     */

 */
package org.codelibs.fess.crawler.entity;

import org.codelibs.core.lang.StringUtil;

/**
 * Represents a URL entry within a sitemap.
 *
 * <p>
 * This class encapsulates the properties of a URL as defined in the sitemap XML format,
 * including its location, last modification date, change frequency, and priority.
 * It implements the {@link Sitemap} interface.
 * </p>
 *
 * <p>

/**
 * Abstract base class for CrawlerClient implementations.
 * Provides common functionality for handling initialization parameters,
 * content length checks, and default method implementations.
 * It defines the basic structure and configuration options for crawler clients.
 */
public abstract class AbstractCrawlerClient implements CrawlerClient {

import java.util.List;
import java.util.Map;
import java.util.regex.Pattern;

import org.codelibs.core.lang.StringUtil;

/**
 * Represents a robots.txt file parser and handler.
 * This class manages the rules defined in a robots.txt file, including user agent directives,
 * allowed/disallowed paths, crawl delays, and sitemap URLs.
 *
 * <p>The robots.txt protocol is implemented according to the standard specification,