* It is used to indicate exceptions related to MIME type handling during the crawling process.
 * This exception can be thrown with a message, a cause, or both.
 */
public class MimeTypeException extends CrawlerSystemException {

    private static final long serialVersionUID = 1L;

    /**
     * Creates a new MimeTypeException with the specified detail message and cause.

/**
 * This class represents a custom exception for the suggester component.
 * It extends the {@link RuntimeException} class and provides constructors
 * to create an exception instance with a message, a cause, or both.
 *
 * <p>Usage examples:</p>
 * <pre>
 * throw new SuggesterException("An error occurred");
 * throw new SuggesterException(new IOException("IO error"));

/**
 * The class CrawlerSystemException is a runtime exception that is thrown when a system error occurs during crawling.
 * It extends the RuntimeException class and provides constructors for creating exceptions with a message, a cause, or both.
 * It also provides a protected constructor that allows to specify whether or not suppression is enabled or stack trace is writable.
 */
public class CrawlerSystemException extends RuntimeException {

/**
 * Exception thrown when there is an issue with the suggestion index.
 * This exception extends {@link SuggesterException}.
 *
 * <p>This exception can be thrown with a message, a cause, or both.</p>
 *
 * <pre>
 * Example usage:
 * throw new SuggestIndexException("Indexing error occurred");
 * throw new SuggestIndexException(new IOException("IO error"));

 *
 * <p>
 * This exception can be thrown when there are problems accessing URLs, files, or any other resources needed for crawling.
 * It includes constructors to handle messages, causes, or both.
 * </p>
 *
 * <p>
 * The log level can be set to DEBUG, INFO, WARN, or ERROR, and the class provides methods to check if a specific log level is enabled.
 * </p>
 *
 * <p>
 * Example usage:
 * </p>
 *

 * Example usage:
 * </p>
 *
 * <pre>
 * {@code
 * RegexRule rule = new RegexRule();
 * rule.addRule("url", "https://example.com/.*");
 * rule.addRule("contentType", "text/html");
 * rule.setAllRequired(true); // Both URL and content type must match
 *
 * ResponseData responseData = new ResponseData();
 * responseData.setUrl("https://example.com/page1");
 * responseData.setContentType("text/html");
 *

 * </ul>
 *
 * <p>
 * The class uses synchronization to ensure thread safety when accessing and modifying the URL queue.
 * It also checks for duplicate URLs before adding them to the queue,
 * both within the queue itself and against previously accessed URLs stored in the {@link MemoryDataHelper}.
 * </p>
 *
 */
/**
 * This class is an implementation of {@link UrlQueueService}.
 */

     * desired, and use YYYY-MM-DD.
     *
     * Note that this tag is separate from the If-Modified-Since (304) header
     * the server can return, and search engines may use the information from
     * both sources differently.
     */
    private String lastmod;

    /**
     * How frequently the page is likely to change. This value provides general