* the most specific (longest) match is used.</p>
 *
 */
public class RobotsTxt {
    private static final String ALL_BOTS = "*";

    /** Map of user agent patterns to their corresponding directives. */
    protected final Map<Pattern, Directive> directiveMap = new LinkedHashMap<>();

    /** List of sitemap URLs found in the robots.txt file. */

        this.cookieSpecRegistry = cookieSpecRegistry;
    }

    /**
     * Sets the cookie date patterns for parsing.
     *
     * @param cookieDatePatterns The cookie date patterns
     */
    public void setCookieDatePatterns(final String[] cookieDatePatterns) {
        this.cookieDatePatterns = cookieDatePatterns;
    }

    /**

        ConditionalAbstractRule conditionalRule = new ConditionalAbstractRule();
        conditionalRule.crawlerContainer = container;
        conditionalRule.setRuleId("conditionalRule");

        // Set patterns
        conditionalRule.setUrlPattern("https?://.*\\.example\\.com/.*");
        conditionalRule.setMimeTypePattern("text/.*");

        // Test matching
        ResponseData responseData1 = new ResponseData();

/**
 * A container implementation that manages the lifecycle and dependency injection of components
 * in a crawler application. This container supports both singleton and prototype component
 * instantiation patterns.
 *
 * <p>The container provides mechanisms for:
 * <ul>
 *   <li>Registering and retrieving components by name</li>
 *   <li>Managing singleton instances with lifecycle hooks</li>

  sources like Apache Nutch, Apache HTTP Server, the file(1) command, etc.

  Notes:
   * Tika supports a wider range of match types than Freedesktop does
   * Glob patterns must be unique, if there's a clash assign to the most
     popular format
   * The main mime type should be the canonical one, use aliases for any
     other widely used forms