# Based on RFC 9309 specification

# Test wildcard patterns
User-agent: WildcardBot
Disallow: /*.pdf$
Disallow: /admin/*.php
Disallow: /*/private/
Allow: /public/*.html

# Test end-of-path ($) patterns
User-agent: EndPathBot
Disallow: /fish$
Disallow: /temp$
Allow: /fishing

# Test complex patterns
User-agent: ComplexBot
Disallow: /
Allow: /$
Allow: /index.html$
Allow: /public/

/**
 * A factory class for managing and creating crawler clients based on URL patterns.
 * This class implements AutoCloseable to properly handle resource cleanup.
 *
 * <p>The factory maintains a map of regular expression patterns to crawler clients,
 * allowing for URL-based client selection. Clients can be added with specific patterns
 * and optionally at specific positions in the processing order.</p>
 *

        private final int priorityLength;

        /**
         * Constructs a new PathPattern from the given robots.txt path pattern.
         * @param pattern the path pattern string from robots.txt (may contain * and $)
         */
        public PathPattern(final String pattern) {
            this.pattern = pattern;

    }

    /**
     * Gets the list of include URL patterns for the specified session.
     *
     * @param sessionId The session ID.
     * @return The list of compiled include patterns.
     * @throws CrawlerSystemException if the patterns cannot be loaded.
     */
    @Override
    public List<Pattern> getIncludeUrlPatternList(final String sessionId) {
        try {

                .getTotalHits()
                .value() > 0);

        // Verify pattern can be retrieved
        final List<Pattern> patterns = urlFilterService.getIncludeUrlPatternList(sessionId);
        assertEquals(1, patterns.size());
        assertTrue(patterns.get(0).matcher("http://example.com/page1").matches());
        assertFalse(patterns.get(0).matcher("http://other.com/page1").matches());

    }

    /**
     * Converts a GSA URL pattern into a regular expression pattern suitable for Fess.
     * Handles various GSA pattern formats including regexp, contains, and URL-based patterns.
     *
     * @param s the input GSA pattern string
     * @return a regular expression pattern string, or empty string for comments/invalid patterns
     */
    protected String getFilterPath(final String s) {

        Pattern[] patterns = (Pattern[]) propMap.get(CRAWLER_METADATA_CONTENT_EXCLUDES);
        if (patterns == null) {
            patterns = split(getCrawlerMetadataContentExcludes(), ",")
                    .get(stream -> stream.filter(StringUtil::isNotBlank).map(Pattern::compile).toArray(n -> new Pattern[n]));
            propMap.put(CRAWLER_METADATA_CONTENT_EXCLUDES, patterns);
        }

     * <p>
     * Patterns can be exact class names or use wildcards with '*' at the end.
     * For example: "com.example.*" allows all classes in the com.example package.
     * </p>
     *
     * @param allowedPatterns the patterns of classes to allow
     * @return an ObjectInputFilter configured with the specified patterns
     */

   - Convert URL patterns to Fess regex patterns
   - Set up LabelType (access control labels) if defined

**Step 3: Verify Imported Configurations**

After import, verify:
- **Crawler > Web**: Check web crawling configurations
- **Crawler > File**: Check file crawling configurations (SMB, FTP)
- **System > General**: Verify URL patterns and filters

**GSA Configuration Mapping**

**Flow**: Poll URL → Validate → Get client → Delay → Check last-modified → Execute → Process → Extract children → Queue children → Delay

### CrawlerClientFactory

Pattern-based client selection using `LinkedHashMap<Pattern, CrawlerClient>`.

**Standard Patterns**:
```java
"^https?://.*"     → httpClient
"^file:.*"         → fileSystemClient
"^ftp://.*"        → ftpClient
"^smb://.*"        → smbClient