*   <li>Normalize the filename to handle special characters.</li>
 *   <li>Configure whether to use the filename for MIME type detection.</li>
 *   <li>Configure whether to use the filename for MIME type detection when the stream is octet-stream.</li>
 * </ul>
 *
 * <p>
 * The MIME type detection is based on the {@code tika-mimetypes.xml} resource, which is loaded during initialization.
 * </p>
 *
 * <p>
 * Usage:
 * </p>

                return l;
            }
        }
        return null;
    }

    /**
     * Sets the language detector.
     *
     * @param detector The language detector.
     */
    public void setDetector(final LanguageDetector detector) {
        this.detector = detector;
    }

    /**
     * Creates a script for updating a document with language information.
     *

    }

    /**
     * Returns the preload size for charset detection.
     * @return The preload size for charset detection.
     */
    public int getPreloadSizeForCharset() {
        return preloadSizeForCharset;
    }

    /**
     * Sets the preload size for charset detection.
     * @param preloadSizeForCharset The preload size for charset detection to set.
     */

```

### Helpers

**RobotsTxtHelper**: RFC 9309 parsing, user-agent matching, crawl-delay, sitemaps
**SitemapsHelper**: Sitemap XML parsing, index handling
**MimeTypeHelper**: MIME detection via Tika
**EncodingHelper**: Charset detection with BOM
**UrlConvertHelper**: URL normalization

---

## Development Workflow

### Build Commands

```bash
mvn clean install              # Build all

      matrix:
        # Override automatic language detection by changing the below list
        # Supported options are ['csharp', 'cpp', 'go', 'java', 'javascript', 'python']
        language: ['java']
        # Learn more...
        # https://docs.github.com/en/github/finding-security-vulnerabilities-and-errors-in-your-code/configuring-code-scanning#overriding-automatic-language-detection

    steps:
    - name: Checkout repository

      matrix:
        # Override automatic language detection by changing the below list
        # Supported options are ['csharp', 'cpp', 'go', 'java', 'javascript', 'python']
        language: ['java', 'javascript']
        # Learn more...
        # https://docs.github.com/en/github/finding-security-vulnerabilities-and-errors-in-your-code/configuring-code-scanning#overriding-automatic-language-detection

    steps:
    - name: Checkout repository

    private ModelVersionUtils() {
        // Utility class
    }

    /**
     * Detects the model version from a POM document.
     * Uses both the modelVersion element and namespace URI for detection.
     *
     * @param pomDocument the POM document (domtrip Document)
     * @return the detected model version
     */
    public static String detectModelVersion(Document pomDocument) {

 * This extractor provides better structured data extraction compared to Tika's generic text extraction.
 *
 * <p>Features:
 * <ul>
 *   <li>Automatic delimiter detection (comma, tab, semicolon, pipe)</li>
 *   <li>Header row detection and extraction</li>
 *   <li>Column name to data value association</li>
 *   <li>Quoted field handling</li>
 *   <li>Column names as metadata</li>

            }
            failureConditions {
                // We have test-retry to handle the crash in tests
                javaCrash = false
                // We want to see the flaky tests for flakiness detection
                supportTestRetry = (performanceTestBuildSpec.type != PerformanceTestType.FLAKINESS_DETECTION)
            }
            if (testProjects.isNotEmpty()) {
                steps {

    /**
     * Checks if automatic UTF-8 encoding detection is enabled.
     *
     * @return True if auto-detection is enabled, false otherwise
     */
    public boolean isAutodetectEncoding() {
        return autodetectEncoding;
    }

    /**
     * Sets whether automatic UTF-8 encoding detection is enabled.
     *
     * @param autodetectEncoding True to enable auto-detection, false to disable
     */