}

        @Override
        public String getName() {
            return name;
        }
    }

    /**
     * Transformer that throws exceptions for testing error handling
     */
    public static class ExceptionThrowingTransformer implements Transformer {
        private final String name;
        private boolean throwInTransform = false;
        private boolean throwInGetData = false;

 *   <li>Starting and cleaning up crawling</li>
 *   <li>Handling unsupported URLs</li>
 *   <li>Checking last modified dates</li>
 *   <li>Getting content</li>
 *   <li>Handling redirects</li>
 *   <li>Processing responses</li>
 *   <li>Handling exceptions during crawling and child URL processing</li>
 *   <li>Handling cases where no URL is in the queue</li>
 *   <li>Handling cases where no response processor or rule is found</li>

 * </p>
 *
 * <p>
 * This class provides methods to extract text from an input stream, handling different scenarios such as:
 * </p>
 * <ul>
 *   <li>Normalizing text content</li>
 *   <li>Handling resource names and content types</li>
 *   <li>Retrying extraction without resource name or content type if the initial attempt fails</li>

│   ├── entity/
│   │   ├── SuggestItem.java        # Core suggestion data structure
│   │   └── ElevateWord.java        # Promoted words configuration
│   ├── request/                    # Request/Response handling
│   │   ├── suggest/                # Suggestion requests
│   │   └── popularwords/           # Popular words requests
│   ├── index/                      # Indexing functionality

    protected String charsetName = Constants.UTF_8;

    /**
     * A directory to store downloaded files.
     */
    protected File baseDir;

    /**
     * Creates a file with the specified path, handling directory creation and duplicate names.
     *
     * @param path the file path to create
     * @return the created file
     * @throws CrawlerSystemException if directory creation fails
     */

 * Katakana string into a list of possible Alphabet readings. It uses a predefined mapping of Katakana
 * characters to their Alphabet equivalents, handling both single and double Katakana character combinations.
 * </p>
 *
 * <p>
 * The conversion process involves iterating through the input string, identifying Katakana characters,

 * and to parse an input stream into a {@link SitemapSet} object.
 * It uses SAX parser for XML sitemaps and XML sitemap indexes,
 * and handles potential exceptions during parsing.
 * The class also includes inner classes for handling XML sitemap and sitemap index parsing.
 */
public class SitemapsHelper {
    private static final Logger logger = LogManager.getLogger(SitemapsHelper.class);

- **Comprehensive Content Extraction**: Office documents, PDFs, archives, images, audio/video files
- **Multi-Threading**: Configurable thread pools for high-performance crawling
- **Fault Tolerance**: Built-in retry mechanisms and error handling
- **Flexible Configuration**: XML-based dependency injection with LastaFlute DI
- **Extensible Architecture**: Plugin system for custom extractors, transformers, and clients

 * and cache file size to optimize the extraction process.
 *
 * <p>
 * The main purpose of this class is to simplify the extraction process by providing a fluent interface
 * for configuring the extraction parameters and handling the underlying complexities of content processing,
 * such as MIME type detection, extractor selection, and content length validation.
 * </p>
 *
 * <p>
 * Example usage:
 * </p>
 *
 * <pre>
 * {@code

        assertFalse(urlFilter.match("https://example.com/document.pdf"));
        assertTrue(urlFilter.match("https://example.com/document.PDF"));
    }

    /**
     * Test very long URL handling
     */
    public void test_veryLongUrl() {
        String sessionId = "test-session-020";
        urlFilter.init(sessionId);

        // Create a very long URL