*   <li>Starting and finishing threads</li>
 *   <li>Starting and cleaning up crawling</li>
 *   <li>Handling unsupported URLs</li>
 *   <li>Checking last modified dates</li>
 *   <li>Getting content</li>
 *   <li>Handling redirects</li>
 *   <li>Processing responses</li>
 *   <li>Handling exceptions during crawling and child URL processing</li>
 *   <li>Handling cases where no URL is in the queue</li>

     */
    public void setRoutePlanner(final HttpRoutePlanner routePlanner) {
        this.routePlanner = routePlanner;
    }

    /**
     * Sets whether HTTP redirects are enabled.
     *
     * @param redirectsEnabled True to enable redirects, false otherwise
     */
    public void setRedirectsEnabled(final boolean redirectsEnabled) {
        this.redirectsEnabled = redirectsEnabled;
    }

    /**

<tr align="left">
<td align="left" width="15" style="width:15px;padding:0;margin:0;line-height:1px;font-size:1px;"></td>

    }

    /**
     * Gets the redirect location if the response is a redirect.
     *
     * @return the redirect location URL, or null if not a redirect
     */
    public String getRedirectLocation() {
        return redirectLocation;
    }

    /**
     * Sets the redirect location if the response is a redirect.
     *
     * @param redirectLocation the redirect location URL to set
     */

    CHECK_LAST_MODIFIED,
    /** Indicates the resource has not been modified. */
    NOT_MODIFIED,
    /** Indicates getting content from a resource. */
    GET_CONTENT,
    /** Indicates a redirect location was found. */
    REDIRECT_LOCATION,
    /** Indicates processing a response. */
    PROCESS_RESPONSE,
    /** Indicates the crawling process has finished. */
    FINISHED_CRAWLING,

     */
    public void setMaxOutputLine(final int maxOutputLine) {
        this.maxOutputLine = maxOutputLine;
    }

    /**
     * Sets whether to redirect standard output.
     * @param standardOutput true to redirect standard output, false otherwise.
     */
    public void setStandardOutput(final boolean standardOutput) {
        this.standardOutput = standardOutput;
    }

            }
            if (link.startsWith("/")) {
                redirect = ftpInfo.toUrl(file.getLink());
            } else if (link.startsWith("../")) {
                redirect = ftpInfo.toChildUrl(file.getLink());
            } else {
                redirect = ftpInfo.toChildUrl("../" + file.getLink());
            }
            if (!uri.equals(redirect)) {
                responseData.setHttpStatusCode(Constants.OK_STATUS);

crawler.urlFilter.addExclude(".*\\.js$");
crawler.urlFilter.addExclude(".*login.*");
```

## Supported Protocols and Formats

### Protocols
- **HTTP/HTTPS**: Full web crawling support with cookies, authentication, redirects
- **File System**: Local and network file system access
- **FTP**: FTP server crawling with authentication
- **SMB/CIFS**: Windows network shares
- **Storage**: Cloud storage systems (MinIO, S3-compatible)

    /**
     * Constructs a new HtmlTransformer.
     */
    public HtmlTransformer() {
        // Default constructor
    }

    /** Header name for location redirects. */
    protected static final String LOCATION_HEADER = "Location";

    /** The crawler container for dependency injection. */
    @Resource
    protected CrawlerContainer crawlerContainer;

                            } else {
                                log(logHelper, LogType.REDIRECT_LOCATION, crawlerContext, urlQueue, responseData);
                                // redirect
                                storeChildUrl(responseData.getRedirectLocation(), urlQueue.getUrl(), urlQueue.getWeight(),
                                        urlQueue.getDepth() == null ? 1 : urlQueue.getDepth() + 1);