* <ul>
 *   <li>{@link #expired()}: Handles the expiration of connections by closing expired and idle connections.</li>
 * </ul>
 *
 * <p>Logging:</p>
 * <ul>
 *   <li>Logs a warning if the clientConnectionManager is null.</li>
 *   <li>Logs a warning if an exception occurs during connection monitoring.</li>
 * </ul>
 *
 */
public class HcConnectionMonitorTarget implements TimeoutTarget {

     */
    public static final String DEBUG = "DEBUG";

    /**
     * Log level constant for info messages.
     */
    public static final String INFO = "INFO";

    /**
     * Log level constant for warning messages.
     */
    public static final String WARN = "WARN";

    /**
     * Log level constant for error messages.
     */
    public static final String ERROR = "ERROR";

 * This class provides a default way to manage delays between crawler operations.
 * It allows setting delays before processing, after processing, when no URLs are in the queue,
 * and when waiting for new URLs.
 * The delays are configurable via constructor parameters.
 *
 */
public class DefaultIntervalController extends AbstractIntervalController {

    /** Delay in milliseconds after processing a URL */

 *
 * @author bowez
 * @author shinsuke
 *
 */
public class RobotsTxtHelper {

    /** Pattern for parsing user-agent records. */
    protected static final Pattern USER_AGENT_RECORD =
            Pattern.compile("^user-agent:\\s*([^\\t\\n\\x0B\\f\\r]+)\\s*$", Pattern.CASE_INSENSITIVE);

    /** Pattern for parsing disallow records. */

 *   <li>{@code POST_PROCESSING} - Represents the post-processing state.</li>
 *   <li>{@code NO_URL_IN_QUEUE} - Indicates that there are no URLs in the queue.</li>
 *   <li>{@code WAIT_NEW_URL} - Indicates that the crawler is waiting for new URLs.</li>
 * </ul>
 */
public interface IntervalController {
    /** Constant representing the pre-processing state. */
    int PRE_PROCESSING = 1;

    /** Constant representing the post-processing state. */

 * It uses SAX parser for XML sitemaps and XML sitemap indexes,
 * and handles potential exceptions during parsing.
 * The class also includes inner classes for handling XML sitemap and sitemap index parsing.
 */
public class SitemapsHelper {
    private static final Logger logger = LogManager.getLogger(SitemapsHelper.class);

    /** The size of the preload buffer for checking file format. */

## Technology Stack

- **Java**: 21+ (configured via parent POM)
- **OpenSearch**: Latest (provided scope)
- **Apache Lucene**: Query parsing and text analysis
- **ICU4J**: Unicode text processing and normalization
- **JUnit 4**: Testing framework
- **Maven**: Build and dependency management

## Quick Start

### Prerequisites

- Java 21 or higher

                responseData.addMetaData(SMB_CREATE_TIME, new Date(file.createTime()));
                try {
                    if (logger.isDebugEnabled()) {
                        logger.debug("Parsing SmbFile Owner: {}", filePath);
                    }
                    final SID ownerUser = file.getOwnerUser();
                    if (ownerUser != null) {

 *   <li>After processing a URL ({@link #delayAfterProcessing()})</li>
 *   <li>When there are no URLs in the queue ({@link #delayAtNoUrlInQueue()})</li>
 *   <li>While waiting for new URLs to be added to the queue ({@link #delayForWaitingNewUrl()})</li>
 * </ul>
 *
 * <p>
 * Subclasses are responsible for implementing the abstract methods to define the actual delay

     */
    public void setExecutionTimeout(final long executionTimeout) {
        this.executionTimeout = executionTimeout;
    }

    /**
     * Sets the working directory for the command.
     * @param workingDirectory The working directory.
     */
    public void setWorkingDirectory(final File workingDirectory) {
        this.workingDirectory = workingDirectory;
    }

    /**