A "middleware" is a function that works with every **request** before it is processed by any specific *path operation*. And also with every **response** before returning it.

* It takes each **request** that comes to your application.
* It can then do something to that **request** or run any needed code.
* Then it passes the **request** to be processed by the rest of the application (by some *path operation*).

process-ram.drawio.svg...

```

### Monitoring

```java
// Monitor crawling progress
while (crawler.crawlerContext.getStatus() == CrawlerStatus.RUNNING) {
    int processed = dataService.getCount(sessionId);
    System.out.println("Processed: " + processed + " URLs");
    Thread.sleep(5000);
}
```

## Contributing

We welcome contributions to Fess Crawler! Please follow these guidelines:

import org.codelibs.fess.crawler.processor.ResponseProcessor;

/**
 * The Rule interface defines the contract for implementing rules that can be applied to
 * response data in a web crawler. Implementations of this interface should provide logic
 * to determine if a given response data matches the rule, retrieve the rule's identifier,
 * and obtain the associated response processor.
 */
public interface Rule extends Serializable {

    /**
     * Artifact type name for a JAR file that can be placed either on the annotation processor class path
     * or module path. The path (classes or modules) is chosen by the plugin, possibly using heuristic rules.
     */
    String PROCESSOR = "processor";

    /**
     * Artifact type name for a JAR file to unconditionally place on the annotation processor class path.
     * If the JAR is modular, its module information are ignored.
     */

     * @return the parsed invoker request. Caller must start by checking {@link InvokerRequest#parsingFailed()} as
     * if there are parser errors, this request may not be fully processed and should immediately be failed.
     */
    @Nonnull
    InvokerRequest parseInvocation(@Nonnull ParserRequest parserRequest);

    /**
     * Current status of the crawler.
     */
    protected volatile CrawlerStatus status = CrawlerStatus.INITIALIZING;

    /**
     * Filter for URLs to control which URLs are processed.
     */
    protected UrlFilter urlFilter;

    /**
     * Manager for crawling rules and configurations.
     */
    protected RuleManager ruleManager;

    /**

		<property name="ruleId">"sitemapsRule"</property>
		<property name="responseProcessor">
			<component
				class="org.codelibs.fess.crawler.processor.impl.SitemapsResponseProcessor">
			</component>
		</property>
		<postConstruct name="addRule">
			<arg>"url"</arg>
			<arg>".*sitemap.*"</arg>
		</postConstruct>
	</component>

	processIOReadBytesMD              = NewCounterMD(processIOReadBytes, "Total bytes read by the process from the underlying storage system, /proc/[pid]/io read_bytes")

    /**
     * The path identified by the Java {@code --processor-path} option.
     * The Java tools location is {@link StandardLocation#ANNOTATION_PROCESSOR_PATH}.
     */
    PROCESSOR_CLASSES(StandardLocation.ANNOTATION_PROCESSOR_PATH, "--processor-path"),

    /**
     * The path identified by the Java {@code --processor-module-path} option.