---

## Architecture

### Module Structure

```
fess-crawler-parent/
├── fess-crawler/              # Core framework
├── fess-crawler-lasta/        # LastaFlute DI integration
└── fess-crawler-opensearch/   # OpenSearch backend
```

### Key Design Patterns

**Factory**: `CrawlerClientFactory`, `ExtractorFactory` - protocol/format-specific component selection

mvn test

# Run single unit test
mvn test -Dtest=SearchEngineClientTest

# Run integration tests (*Tests.java in src/test/java/org/codelibs/fess/it/)
# Requires Fess server and OpenSearch running
mvn test -P integrationTests -Dtest.fess.url="http://localhost:8080" -Dtest.search_engine.url="http://localhost:9201"

# Run single integration test

    $ mvn jdeb:jdeb # .deb package

### Generate Source Code

    $ mvn dbflute:download # (one time command)
    $ mvn dbflute:freegen
    $ mvn license:format

### Integration Tests

Integration tests require a running Fess server with OpenSearch. Follow these steps:

#### 1. Build Fess

    $ mvn antrun:run  # Download OpenSearch plugins (if not done)
    $ mvn package     # Build the package

import org.opensearch.action.bulk.BulkRequestBuilder;
import org.opensearch.action.bulk.BulkResponse;
import org.opensearch.action.get.GetResponse;
import org.opensearch.action.index.IndexAction;
import org.opensearch.action.index.IndexRequest;
import org.opensearch.action.index.IndexRequestBuilder;
import org.opensearch.index.query.QueryBuilder;
import org.opensearch.transport.client.Client;

/**

import org.junit.AfterClass;
import org.junit.Before;
import org.junit.BeforeClass;
import org.junit.Test;
import org.opensearch.action.bulk.BulkRequestBuilder;
import org.opensearch.action.index.IndexAction;
import org.opensearch.action.index.IndexRequestBuilder;
import org.opensearch.index.IndexNotFoundException;
import org.opensearch.search.sort.SortBuilders;
import org.opensearch.search.sort.SortOrder;

import org.apache.pdfbox.pdmodel.common.filespecification.PDEmbeddedFile;
import org.apache.pdfbox.pdmodel.common.filespecification.PDFileSpecification;
import org.apache.pdfbox.pdmodel.interactive.annotation.PDAnnotation;
import org.apache.pdfbox.pdmodel.interactive.annotation.PDAnnotationFileAttachment;
import org.apache.pdfbox.text.PDFTextStripper;
import org.codelibs.core.lang.ThreadUtil;
import org.codelibs.fess.crawler.entity.ExtractData;

    - name: Build with Maven
      run: mvn -B source:jar javadoc:jar package --file pom.xml
    - name: Run Fess
      run: bash src/test/resources/before_script.sh
    - name: Run Integration Test

- **Crawl Sources**: URLs, file shares, databases being indexed
- **Custom Configurations**: URL patterns, security rules, metadata fields
- **Search Features**: Facets, suggestions, security filtering, ranking rules
- **Integration Points**: Applications consuming search APIs
- **User Permissions**: Access control and label configurations

### 2. Infrastructure Requirements

Ensure your Fess environment meets these requirements:

 * modification checking, and integration with the Fess search engine backend.
 *
 * <p>Key features include:</p>
 * <ul>
 * <li>Incremental crawling support with last-modified timestamp checking</li>
 * <li>Document expiration handling</li>
 * <li>Child URL extraction and queueing</li>
 * <li>Integration with Fess configuration and permission systems</li>

     * Controls the level of detail in system log messages.
     */
    @Size(max = 10)
    public String logLevel;

    /**
     * Storage service endpoint URL for cloud storage integration.
     * Used for storing files in cloud storage services like S3.
     */
    @Size(max = 1000)
    public String storageEndpoint;

    /**
     * Access key for cloud storage authentication.