import org.apache.lucene.util.BytesRef;
import org.apache.lucene.util.CharsRef;
import org.apache.lucene.util.UnicodeUtil;
import org.apache.lucene.util.fst.FST;

// https://issues.apache.org/jira/browse/LUCENE-5252
public final class NGramSynonymTokenizer extends Tokenizer {

    public static final int DEFAULT_N_SIZE = 2;

    public static final String DEFAULT_DELIMITERS = " 　\t\n\r";

## 다른 검색 제공자에서의 마이그레이션

[MIGRATION.md](MIGRATION.md)를 참조하십시오.

## 데이터 저장소

현재 Fess는 다음 [저장소 위치 및 API](https://fess.codelibs.org/15.3/admin/dataconfig-guide.html)를 크롤링할 수 있습니다:

 - [Confluence/Jira](https://github.com/codelibs/fess-ds-atlassian)
 - [Box](https://github.com/codelibs/fess-ds-box)
 - [CSV](https://github.com/codelibs/fess-ds-csv)
 - [Database](https://github.com/codelibs/fess-ds-db)

## Data Store

Actuellement, Fess prend en charge le parcours des [emplacements de stockage et API](https://fess.codelibs.org/15.3/admin/dataconfig-guide.html) suivants :

 - [Confluence/Jira](https://github.com/codelibs/fess-ds-atlassian)
 - [Box](https://github.com/codelibs/fess-ds-box)
 - [CSV](https://github.com/codelibs/fess-ds-csv)
 - [Base de données](https://github.com/codelibs/fess-ds-db)

Consulte [MIGRATION.md](MIGRATION.md).

## Data Store

Atualmente, o Fess suporta o rastreamento dos seguintes [locais de armazenamento e APIs](https://fess.codelibs.org/15.3/admin/dataconfig-guide.html):

 - [Confluence/Jira](https://github.com/codelibs/fess-ds-atlassian)
 - [Box](https://github.com/codelibs/fess-ds-box)
 - [CSV](https://github.com/codelibs/fess-ds-csv)
 - [Database](https://github.com/codelibs/fess-ds-db)

## Almacenamiento de Datos

Actualmente, Fess admite el rastreo de las siguientes [ubicaciones de almacenamiento y APIs](https://fess.codelibs.org/15.3/admin/dataconfig-guide.html):

 - [Confluence/Jira](https://github.com/codelibs/fess-ds-atlassian)
 - [Box](https://github.com/codelibs/fess-ds-box)
 - [CSV](https://github.com/codelibs/fess-ds-csv)
 - [Database](https://github.com/codelibs/fess-ds-db)

3. **Fallback**: English (from `fess_label.properties` and `fess_message.properties`)

### Document Language Detection

During crawling and indexing, Fess:

1. Detects language from document content using Apache Tika
2. Validates against `supported.languages` list
3. Creates language-specific fields (e.g., `content_ja`, `title_en`, `content_sv`)
4. Applies language-specific analyzers for better search results

## Technology Stack

- **Java**: 21+ (requires Java 21 or higher)
- **Build System**: Maven 3.x
- **DI Container**: LastaFlute DI
- **HTTP Client**: Apache HttpComponents
- **Content Extraction**: Apache Tika, Apache POI, PDFBox
- **Testing**: JUnit 4, UTFlute, Testcontainers
- **Storage Backends**: OpenSearch, Memory-based

## Quick Start

### Prerequisites

- Java 21 or higher
- Maven 3.6 or higher

## 他の検索プロバイダーからの移行

[MIGRATION.md](MIGRATION.md)を参照してください。

## データストア

現在、Fessは以下の[ストレージロケーションとAPI](https://fess.codelibs.org/15.3/admin/dataconfig-guide.html)のクロールをサポートしています：

 - [Confluence/Jira](https://github.com/codelibs/fess-ds-atlassian)
 - [Box](https://github.com/codelibs/fess-ds-box)
 - [CSV](https://github.com/codelibs/fess-ds-csv)
 - [Database](https://github.com/codelibs/fess-ds-db)

        } else {
            cmdList.add("-Dfess.log.level=" + logLevel);
            if ("debug".equalsIgnoreCase(logLevel)) {
                cmdList.add("-Dorg.apache.tika.service.error.warn=true");
            }
        }
        stream(fessConfig.getJvmCrawlerOptionsAsArray())
                .of(stream -> stream.filter(StringUtil::isNotBlank).forEach(value -> cmdList.add(value)));

        final String content = extractData.getContent();
        CloseableUtil.closeQuietly(in);
        logger.info(content);
        assertTrue(content.contains("テスト"));
    }

    // TODO tika needs to support pdfbox 2.0
    //    public void test_getTika_pdf() {
    //        final InputStream in = ResourceUtil
    //                .getResourceAsStream("extractor/test.pdf");