### Helpers

- **RobotsTxtHelper**: RFC 9309 parsing, user-agent matching, crawl-delay, sitemaps
- **SitemapsHelper**: Sitemap XML parsing, index handling
- **MimeTypeHelper**: MIME detection via Tika
- **EncodingHelper**: Charset detection with BOM
- **UrlConvertHelper**: URL normalization
- **ContentLengthHelper**: Content length limits per MIME type

---

## Development Workflow

### Build Commands

<!DOCTYPE components PUBLIC "-//DBFLUTE//DTD LastaDi 1.0//EN"
	"http://dbflute.org/meta/lastadi10.dtd">
<components namespace="fessCrawler">
	<include path="crawler/container.xml" />

	<component name="encodingHelper" class="org.codelibs.fess.crawler.helper.EncodingHelper">
		<postConstruct name="addEncodingMapping">
			<arg>"unicode"</arg>
			<arg>"UTF-16LE"</arg>
		</postConstruct>
	</component>