instance="prototype">
		<property name="connectionTimeout">15000</property>
		<property name="soTimeout">30000</property>
	</component>
	<component name="httpClient"
		class="org.codelibs.fess.crawler.client.FaultTolerantClient" instance="prototype">
		<property name="crawlerClient">internalHttpClient</property>
		<property name="maxRetryCount">5</property>
		<property name="retryInterval">500</property>
	</component>

- `DeferredFileOutputStream` for large responses (temp files for >1MB)
- Connection pooling with limits
- Background temp file deletion via `FileUtil.deleteInBackground()`

**Fault Tolerance**:
- `FaultTolerantClient` wrapper (retry, circuit breaker)
- Graceful degradation (e.g., robots.txt parsing continues on errors)

---

## Key Components

### Crawler (`Crawler.java`)

Main orchestrator for crawling operations.

import java.util.Map;

import org.apache.http.client.config.CookieSpecs;
import org.codelibs.core.io.ResourceUtil;
import org.codelibs.fess.crawler.client.CrawlerClientFactory;
import org.codelibs.fess.crawler.client.FaultTolerantClient;
import org.codelibs.fess.crawler.client.fs.FileSystemClient;
import org.codelibs.fess.crawler.client.http.HcHttpClient;
import org.codelibs.fess.crawler.container.StandardCrawlerContainer;