return fessConfig.isCrawlerDocumentDuplicateTermRemoved();
    }

    /**
     * Gets the array of space character codes from configuration.
     *
     * @return array of character codes to be treated as spaces
     */
    protected int[] getSpaceChars() {
        final FessConfig fessConfig = ComponentUtil.getFessConfig();

      // Server sent a 100-continue even though we did not request one. Try again to read the
      // actual response status.
      code == 100 -> true

      // Handle Processing (102) & Early Hints (103) and any new codes without failing
      // 100 and 101 are the exceptions with different meanings
      // But Early Hints not currently exposed
      code in (102 until 200) -> true

      else -> false
    }

- `CurlTest`: Tests for static factory methods and basic HTTP operations
- `CurlRequestTest`: Tests for request builder, headers, parameters, body content, SSL, proxies, timeouts
- `CurlResponseTest`: Tests for response handling, status codes, content retrieval, resource management
- `CurlExceptionTest`: Tests for exception handling and error scenarios

If a translation is missing in `fess_label_[locale].properties`, Fess automatically falls back to the base `fess_label.properties` (English).

### Language Code Format

Use standard locale codes:
- ISO 639-1 language codes: `en`, `ja`, `de`, `fr`, `sv`, etc.
- With region when needed: `pt_BR`, `zh_CN`, `zh_TW`, `en_IE`

### Cache Behavior

- Language items are cached for 1 hour by `SystemHelper`

        int[] codes = (int[]) propMap.get(CRAWLER_FAILURE_URL_STATUS_CODES);
        if (codes == null) {
            codes = split(getCrawlerFailureUrlStatusCodes(), ",")
                    .get(stream -> stream.filter(StringUtil::isNotBlank).mapToInt(Integer::parseInt).toArray());
            propMap.put(CRAWLER_FAILURE_URL_STATUS_CODES, codes);
        }
        for (final int v : codes) {

    // Make sure that hash codes are not computed per-instance.
    assertEqualHashCode(Predicates.not(p1), Predicates.not(p1));

    assertEqualHashCode(Predicates.and(p1, p2), Predicates.and(p1, p2));

    assertEqualHashCode(Predicates.or(p1, p2), Predicates.or(p1, p2));

    // While not a contractual requirement, we'd like the hash codes for ands
    // & ors of the same predicates to not collide.

     * log n) on average.
     *
     * <p>The online hash flooding detecting in RegularSetBuilderImpl.add can detect e.g. many
     * exactly matching hash codes, which would cause construction to take O(n^2), but can't detect
     * e.g. hash codes adversarially designed to go into ascending table locations, which keeps
     * construction O(n) (as desired) but then can have O(n) queries later.
     *

    // Make sure that hash codes are not computed per-instance.
    assertEqualHashCode(Predicates.not(p1), Predicates.not(p1));

    assertEqualHashCode(Predicates.and(p1, p2), Predicates.and(p1, p2));

    assertEqualHashCode(Predicates.or(p1, p2), Predicates.or(p1, p2));

    // While not a contractual requirement, we'd like the hash codes for ands
    // & ors of the same predicates to not collide.

    /**
     * Processes the response data from a crawled URL, including failure handling.
     * This method extends the base response processing to handle Fess-specific failure
     * URL tracking when certain HTTP status codes are encountered.
     *
     * @param urlQueue the URL queue item that was processed
     * @param responseData the response data from the crawl operation
     */
    @Override

    protected int connectionCheckInterval = 5; // sec

    /** Idle connection timeout in milliseconds */
    protected long idleConnectionTimeout = 60 * 1000L; // 1min

    /** Pattern for matching HTTP redirect status codes */
    protected Pattern redirectHttpStatusPattern = Pattern.compile("[3][0-9][0-9]");

    /** Whether to use robots.txt disallow rules */
    protected boolean useRobotsTxtDisallows = true;