urlEncodeDisabled.set(true);
                }
            }));

            // failure url
            final List<String> excludedUrlList = ComponentUtil.getCrawlingConfigHelper().getExcludedUrlList(webConfig.getConfigId());
            if (excludedUrlList != null) {
                excludedUrlList.stream().filter(StringUtil::isNotBlank).map(String::trim).distinct().forEach(u -> {

     *
     * @param configId the configuration ID to get excluded URLs for
     * @return a list of URLs that should be excluded from crawling, or an empty list if none
     */
    public List<String> getExcludedUrlList(final String configId) {
        final FessConfig fessConfig = ComponentUtil.getFessConfig();
        final int failureCount = fessConfig.getFailureCountThreshold();