Search Options

Display Count
Sort
Preferred Language
Advanced Search

Results 1 - 3 of 3 for robotsTxtEnabled (0.07 seconds)

  2. github.com/codelibs/fess

    src/test/java/org/codelibs/fess/opensearch/config/exentity/WebConfigTest.java 

            assertEquals(0, ((WebAuthenticationConfig[]) initParamMap.get(HcHttpClient.AUTHENTICATIONS_PROPERTY)).length);
        assertTrue(Boolean.valueOf(initParamMap.get("robotsTxtEnabled").toString()).booleanValue());
    }

    @Test
    public void test_initializeClientFactoryWithConfigParameter() {
        final Map<String, String> systemPropMap = new HashMap<>();
    Created: Tue Mar 31 13:07:34 GMT 2026
    - Last Modified: Fri Mar 13 23:01:26 GMT 2026
    - 7.7K bytes
    - Click Count (0)
  3. github.com/codelibs/fess-crawler

    fess-crawler/src/main/java/org/codelibs/fess/crawler/client/http/HcHttpClient.java 

        public static final String USER_AGENT_PROPERTY = "userAgent";

    /** Property name for robots.txt enabled setting */
    public static final String ROBOTS_TXT_ENABLED_PROPERTY = "robotsTxtEnabled";

    /** Property name for web authentications setting */
    public static final String AUTHENTICATIONS_PROPERTY = "webAuthentications";

    /** Property name for request headers setting */
    Created: Sun Apr 12 03:50:13 GMT 2026
    - Last Modified: Thu Jan 08 04:17:06 GMT 2026
    - 4.8K bytes
    - Click Count (0)
  4. github.com/codelibs/fess-crawler

    CLAUDE.md 

        └── Hc5HttpClient
```

Switch via system property: `-Dfess.crawler.http.client=hc4` or `hc5` (default)

**Key Properties**: `connectionTimeout`, `soTimeout`, `proxyHost`, `proxyPort`, `userAgent`, `robotsTxtEnabled`, `ignoreSslCertificate`, `maxTotalConnection`, `defaultMaxConnectionPerRoute`

### CrawlerClientFactory

Pattern-based client selection (from `crawler/client.xml`):
- `http:.*`, `https:.*` → SwitchableHttpClient
    Created: Sun Apr 12 03:50:13 GMT 2026
    - Last Modified: Thu Mar 12 03:39:20 GMT 2026
    - 8.1K bytes
    - Click Count (0)
Back to Top