Search Options

Display Count
Sort
Preferred Language
Advanced Search

Results 1 - 5 of 5 for robotsTxtEnabled (0.09 seconds)

  1. src/test/java/org/codelibs/fess/opensearch/config/exentity/WebConfigTest.java

            assertEquals(0, ((WebAuthenticationConfig[]) initParamMap.get(HcHttpClient.AUTHENTICATIONS_PROPERTY)).length);
            assertTrue(Boolean.valueOf(initParamMap.get("robotsTxtEnabled").toString()).booleanValue());
        }
    
        @Test
        public void test_initializeClientFactoryWithConfigParameter() {
            final Map<String, String> systemPropMap = new HashMap<>();
    Created: Tue Mar 31 13:07:34 GMT 2026
    - Last Modified: Fri Mar 13 23:01:26 GMT 2026
    - 7.7K bytes
    - Click Count (0)
  2. fess-crawler/src/main/java/org/codelibs/fess/crawler/client/http/HcHttpClient.java

        public static final String USER_AGENT_PROPERTY = "userAgent";
    
        /** Property name for robots.txt enabled setting */
        public static final String ROBOTS_TXT_ENABLED_PROPERTY = "robotsTxtEnabled";
    
        /** Property name for web authentications setting */
        public static final String AUTHENTICATIONS_PROPERTY = "webAuthentications";
    
        /** Property name for request headers setting */
    Created: Sun Apr 12 03:50:13 GMT 2026
    - Last Modified: Thu Jan 08 04:17:06 GMT 2026
    - 4.8K bytes
    - Click Count (0)
  3. CLAUDE.md

        └── Hc5HttpClient
    ```
    
    Switch via system property: `-Dfess.crawler.http.client=hc4` or `hc5` (default)
    
    **Key Properties**: `connectionTimeout`, `soTimeout`, `proxyHost`, `proxyPort`, `userAgent`, `robotsTxtEnabled`, `ignoreSslCertificate`, `maxTotalConnection`, `defaultMaxConnectionPerRoute`
    
    ### CrawlerClientFactory
    
    Pattern-based client selection (from `crawler/client.xml`):
    - `http:.*`, `https:.*` → SwitchableHttpClient
    Created: Sun Apr 12 03:50:13 GMT 2026
    - Last Modified: Thu Mar 12 03:39:20 GMT 2026
    - 8.1K bytes
    - Click Count (0)
  4. fess-crawler/src/main/java/org/codelibs/fess/crawler/client/http/Hc4HttpClient.java

            }
    
            super.init();
    
            // robots.txt parser
            final Boolean robotsTxtEnabled = getInitParameter(ROBOTS_TXT_ENABLED_PROPERTY, Boolean.TRUE, Boolean.class);
            if (robotsTxtHelper != null) {
                robotsTxtHelper.setEnabled(robotsTxtEnabled);
            }
    
            // httpclient
            final RequestConfig.Builder requestConfigBuilder = RequestConfig.custom();
    Created: Sun Apr 12 03:50:13 GMT 2026
    - Last Modified: Fri Jan 09 23:46:52 GMT 2026
    - 54.4K bytes
    - Click Count (0)
  5. fess-crawler/src/main/java/org/codelibs/fess/crawler/client/http/Hc5HttpClient.java

            }
    
            super.init();
    
            // robots.txt parser
            final Boolean robotsTxtEnabled = getInitParameter(ROBOTS_TXT_ENABLED_PROPERTY, Boolean.TRUE, Boolean.class);
            if (robotsTxtHelper != null) {
                robotsTxtHelper.setEnabled(robotsTxtEnabled);
            }
    
            // httpclient
            final RequestConfig.Builder requestConfigBuilder = RequestConfig.custom();
    Created: Sun Apr 12 03:50:13 GMT 2026
    - Last Modified: Sat Jan 31 12:23:29 GMT 2026
    - 62.2K bytes
    - Click Count (0)
Back to Top