Search Options

Results per page
Sort
Preferred Languages
Advance

Results 1 - 2 of 2 for convertRobotsTxtPathPattern (0.1 sec)

  2. github.com/codelibs/fess-crawler

    fess-crawler/src/test/java/org/codelibs/fess/crawler/client/http/HcHttpClientTest.java 

            assertEquals("/.*\\?.*", httpClient.convertRobotsTxtPathPattern("/*?*"));
        assertEquals("/.*", httpClient.convertRobotsTxtPathPattern("/"));
        assertEquals("/index\\.html$", httpClient.convertRobotsTxtPathPattern("/index.html$"));
        assertEquals(".*index\\.html$", httpClient.convertRobotsTxtPathPattern("index.html$"));
        assertEquals("/\\..*", httpClient.convertRobotsTxtPathPattern("/."));
    Registered: Sun Nov 10 03:50:12 UTC 2024
    - Last Modified: Thu May 09 09:28:25 UTC 2024
    - 8.8K bytes
    - Viewed (0)
  3. github.com/codelibs/fess-crawler

    fess-crawler/src/main/java/org/codelibs/fess/crawler/client/http/HcHttpClient.java 

                                    for (String urlPattern : directive.getDisallows()) {
                                    if (StringUtil.isNotBlank(urlPattern)) {
                                        urlPattern = convertRobotsTxtPathPattern(urlPattern);
                                        final String urlValue = hostUrl + urlPattern;
                                        crawlerContext.getUrlFilter().addExclude(urlValue);
    Registered: Sun Nov 10 03:50:12 UTC 2024
    - Last Modified: Thu May 09 09:29:26 UTC 2024
    - 41K bytes
    - Viewed (0)
Back to top