Search Options

Results per page
Sort
Preferred Languages
Advance

Results 1 - 2 of 2 for convertRobotsTxtPatternToRegex (0.56 sec)

  2. github.com/codelibs/fess-crawler

    fess-crawler/src/test/java/org/codelibs/fess/crawler/client/http/HcHttpClientTest.java 

            assertEquals("/.*\\?.*", httpClient.convertRobotsTxtPatternToRegex("/*?*"));
        assertEquals("/.*", httpClient.convertRobotsTxtPatternToRegex("/"));
        assertEquals("/index\\.html$", httpClient.convertRobotsTxtPatternToRegex("/index.html$"));
        assertEquals(".*index\\.html$", httpClient.convertRobotsTxtPatternToRegex("index.html$"));
    Registered: Sun Sep 21 03:50:09 UTC 2025
    - Last Modified: Sat Sep 06 04:15:37 UTC 2025
    - 11.7K bytes
    - Viewed (0)
  3. github.com/codelibs/fess-crawler

    fess-crawler/src/main/java/org/codelibs/fess/crawler/client/http/HcHttpClient.java 

                                    for (String urlPattern : directive.getDisallows()) {
                                    if (StringUtil.isNotBlank(urlPattern)) {
                                        urlPattern = convertRobotsTxtPatternToRegex(urlPattern);
                                        final String urlValue = hostUrl + urlPattern;
                                        crawlerContext.getUrlFilter().addExclude(urlValue);
    Registered: Sun Sep 21 03:50:09 UTC 2025
    - Last Modified: Thu Aug 07 02:55:08 UTC 2025
    - 52.2K bytes
    - Viewed (0)
Back to top