Search Options

Display Count
Sort
Preferred Language
Advanced Search

Results 1 - 2 of 2 for getRobotsTxtUrlSet (0.07 seconds)

  2. github.com/codelibs/fess-crawler

    fess-crawler/src/main/java/org/codelibs/fess/crawler/client/http/Hc4HttpClient.java 

            } else {
            hostUrl = url;
        }
        final String robotTxtUrl = hostUrl + "/robots.txt";

        // check url
        if (crawlerContext.getRobotsTxtUrlSet().contains(robotTxtUrl)) {
            if (logger.isDebugEnabled()) {
                logger.debug("{} is already visited.", robotTxtUrl);
            }
            return;
        }
    Created: Sun Apr 12 03:50:13 GMT 2026
    - Last Modified: Fri Jan 09 23:46:52 GMT 2026
    - 54.4K bytes
    - Click Count (0)
  3. github.com/codelibs/fess-crawler

    fess-crawler/src/main/java/org/codelibs/fess/crawler/client/http/Hc5HttpClient.java 

            } else {
            hostUrl = url;
        }
        final String robotTxtUrl = hostUrl + "/robots.txt";

        // check url
        if (crawlerContext.getRobotsTxtUrlSet().contains(robotTxtUrl)) {
            if (logger.isDebugEnabled()) {
                logger.debug("{} is already visited.", robotTxtUrl);
            }
            return;
        }
    Created: Sun Apr 12 03:50:13 GMT 2026
    - Last Modified: Sat Jan 31 12:23:29 GMT 2026
    - 62.2K bytes
    - Click Count (0)
Back to Top