- Sort Score
- Result 10 results
- Languages All
Results 1 - 2 of 2 for getRobotsTxtUrlSet (0.15 sec)
-
fess-crawler/src/test/java/org/codelibs/fess/crawler/client/http/HcHttpClientTest.java
CrawlingParameterUtil.setCrawlerContext(crawlerContext); httpClient.init(); httpClient.processRobotsTxt(url); assertEquals(1, crawlerContext.getRobotsTxtUrlSet().size()); assertTrue(crawlerContext.getRobotsTxtUrlSet().contains("http://localhost:7070/robots.txt")); assertFalse(urlFilter.match("http://localhost:7070/admin/"));
Registered: Wed Jun 12 15:17:51 UTC 2024 - Last Modified: Thu May 09 09:28:25 UTC 2024 - 8.8K bytes - Viewed (0) -
fess-crawler/src/main/java/org/codelibs/fess/crawler/client/http/HcHttpClient.java
Registered: Wed Jun 12 15:17:51 UTC 2024 - Last Modified: Thu May 09 09:28:25 UTC 2024 - 41K bytes - Viewed (0)