Search Options

Results per page
Sort
Preferred Languages
Advance

Results 1 - 1 of 1 for robot (0.01 sec)

  1. fess-crawler-opensearch/src/test/java/org/codelibs/fess/crawler/util/CrawlerWebServer.java

                buf.append("User-agent: *").append('\n');
                buf.append("Disallow: /admin/").append('\n');
                buf.append("Disallow: /websvn/").append('\n');
                final File robotTxtFile = new File(tempDir, "robots.txt");
                FileUtil.writeBytes(robotTxtFile.getAbsolutePath(), buf.toString().getBytes("UTF-8"));
                robotTxtFile.deleteOnExit();
    
                generateContents(tempDir, count);
    
    Registered: Sun Nov 10 03:50:12 UTC 2024
    - Last Modified: Thu Nov 07 04:44:10 UTC 2024
    - 5K bytes
    - Viewed (0)
Back to top