Search Options

Results per page
Sort
Preferred Languages
Advance

Results 1 - 2 of 2 for setCrawlDelay (0.04 sec)

  1. fess-crawler/src/main/java/org/codelibs/fess/crawler/entity/RobotsTxt.java

                this.userAgent = userAgent;
            }
    
            /**
             * Sets the crawl delay for this directive.
             * @param crawlDelay the crawl delay in seconds
             */
            public void setCrawlDelay(final int crawlDelay) {
                this.crawlDelay = crawlDelay;
            }
    
            /**
             * Gets the crawl delay for this directive.
             * @return the crawl delay in seconds
             */
    Registered: Sun Sep 21 03:50:09 UTC 2025
    - Last Modified: Sun Jul 06 02:13:03 UTC 2025
    - 10K bytes
    - Viewed (0)
  2. fess-crawler/src/main/java/org/codelibs/fess/crawler/helper/RobotsTxtHelper.java

                                    final int crawlDelay = Integer.parseInt(value);
                                    for (final Directive directive : currentDirectiveList) {
                                        directive.setCrawlDelay(Math.max(0, crawlDelay));
                                    }
                                } catch (final NumberFormatException e) {
                                    // ignore invalid crawl-delay values
    Registered: Sun Sep 21 03:50:09 UTC 2025
    - Last Modified: Sun Jul 06 02:13:03 UTC 2025
    - 7.7K bytes
    - Viewed (0)
Back to top