* you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,
 * either express or implied. See the License for the specific language

 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,
 * either express or implied. See the License for the specific language

     */
    public RobotsTxt() {
        // Default constructor
    }

    /**
     * Checks if access to a given path is allowed for a specific user agent according to robots.txt rules.
     *
     * @param path The path to check for access permission
     * @param userAgent The user agent string to check against robots.txt directives
     * @return true if access is allowed, false if access is disallowed by robots.txt rules.

    }

    /**
     * Sets whether to match the first word.
     * @param matchWordFirst True to match the first word, false otherwise.
     * @return This builder instance.
     */
    public SuggestRequestBuilder setMatchWordFirst(final boolean matchWordFirst) {
        request.setMatchWordFirst(matchWordFirst);
        return this;
    }

    /**
     * Sets whether to skip duplicate words.

/**
 * Utility class for managing crawling parameters using ThreadLocal variables.
 * This class provides methods to set and get various parameters related to the crawling process.
 *
 * <p>This class is final and cannot be instantiated.</p>
 *
 * <p>The following parameters are managed:</p>
 * <ul>
 *   <li>{@link UrlQueue} - The queue of URLs to be crawled.</li>
 *   <li>{@link CrawlerContext} - The context of the current crawling process.</li>

public interface SuggestWriter {
    /**
     * Writes the given suggest items to the specified index.
     *
     * @param client   the client used to interact with the search engine
     * @param settings the settings for the suggest feature
     * @param index    the name of the index where the suggest items will be written
     * @param items    an array of suggest items to be written to the index

 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,
 * either express or implied. See the License for the specific language

            for (final Future<Integer> future : futures) {
                future.get();
            }
        } catch (final InterruptedException e) {
            // no thing to do
        } catch (final ExecutionException e) {
            // no thing to do
        }

        long elapsed = (System.nanoTime() - time) / 1000000;
        long wait = waittime * (numTasks - 1);

     * they crawl the page. Valid values are:
     * <ul>
     * <li>always</li>
     * <li>hourly</li>
     * <li>daily</li>
     * <li>weekly</li>
     * <li>monthly</li>
     * <li>yearly</li>
     * <li>never</li>
     * </ul>
     * The value "always" should be used to describe documents that change each
     * time they are accessed. The value "never" should be used to describe

 * It allows defining multiple regular expressions for different fields of the ResponseData.
 * The rule can be configured to require all regular expressions to match (allRequired = true) or
 * only one of them (allRequired = false). It also supports a default rule that always matches.
 *
 * <p>
 * The class uses a map of field names to Pattern objects to store the regular expressions.