*/
    @Size(max = 10000)
    public String virtualHostValue;

    /**
     * Enable or disable appending query parameters to search URLs.
     * When enabled, additional parameters are added to search result URLs.
     */
    @Size(max = 10)
    public String appendQueryParameter;

    /**
     * Enable or disable login requirement for search access.

    }

    /**
     * Creates a URL object from the current URL and a relative or absolute URL string.
     *
     * @param currentUrl the current URL as base
     * @param url the URL string to process
     * @return the URL object
     * @throws MalformedURLException if the URL is malformed
     */
    protected URL getURL(final String currentUrl, final String url) throws MalformedURLException {

            "type" : "integer"
          },
          "updatedBy" : {
            "type" : "keyword"
          },
          "updatedTime" : {
            "type" : "long"
          },
          "urls" : {
            "type" : "keyword"
          },
          "userAgent" : {
            "type" : "keyword"
          },
          "virtualHosts" : {
            "type" : "keyword"
          }
        }

Isso pode ser útil se, por exemplo, você mora em um país que restringe algumas URLs.

### Desativar a documentação automática

O primeiro passo é desativar a documentação automática, pois por padrão, ela usa o CDN padrão.

Para desativá-los, defina suas URLs como `None` ao criar seu aplicativo `FastAPI`:

{* ../../docs_src/custom_docs_ui/tutorial001.py hl[8] *}

 *
 * <p>
 * It provides methods to set and retrieve the server address, port, username,
 * password, and domain. Additionally, it offers a method to construct a path
 * prefix for SMB URLs based on the configured server and port.
 * </p>
 *
 * <p>
 * The path prefix is in the format "smb://server:port/", where the port is
 * included only if it's greater than 0. If the server is not set, the path

        }
    }

    /**
     * Registers a client component with a regular expression.
     * The component will be loaded into all registered CrawlerClientFactories.
     * @param regex The regular expression to match URLs.
     * @param componentName The name of the component to register.
     */
    public synchronized void register(final String regex, final String componentName) {
        clientMap.put(regex, componentName);

Você pode defini-los da seguinte maneira:

    }

    /**
     * Gets the URL of the crawled resource.
     *
     * @return the URL
     */
    public String getUrl() {
        return url;
    }

    /**
     * Sets the URL of the crawled resource.
     *
     * @param url the URL to set
     */
    public void setUrl(final String url) {
        this.url = url;
    }

    /**

```java
// Set maximum number of URLs to crawl
crawler.crawlerContext.setMaxAccessCount(1000);

// Set number of crawler threads
crawler.crawlerContext.setNumOfThread(10);

// Set maximum crawl depth
crawler.crawlerContext.setMaxDepth(3);

// Set request interval (politeness)
crawler.crawlerContext.setDefaultIntervalTime(1000); // 1 second
```

### URL Filtering

```java
// Include patterns

      } catch (Exception ignore) {
        // probably URISyntaxException or IllegalArgumentException
        // fall back to copying URLs to files in the testDir == null block below
      }
    }

    if (testDir == null) {
      // testdata resources aren't file:// urls, so create a directory to store them in and then
      // copy the resources to the filesystem as needed
      testDir = createTempDir();
    }