:root {
    --md-code-font: "Fira Code", monospace, "Noto Color Emoji";
}

/* Override default regular font in Material for MkDocs to include Noto Color Emoji */
:root {
    --md-text-font: "Roboto", "Noto Color Emoji";
}

.termynal-comment {
  color: #4a968f;
  font-style: italic;
  display: block;
}

.termy {
  /* For right to left languages */
  direction: ltr;
}

labels.crawling_info_thread_dump=Vidage de thread
labels.crawling_info_CrawlerStartTime=Heure de début du robot d'exploration
labels.crawling_info_CrawlerEndTime=Heure de fin du robot d'exploration
labels.crawling_info_CrawlerExecTime=Temps d'exécution du robot d'exploration
labels.crawling_info_CrawlerStatus=Statut du robot d'exploration
labels.crawling_info_WebFsCrawlExecTime=Temps d'exécution de l'exploration (Web/Fichier)

    /** XPath expression for extracting robots content from meta tags */
    private static final String META_NAME_ROBOTS_CONTENT = "//META[@name=\"robots\" or @name=\"ROBOTS\"]/@content";

    /** Robots tag value indicating no indexing or following */
    private static final String ROBOTS_TAG_NONE = "none";

    /** Robots tag value indicating no indexing */

- **Testing**: JUnit 4, UTFlute, Mockito 5.7.0
- **Storage**: In-memory (default), OpenSearch (optional)

### Protocols

- **HTTP/HTTPS**: Full crawling, cookies, auth, robots.txt
- **File**: Local/network file systems
- **FTP**: With authentication
- **SMB/CIFS**: Windows shares (SMB1/SMB2+)
- **Storage**: MinIO/S3-compatible

### Content Formats

¿Quieres solo tener un `str`? ¿O solo un `dict`? ¿O un instance de clase modelo de base de datos directamente? Todo funciona de la misma manera.

¿En realidad no tienes usuarios que inicien sesión en tu aplicación sino robots, bots u otros sistemas, que solo tienen un token de acceso? Una vez más, todo funciona igual.

    with these curl arguments:

    ```
    curl \
      --http2-prior-knowledge \
      --proxy localhost:8888 \
      --proxytunnel \
      http://squareup.com/robots.txt
    ```

 *  Fix: Support executing OkHttp on kotlin-stdlib versions as old as 1.4. The library still builds
    on up-to-date Kotlin releases (1.6.21) but no longer needs that version as a runtime dependency.

     * Get the value for the key 'crawler.ignore.robots.txt'. <br>
     * The value is, e.g. false <br>
     * comment: Whether to ignore robots.txt during crawling.
     * @return The value of found property. (NotNull: if not found, exception but basically no way)
     */
    String getCrawlerIgnoreRobotsTxt();

    /**
     * Is the property for the key 'crawler.ignore.robots.txt' true? <br>
     * The value is, e.g. false <br>