assertTrue(robotsTxt.allows("/priceinfo", "DollarBot"));

        // Test sitemaps
        String[] sitemaps = robotsTxt.getSitemaps();
        assertEquals(1, sitemaps.length);
        assertEquals("http://www.example.com/sitemap.xml", sitemaps[0]);
    }

    public void testParse_malformed() {
        RobotsTxt robotsTxt;

    }

    /**
     * Adds sitemaps to the thread-local storage.
     * @param sitemaps An array of sitemap URLs.
     */
    public void addSitemaps(final String[] sitemaps) {
        sitemapsLocal.set(sitemaps);
    }

    /**
     * Removes sitemaps from the thread-local storage and returns them.
     * @return An array of sitemap URLs, or null if none were present.
     */

        responseData.setResponseBody(file, false);
        return responseData;
    }

    private ResponseData getTestData3_OK() {
        final ResponseData responseData = new ResponseData();
        responseData.setUrl("http://example.com/sitemap.txt");
        File file = ResourceUtil.getResourceAsFile("sitemaps/sitemap1.txt");
        responseData.setResponseBody(file, false);
        return responseData;
    }

De esta manera, puedes crear un token con una expiración de, digamos, 1 semana. Y luego, cuando el usuario regresa al día siguiente con el token, sabes que el usuario todavía está registrado en tu sistema.

    controller.setDefaultIntervalTime(1000);
});
```

### Sitemap Support

```java
// Enable sitemap processing
container.singleton("sitemapsRule", SitemapsRule.class, rule -> {
    rule.addRule("url", ".*sitemap.*");
});

// Add sitemap URL
crawler.addUrl("https://example.com/sitemap.xml");
```

## Data Access and Storage

### Accessing Crawled Data

```java

| Facet Fields | Configure in Fess search settings |
| Highlighting | Built-in in Fess search results |

---

### Other Search Systems

For proprietary or custom search systems:

#### API-Based Migration

**Step 1: Extract Documents**

Use your current system's API or export functionality to extract:
- Document URLs or identifiers
- Document content/text
- Metadata (title, author, date, etc.)

Te mostraré cómo **construir tu propia imagen desde cero** para ejecutar un solo proceso de Uvicorn. Es un proceso sencillo y probablemente es lo que querrías hacer al usar un sistema de gestión de contenedores distribuido como **Kubernetes**.

## Resumen { #recap }

 * </p>
 *
 * @see <a href="https://developers.google.com/search/docs/crawling-indexing/sitemaps/video-sitemaps">Video Sitemaps</a>
 */
public class SitemapVideo implements Serializable {

    private static final long serialVersionUID = 1L;

    /**
     * A URL pointing to the video thumbnail image file.
     * Images must be at least 160x90 pixels and at most 1920x1080 pixels.
     */

Isso significa que no lugar do processo normal de seus usuários enviarem requisições para a sua API, é a **sua API** (ou sua aplicação) que poderia **enviar requisições para o sistema deles** (para a API deles, a aplicação deles).

Isso normalmente é chamado de **webhook**.

## Etapas dos webhooks { #webhooks-steps }

User-agent: Bot1
User-agent: Bot2
User-agent: Bot3
Disallow: /shared/

# Case 11: Sitemap with various formats
Sitemap: http://example.com/sitemap.xml
sitemap: http://example.com/sitemap2.xml
SITEMAP: http://example.com/sitemap3.xml
Sitemap:    # empty sitemap (should be ignored)
Sitemap: not-a-valid-url

# Case 12: Malformed lines that should be completely ignored
This line is completely invalid
:NoKey