crawler2.awaitTermination();
```

### Custom Interval Control

```java
// Configure politeness policy
container.singleton("intervalController", DefaultIntervalController.class, controller -> {
    controller.setDelayMillisForWaitingNewUrl(5000);
    controller.setDefaultIntervalTime(1000);
});
```

### Sitemap Support

```java
// Enable sitemap processing
container.singleton("sitemapsRule", SitemapsRule.class, rule -> {

        domain = domain.toLowerCase();
        return domains.get(domain) != null;
    }

    /**
     * Gets a domain controller transport for the specified domain
     * @param domain the domain name
     * @param auth the authentication credentials
     * @return an SMB transport to the domain controller
     * @throws SmbAuthException if authentication fails
     */

     * Before upgrading to 1.2, <strong>delete all Deployment alpha-version resources</strong>, including the Replication Controllers and Pods the Deployment manages. Then
create Deployment Beta resources after upgrading to 1.2. Not deleting the
Deployment objects may cause the deployment controller to mistakenly match
other pods and delete them, due to the selector API change.

acquire TLS certificates from a CA or create self-signed certificates. While creating / acquiring certificates ensure the corresponding domain names are set as per the standard [DNS naming conventions](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#pod-identity) in a Kubernetes StatefulSet (for a distributed MinIO setup). Then create a secret using ```bash kubectl create secret generic tls-ssl-minio --from-file=path/to/private.key --from-file=path/to/public.crt ``` Then install...

acquire TLS certificates from a CA or create self-signed certificates. While creating / acquiring certificates ensure the corresponding domain names are set as per the standard [DNS naming conventions](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#pod-identity) in a Kubernetes StatefulSet (for a distributed MinIO setup). Then create a secret using ```bash kubectl create secret generic tls-ssl-minio --from-file=path/to/private.key --from-file=path/to/public.crt ``` Then install...

			true,
		},
		{
			[]string{"data/controller{1...11}/export{1...8}"},
			[]uint64{88},
			[][]uint64{{11, 11, 11, 11, 11, 11, 11, 11}},
			true,
		},
		{
			[]string{"data{1...4}"},
			[]uint64{4},
			[][]uint64{{4}},
			true,
		},
		{
			[]string{"data/controller1/export{1...10}, data/controller2/export{1...10}, data/controller3/export{1...10}"},
			[]uint64{10, 10, 10},

acquire TLS certificates from a CA or create self-signed certificates. While creating / acquiring certificates ensure the corresponding domain names are set as per the standard [DNS naming conventions](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#pod-identity) in a Kubernetes StatefulSet (for a distributed MinIO setup). Then create a secret using ```bash kubectl create secret generic tls-ssl-minio --from-file=path/to/private.key --from-file=path/to/public.crt ``` Then install...

acquire TLS certificates from a CA or create self-signed certificates. While creating / acquiring certificates ensure the corresponding domain names are set as per the standard [DNS naming conventions](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#pod-identity) in a Kubernetes StatefulSet (for a distributed MinIO setup). Then create a secret using ```bash kubectl create secret generic tls-ssl-minio --from-file=path/to/private.key --from-file=path/to/public.crt ``` Then install...

acquire TLS certificates from a CA or create self-signed certificates. While creating / acquiring certificates ensure the corresponding domain names are set as per the standard [DNS naming conventions](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#pod-identity) in a Kubernetes StatefulSet (for a distributed MinIO setup). Then create a secret using ```bash kubectl create secret generic tls-ssl-minio --from-file=path/to/private.key --from-file=path/to/public.crt ``` Then install...

acquire TLS certificates from a CA or create self-signed certificates. While creating / acquiring certificates ensure the corresponding domain names are set as per the standard [DNS naming conventions](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#pod-identity) in a Kubernetes StatefulSet (for a distributed MinIO setup). Then create a secret using ```bash kubectl create secret generic tls-ssl-minio --from-file=path/to/private.key --from-file=path/to/public.crt ``` Then install...