**Fault Tolerance**:
- `FaultTolerantClient` wrapper (retry, circuit breaker)
- Graceful degradation (e.g., robots.txt parsing continues on errors)

---

## Key Components

### Crawler (`Crawler.java`)

Main orchestrator for crawling operations.

**Key Methods**:
```java
String execute()                // Start crawling, return session ID

    <exportedPackage>org.codehaus.plexus.util.xml.pull.XmlSerializer</exportedPackage>

    <!-- plexus-container, plexus-component-annotations -->
    <exportedPackage>org.codehaus.plexus.*</exportedPackage>
    <exportedPackage>org.codehaus.plexus.component</exportedPackage>
    <exportedPackage>org.codehaus.plexus.configuration</exportedPackage>
    <exportedPackage>org.codehaus.plexus.container</exportedPackage>

 *
 * Percent encoding is used in every URL component except for the hostname. But the set of
 * characters that need to be encoded is different for each component. For example, the path
 * component must escape all of its `?` characters, otherwise it could be interpreted as the
 * start of the URL's query. But within the query and fragment components, the `?` character
 * doesn't delimit anything and doesn't need to be escaped.

import org.apache.maven.plugin.version.PluginVersionResolutionException;
import org.apache.maven.project.MavenProject;

/**
 * A facade that provides lifecycle services to components outside maven core.
 * <p>
 * Note that this component is not normally used from within core itself.
 *
 */
@Named
@Singleton
public class DefaultLifecycleExecutor implements LifecycleExecutor {

        }

        /**
         * Gets the directory path component.
         *
         * @return the path component
         */
        public String getPath() {
            return path;
        }

        /**
         * Gets the file or directory name component.
         *
         * @return the name component
         */
        public String getName() {
            return name;

import org.codehaus.plexus.component.composition.CycleDetectedInComponentGraphException;
import org.codehaus.plexus.component.configurator.ComponentConfigurationException;
import org.codehaus.plexus.component.configurator.ComponentConfigurator;
import org.codehaus.plexus.component.configurator.ConfigurationListener;
import org.codehaus.plexus.component.configurator.expression.ExpressionEvaluationException;

import org.apache.maven.model.Plugin;
import org.apache.maven.plugin.lifecycle.io.LifecycleStaxReader;
import org.codehaus.plexus.classworlds.realm.ClassRealm;
import org.codehaus.plexus.component.repository.ComponentDescriptor;
import org.codehaus.plexus.component.repository.ComponentSetDescriptor;
import org.eclipse.aether.graph.DependencyNode;

/**
 */
public class PluginDescriptor extends ComponentSetDescriptor implements Cloneable {

* Caddy
    * Automatically handles certificates renewals ✨
* Nginx
    * With an external component like Certbot for certificate renewals
* HAProxy
    * With an external component like Certbot for certificate renewals
* Kubernetes with an Ingress Controller like Nginx
    * With an external component like cert-manager for certificate renewals
* Handled internally by a cloud provider as part of their services (read below 👇)

* Caddy
    * Handhabt automatisch Zertifikat-Erneuerungen ✨
* Nginx
    * Mit einer externen Komponente wie Certbot für Zertifikat-Erneuerungen
* HAProxy
    * Mit einer externen Komponente wie Certbot für Zertifikat-Erneuerungen
* Kubernetes mit einem Ingress Controller wie Nginx
    * Mit einer externen Komponente wie cert-manager für Zertifikat-Erneuerungen

                            "content": {
                                "application/json": {
                                    "schema": {
                                        "$ref": "#/components/schemas/HTTPValidationError"
                                    }
                                }
                            },
                        },