# Run tests for specific module
mvn test -pl fess-crawler
```

## Examples

### Web Crawling with Custom Rules

```java
// Create crawler with custom configuration
StandardCrawlerContainer container = new StandardCrawlerContainer();

// Configure HTTP client with custom settings
container.singleton("httpClient", HcHttpClient.class, client -> {
    client.setUserAgent("MyBot/1.0");

            // BUILTIN with unknown type -> falls back to numeric toString
            SID unknownBuiltin = new SID(buildSidT((byte) 1, ident, 544), jcifs.SID.SID_TYPE_UNKNOWN, "BUILTIN", "ignored", false);
            assertEquals(unknownBuiltin.toString(), unknownBuiltin.toDisplayString());

            // BUILTIN with other type also returns acctName

        final int start = dstIndex;
        dstIndex += this.info.encode(dst, dstIndex);

        /* 6 zeros observed with NT */
        SMBUtil.writeInt8(0L, dst, dstIndex);
        dstIndex += 6;

        /*
         * Also observed 4 byte alignment but we stick
         * with the default for jCIFS which is 2
         */

        return dstIndex - start;
    }

    @Override

 * or more contributor license agreements.  See the NOTICE file
 * distributed with this work for additional information
 * regarding copyright ownership.  The ASF licenses this file
 * to you under the Apache License, Version 2.0 (the
 * "License"); you may not use this file except in compliance
 * with the License.  You may obtain a copy of the License at
 *
 *   http://www.apache.org/licenses/LICENSE-2.0
 *

Any metadata changes on the source object version, such as metadata updates via PutObjectTagging, PutObjectRetention, PutObjectLegalHold and COPY api are replicated in a similar manner to target version, with the `X-Amz-Replication-Status` again cycling through the same states.

            return addrs;
        } catch (final UnknownHostException uhe) {
            throw new UnknownHostException("no name with type 0x" + Hexdump.toHexString(addr.hostName.hexCode, 2)
                    + (addr.hostName.scope == null || addr.hostName.scope.length() == 0 ? " with no scope"
                            : " with scope " + addr.hostName.scope)
                    + " for host " + addr.getHostAddress());
        }
    }

    /**

- ✅ Does not consume or modify stream
- ✅ Consistent behavior across multiple calls
- ✅ Works with various InputStream types

---

### 5. TextExtractorEnhancedTest.java
**Purpose**: Test TextExtractor improvements and encoding handling.

**Key Test Areas**:
- Improved error messages with encoding
- Various encoding support
- Edge cases (empty, large, Unicode)
- Special characters handling

	  # tags:
	  #   - key: "name"
	  #     value: "pick*" # match objects with tag 'name', with all values starting with 'pick'

	  ## NOTE: metadata filter not supported when "source" is non MinIO.
	  # metadata:
	  #   - key: "content-type"
	  #     value: "image/*" # match objects with 'content-type', with all values starting with 'image/'

	notify:

     * <ul>
     *   <li>&amp; with &amp;amp;</li>
     *   <li>&lt; with &amp;lt;</li>
     *   <li>&gt; with &amp;gt;</li>
     *   <li>" with &amp;quot;</li>
     *   <li>' with &amp;apos;</li>
     * </ul>
     * Additionally, it strips invalid XML characters from the input string.
     *
     * @param value the input string to be escaped

 *
 * The easiest way to pin a host is turn on pinning with a broken configuration and read the
 * expected configuration when the connection fails. Be sure to do this on a trusted network, and
 * without man-in-the-middle tools like [Charles][charles] or [Fiddler][fiddler].
 *
 * For example, to pin `https://publicobject.com`, start with a broken configuration:
 *
 * ```java
 * String hostname = "publicobject.com";