}
        }
        return null;
    }

    /**
     * Adds a directive to the robots.txt rules.
     * The user-agent pattern in the directive is converted to a regular expression pattern,
     * where '*' is replaced with '.*' for pattern matching, and stored case-insensitively.
     *
     * @param directive The directive to add to the robots.txt rules
     */

 *       specified in the HTML content.</li>
 *   <li><b>preloadSizeForCharset:</b> The number of bytes to read from the input
 *       stream to determine the character set encoding.</li>
 *   <li><b>invalidUrlPattern:</b> A regular expression pattern used to identify
 *       invalid URLs.</li>
 * </ul>
 *
 * <p>
 * <b>Usage:</b>
 * </p>
 * <p>
 * The {@code transform} method is the main entry point for transforming an HTML

        } finally {
            EntityUtils.consumeQuietly(httpEntity);
        }
    }

    /**
     * Converts a robots.txt pattern to a regular expression.
     *
     * @param path The robots.txt pattern to convert
     * @return The equivalent regular expression
     */
    protected String convertRobotsTxtPatternToRegex(final String path) {