Search Options

Results per page
Sort
Preferred Languages
Advance

Results 1 - 1 of 1 for stripComment (0.11 sec)

  1. fess-crawler/src/main/java/org/codelibs/fess/crawler/helper/RobotsTxtHelper.java

                boolean isGroupRecordStarted = false;
    
                while ((line = reader.readLine()) != null) {
                    try {
                        // Strip comments and trim whitespace
                        line = stripComment(line).trim();
                        if (StringUtil.isEmpty(line)) {
                            continue;
                        }
    
                        // Try to parse as User-agent directive
    Registered: Sat Dec 20 11:21:39 UTC 2025
    - Last Modified: Fri Nov 14 12:52:01 UTC 2025
    - 11.4K bytes
    - Viewed (0)
Back to top