Search Options

Results per page
Sort
Preferred Languages
Advance

Results 1 - 1 of 1 for stripComment (0.14 sec)

  1. fess-crawler/src/main/java/org/codelibs/fess/crawler/helper/RobotsTxtHelper.java

                final List<Directive> currentDirectiveList = new ArrayList<>();
                boolean isGroupRecodeStarted = false;
                while ((line = reader.readLine()) != null) {
                    line = stripComment(line).trim();
                    if (StringUtil.isEmpty(line)) {
                        continue;
                    }
    
                    String value = getValue(USER_AGENT_RECORD, line);
                    if (value != null) {
    Registered: Sun Nov 10 03:50:12 UTC 2024
    - Last Modified: Sat Oct 12 01:40:57 UTC 2024
    - 6.1K bytes
    - Viewed (0)
Back to top