Search Options

Results per page
Sort
Preferred Languages
Advance

Results 1 - 8 of 8 for Tokenizer (0.34 sec)

  1. src/main/java/org/codelibs/core/text/Tokenizer.java

        static {
            setup(defaultCtype);
        }
    
        /**
         * Creates a {@link Tokenizer}.
         *
         * @param str
         *            The string. Must not be {@literal null}.
         */
        public Tokenizer(final String str) {
            this(str, defaultCtype);
        }
    
        /**
         * Creates a {@link Tokenizer}.
         *
         * @param str
         *            The string. Must not be {@literal null}.
    Registered: Sat Dec 20 08:55:33 UTC 2025
    - Last Modified: Sat Jul 05 00:11:05 UTC 2025
    - 8.8K bytes
    - Viewed (0)
  2. src/main/java/jcifs/smb1/http/Handler.java

                if (handler == null) {
                    final String path = System.getProperty(HANDLER_PKGS_PROPERTY);
                    final StringTokenizer tokenizer = new StringTokenizer(path, "|");
                    while (tokenizer.hasMoreTokens()) {
                        final String provider = tokenizer.nextToken().trim();
                        if (provider.equals("jcifs.smb1")) {
                            continue;
                        }
    Registered: Sat Dec 20 13:44:44 UTC 2025
    - Last Modified: Sat Aug 16 01:32:48 UTC 2025
    - 6.1K bytes
    - Viewed (0)
  3. src/main/java/org/codelibs/fess/suggest/converter/KatakanaConverter.java

     * use a tokenizer to process the input.
     *
     * <p>
     * The class provides methods to initialize the converter, convert strings, and
     * check if a tokenizer is enabled. It also includes methods to create a token
     * stream and extract reading information from the stream's attributes, although
     * the tokenizer-related functionality is currently commented out.
     * </p>
     */
    Registered: Sat Dec 20 13:04:59 UTC 2025
    - Last Modified: Fri Jul 04 14:00:23 UTC 2025
    - 6.1K bytes
    - Viewed (0)
  4. src/main/java/jcifs/http/Handler.java

                if (handler == null) {
                    final String path = System.getProperty(HANDLER_PKGS_PROPERTY);
                    final StringTokenizer tokenizer = new StringTokenizer(path, "|");
                    while (tokenizer.hasMoreTokens()) {
                        final String provider = tokenizer.nextToken().trim();
                        if (provider.equals("jcifs")) {
                            continue;
                        }
    Registered: Sat Dec 20 13:44:44 UTC 2025
    - Last Modified: Sat Aug 16 01:32:48 UTC 2025
    - 6.9K bytes
    - Viewed (0)
  5. src/test/java/org/codelibs/fess/suggest/converter/AnalyzerConverterTest.java

                    .put("analysis.analyzer.test_reading_analyzer.tokenizer", "standard")
                    .put("analysis.analyzer.test_reading_term_analyzer.type", "custom")
                    .put("analysis.analyzer.test_reading_term_analyzer.tokenizer", "standard")
                    .put("analysis.analyzer.test_reading_analyzer_ja.type", "custom")
                    .put("analysis.analyzer.test_reading_analyzer_ja.tokenizer", "keyword")
    Registered: Sat Dec 20 13:04:59 UTC 2025
    - Last Modified: Mon Sep 01 13:33:03 UTC 2025
    - 12.5K bytes
    - Viewed (0)
  6. README.md

    String escaped = JsonUtil.escape("Hello \"World\" with special chars");
    String unescaped = JsonUtil.unescape(escaped);
    
    // Text tokenization
    Tokenizer tokenizer = new Tokenizer("field1,field2,field3", ",");
    while (tokenizer.hasMoreTokens()) {
        String token = tokenizer.nextToken();
        // Process each token
    }
    
    // Decimal formatting
    DecimalFormat format = DecimalFormatUtil.getDecimalFormat("###,###.00");
    ```
    
    Registered: Sat Dec 20 08:55:33 UTC 2025
    - Last Modified: Sun Aug 31 02:56:02 UTC 2025
    - 12.7K bytes
    - Viewed (0)
  7. src/test/java/org/codelibs/fess/suggest/analysis/SuggestAnalyzerTest.java

            List<AnalyzeToken> tokens = analyzer.analyzeAndReading(text, field, lang);
    
            assertNotNull(tokens);
            assertEquals(2, tokens.size()); // "Test123" and "ABC-456" (hyphen is not a split char in our tokenizer)
            assertEquals("Test123", tokens.get(0).getTerm());
            assertEquals("ABC-456", tokens.get(1).getTerm());
            assertEquals("TEST123", ((TestAnalyzeToken) tokens.get(0)).getReading());
    Registered: Sat Dec 20 13:04:59 UTC 2025
    - Last Modified: Mon Sep 01 13:33:03 UTC 2025
    - 15.7K bytes
    - Viewed (0)
  8. ADDING_NEW_LANGUAGE.md

          },
          "swedish_stemmer": {
            "type": "stemmer",
            "language": "swedish"
          }
        },
        "analyzer": {
          "sv_analyzer": {
            "type": "custom",
            "tokenizer": "standard",
            "filter": ["lowercase", "swedish_stop", "swedish_stemmer"]
          }
        }
      }
    }
    ```
    
    ### 2. Add Custom Dictionaries (Optional)
    
    Registered: Sat Dec 20 09:19:18 UTC 2025
    - Last Modified: Thu Nov 06 11:36:30 UTC 2025
    - 10.4K bytes
    - Viewed (0)
Back to top