nofollow - Code Search

src/main/java/org/codelibs/fess/crawler/transformer/FessXpathTransformer.java

                    }
                    if (content.contains(ROBOTS_TAG_NOFOLLOW)) {
                        nofollow = true;
                    }
                }
                if (noindex && nofollow) {
                    logger.info("META(robots=noindex,nofollow): {}", responseData.getUrl());
                    throw new ChildUrlsException(Collections.emptySet(), "#processMetaRobots");
                }

Registered: Thu Oct 31 13:40:30 UTC 2024

- Last Modified: Thu Oct 24 13:01:38 UTC 2024

- 42.9K bytes

- Viewed (0)

github.com/codelibs/fess-crawler

fess-crawler/src/main/java/org/codelibs/fess/crawler/entity/ResponseData.java

    }

    public void setStatus(final int status) {
        this.status = status;
    }

    public void setNoFollow(final boolean value) {
        noFollow = value;
    }

    public boolean isNoFollow() {
        return noFollow;
    }

    public void addMetaData(final String name, final Object value) {
        metaDataMap.put(name, value);
    }

Registered: Sun Nov 10 03:50:12 UTC 2024

- Last Modified: Sat Oct 12 01:40:57 UTC 2024

- 5.8K bytes

- Viewed (0)

github.com/codelibs/fess

src/test/java/org/codelibs/fess/mylasta/direction/FessPropTest.java

        FessConfig fessConfig = new FessConfig.SimpleImpl() {
            @Override
            public String getCrawlerDocumentHtmlPrunedTags() {
                return "script,div#main,p.image,a[rel=nofollow],div[x-y=a-.:_0]";
            }
        };

        PrunedTag[] tags = fessConfig.getCrawlerDocumentHtmlPrunedTagsAsArray();
        assertTrue(matchesTag(tags[0], "<script></script>"));

Registered: Thu Oct 31 13:40:30 UTC 2024

- Last Modified: Thu Feb 22 01:37:57 UTC 2024

- 12.7K bytes

- Viewed (0)

github.com/codelibs/fess

src/test/java/org/codelibs/fess/crawler/transformer/FessXpathTransformerTest.java

            };
        };

        final ResponseData responseData = new ResponseData();
        responseData.addMetaData("X-Robots-Tag", "nofollow");

        transformer.processXRobotsTag(responseData, new ResultData());
        assertTrue(responseData.isNoFollow());
    }

    public void test_processMetaRobots_no() throws Exception {
        final String data = "<html><body>foo</body></html>";

Registered: Thu Oct 31 13:40:30 UTC 2024

- Last Modified: Thu Oct 24 13:01:38 UTC 2024

- 41.3K bytes

- Viewed (0)

github.com/codelibs/fess

src/main/resources/fess_config.properties

crawler.document.html.digest.xpath=//META[@name='description']/@content
crawler.document.html.canonical.xpath=//LINK[@rel='canonical'][1]/@href
crawler.document.html.pruned.tags=noscript,script,style,header,footer,aside,nav,a[rel=nofollow]
crawler.document.html.max.digest.length=120
crawler.document.html.default.lang=
crawler.document.html.default.include.index.patterns=

Registered: Thu Oct 31 13:40:30 UTC 2024

- Last Modified: Tue Oct 01 14:13:38 UTC 2024

- 30.9K bytes

- Viewed (0)

github.com/codelibs/fess

src/main/java/org/codelibs/fess/mylasta/direction/FessConfig.java

    String CRAWLER_DOCUMENT_HTML_CANONICAL_XPATH = "crawler.document.html.canonical.xpath";

    /** The key of the configuration. e.g. noscript,script,style,header,footer,aside,nav,a[rel=nofollow] */
    String CRAWLER_DOCUMENT_HTML_PRUNED_TAGS = "crawler.document.html.pruned.tags";

    /** The key of the configuration. e.g. 120 */

Registered: Thu Oct 31 13:40:30 UTC 2024

- Last Modified: Fri Oct 11 21:11:58 UTC 2024

- 468.5K bytes

- Viewed (1)

github.com/square/okhttp

okhttp/src/main/kotlin/okhttp3/internal/connection/RoutePlanner.kt

import okhttp3.HttpUrl

/**
 * Policy on choosing which connection to use for an exchange and any retries that follow. This uses
 * the following strategies:
 *
 *  1. If the current call already has a connection that can satisfy the request it is used. Using
 *     the same connection for an initial exchange and its follow-ups may improve locality.
 *

Registered: Fri Nov 01 11:42:11 UTC 2024

- Last Modified: Wed Dec 20 23:27:07 UTC 2023

- 4.2K bytes

- Viewed (0)

github.com/square/okhttp

okhttp/src/main/kotlin/okhttp3/internal/http/RetryAndFollowUpInterceptor.kt

    return true
  }

  /**
   * Figures out the HTTP request to make in response to receiving [userResponse]. This will
   * either add authentication headers, follow redirects or handle a client request timeout. If a
   * follow-up is either unnecessary or not applicable, this returns null.
   */
  @Throws(IOException::class)
  private fun followUpRequest(
    userResponse: Response,
    exchange: Exchange?,

Registered: Fri Nov 01 11:42:11 UTC 2024

- Last Modified: Mon Apr 15 13:24:48 UTC 2024

- 12.1K bytes

- Viewed (0)

github.com/gradle/gradle

architecture/standards/0001-use-architectural-decision-records.md

We use *Specification* and *Discovery* documents stored in Google Drive, but they present some downsides:

* They are rarely updated after creation and initial review, and then become hard to follow, especially after important decisions are made
* They are not synced with the code to reflect the eventual solution that is committed
* Google Docs is not a "code-oriented" tool, like asciidoc can be

Registered: Wed Nov 06 11:36:14 UTC 2024

- Last Modified: Sat Mar 02 21:54:40 UTC 2024

- 2.8K bytes

- Viewed (0)

github.com/square/okhttp

CONTRIBUTING.md

============

Keeping the project small and stable limits our ability to accept new contributors. We are not
seeking new committers at this time, but some small contributions are welcome.

If you've found a security problem, please follow our [bug bounty][security] program.

If you've found a bug, please contribute a failing test case so we can study and fix it.

If you have a new feature idea, please build it in an external library. There are

Registered: Fri Nov 01 11:42:11 UTC 2024

- Last Modified: Tue Feb 14 08:26:50 UTC 2023

- 2K bytes

- Viewed (0)

Search Options