nofollow - Code Search

src/main/java/org/codelibs/fess/crawler/transformer/FessXpathTransformer.java

                    }
                    if (content.contains(ROBOTS_TAG_NOFOLLOW)) {
                        nofollow = true;
                    }
                }
                if (noindex && nofollow) {
                    logger.info("META(robots=noindex,nofollow): {}", responseData.getUrl());
                    throw new ChildUrlsException(Collections.emptySet(), "#processMetaRobots");
                }

Registered: Thu Oct 31 13:40:30 UTC 2024

- Last Modified: Thu Oct 24 13:01:38 UTC 2024

- 42.9K bytes

- Viewed (0)

github.com/codelibs/fess-crawler

fess-crawler/src/main/java/org/codelibs/fess/crawler/entity/ResponseData.java

    }

    public void setStatus(final int status) {
        this.status = status;
    }

    public void setNoFollow(final boolean value) {
        noFollow = value;
    }

    public boolean isNoFollow() {
        return noFollow;
    }

    public void addMetaData(final String name, final Object value) {
        metaDataMap.put(name, value);
    }

Registered: Sun Nov 10 03:50:12 UTC 2024

- Last Modified: Sat Oct 12 01:40:57 UTC 2024

- 5.8K bytes

- Viewed (0)

github.com/codelibs/fess

src/test/java/org/codelibs/fess/mylasta/direction/FessPropTest.java

        FessConfig fessConfig = new FessConfig.SimpleImpl() {
            @Override
            public String getCrawlerDocumentHtmlPrunedTags() {
                return "script,div#main,p.image,a[rel=nofollow],div[x-y=a-.:_0]";
            }
        };

        PrunedTag[] tags = fessConfig.getCrawlerDocumentHtmlPrunedTagsAsArray();
        assertTrue(matchesTag(tags[0], "<script></script>"));

Registered: Thu Oct 31 13:40:30 UTC 2024

- Last Modified: Thu Feb 22 01:37:57 UTC 2024

- 12.7K bytes

- Viewed (0)

github.com/codelibs/fess

src/test/java/org/codelibs/fess/crawler/transformer/FessXpathTransformerTest.java

            };
        };

        final ResponseData responseData = new ResponseData();
        responseData.addMetaData("X-Robots-Tag", "nofollow");

        transformer.processXRobotsTag(responseData, new ResultData());
        assertTrue(responseData.isNoFollow());
    }

    public void test_processMetaRobots_no() throws Exception {
        final String data = "<html><body>foo</body></html>";

Registered: Thu Oct 31 13:40:30 UTC 2024

- Last Modified: Thu Oct 24 13:01:38 UTC 2024

- 41.3K bytes

- Viewed (0)

github.com/codelibs/fess

src/main/resources/fess_config.properties

crawler.document.html.digest.xpath=//META[@name='description']/@content
crawler.document.html.canonical.xpath=//LINK[@rel='canonical'][1]/@href
crawler.document.html.pruned.tags=noscript,script,style,header,footer,aside,nav,a[rel=nofollow]
crawler.document.html.max.digest.length=120
crawler.document.html.default.lang=
crawler.document.html.default.include.index.patterns=

Registered: Thu Oct 31 13:40:30 UTC 2024

- Last Modified: Tue Oct 01 14:13:38 UTC 2024

- 30.9K bytes

- Viewed (0)

github.com/codelibs/fess

src/main/java/org/codelibs/fess/mylasta/direction/FessConfig.java

    String CRAWLER_DOCUMENT_HTML_CANONICAL_XPATH = "crawler.document.html.canonical.xpath";

    /** The key of the configuration. e.g. noscript,script,style,header,footer,aside,nav,a[rel=nofollow] */
    String CRAWLER_DOCUMENT_HTML_PRUNED_TAGS = "crawler.document.html.pruned.tags";

    /** The key of the configuration. e.g. 120 */

Registered: Thu Oct 31 13:40:30 UTC 2024

- Last Modified: Fri Oct 11 21:11:58 UTC 2024

- 468.5K bytes

- Viewed (1)

github.com/square/okhttp

okhttp/src/main/kotlin/okhttp3/internal/connection/RoutePlanner.kt

import okhttp3.HttpUrl

/**
 * Policy on choosing which connection to use for an exchange and any retries that follow. This uses
 * the following strategies:
 *
 *  1. If the current call already has a connection that can satisfy the request it is used. Using
 *     the same connection for an initial exchange and its follow-ups may improve locality.
 *

Registered: Fri Nov 01 11:42:11 UTC 2024

- Last Modified: Wed Dec 20 23:27:07 UTC 2023

- 4.2K bytes

- Viewed (0)

github.com/minio/minio

docs/integrations/veeam/README.md

- Follow through the wizard as above for Veeam Backup and Replication as the steps are the same between both products

- Under Backup Infrastructure -> Backup Repositories, right click and "Add Backup Repository"

- Follow the wizard.  Under the "Object Storage Backup Repository" section, choose the MinIO object storage you created above

Registered: Sun Nov 03 19:28:11 UTC 2024

- Last Modified: Thu Sep 29 04:28:45 UTC 2022

- 5.4K bytes

- Viewed (0)

github.com/square/okhttp

okhttp/src/main/kotlin/okhttp3/internal/http/RetryAndFollowUpInterceptor.kt

    return true
  }

  /**
   * Figures out the HTTP request to make in response to receiving [userResponse]. This will
   * either add authentication headers, follow redirects or handle a client request timeout. If a
   * follow-up is either unnecessary or not applicable, this returns null.
   */
  @Throws(IOException::class)
  private fun followUpRequest(
    userResponse: Response,
    exchange: Exchange?,

Registered: Fri Nov 01 11:42:11 UTC 2024

- Last Modified: Mon Apr 15 13:24:48 UTC 2024

- 12.1K bytes

- Viewed (0)

github.com/gradle/gradle

architecture/standards/0001-use-architectural-decision-records.md

We use *Specification* and *Discovery* documents stored in Google Drive, but they present some downsides:

* They are rarely updated after creation and initial review, and then become hard to follow, especially after important decisions are made
* They are not synced with the code to reflect the eventual solution that is committed
* Google Docs is not a "code-oriented" tool, like asciidoc can be

Registered: Wed Nov 06 11:36:14 UTC 2024

- Last Modified: Sat Mar 02 21:54:40 UTC 2024

- 2.8K bytes

- Viewed (0)

Search Options