SYSTEM AND METHOD FOR IMPROVING WEBPAGE INDEXING AND OPTIMIZATION

• US 20120016897A1
• Filed: 07/15/2011
• Published: 01/19/2012
• Est. Priority Date: 07/16/2010
• Status: Abandoned Application

• Current Assignee: SDX Acquisition, LLC
• Original Assignee: Altruik, Inc. (SearchDex, Inc.)
• Inventors: TULUMBAS, Gregory, Batista Reyes, Hamlet
• Application Number:
  US13/184,245
• Publication Number:
  US 20120016897A1
• Time in Patent Office: Days
• US Class Current: 707/759
• CPC Class Codes: G06F 16/9566 URL specific, e.g. using aliases, detecting broken or misspelled links

Abstract

A system and method may include a processor that normalizes dynamic URLs by sorting URL parameters and removing duplicative URL parameters. The processor may additionally or alternatively provide redirects from one URL to another, where the two URLs are associated with duplicative content. The processor may additionally or alternatively insert a canonical tag into content associated with a URL, where the canonical tag points to another URL whose content is a near duplicate of the content associated with the first URL. The processor may additionally or alternatively apply transformation rules to content of a webpage based on the matching of portions of the URL of the webpage to various character strings.

Claims

1. A computer-implemented page request normalization method, comprising:

responsive to receipt of a page request from a requesting entity, modifying, by a computer processor, the request by at least one of (a) removing one or more of duplicative parameters included in the request, and (b) changing an order of parameters of the request; and

returning, by the processor and to the requesting entity, the modified request as a page redirect.

2. The method of claim 1, wherein the requests are in the form of Uniform Resource Locator (URL).

3. The method of claim 2, wherein the URL of the received request refers to a dynamic webpage.

4. The method of claim 3, wherein the modifying includes sorting query parameters of the URL by one or more sort keys.

5. The method of claim 4, wherein the modifying further includes:

comparing pairs of the query of the query parameters to determine whether they are duplicates of each other; and

for each of the pairs of compared query parameters determined to be duplicates of each other, removing one of the parameters of the respective pair.

6. The method of claim 3, wherein the modifying includes sorting query parameters of the URL by alphanumeric order.

7. The method of claim 6, wherein the modifying further includes, subsequent to the sorting:

comparing pairs of the query of the query parameters to determine whether they are duplicates of each other; and

for each of the pairs of compared query parameters determined to be duplicates of each other, removing one of the parameters of the respective pair.

8. The method of claim 3, wherein each of at least one of the parameters of the URL of the received request includes a respective key and a respective value for the key.

9. A computer-implemented page request handling method, comprising:

where different ones of a plurality of received webpage requests differ with respect to at least one of (a) a number of included copies of a query parameter and (b) an order of included query parameters, and where each of the plurality of received webpage requests includes at least one copy of each query parameter of each of all others of the plurality of received webpage requests, transmitting, for all of the plurality of received webpage requests, by a computer processor, and to a web server, a respective normalized webpage request, wherein all of the normalized webpage requests include an identical number of query parameters in an identical order.

10. A computer-implemented page link normalization method, comprising:

responsive to receipt of a webpage addressed to a receiving entity and including a webpage link, modifying, by a computer processor, the webpage by at least one of (a) removing one or more of duplicative parameters included in the link, and (b) changing an order of parameters of the link; and

forwarding, by the processor and to the receiving entity, the modified webpage.

11. A computer-implemented method for duplicate content connection, comprising:

comparing, by a computer processor, fingerprints, each associated with a different one of a plurality of page source identifiers;

for a subset of the plurality of page source identifiers for which it is determined in the comparing that the fingerprints of the subset are identical, recording, by the processor, a selection of one of the page source identifiers of the subset as authoritative; and

responsive to a page request using one of the subset of page source identifiers other than the one selected as authoritative, returning a page redirect with the page source identifier selected as authoritative.

12. The method of claim 11, wherein each of at least one of the plurality of page source identifiers is a Uniform Resource Locator (URL).

13. The method of claim 11, further comprising:

generating the fingerprints based on respective content obtainable by the respective page source identifiers.

14. The method of claim 13, wherein the content on which the fingerprints are based is limited to content that is displayed on a user interface in response to respective page requests.

15. The method of claim 11, wherein the fingerprints are checksum values.

16. The method of claim 11, further comprising:

determining which of the subset of page source identifiers is the shortest, wherein the shortest of the subset of page source identifiers is selected as the authoritative page source identifier.

17. The method of claim 11, further comprising:

determining which of the subset of page source identifiers is most frequently used in page requests, wherein the most frequently used of the subset of page source identifiers is selected as the authoritative page source identifier.

18. The method of claim 17, wherein the one of the subset of page source identifiers recorded as the authoritative page source identifier changes over time.

19. The method of claim 11, wherein the recordation is based on a user selection.

20. The method of claim. 11, wherein the selection is based on at least one of sizes of respective ones of the subset of page source identifiers and frequencies of use of the respective ones of the subset of page source identifiers.

21. A computer-implemented method for near-duplicate content correction, comprising:

determining, by a computer processor, that content associated with a subset of a plurality of page source identifiers is similar;

recording, by the processor, a selection of one of the page source identifiers of the subset as authoritative; and

providing, by the processor, a canonical tag to the authoritative page source identifier to each of the other page source identifiers of the subset.

22. The method of claim 21, wherein the canonical tags are inserted into respective hyper-text markup language (HTML) headers of respective pages associated with the respective other page source identifiers of the subset.

23. The method of claim 21, wherein respective ones of the canonical tags are provided to respective ones of the other page source identifiers in response to respective page requests using the respective ones of the other page source identifiers.

24. The method of claim 21, wherein the determination is based on a comparison of simhash values associated with the plurality of page source identifiers.

25. A computer-implemented page link optimization method, comprising:

responsive to receipt of a webpage addressed to a receiving entity and including a first webpage link;

determining, by a computer processor, that the first webpage link is part of a group of webpage links for which a second webpage link is recorded as being authoritative;

in accordance with the determination, modifying, by the processor, the webpage by replacing the first webpage link with the second webpage link; and

forwarding, by the processor and to the receiving entity, the modified webpage;

wherein the webpage links of the group are included in the group in response to a determination that content associated with the webpage links of the group are duplicative.

26. A computer-implemented page optimization method, comprising:

determining, by a computer processor, that a page source identifier includes one or more of a plurality of character strings that are each associated with a respective transformation rule set; and

in accordance with the determination, modifying, by the processor, content of a page identified by the page source identifier by application of each of the respective one or more transformation rule sets.

27. The method of claim 26, wherein, in response to a page request from a requesting entity, the modifying is performed and the modified page is provided to the requesting entity.

28. The method of claim 27, wherein the processor forwards the page request to a web server, obtains the page from the web server in response to the forwarded page request, and performs the modification to the page obtained from the web server in response to the forwarded request.