System and method for content selection for web page indexing

• US 10,324,984 B2
• Filed: 11/02/2011
• Issued: 06/18/2019
• Est. Priority Date: 05/05/2009
• Status: Active Grant

• Current Assignee: Oracle America, Inc. (Oracle Corporation)
• Original Assignee: Oracle America, Inc. (Oracle Corporation)
• Inventors: Hauser, Robert R.
• Primary Examiner(s): Savla, Arpan P.
• Assistant Examiner(s): Davanlou, Soheila (Gina)
• Application Number:
  US13/287,568
• Publication Number:
  US 20120047431A1
• Time in Patent Office: 2,785 Days
• Field of Search: 707601, 707793, 707705, 707796, 707741, 707748, 707734, 1741, 1748, 1734
• CPC Class Codes: G06F 16/81 Indexing, e.g. XML tags; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

An indexing system for documents such as web pages divides a document into elements, such as document object model elements. User attention data from prior interactions with the document are analyzed to determine those elements of a document that satisfy a threshold requirement of user attention. Elements meeting the user attention threshold requirement are added to a set of indexable content for the document. Furthermore, document sections are determined based on attention data and each section is indexed separately. Indexing is per section and based only on the indexable content, thereby enhancing the index relevance, increasing the efficiency of search engines and reducing spamdexing.

Claims

1. A method of selecting content of a web page for indexing, the method comprising:

transmitting, by an indexer server, a request to retrieve web page content from a web page data store, the request including a uniform resource locator (URL) of a web page, wherein the indexer server comprises one or more computer systems configured to performing indexing of web page documents within web page data stores;

receiving, by the indexer server, the web page content from the web page data store, based on the URL;

requesting, by the indexer server, interaction data associated with the URL from an interaction data store;

receiving, by the indexer server, interaction data from the interaction data store for the URL that identifies one or more attention events, each attention event based on a previous user interface event detected within the web page content of the URL, and each attention event including a location within the web page where the respective user interface event was detected;

dividing, by the indexer server, the web page content into a plurality of non-overlapping content sections;

for each particular content section of the plurality of content sections;

analyzing, by the indexer server, the locations of the one or more attention events in the received interaction data to determine whether the particular content section is associated with one or more of the attention events;

determining whether the particular content section of the web page is found within one or more additional web pages stored within the web page data store;

in response to determining that the particular content section of the web page is found within one or more additional web pages stored within the web page data store;

(a) analyzing the locations of a plurality of additional attention events associated with the one or more additional web pages, to determine whether the particular content section is associated with one or more of the plurality of additional attention events; and

(b) summing the number of attention events associated with the particular content section within the web page, and the number of additional attention events associated with the particular content section within the one or more additional web pages; and

in response to a determination that the sum of the numbers of attention events associated with the particular content section within the web page and within the additional web pages, is greater than a threshold of attention events, adding the particular content section to an indexable content of the web page; and

outputting, by the indexer server, the indexable content corresponding to the web page.

2. The method of claim 1, further comprising:

indexing the web page based on the indexable content corresponding to the web page.

3. The method of claim 1, wherein dividing the web page content into the plurality of non-overlapping content sections comprises dividing the web page into a plurality of document object elements associated with a corresponding document object model (DOM), and wherein the method further comprises:

retrieving a plurality of DOM element identifiers (IDs) for all logged interactions;

analyzing the plurality of DOM element IDs; and

selecting a DOM element of the web page.

4. The method of claim 3, further comprising:

determining whether the selected DOM element is found in an interaction data set of the plurality of DOM element IDs;

selecting a next DOM element for the web page if the selected DOM element is found in the interaction data set of DOM element IDs;

analyzing the interaction data for the DOM element to determine if the DOM element is associated with attention events above the threshold if the interaction data does exist for the selected DOM element; and

adding the DOM element to an indexable content of the web page if the DOM element satisfies the threshold.

5. The method of claim 4, wherein the threshold is based on human classified attention events.

6. The method of claim 1, further comprising:

processing each of the particular content sections separately and indexing the corresponding web page sections into separate section indexes separate from other section indexes.

7. The method of claim 6, further comprising:

identifying an attention history of each of a plurality of elements of each section; and

selecting each of the plurality of elements which meet a predefined attention history requirement; and

adding the plurality of selected elements to indexable content for a respective content section of the web page.

8. An apparatus configured to select content of a web page, the apparatus comprising:

a transmitter configured to transmit a request to retrieve web page content from a web page data store, the request including a uniform resource locator (URL) of a web page; and

a processor configured to

receive the web page content based on the URL;

request interaction data associated with the URL from an interaction data store;

receive interaction data for the URL that identifies one or more attention events, each attention event based on a previous user interface event detected within the web page content of the URL, and each attention event including a location within the web page where the respective user interface event was detected;

divide the web page content into a plurality of non-overlapping content sections;

for each particular content section of the plurality of content sections;

analyzing the locations of the one or more attention events in the received interaction data to determine whether the particular content section is associated with one or more of the attention events;

determining whether the particular content section of the web page is found within one or more additional web pages stored within the web page data store;

in response to determining that the particular content section of the web page is found within one or more additional web pages stored within the web page data store;

(a) analyzing the locations of a plurality of additional attention events associated with the one or more additional web pages, to determine whether the particular content section is associated with one or more of the plurality of additional attention events; and

(b) summing the number of attention events associated with the particular content section within the web page, and the number of additional attention events associated with the particular content section within the one or more additional web pages; and

in response to a determination that the sum of the numbers of attention events associated with the particular content section within the web page and within the additional web pages, is greater than a threshold of attention events, add the particular content section to an indexable content of the web page; and

output the indexable content corresponding to the web page.

9. The apparatus of claim 8, wherein the processor is further configured to:

index the web page based on the indexable content corresponding to the web page.

10. The apparatus of claim 8, wherein dividing the web page content into the plurality of non-overlapping content sections comprises dividing the web page into a plurality of document object elements associated with a corresponding document object model (DOM), and wherein the processor is further configured to:

retrieve a plurality of DOM element identifiers (IDs) for all logged interactions;

analyze the plurality of DOM element IDs; and

select a DOM element of the web page.

11. The apparatus of claim 10, wherein the processor is further configured to:

determine whether the selected DOM element is found in an interaction data set of the plurality of DOM element IDs;

select a next DOM element for the web page if the selected DOM element is found in the interaction data set of DOM element IDs;

analyze the interaction data for the DOM element to determine if the DOM element is associated with attention events above the threshold if the interaction data does exist for the selected DOM element; and

add the DOM element to an indexable content of the web page if the DOM element satisfies the threshold.

12. The apparatus of claim 11, wherein the threshold is based on human classified attention events.

13. The apparatus of claim 8, wherein the processor is further configured to:

process each of the particular content sections separately and index the corresponding web page sections into separate section indexes separate from other section indexes.

14. The apparatus of claim 13, wherein the processor is further configured to:

identify an attention history of each of a plurality of elements of each section; and

select each of the plurality of elements which meet a predefined attention history requirement; and

add the plurality of selected elements to indexable content for a respective content section of the web page.

15. A non-transitory computer readable storage medium storing instructions thereon, that when executed by one or more processors, cause the one or more processors to select content of a web page, by:

transmitting a request to retrieve web page content from a web page data store, the request including a uniform resource locator (URL) of a web page;

receiving the web page content based on the URL;

requesting interaction data associated with the URL from an interaction data store;

receiving interaction data for the URL that identifies one or more attention events, each attention event based on a previous user interface event detected within the web page content of the URL, and each attention event including a location within the web page where the respective user interface event was detected;

dividing the web page content into a plurality of non-overlapping content sections;

for each particular content section of the plurality of content sections;

analyzing the locations of the one or more attention events in the received interaction data to determine whether the particular content section is associated with one or more of the attention events; and

determining whether the particular content section of the web page is found within one or more additional web pages stored within the web page data store;

in response to determining that the particular content section of the web page is found within one or more additional web pages stored within the web page data store;

(a) analyzing the locations of a plurality of additional attention events associated with the one or more additional web pages, to determine whether the particular content section is associated with one or more of the plurality of additional attention events; and

(b) summing the number of attention events associated with the particular content section within the web page, and the number of additional attention events associated with the particular content section within the one or more additional web pages, and

in response to a determination that the sum of the numbers of attention events associated with the particular content section within the web page and within the additional web pages, is greater than a threshold of attention events, adding the particular content section to an indexable content of the web page; and

outputting the indexable content corresponding to the web page.

16. The non-transitory computer readable storage medium of claim 15, wherein the processor is further configured to perform:

indexing the web page based on the indexable content corresponding to the web page.

17. The non-transitory computer readable storage medium of claim 15, wherein dividing the web page content into the plurality of non-overlapping content sections comprises dividing web page document into a plurality of document object elements associated with a corresponding document object model (DOM), and wherein the instructions further causing the one or more processors to perform:

retrieving a plurality of DOM element identifiers (IDs) for all logged interactions;

analyzing the plurality of DOM element IDs; and

selecting a DOM element of the web page.

18. The non-transitory computer readable storage medium of claim 17, wherein the processor is further configured to perform:

determining whether the selected DOM element is found in an interaction data set of the plurality of DOM element IDs;

selecting a next DOM element for the web page if the selected DOM element is found in the interaction data set of DOM element IDs;

analyzing the interaction data for the DOM element to determine if the DOM element is associated with attention events above the threshold if the interaction data does exist for the selected DOM element; and

adding the DOM element to an indexable content of the web page if the DOM element satisfies the threshold.

19. The non-transitory computer readable storage medium of claim 18, wherein the threshold is based on human classified attention events.

20. The non-transitory computer readable storage medium of claim 15, wherein the processor is further configured to perform:

processing each of the particular content sections separately and indexing the corresponding web page sections into separate section indexes separate from other section indexes.