Push-based web site content indexing

• US 20020078134A1
• Filed: 12/18/2000
• Published: 06/20/2002
• Est. Priority Date: 12/18/2000
• Status: Abandoned Application

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Mazza, Samuel, Stone, Alan E.
• Application Number:
  US09/737,948
• Publication Number:
  US 20020078134A1
• Time in Patent Office: Days
• US Class Current: 709/202
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

Various embodiment of a technique for pushed-based indexing of web content are described.

Claims

1. A method comprising:

assigning at least one domain indexer to each of a plurality of web domains;

each of the at least one domain indexers indexing web content of the associated web domain; and

one or more of the domain indexers sending an index for the associated web domain to a predetermined destination.

2. The method of claim 1 and further comprising:

each of the domain indexers detecting changes in the web content of the associated web domain; and

sending the web content changes to the predetermined destination.

3. The method of claim 1 and further comprising using the web indexes for each of the web domains to generate a master web index.

4. The method of claim 1 wherein sending the index comprises sending an index for the associated web domain to an index aggregator so that each index can be used to generate a master index.

5. The method of claim 2 wherein the web content changes are sent as one or more of:

updated or changed web pages; and

differences between old and new web pages.

6. The method of claim 2 wherein detecting changes in the web content of the associated web domain comprises:

comparing a new digest for the web page to an old digest for the web page.

7. The method of claim 2 wherein detecting changes in the web content of the associated web domain comprises:

generating an old digest for a web page;

generating a new digest for a later version of the web page; and

comparing the new digest to the old digest, wherein a difference between the two digests indicates that the web page has changed.

8. A method comprising:

comparing a content indicator of a new version of a file to a content indicator of an older version of the file;

determining whether the content of the file has changed based on the comparing;

sending updated file content information for the file to a predetermined location if the file has changed.

9. The method of claim 8 wherein the comparing comprises comparing an index of a new version of a file to an index of an older version of the file.

10. The method of claim 8 and further comprising generating an updated master index based on updated file content information.

11. The method of claim 8 wherein the sending comprises sending either the new version of the file or differences between new and old versions of the file to a predetermined location if the file has changed.

13. The apparatus of claim 12 wherein the content indicators comprise file digests.

14. The apparatus of claim 12 wherein the content indicator comprises one or more of:

an indication of file size;

a time and/or date of when the file was updated; and

a file digest.

15. The apparatus of claim 12 wherein the updated file content information comprises at least one of:

the new version of the file; and

differences between new and old versions of the file

17. The system of claim 16 wherein the content indicators comprise file digests.

18. The apparatus of claim 16 wherein the content indicator comprises one or more of:

an indication of file size;

a time and/or date of when the file was updated; and

a file digest.

19. The system of claim 16 and further comprising;

an index aggregator to receive the updated file content information from one or more index aggregators; and

an update program to update ate a master web index baseUupdated file content information from the one or more index aggregators.

20. The system of claim 16 wherein each of the web domains comprise one or more of the following:

servers at a physical location;

web content at a physical location;

addressable web content associated with a particular address or Uniform Resource Locator;

web content at a specific web site; and

web content stored within a specific geographic region.

12. An apparatus comprising a domain indexer to compare a content indicator of a new version of a file to a content indicator of an older version of the file, to determine whether the content of the file has changed based on the comparing, and to send updated file content information for the file to a predetermined location if the file has changed.

16. A system comprising a plurality of domain indexers, at least one domain indexer provided for each of a plurality of web domains, each domain indexer to compare a content indicator of a new version of a file to a content indicator of an older version of the file, to determine whether the content of the file has changed based on the comparing, and to send updated file content information for the file to a predetermined location if the file has changed.

21. An apparatus comprising a domain indexer that is assigned to a local web domain to perform web page indexing for the web content of the web domain, to send the web index to a predetermined location or address, to detect changes in the web content at the web domain, and to send the web content changes to the predetermined location or address.

22. The apparatus of claim 21 wherein the web domain comprises all or part of the addressable web content within a particular URL or address.

23. The apparatus of claim 21 wherein the web domain comprises all or part of the web content provided within a specific physical location.

24. The apparatus of claim 21 wherein the domain indexer is located at the same location or region as at least a portion of the web content for the web domain.

25. The apparatus of claim 21 wherein the web domain comprises all or part of the web content provided within a specific physical location.

27. The apparatus of claim 26 wherein the detecting comprises:

comparing a content indicator of a new version of a file to a content indicat an older version of the file; and

determining whether the content of the file has changed based on the comparing.

28. The apparatus of claim 26 wherein the sending comprises sending the web content changes to an index aggregator.

29. The apparatus of claim 26 wherein the detecting comprises comparing a new digest of a plurality of files to a previous digest of the plurality of files.

26. An apparatus comprising a storage readable media having instructions stored thereon, the instructions resulting in the following when executed by a machine that is assigned to a local web domain:

performing web page indexing for the web content of the web domain;

sending the web index to a predetermined location or address;

detecting changes in the web content at the web domain; and

sending the web content changes to the predetermined location or address.