Aggregating data from difference sources

• US 20070245118A1
• Filed: 04/12/2006
• Published: 10/18/2007
• Est. Priority Date: 04/12/2006
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Liu, Yue, Wu, Qiang, Suponau, Dzmitry, Girotto, Jay, Wad, Rohit
• Granted Patent: US 7,509,477 B2
• Time in Patent Office: Days
• US Class Current: 711/216
• CPC Class Codes: G06F 16/313 Selection or weighting of terms for indexing
• CPC Class Codes: G06F 16/325 Hash tables
• CPC Class Codes: Y10S 707/99931 Database or file accessing
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching

Abstract

A method and system that aggregates data associated with one or more entities from different data sources are provided. The data sources include documents, web pages, or images that have information about one or more entities. The information is extracted from the data sources based on criteria that define the entities. The extracted information is utilized to generate a hash identifier that corresponds to each entity and one or more storage locations. The one or more storage locations and associated hash identifiers are utilized to store the extracted information corresponding to the entities, and the extracted information for each entity is structured as a virtual page that is stored in an index having references to the data sources.

Claims

1. A method to aggregate data from different sources, the method comprising:

receiving a collection of criteria that represent an entity;

crawling a corpus of documents having data related to the entity;

storing each document or a reference to each document in an index;

extracting information from each document based on the collection of criteria;

generating a hash identifier from the collection of criteria;

creating a storage location for the hash identifier;

storing extracted information in the corresponding storage locations based on the hash identifier; and

incorporating the extracted information into the index.

2. The method according to claim 1, wherein the collection of criteria is defined by an application owner.

3. The method according to claim 1, wherein the hash identifier represents a normalized subset of the collection of criteria.

4. The method according to claim 1, wherein two or more storage locations are reserved when the hash identifier requires large storage areas.

5. The method according to claim 4, wherein the two or more storage locations are associated with the hash identifier.

6. The method according to claim 1, wherein incorporating the extracted information into the index further comprises,

generating a virtual page to organize the extracted information, and

storing each virtual page or a reference to each virtual page in the index.

7. The method according to claim 6, further comprising:

ranking each virtual page based on references for virtual pages or documents in the corpus of documents.

8. A method to generate virtual pages, the method comprising:

receiving a set of criteria;

parsing a corpus of documents based on the set of criteria;

generating a hash identifier based on the set of criteria;

associating the hash identifier with a storage location;

aggregating parsed information based on the hash identifier; and

generating virtual pages for each hash identifier.

9. The method according to claim 8, wherein the set of criteria represents an entity.

10. The method according to claim 9, wherein a subset of the set of criteria associated with the entity is hashed to generate the hash identifier for the entity.

11. The method according to claim 8, wherein the storage location is a storage device.

12. The method according to claim 8, wherein the storage location stores the parsed information.

13. The method according to claim 12, wherein the parsed information includes data extracted from the corpus documents based on the set of criteria.

14. The method according to claim 8, wherein the virtual page is a structured aggregation of the parsed information

15. The method according to claim 14, wherein a markup language is utilized to structure the virtual page.

16. A system to generate virtual pages, the system comprising:

a set of crawlers to gather information about a collection of documents that provides information about one or more entities,

an extracting component to extract data from the collection of documents, wherein predefined criteria associated with the one or more entities specify what information is extracted from the documents;

a hashing component to generate hash identifiers and associate the hash identifiers with storage locations that store information about the one more entities;

a virtual page component to aggregate and structure the extracted information into virtual pages based on the hash identifiers; and

an index to provide access to the virtual pages.

17. The system according to claim 16, wherein updates to information associated with the entities are propagated from the collection of documents to the corresponding virtual pages.

18. The system according to claim 16, wherein the hashing component normalizes a subset of the extracted information

19. The system according to claim 18, wherein the subset of the extracted information is selected from the group consisting of URLs, authors, product names and ISBNs.

20. The system according to claim 16, wherein the virtual page is structured by a markup language and ranked based on the number of references associated with virtual pages or documents from the collection of documents.