Method and apparatus to retrieve information from a network

• US 6,584,468 B1
• Filed: 09/29/2000
• Issued: 06/24/2003
• Est. Priority Date: 09/29/2000
• Status: Expired due to Term

• Current Assignee: NineSigma Holdings, Inc.
• Original Assignee: NineSigma
• Inventors: Umbel, Christopher M., Lenhart, Joel, Indianer, Evan M., Gabriel, Kaigham J.
• Primary Examiner(s): Metjahic, Safet
• Assistant Examiner(s): NGUYEN, MERILYN P
• Application Number:
  US09/675,594
• Time in Patent Office: 998 Days
• Field of Search: 707/3, 707/5, 707/10, 707/4, 707/100, 707/104.1, 707/1, 709/223, 709/225
• US Class Current: 1/1
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9538 Presentation of query results
• CPC Class Codes: Y10S 707/99935 Query augmenting and refining, e.g. inexact access

Abstract

A method and apparatus to index network information is described. A network is searched for files of information relevant to people and resources in a particular field using a search list of weighted links to said files. The information is parsed into content and additional links to additional files. The content is weighted and copied to memory (such as a database). A determination is made as to whether the additional links are relevant to the people and resources in the given technical field. Those additional links that are relevant are weighted using a predetermined weighting algorithm. The relevant additional weighted links are copied to the search list. This process continues until an ending condition occurs.

Claims

1. A method to index network information, comprising:

(a) establishing a list of parameters to be searched over a network;

(b) establishing a search list of weighted links;

(c) assigning a predetermined initial weight to each parent link in said search list;

(d) searching said network for files of information containing one or more of said parameters using said search list of weighted links to said files;

(e) parsing said information into content and additional links to additional files;

(f) weighting said content;

(g) copying said weighted content into a memory;

(h) comparing each of said additional links to a locally-stored excluded domain file, wherein said excluded domain file contains a list of irrelevant links;

(i) identifying those of said additional links that are found relevant in accordance with said comparison using said excluded domain file;

(j) assigning a predetermined link weight to each said relevant additional link, wherein each said relevant additional link is initially assigned an identical link weight that is different from said initial weight of a corresponding parent link;

(k) adjusting said link weight of each said relevant additional link to be more than, less than or equal to said initial weight of said corresponding parent link depending on at least one of the following;

whether said relevant additional link has been previously processed,

whether said relevant additional link has been previously unprocessed, and

a number of said parameters present in said content corresponding to said relevant additional link;

(l) copying said relevant additional weighted links to said search list; and

(m) performing steps (d)-(l) until an ending condition occurs.

2. The method of claim 1, wherein said ending condition comprises one of said search list becoming empty, a predetermined time limit is met, and receiving a request to terminate.

3. The method of claim 1, wherein said network uses an Internet protocol.

4. The method of claim 1, wherein said network uses at least one of a World Wide Web protocol and a File Transfer Protocol.

5. The method of claim 1, wherein said links each comprise a universal resource locator.

6. The method of claim 1, wherein said files each comprise at least one of a text file, a Portable Document Format (PDF) file, a hypertext markup language (HTML) file, an Extensible HTML (XHTML) file, an Extensible Markup Language (XML) file, an Excel Table Markup Language (XTML) file, a Virtual Reality Modeling Language (VRML) and a VHML file.

7. The method of claim 1, wherein said content comprises at least one of a group comprising text, images, data files, executable files, data, charts, tables, meta tags, program instructions and graphics.

8. The method of claim 1, wherein searching said network comprises:

selecting a weighted link from said search list based on a predetermined ranking of each said weighted link within said search list;

retrieving a selected file of information corresponding to said selected weighted link; and

removing said selected weighted link from said search list.

9. The method of claim 8, wherein said retrieving said selected file comprises:

determining whether said searching is for a master build process;

retrieving said selected file of information from said network if said searching is for said master build process;

determining whether said selected file of information has been previously indexed into said memory if said searching is not for said master build process;

retrieving said selected file of information from said memory if said selected file of information has been previously indexed into said memory; and

retrieving said selected file of information from said network if said selected file of information has not been previously indexed into said memory.

10. The method of claim 8, wherein said selecting comprises:

ranking each weighted link from lowest weight to highest weight; and

selecting a weighted link with a lowest weight.

11. The method of claim 8, wherein said selecting comprises:

ranking each weighted link from highest weight to lowest weight; and

selecting a weighted link with a highest weight.

12. The method of claim 1, further comprising copying said additional links to a parent-child table.

13. The method of claim 1, wherein said parameters represents at least one of the following categories of information:

text;

images;

data;

meta tags;

program instructions; and

graphics.

14. A machine-readable medium whose contents cause a computer system to index network information by performing the following:

(a) establishing a list of parameters to be searched over a network;

(b) establishing a search list of weighted links;

(c) assigning a predetermined initial weight to each parent link in said search list;

(d) searching said network for files of information containing one or more of said parameters using said search list of weighted links to said files;

(e) parsing said information into content and additional links to additional files;

(f) weighting said content;

(g) copying said weighted content into a memory;

(h) comparing each of said additional links to a locally-stored excluded domain file, wherein said excluded domain file contains a list of irrelevant links;

(i) identifying those of said additional links that are found relevant in accordance with said comparison using said excluded domain file;

(j) assigning a predetermined link weight to each said relevant additional link, wherein each said relevant additional link is initially assigned an identical link weight that is different from said initial weight of a corresponding parent link;

(k) adjusting said link weight of each said relevant additional link to be more than, less than or equal to said initial weight of said corresponding parent link depending on at least one of the following;

whether said relevant additional link has been previously processed,

whether said relevant additional link has been previously unprocessed, and

a number of said parameters present in said content corresponding to said relevant additional link;

(l) copying said relevant additional weighted links to said search list; and

(m) performing steps (d)-(l) until an ending condition occurs.

15. The machine-readable medium of claim 14, wherein said ending condition comprises one of said search list becoming empty, a predetermined time limit is met, and receiving a request to terminate.

16. The machine-readable medium of claim 14, wherein said network uses an Internet protocol.

17. The machine-readable medium of claim 14, wherein said network uses at least one of a World Wide Web protocol and a File Transfer Protocol.

18. The machine-readable medium of claim 14, wherein said links each comprise a universal resource locator.

19. The machine-readable medium of claim 14, wherein said files each comprise at least one of a text file, a Portable Document Format (PDF) file, a hypertext markup language (HTML) file, an Extensible HTML (XHTML) file, an Extensible Markup Language (XML) file, an Excel Table Markup Language (XTML) file, a Virtual Reality Modeling Language (VRML) and a VHML file.

20. The machine-readable medium of claim 14, wherein said content comprises at least one of a group comprising text, images, data files, executable files, data, charts, tables, meta tags, program instructions and graphics.

21. The machine-readable medium of claim 14, wherein searching said network comprises:

selecting a weighted link from said search list based on a predetermined ranking of each said weighted link within said search list;

retrieving a selected file of information corresponding to said selected weighted link; and

removing said selected weighted link from said search list.

22. The machine-readable medium of claim 21, wherein said retrieving said selected file comprises:

determining whether said searching is for a master build process;

retrieving said selected file of information from said network if said searching is for said master build process;

determining whether said selected file of information has been previously indexed into said memory if said searching is not for said master build process;

retrieving said selected file of information from said memory if said selected file of information has been previously indexed into said memory; and

retrieving said selected file of information from said network if said selected file of information has not been previously indexed into said memory.

23. The machine-readable medium of claim 21, wherein said selecting comprises:

ranking each weighted link from lowest weight to highest weight; and

selecting a weighted link with a lowest weight.

24. The machine-readable medium of claim 21, wherein said selecting comprises:

ranking each weighted link from highest weight to lowest weight; and

selecting a weighted link with a highest weight.

25. The machine-readable medium of claim 14, further comprising copying said additional links to a parent-child table.

26. The machine-readable medium of claim 14, wherein said parameters represent at least one of the following categories of information:

text;

images;

data;

meta tags;

program instructions; and

graphics.

27. An apparatus to index network information, comprising:

(a) means for establishing a list of parameters to be searched over a network;

(b) means for establishing a search list of weighted links;

(c) means for assigning a predetermined initial weight to each parent link in said search list;

(d) means for searching said network for files of information containing one or more of said parameters using said search list of weighted links to said files;

(e) means for parsing said information into content and additional links to additional files;

(f) means for weighting said content;

(g) means for copying said weighted content into a memory;

(h) means for comparing each of said additional links to a locally-stored excluded domain file, wherein said excluded domain file contains a list of irrelevant links;

(i) means for identifying those of said additional links that are found relevant in accordance with said comparison using said excluded domain file;

(j) means for assigning a predetermined link weight to each said relevant additional link, wherein each said relevant additional link is initially assigned an identical link weight that is different from said initial weight of a corresponding parent link;

(k) means for adjusting said link weight of each said relevant additional link to be more than, less than or equal to said initial weight of said corresponding parent link depending on at least one of the following;

whether said relevant additional link has been previously processed,

whether said relevant additional link has been previously unprocessed, and

a number of said parameters present in said content corresponding to said relevant additional link;

(l) means for copying said relevant additional weighted links to said search list; and

(m) means for performing functions in (d)-(l) until an ending condition occurs.

28. The apparatus of claim 27, wherein said parameters represent at least one of the following categories of information:

text;

images;

data;

meta tags;

program instructions; and

graphics.