System and method for efficient control and capture of dynamic database content

• US 7,676,555 B2
• Filed: 12/04/2006
• Issued: 03/09/2010
• Est. Priority Date: 07/24/2001
• Status: Expired due to Fees

• Current Assignee: Brightplanet Corp II, Inc.
• Original Assignee: Brightplanet Corp. (Brightplanet Corp II, Inc.)
• Inventors: Tiahrt, Thomas W., Bushee, William J.
• Primary Examiner(s): Meky; Moustafa M
• Application Number:
  US11/633,673
• Publication Number:
  US 20070192442A1
• Time in Patent Office: 1,191 Days
• Field of Search: 709200-203, 709217-227, 709228-232, 709 6- 8, 707 2- 5, 707 9- 10, 707/102, 707 6- 8
• US Class Current: 709/219
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/953 Querying, e.g. by the use of web search engines
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching

Abstract

A system and method for efficient control and capture of dynamic database content for rapidly providing a user with a highly relevant collection of documents related to a query. The system and method for efficient control and capture of dynamic database content includes a computer system with a storage means for facilitating the retention and recall of dynamic database content and a communications means for facilitating bi-directional communication of the computer system with local or distributed networks; an executory module operationally coupled to the computer system for controlling the storage means and the communications means as well as directing the system for the efficient control and capture of dynamic database content to a plurality of pre-selected Internet sites; and a capture module which is in communication with the executory module and facilitates selection of the plurality of Internet sites associated with a query submitted by a user of the system.

Claims

1. A method for efficiently controlling and capturing dynamic database content comprising:

obtaining a query from a user, said query to be used to search for relevant information in a specified area;

preprocessing said query;

routing said query to a plurality of databases, each one of said databases using said query to locate relevant information for the user;

obtaining results from each one of said plurality of databases;

processing said results; and

storing said results for use and analysis by the user;

wherein said preprocessing further comprises;

invoking a thread manager to facilitate the submission of said query to said plurality of databases, said thread manager coordinating communication between an executory module and a capture module, said executory module controlling bi-directional communications with said databases;

obtaining a thread count parameter for limiting the maximum number of coexisting threads allowed by said thread manager;

establishing a plurality of threads between said capture module and said executory module such than a plurality of coexisting sequential control flows are created between said capture module and said executory module;

establishing a plurality of sockets for said executory module to generate bi-directional connectivity between said executory module and said plurality of appropriate databases;

assigning each one of said plurality of databases to a specified socket;

obtaining a query from a queue;

routing said query from said capture module through an associated one of said plurality of threads to said executory module and from said executory module through an associated one of said plurality of sockets to an associated one of said plurality of appropriate databases;

waiting for a response from said one of said plurality of appropriate databases; and

capturing results returned by each one of said plurality of databases for processing and storage.

2. A method for efficiently controlling and capturing dynamic database content comprising:

obtaining a query from a user, said query to be used to search for relevant information in a specified area;

preprocessing said query;

routing said query to a plurality of databases, each one of said databases using said query to locate relevant information for the user;

obtaining results from each one of said plurality of databases;

processing said results; and

storing said results for use and analysis by the user;

wherein said obtaining results further comprises;

capturing links to relevant documents returned from one of said databases via a results page, said links providing a URL path to obtain each one of said documents;

examining each one of said URLs to determine if a redirection is used to reach an ultimate location for anyone of said documents;

following each one of said URLs to determine if subsequent redirects are used;

capturing one of said documents from said database;

capturing an ultimate URL portion associated with said document;

capturing a source URL portion identifying which one of said databases was a referring source of said document; and

deleting any intervening URLs between said source url portion and said ultimate URL portion.

3. A method for efficiently controlling and capturing dynamic database content comprising:

obtaining a query from a user, said query to be used to search for relevant information in a specified area;

preprocessing said query;

routing said query to a plurality of databases, each one of said databases using said query to locate relevant information for the user;

obtaining results from each one of said plurality of databases;

processing said results; and

storing said results for use and analysis by the user;

wherein said obtaining results further comprises;

obtaining an aging parameter from the user, said aging parameter defining the maximum age of a document to be considered current;

checking each URL for a document returned as a result by one of said databases with documents stored in a document storage and retrieval portion, said document storage and retrieval portion storing all documents captured during a search;

determining if said document is located in said document storage and retrieval portion;

downloading said document if said document is not located in said document storage and retrieval portion;

examining record related information for said document if said document is located within said document storage and retrieval portion;

verifying an age of said document from said record related information;

comparing said age of said document with said aging parameter;

querying a server associated with said document for the existence of modifications to said document if said document IS older than the maximum age determined by said aging

parameter; and

downloading said document if said document has been modified since the last time the document was downloaded.