Method and apparatus for performing data collection, interpretation and analysis, in an information platform

• US 6,078,924 A
• Filed: 01/30/1998
• Issued: 06/20/2000
• Est. Priority Date: 01/30/1998
• Status: Expired due to Fees

• Current Assignee: EoExchange, Inc.
• Original Assignee: EoExchange, Inc.
• Inventors: Ludin, Stephen L., Putterman, Daniel S., Al Hussein, Hussein K., Hinnant, Michael C., Tejada, Wilfredo M., Shotton, Frederick R., Ainsbury, Robert D., Lahham, Muwaffaq M.
• Primary Examiner(s): Amsbury, Wayne
• Assistant Examiner(s): PARDO, THUY N
• Application Number:
  US09/016,758
• Time in Patent Office: 872 Days
• Field of Search: 709/300, 382/103, 382/173, 707/1, 707/2, 707/3, 707/5, 707/103, 707/104, 707/200, 707/500, 707/101, 707/102
• US Class Current: 1/1
• CPC Class Codes: G06F 16/34 Browsing; Visualisation therefor
• CPC Class Codes: G06F 16/38 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/383 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/93 Document management systems
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: Y10S 707/99942 Manipulating data structure, e.g. compression, compaction, compilation
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface

Abstract

An information platform automates the collection of data, provides a method for organizing the library of information and provides analysis using multiple content-types, thereby providing a user with a market understanding necessary to execute rapid and knowledgeable decision making. The information platform collects and integrates data, observations and intelligence; provides controls for multiple methods of information navigation and analysis; and allows details to be digested in the context of other data, regardless of its type. The information platform is a client/server implementation that is subdivided into four major sections, including: (1) Data Retrieval, which provides a sophisticated catalog for finding internal and external information and collection agents which retrieve specific information without user intervention; (2) Data Classification and Storage which handles the storage of the information once it has been gathered from a source; (3) Information Browsing, Query, Analysis, and Report Creation which provides information browsing, reporting, and analysis tools; and (4) Desktop Integration where the information platform takes information from a wide variety of formats (HTML, text, spreadsheet) and combines them all into a single format (HTML, text, spreadsheet).

Claims

1. An information platform for performing data collection, interpretation and analysis, comprising:

a data retrieval module comprising;

a catalog including a data store for collecting internal and external information from relevant sources;

a geometry recognition module for analyzing multiple sources and recognizing particular patterns within each source; and

a page analyzer for scanning a source document, breaking said source document into blocks and sub-blocks of information, and returning granular pieces for aggregation in said data store;

a data classification and storage module;

an information browsing, query, analysis, and report creation module said information browsing, query, analysis, and report creation module comprising;

a classification subsystem for classifying data according to a specific language, wherein said classification allows said data to be archived and tracked in an object store, and wherein said classification allows said object store to manage complex relationships between a plurality of items whereby, once classified, an item is associated to several other data types by any of one or more characteristics; and

a desktop integration module.

2. The information platform of claim 1, wherein said classification subsystem uses object-oriented meta data technology.

3. The information platform of claim 1, wherein said information browsing, query, analysis, and report creation module comprises:

visualization controls that include any of a time-span slider bar to allow for comparisons of data over time;

a population quadrant to compare data frequency across multiple axis; and

a cross tabular chart to analyze information.

4. An information platform for performing data collection, interpretation and analysis, comprising:

a data retrieval module comprising;

a catalog including a data store for collecting internal and external information from relevant sources; and

a parsing engine for interpreting the format of a stream of information and then returning requested elements to a user by reading a source document and determining said source document page geometry, wherein said parsing engine locates an element by finding a specific string or pattern within a source document, where regular expressions are character strings in which plain text indicates that that text must exist in a target string, and special characters are used to indicate what variability is allowed in said target strings, and wherein said parsing engine performs any of;

a simple content match which looks for a sub-string on regular expression in said source document and returns a primary document element containing the match;

a bounded content match in which the search scope is limited to one contiguous part of said source document; and

a simple/bounded content match in element type which is a bounded or document wide search that look for a sub-string within a specific element type; and

a data classification and storage module; and

an information browsing, query, analysis, and report creation module.

5. The information platform of claim 4, wherein said parsing engine subdivides a source document into elements at include any of a section, a sub-section, a paragraph including any of a sentence, phrase, and word;

a table including any of a title, column, row, column header, spanned column header, header, row header, spanned row header, and cell;

an image, a link, form, and a line, optionally within a range.

6. The information platform of claim 4, wherein said parsing engine identifies an element by either of its ordinal position and by searching for specific content and pattern matching.

7. The information platform of claim 4, wherein said parsing engine parses one contiguous span of information in a source document that is identified by specifying beginning and ending elements, wherein said specified elements can be defined using any coordinate specification techniques.

8. The information platform of claim 7, wherein said parsing engine further comprises:

a parsing subsystem is implemented as an object with interfaces that support any of the following operations;

Set source stream;

Set/get operation bounds (passing begin and ending element coordinates or ids);

Get source stream in specific format (pass desired format type);

Get document tree structure (passing handle to a window tree control);

Get element stream (passing element coordinates, or ID);

Get element ID (passing element coordinates);

Get element count (passing element type);

Get first element stream (passing pattern based element coordinates, optional element type);

Get next element stream;

Get last element stream;

Get table size (passing element coordinates or ID; and

Get header coordinates (passing table coordinates/ID plus col/row of header).

9. The information platform of claim 8, wherein said parsing subsystem further comprises any of the following parsing classes:

CBlock which parses high level blocks with methods to access different kind of information about them and methods to create a fully structured document;

CHtmlParse which has a container of virtual and non-virtual HTML tags and methods to access different kind of structured information about them and where an input stream which is either structured, quasi-structured, or non-structured HTML or Text document is used to create a container of tags and provide methods to access information and to create a new HTML structured document;

CTextParse which takes as an input a segment of a stream document which has no structured information and creates a container of virtual tags and structured information about said stream segment where said container has any of tables, rows, columns, table cells, table headers, paragraphs, section, and subsection headers, and page numbers and where hyper-text links may be added;

CParag which takes a segment of unstructured text that ends with a blank line or end of document and converts it to HTML by deciding if it should be a table, section header, subsection header, paragraph, or page number;

CLine which takes one line of unstructured text and creates a list of phrases in it with information about the phrases and which also provides a line calculator to be able to merge two lines together if needed, or compare lines together to see if they line up in a table, to decide if the line is a header row;

CPhrase which takes one phrase of unstructured text and provides information about it and where it starts and ends in terms of characters and column cells and which also provides a phrase calculator to merge phrases, compare phrases;

CTag which creates an HTML tag structure with information about the attributes of a tag and which is used to create HTML code from the container of the blocks; and

CHtmlTags which provides the latest syntax of the HTML language with its attributes and which is used in the parsing of HTML documents and in the creation of structured HTML streams.

10. A method for performing data collection, interpretation and analysis, in an information platform, said method comprising the steps of:

finding tags starting and ending points in a source document;

identifying tags that have structural information;

providing a text parser for a text segment that has raw text or text with tags that have no structural information;

dividing said segment into paragraphs with said text parser;

parsing each paragraph using a paragraph parser, wherein said paragraph parser studies the lines of a paragraph using a line parser and decides if said paragraph is a regular text paragraph, header, page number, or table, wherein said line parser divides a line into phrases and calls a phrase parser to get information about each phrase, wherein said paragraph parser then uses line and phrase calculators to identify the structure of said paragraph;

wherein if said paragraph is a table, said paragraph parser creates a table;

wherein said paragraph parser generates virtual tags and returns a list of such tags to said text parser;

combining all of said virtual tags from all of said parsed paragraphs with said text parser and then passing said virtual tags back to an HTML parser;

creating one container for all the virtual and non-virtual tags with said HTML parser and enumerating all the tags properly in said source document; and

creating a list of high level blocks from said virtual and non-virtual tags;

wherein a user can retrieve a generated, fully structured document of said source document; and

wherein said user can retrieve information about any block in said source document using regular expressions.

11. An apparatus for performing data collection, interpretation and analysis, in an information platform, comprising:

a parse engine for finding tags starting and ending points in a source document and for identifying tags that have structural information;

a text parser for a text segment that has raw text or text with tags that have no structural information;

said text parser dividing said segment into paragraphs; and

a paragraph parser for parsing each paragraph;

wherein said paragraph parser studies the lines of a paragraph using a line parser and decides if said paragraph is a regular text paragraph, header, page number, or table, wherein said line parser divides a line into phrases;

wherein said paragraph parser calls a phrase parser to get information about each phrase;

wherein said paragraph parser then uses line and phrase calculators to identify the structure of said paragraph;

wherein if said paragraph is a table, said paragraph parser creates a table;

wherein said paragraph parser generates virtual tags and returns a list of such tags to said text parser;

wherein said text parser combines all of said virtual tags from all of said parsed paragraphs and then passes said virtual tags back to an HTML parser;

wherein said HTML parser creates one container for all the virtual and non-virtual tags, enumerates all the tags properly in said source document, and creates a list of high level blocks from said virtual and non-virtual tags;

wherein a user can retrieve a generated, fully structured document of said source document; and

wherein said user can retrieve information about any block in said source document using regular expressions.