System and method for managing knowledge

• US 7,685,083 B2
• Filed: 07/10/2006
• Issued: 03/23/2010
• Est. Priority Date: 02/01/2002
• Status: Active Grant

• Current Assignee: John Fairweather
• Original Assignee: John Fairweather
• Inventors: Fairweather, John
• Primary Examiner(s): Holmes; Michael B
• Application Number:
  US11/484,220
• Publication Number:
  US 20070112714A1
• Time in Patent Office: 1,352 Days
• Field of Search: 706/46, 706/45
• US Class Current: 706/45
• CPC Class Codes: G06F 8/427 Parsing
• CPC Class Codes: G06F 9/4493 Object persistence
• CPC Class Codes: G06K 13/0825 the ejection arrangement being of the push-push kind
• CPC Class Codes: Y10S 707/913 Multimedia
• CPC Class Codes: Y10S 707/966 Distributed
• CPC Class Codes: Y10S 707/99931 Database or file accessing
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching
• CPC Class Codes: Y10S 707/99942 Manipulating data structure, e.g. compression, compaction, compilation

Abstract

An intelligence system is provided that is comprised of several basic components: a system for converting incoming unstructured data into a well described normalized form supported by a dedicated ‘mining’ language tied intimately to a system ontology; a system for accessing and manipulating data held in memory or in persistent storage in its normalized binary form; an ‘ontology’ that represents and contains the items and fields necessary for the target system to perform its function; a memory system tied to the ontology; a memory management system for splitting incoming data into those portions to be directed to each container; a query system for querying each container to retrieve portions of composite objects; a UI to display and interact with data within the system; a memory system that forms collections of datums and enables manipulation and exchange of these collections both within the local machine as well as across the network.

Claims

1. A method for facilitating meta-analysis of data captured for intelligence purposes using a computer network and implemented as an unconstrained system, the method comprising the steps of:

(a) establishing a distributed acquisition server architecture within the computer network responsive to a data-flow driven environment;

(b) sampling a plurality of streams of unstructured data by said distributed acquisition server architecture;

(c) converting said plurality of streams of unstructured data into a well described normalized form of binary data via a dedicated mining language tied to a current system ontology;

(d) storing said converted binary data in a memory system tied to said current system ontology within said computer network, wherein said memory system defines a plurality of persistent storage containers required to contain said converted binary data;

(e) directing said storing step with a memory management system which splits said converted binary data into an appropriate one of said plurality of persistent storage containers;

(f) executing one or more control and/or data-flow based programs, called widgets, on said converted binary data stored in said plurality of persistent storage containers, wherein execution of said one or more widgets begins when a matching set of data objects or tokens from said converted binary data appear on an input data-flow pin of said one or more widgets;

(g) producing a set of resultant data tokens on an output data-flow pin of said one or more widgets, wherein said set of resultant data tokens become part of said data-flow driven environment in said persistent storage containers or in a memory of a computer within the computer network;

(h) querying a registered search capability of one or more said plurality of persistent storage containers producing a list of hits;

(i) querying said list of hits with Boolean and other operators to specify logical combinations of said list of hits;

(j) displaying and interacting with said plurality of streams of unstructured data, said list of hits, and said logical combinations of said list of hits through a user interface on a display device within the computer network;

(k) forming collections of datums from said logical combinations of said list of hits through a memory collections system that forms and enables manipulation and exchange of said collections of datums both within a local computer as well as across the computer network;

(l) delivering said collections of datums for meta-analysis to a user accessing the computer network through said user interface; and

(m) based upon said meta-analysis by said user, revising said querying steps (h) and (i) repeating steps (j), (k) and (l).

2. The method according to claim 1 wherein said establishing a distributed acquisition server architecture step (a) further comprises the steps of:

establishing one or more servers;

logically connecting to the one or more servers a mass storage system;

logically connecting to the one or more servers a types system for defining data types at a binary level; and

logically connecting to the one or more servers a query system for executing data queries on the one or more servers mapped to the data type being queried.

3. The method according to claim 1 wherein said establishing a data-flow driven environment step (a) further comprises the steps of:

establishing a data-flow based scheduling environment for managing an execution of one or more control-flow based functional building blocks;

establishing a visual programming environment to build and control a flow of data collections between one or more of the building blocks within the scheduling environment;

establishing a pin-based application programming interface for accessing the contents from an executing code within the one or more building blocks through one or more widget input pins; and

establishing a strongly-typed run-time discoverable types systems for defining types of the flow of data collections presented to the one or more widget input pins at run-time.

4. The method according to claim 1 wherein said converting step (c) via said dedicated mining language further comprises the steps of:

receiving a first source data for mining by the computer network;

parsing the first source data by the computer network;

creating, as a result of the parsing step, a first collection of records conformed to a structured target data model described by an ontology description language,

storing the first collection of records conformed to the structured target data model in the memory system; and

retrieving the first collection of records for the further processing by the computer network.

5. The method according to claim 1 wherein said converting step (c) via said current system ontology further comprises the steps of:

establishing an ontology description language, or ODL, wherein the ODL is derived by extensions to a standard computer programming base language as implemented using a types system;

registering a plurality of data containing with a collections system via a plug-in registry;

automatically generating and handling, with a database creation engine, one or more persistent storage tables necessary in the data containers that have been registered with the collections system, wherein the database creation engine uses specifications given in the ODL; and

automatically generating, with a user interface creation engine using the ODL, a user interface that permits display, interaction with, and querying of the data residing in the persistent storage containers.

6. The method according to claim 1 wherein said converting step (c) further comprises:

processing said plurality of streams of unstructured data with a two-phase lexical analyzer yielding a plurality of tokens, wherein said processing by the two-phase lexical analyzer further comprises the steps of;

creating a first table in the memory, wherein the first table describes one or more single ocharacter transitions using records of a first type;

creating a second table in the memory, wherein the second table is an ordered series of records of a second type;

receiving a text input into the lexical analyzer;

searching the records in the first table for a matching record against each successive character of the text input;

if the matching record for the text input is found in the first table, outputting a token associated with the matching record;

responsive to a failure to find the matching record in the first table, searching the records in the second table from the beginning for the matching record against the each successive character of the text input, wherein the matching record is found when a current state of the lexical analyzer lies between an upper state bound and a lower state bound and the each successive character of the text input lies between an upper character bound and a lower character bound as specified in each said record being searched in the second table; and

if the matching record is found in the second table, assigning a current state of the lexical analyzer a value of an ending state field of the matching record.

7. The method according to claim 6 wherein said processing step further comprises:

parsing said plurality of tokens through a predictive parser, wherein said parsing by the predictive parser further comprises the steps of;

specifying a specific source language syntax to be parsed to the predictive parser at run-time via a parser specification using a specification language describing not only parser productions in response to input tokens and syntax, but also one or more registered plug-in operators to be called at specified points in the parsing process determined by when said one or more registered plug-in operators are popped off a parser stack associated with the predictive parser;

converting the parser specification into one or more parser tables to drive operation of the predictive parser that is otherwise unmodified and source language independent;

calling by the predictive parser a registered resolver in order to obtain a series of tokens from an input token stream, passing a ‘

no action’

mode parameter to indicate an input token request wherein the registered resolver may at any time it is called (regardless of the mode parameter), choose to alter either a subseiuent token stream returned, a state of the parser stack, or a state of an evaluation stack associated with the predictive parser; and

when a one of the series of tokens has a value within a first defined range, pushing by the predictive parser the one of said series of tokens onto said evaluation stack as an un-resolved symbol referencing a text string of the one of the series of tokens.

8. The method according to claim 1 wherein said storing said converted binary data in a memory system step (d) further comprises steps of:

obtaining a reference to a block of physical memory from a standard operating system supplied heap allocation facility or other standard memory allocation scheme;

creating one or more memory structures to be stored within the block of physical memory, the memory structures each having a space allocated for a header and a data portion;

creating the header for said memory structures, wherein the header includes a field for linking to a next said memory structure in the block of physical memory based on a relative memory offset between a referencing header and a referenced header within the block of physical memory, and further wherein the header includes a field for identifying additional data structures unique to a particular type of said memory structure; and

storing the header within a corresponding said memory structure.

9. The method according to claim 1 wherein said directing said storing step with a memory management system step (e) further comprises the steps of:

populating a plurality of databases with a binary type and field descriptions;

generating type databases with a run-time modifiable type compiler that is capable of either explicit API calls or by compilation of unmodified header files or individual type definitions in a standard programming language;

reading and writing the types with a complete Application Programming Interface suite for accessing the type information as well as full support for type relationships and inheritance, and type fields, given knowledge of a unique numeric type ID and a field name/path; and

converting the type names to unique type IDs with a hashing process which may also incorporate a number of logical flags relating to the nature of the type.

10. The method according to claim 1 wherein said displaying step (j) through said user interface further comprises the step of:

translating in real-time tokens floin a base language to a foreign language, without requiring the tokens to be obtained through specialized Application Programming Interfaces (“

APIs”

) from localized resources, by modifying the standard rendering chain to intercept all rendering calls for the tokens in the base language and invoking processing instructions necessary to perform the mapping to the foreign language.

11. The method according to claim 10 further comprising:

providing a dynamic hyper-linking architecture under the control of said user within said user interface, wherein said providing the dynamic hyper-linking architecture step further comprises the steps of;

providing a threaded environment;

associating arbitrary data with threads in the threaded environment, wherein the arbitrary data is function registries;

hierarchically nesting the thread contexts with corresponding user interface context relationships;

passing ‘

events’

containing messages between the threads;

invoking transparently certain environment supplied events; and

looking-up the threads based on a unique thread ID,

wherein the dynamic hyper-linking architecture uses both the threaded environment and symbolic functions to dynamically create links to data and functions that are displayed and/or executed responsive to user selection of a link.

12. The method according to claim 1 wherein said forming step (k) through said memory collections system further comprises the steps of:

instantiating arbitrarily complex structures in a ‘

flat’

data model within a single memory allocation;

defining and accessing binary strongly-typed data in a run-time type system;

encoding information in a set of ‘

containers’

in a memory resident form, a file-based form, and a server-based form;

intepreting and executing all necessary collection manipulations remotely in a client/server environment tied to a types system;

providing a basic aggregation structure having at a minimum a ‘

parent,’

‘

child,’ and

‘

sibling’

links or equivalents; and

attaching strongly typed data to a data attachment structure whose size may vary and which is associated with and possibly identical to a containing aggregation node in the collection.

13. A system for facilitating meta-analysis of data captured for intelligence purposes within a computer network, which is implemented as an unconstrained system, the system comprising:

a distributed acquisition server architecture within the computer network responsive to a data-flow driven environment;

a plurality of streams of unstructured data which are sampled by said distributed acquisition server architecture;

a dedicated mining language tied to a current system ontology for converting said plurality of streams of unstructured data into a well described normalized form of binary data;

a memory system tied to said current system ontology within said computer network for storing said converted binary data, wherein said memory system defines a plurality of persistent storage containers required to contain said converted binary data;

a memory management system for splitting and directing said converted binary data into an appropriate one of said plurality of persistent storage containers;

one or more control and/or data-flow based programs, called widgets, each said widget having at least one input data-flow pin and at least one output data-flow pin, wherein said one or more widgets are executed on said converted binary data stored in said plurality of persistent storage containers when a matching set of data objects or tokens from said converted binary data appear on said at least one input data-flow pin of said one or more widgets;

a set of resultant data tokens produced on said output data-flow pins of said one or more widgets, wherein said set of resultant data tokens become part of said data-flow driven environment in said persistent storage containers or in a memory of a computer within the computer network;

a user interface having a lower querying layer and an upper querying layer, wherein said lower querying layer queries one or more registered search capability for each of said plurality of persistent storage containers which produces a list of hits, and further wherein said upper querying layer queries said list of hits with Boolean and other operators to specify logical combinations of said list of hits;

a display device within the computer network for displaying and interacting with said plurality of streams of unstructured data, said list of hits, and said logical combinations of said list of hits through said user interface; and

a memory collections system that forms collections of datums from said logical combinations of said list of hits and enables manipulation and exchange of said collections of datums both within a local computer as well as across the computer network, wherein a user accesses through said user interface said collections of datums for meta-analysis, and based upon said meta-analysis by said user, said user can revise said queries to refine said collections of datums.

14. The system according to claim 13, wherein said distributed acquisition server architecture further comprises:

one or more servers;

a mass storage system logically connected to the one or more servers;

a types system logically connected to the one or more servers for defining data types at a binary level; and

a query system logically connected to the one or more servers for executing data queries on the one or more servers mapped to the data type being queried.

15. The system according to claim 13, wherein said data-flow driven environment further comprises:

a data-flow based scheduling environment for managing an execution of one or more control-flow based functional building blocks;

a visual programming environment to build and control a flow of data collections between one or more of the building blocks within the scheduling environment;

a pin-based application programming interface for accessing the contents from an executing code within the one or more building blocks through one or more widget input pins; and

a strongly-typed run-time discoverable types system for defining types of the flow of data collections presented to the one or more widget input pins at run-time.

16. The system according to claim 13, wherein said dedicated mining language further comprises:

a first source data for mining by the computer network;

a parser for parsing the first source data by the computer network, wherein the parser further comprises an outer parser having an embedded inner parser; and

a first collection of records created by the parser that are conformed to a structured target data model described by an ontology description language;

wherein the first collection of records conformed to the structured target data model are stored in the memory system and may be retrieved for further processing by the computer net work.

17. The system according to claim 13, wherein said system ontology further comprises:

an ontology description language, or ODL, wherein the ODL is derived by extensions to a standard computer programming base language as implemented using a types system;

a plurality of data containers with a collections system registered via a plug-in registry;

a database creation engine wherein said database creation engine uses specifications given in the ODL to automatically generate and handle one or more persistent storage tables necessary in the data containers that have been registered with the collections system; and

a user interface creation engine, wherein the user interface creation engine uses the ODL to automatically generate a user interface that permits display, interaction with, and querying of the data residing in the persistent storage in the one or more storage devices.

18. The system according to claim 13 further comprising:

a two-phase lexical analyzer for processing said plurality of streams of unstructured data yielding a plurality of tokens, wherein said two-phase lexical analyzer further comprises;

a first table created in the memory, wherein the first table describes one or more single character transitions using records of a first type;

a second table created in the memory, wherein the second table is an ordered series of records of a second type; and

a text input received into the lexical analyzer;

wherein the records in the first table are searched for a matching record against each successive character of the text input, and if the matching record for the text input is found in the first table, a token associated with the matching record is output, and further wherein if a matching record in the first table is not found, the records in the second table are searched from the beginning for the matching record against the each successive character of the text input, wherein the matdhing record is found when a current state of the lexical analyzer lies between an upper state bound and a lower state bound and the each successive character of the text input lies between an upper character bound and a lower character bound as specified in each said record being searched in the second table, and if the matching record is found in the second table, a current state of the lexical analyzer is assigned a value of an ending state field of the matching record.

19. The system according to claim 18 further comprising:

a predictive parser for parsing said plurality of tokens, wherein said predictive parser further comprises;

an application programming interface, logically connected to the predictive parser, which permits registration and use of one or more plug-ins and one or more resolvers;

a means for choosing and specifying a source grammar to be parsed by the predictive parser, converting the source grammar to an equivalent one or more parsing tables, and logically connecting the one or more parsing tables to the predictive parser for parsing the complex language input;

a means for invoking by the predictive parser the one or more resolvers such that the complex language input is passed by the predictive parser through the one or more resolvers for tokenization into a token stream; and

a means for invoking by the predictive parser the one or more plug-ins, which are logically connected to the one or more resolvers, wherein the one or more plug-ins interpret any reverse-polish operators embedded in the specified source grammar when exposed on a parser stack by said predictive parser.

20. The system according to claim 13, wherein said memory system further comprises:

a reference to a block of physical memory from a standard operating system supplied heap allocation facility or other standard memory allocation scheme;

one or more memory structures stored within the block of physical memory, the memory structures each having a space allocated for a header and a data portion; and

one or more fields within the header for linking to the one or more memory structures that are related to a first memory structure within the block of physical memory, wherein the one or more fields are based on a relative memory offset between a referencing header and a referenced header within the block of physical memory.

21. The system according to claim 13, wherein said memory management system further comprises:

a plurality of databases populated with a binary type and field descriptions;

a compiler capable of accessing the one or more custom binary type and field description databases at run-time and generating or modifying the one or more custom binary type and field description databases;

a complete Application Programming Interface suite for reading and writing the types as well as full support for type relationships and inheritance, and type fields, given knowledge of a unique numeric type ID and a field name/path; and

a hashing process, wherein the hashing process converts type names to unique numeric type IDs which may also incorporate a number of logical flags relating to the nature of the type.

22. The system according to claim 13, wherein said user interface translates in real-time tokens from a base language to a foreign language, without requiring the tokens to be obtained through specialized Application Programming Interfaces (“

APIs”

) from localized resources, by modifying the standard rendering chain to intercept all rendering calls for the tokens in the base language and invoking processing instructions necessary to perform the mapping to the foreign language.

23. The system according to claim 22 further comprising:

a dynamic hyper-linking architecture within said user interface, wherein said dynamic hyper-linking architecture further comprises;

a threaded environment, wherein the threaded environment associates an arbitrary data with one or more threads, which when associated with a User Interface (UI) context are identified by unique thread identification numbers (IDs), and includes a hierarchical nesting of thread contexts with one or more corresponding UI context relationships;

wherein ‘

events’

containing messages are passed between the threads, and certain environment supplied events are invoked transparently. wherein the threads are looked-up based on a unique thread ID and wherein the dynamic hyper-linking architecture uses both the threaded environment and symbolic functions to dynamically create links to data and functions that are displayed and/or executed responsive to user selection of a link.

24. The system according to claim 13, wherein said memory collections system further comprises:

a ‘

flat’

data model for instantiating arbitrarily complex structures within a single memory allocation;

a run-time type system for defining and accessing binary strongly-typed data;

a set of ‘

containers’

for encoding information in a memory resident form, a file-based form, and a server-based form;

;

a client/server environment tied to a types system for interpreting and executing all necessary collection manipulations remotely;

a basic aggregation structure having at a minimum a ‘

parent,’

‘

child,’ and

‘

sibling’

links or equivalents; and

a data attachment structure for attaching strongly typed data whose size may vary and which is associated with and possibly identical to a containing aggregation node in the collection.