System and method for collection, retrieval, and distribution of data

• US 8,819,080 B2
• Filed: 06/13/2007
• Issued: 08/26/2014
• Est. Priority Date: 06/13/2007
• Status: Active Grant

• Current Assignee: The Boeing Co.
• Original Assignee: The Boeing Co.
• Inventors: Modad, Chad, O'Connell, Don M.
• Primary Examiner(s): Btit, Jacob F
• Assistant Examiner(s): TRAN, BAO G
• Application Number:
  US11/762,437
• Publication Number:
  US 20080313207A1
• Time in Patent Office: 2,631 Days
• Field of Search: 707/102, 707/101, 707/3, 707/100, 707/201, 707/999.001, 707/999, 707/999.003, 707/999.201, 707/999.1, 707/661, 707/803, 707/809, 717136-161, 382/100, 348/207.1, 709/204, 709/206, 709/221, 709/225
• US Class Current: 707/803
• CPC Class Codes: G06F 16/24573 using data annotations, e.g. user-defined metadata

Abstract

A system may include a data collector to collect data from at least one data source and to tag each group of collected data with a metadata tag. The metadata tag may include information about the collected data. The system may also include a metadata catalogue for containing a plurality of metadata tags and to allow a search for existence and a location on a network of any data corresponding to the search.

Claims

1. A system for collection, retrieval, and distribution of data, comprising:

a computing device comprising a processor and a memory;

a data collector embodied in the computing device to collect data from at least one data source and to tag each group of collected data with an associated metadata tag, wherein the metadata tag includes information about the collected data;

an object distribution logic module associated with the data collector, the object distribution logic module comprising one or more caches to store data and a distributed caching strategy structure, the distributed caching strategy structure including a cache collapsing module to only send updated portions of data rather than an entire data set in accordance with data consumer submitted information and a store and forward feature to dynamically reduce flow of the data to data consumers;

at least one other computing device;

at least one distributed metadata catalogue embodied in the at least one other computing device for containing a plurality of metadata tags, wherein the metadata tags without any associated groups of collected data are stored on the at least one distributed metadata catalogue, wherein only metadata indicating the existence and location of a requested group of data on a network is sent to a requester in response to a data request corresponding to the requested group of data associated with the metadata tag to avoid sending the requested group of data over the network until needed; and

wherein the at least one distributed metadata catalogue comprises a metadata catalogue query services module configured to receive a query from the data consumer, wherein the query is received by a local metadata catalogue query services module of the at least one distributed metadata catalogue that is a closest distance from the data consumer, the local catalogue query services module is configured to determine if a scope of the query is set to local, the local catalogue query services module responding to the query in response to the scope of the query being set to local and the local catalogue query services module passing the query to another catalogue query services module of another distributed metadata catalogue capable of responding to the query.

2. The system of claim 1, further comprising a collection/tagging policy associated with the data collector for dynamically configuring a behavior of the data collector in collecting data, tagging the collected data and generating the metadata tags.

3. The system of claim 2, wherein the behavior of the data collector comprises:

how often data is collected and stored;

how much data is collected and stored;

a format to be used for the metadata tags;

how often metadata tags are generated;

how specific fields in the metadata tag format will be filled in; and

assignment of a significance or priority to the data collected.

4. The system of claim 2, further comprising a policy distribution manager to set the collection/tagging policy.

5. The system of claim 1, wherein the information about the collected data included in the metadata tag comprises at least one of a description of the data and a summary of the data.

6. The system of claim 1, wherein the metadata tag comprises information for at least one of routing data to a consumer of the data and caching the data according to predetermined criteria.

7. The system of claim 1, further comprising a data store associated with the data collector at a data collection node to store the collected data, wherein the collected data is only moved in accordance with a distributed caching policy.

8. The system of claim 1, further comprising a distributed data collection structure including a plurality of data collection nodes, each data collection node comprising a data collector.

9. The system of claim 1, further comprising:

a metadata node to respond to a metadata catalogue query and to distribute metadata catalogue information to the data consumers; and

a distributed caching policy to configure a behavior of the metadata node.

10. The system of claim 1, wherein the network comprises a delay and fault tolerant network to distribute data to a data consumer, wherein the network is delay tolerant by storing and prioritizing the data to reduce an impact of any delivery delays.

11. The system of claim 1, wherein the metadata catalogue comprises:

a metadata catalogue data store for storing the plurality of metadata tags.

12. The system of claim 1, further comprising a metadata entry/update services module to receive any new and updated metadata tags from the data collector.

13. The system of claim 1, further comprising a data retrieval service module to distribute data collected by the data collector to a data consumer based on a distribution/caching policy.

14. The system of claim 1, further comprising a data streamer to publish selected data collected by the data collector to a data consumer that has subscribed to receive the selected data based on a distribution/caching policy.

15. The system of claim 1, further comprising a mechanism to cache data collected by the data collector and to transport selected data across the network in response to a request, wherein the selected data is cached at a new system location to service subsequent requests.

16. The system of claim 1, further comprising a network resource manager to provide data about current loading and status of the network and for managing use of bandwidth on the network to transfer data from a data collection node to a data consumption node.

17. The system of claim 1, further comprising a network access mechanism for accessing the network and transmitting data across the network, wherein the network access mechanism is capable of being plugged-in between an application and the network to provide seamless network management.

18. The system of claim 1, further comprising a prioritizer to reorder data collected by the data collector and stored in a cache so that data with a higher priority is transmitted first.

19. The system of claim 1, further comprising a store and forward mechanism to hold data in a cache until the network for transmitting the data becomes available in response to a loss of network connectivity and to remove data from the cache that is beyond a predetermined age.

20. The system of claim 1, further comprising a mechanism to adapt to changing network bandwidths using a set of configuration parameters.

21. The system of claim 1, further comprising an event combiner to combine event updates on a single event for transmission and to remove older and any unnecessary events from a queue.

22. The system of claim 1, further comprising:

a configurable distributed caching structure to dynamically reduce a flow of data to a plurality of data consumers based on a set of predetermined strategies and policies, wherein the policies include a set of dynamic, configurable data distribution/caching policies, and wherein the set of predetermined strategies and policies comprises;

cache collapsing to reduce an amount of data to be distributed to data consumers by sending only an updated portion of data rather than an entire data set; and

dynamic flow control to reduce the amount of data to be distributed based on a group of factors including usage of the data source, priority of the data, age of the data, resource management based on data flow, smart caching, and small grained distribution of data.

23. The system of claim 1, wherein the at least one metadata catalogue comprises a plurality of distributed metadata catalogues, the data request being sent to the metadata catalogue that is a shortest distance from the requester.

24. The system of claim 1, wherein the other catalogue query services module is configured to send a response to the query to the local catalogue query services module and the local catalogue query services module is configured to send the response to the query to the data consumer.

25. The system of claim 1, further comprising a metadata notification services module to listen for new metadata entering the metadata catalogue and to notify a data consumer if the new metadata corresponds to a query previously registered with the metadata notification services module by the data consumer, wherein the metadata notification services module is operable in a distributed manner to pass any query to another metadata notification services module that can best respond to the query.

26. A system for collection, retrieval, and distribution of data, comprising:

a plurality of computing devices;

a configurable distributed data collection and caching structure including a plurality of data collection nodes, each data collection node being embodied in a respective one of the plurality of computing devices and each data collection node comprising a data collector to collect data from at least one data source and to tag each group of collected data with an associated metadata tag, wherein the metadata tag includes information about the collected data;

an object distribution logic module associated with the configurable data collection and caching structure, the object distribution logic module comprising one or more caches to store data and a distributed caching strategy structure, the distributed caching strategy structure including a cache collapsing module to only send updated portions of data rather than an entire data set in accordance with data consumer submitted information and a store and forward feature to dynamically reduce flow of the data to data consumers;

at least one data processing device comprising a processor and a memory;

at least one distributed metadata catalogue node embodied in the at least one data processing device, the at least one metadata catalogue node comprising;

a metadata catalogue data store for storing a plurality of metadata tags without any associated groups of collected data, each group of collected data being associated with one of the plurality of metadata tags;

a metadata catalogue query services module to receive and respond to a query from the data consumer, wherein a response to the query includes only metadata indicating the existence and location in the system of data corresponding to the query to avoid moving the data corresponding to the query over a network unless needed; and

wherein the query is received by a local metadata catalogue query services module of the at least one distributed metadata catalogue node that is a closest distance from the data consumer, the local catalogue query services module is configured to determine if a scope of the query is set to local, the local catalogue query services module responding to the query in response to the scope of the query being set to local and the local catalogue query services module passing the query to another catalogue query services module of another distributed metadata catalogue node capable of responding to the query.

27. The system of claim 26, a collection/tagging policy function associated with the data collector for dynamically configuring a behavior of the data collector in collecting data, tagging the collected data and generating the metadata tags.

28. The system of claim 26, wherein each data collection node comprises a data store to store the collected data, wherein the collected data is only moved in accordance with a distribution/caching policy.

29. The system of claim 26, further comprising:

a data retrieval service module to distribute selected data collected by the data collector to a data consumer that has requested the selected data, wherein the selected data is retrieved based on a distribution/caching policy;

a data streamer to publish chosen data collected by the data collector to the data consumer or another data consumer that has subscribed to receive the chosen data based on the distribution/caching policy; and

a policy distribution manager to set the distribution/caching policy.

30. A method for collection, retrieval, and distribution of data, comprising:

collecting data from at least one data source by a data collector embodied in a computing device, the computing device comprising a processor and a memory, wherein an object distribution logic module is associated with the data collector, the object distribution logic module comprising one or more caches to store data and a distributed caching strategy structure, the distributed caching strategy structure including a cache collapsing module to only send updated portions of data rather than an entire data set in accordance with data consumer submitted information and a store and forward feature to dynamically reduce flow of the data to data consumers;

tagging each group of collected data by the data collector with an associated metadata tag that includes at least one of a location of the group of data associated with the metadata tag on a system, a description of the group of data associated with the metadata tag, and a summary of the group of data associated with the metadata tag;

forming at least one distributed metadata catalogue embodied in at least one other computing device, the at least one distributed metadata catalogue containing a plurality of metadata tags without any associated groups of data, wherein only metadata indicating the existence and location of a requested group of data on the system is sent to a requester in response to a data request corresponding to the requested group of data associated with the metadata tag to avoid sending the requested group of data over a network until needed; and

wherein the at least one distributed metadata catalogue comprises a metadata catalogue query services module configured to receive a query from the data consumer, wherein the query is received by a local metadata catalogue query services module of the at least one distributed metadata catalogue that is a closest distance from the data consumer, the local catalogue query services module is configured to determine if a scope of the query is set to local, the local catalogue query services module responding to the query in response to the scope of the query being set to local and the local catalogue query services module passing the query to another catalogue query services module of another distributed metadata catalogue capable of responding to the query.

31. The method of claim 30, further comprising allowing dynamic configuration of a data collector, wherein allowing dynamic configuration of the data collector comprises:

allowing configuration of how often data is collected and stored;

allowing configuration of how much data is collected and stored;

allowing configuration of a format to be used for the metadata tags;

allowing configuration of how often metadata tags are generated;

allowing configuration of how specific fields in the metadata tag format will be filled in; and

allowing assignment of a significance or a priority to each group of data collected.

32. The method of claim 30, further comprising:

storing the collected data at a point of collection; and

moving the data when needed in accordance with a distributed caching policy.

33. A computer program product for collection, retrieval, and distribution of data, the computer program product comprising:

a non-transitory computer readable storage medium having computer usable program code embodied therewith, the computer usable medium comprising;

computer usable program code configured to collect data from at least one data source by a data collector, wherein an object distribution logic module is associated with the data collector, the object distribution logic module comprising one or more caches to store data and a distributed caching strategy structure, the distributed caching strategy structure including a cache collapsing module to only send updated portions of data rather than an entire data set in accordance with data consumer submitted information and a store and forward feature to dynamically reduce flow of the data to data consumers;

computer usable program code configured to tag each group of collected data with an associated metadata tag that includes at least one of a location of the data associated with the metadata tag on the system, a description of the data associated with the metadata tag, and a summary of the data associated with the metadata tag;

computer usable program code configured to form at least one distributed metadata catalogue containing a plurality of metadata tags without any associated groups of data, wherein only metadata indicating the existence and location of a requested group of data on the system is sent to a requester in response to a data request corresponding to the requested group of data associated with the metadata tag to avoid sending the requested group of data over a network until needed; and

wherein the at least one distributed metadata catalogue comprises a metadata catalogue query services module configured to receive a query from the data consumer, wherein the query is received by a local metadata catalogue query services module of the at least one distributed metadata catalogue that is a closest distance from the data consumer, the local catalogue query services module is configured to determine if a scope of the query is set to local, the local catalogue query services module responding to the query in response to the scope of the query being set to local and the local catalogue query services module passing the query to another catalogue query services module of another distributed metadata catalogue capable of responding to the query.

34. The computer program product of claim 33, further comprising computer usable program code configured to allow dynamic configuration of a data collector, wherein the computer usable program code configured to allow dynamic configuration of the data collector comprises:

computer usable program code configured to allow configuration of how often data is collected and stored;

computer usable program code configured to allow configuration of how much data is collected and stored;

computer usable program code configured to allow configuration of a format to be used for the metadata tags;

computer usable program code configured to allow configuration of how often metadata tags are generated;

computer usable program code configured to allow configuration of how specific fields in the metadata tag format will be filled in; and

computer usable program code configured to allow assignment of a significance or a priority to each group of data collected.