SYSTEM AND METHOD FOR COLLECTION, RETRIEVAL, AND DISTRIBUTION OF DATA

• US 20080313207A1
• Filed: 06/13/2007
• Published: 12/18/2008
• Est. Priority Date: 06/13/2007
• Status: Active Grant

• Current Assignee: The Boeing Co.
• Original Assignee: The Boeing Co.
• Inventors: O'Connell, Don M., Modad, Chad
• Granted Patent: US 8,819,080 B2
• Time in Patent Office: Days
• US Class Current: 707/102
• CPC Class Codes: G06F 16/24573 using data annotations, e.g. user-defined metadata

Abstract

A system for collection, retrieval and distribution of data may include a data collector to collect data from at least one data source and to tag each group of collected data with a metadata tag. The metadata tag may include information about the collected data. The system may also include a metadata catalogue for containing a plurality of metadata tags and to allow a search for existence and a location on a network of any data corresponding to the search.

Claims

1. A system for collection, retrieval, and distribution of data, comprising:

a data collector to collect data from at least one data source and to tag each group of collected data with a metadata tag, wherein the metadata tag includes information about the collected data; and

a metadata catalogue for containing a plurality of metadata tags and to allow a search for existence and a location on a network of any data corresponding to the search.

2. The system of claim 1, further comprising a collection/tagging policy associated with the data collector for dynamically configuring a behavior of the data collector in collecting data, tagging the collected data and generating the metadata tags.

3. The system of claim 2, wherein the behavior of the data collector comprises:

how often data is collected and stored;

how much data is collected and stored;

a format to be used for the metadata tags;

how often metadata tags are generated;

how specific fields in the metadata tag format will be filled in; and

assignment of a significance or priority to the data collected.

4. The system of claim 2, further comprising a policy distribution manager to set the collection/tagging policy.

5. The system of claim 1, wherein the information about the collected data included in the metadata tag comprises at least one of a description of the data and a summary of the data.

6. The system of claim 1, wherein the metadata tag comprises information for at least one of routing data to a consumer of the data and caching the data according to predetermined criteria.

7. The system of claim 1, further comprising a data store associated with the data collector at a data collection node to store the collected data, wherein the collected data is only moved in accordance with a distributed caching policy.

8. The system of claim 1, further comprising a distributed data collection structure including a plurality of data collection nodes, each data collection node comprising a data collector.

9. The system of claim 1, further comprising a configurable distributed caching structure to dynamically reduce a flow of data to a plurality of data consumers based on a set of predetermined strategies and policies, wherein the policies include a set of dynamic, configurable data distribution/caching policies.

10. The system of claim 9, wherein the set of predetermined strategies and policies comprises:

cache collapsing to reduce an amount of data to be distributed to data consumers by sending only an updated portion of data rather than an entire data set; and

dynamic flow control to reduce the amount of data to be distributed based on a group of factors including usage of the data source, priority of the data, age of the data, resource management based on data flow, smart caching, and small grained distribution of data.

11. The system of claim 9, further comprising:

a metadata node to respond to a metadata catalogue query and to distribute metadata catalogue information to the data consumers; and

a distributed caching policy to configure a behavior of the metadata node.

12. The system of claim 1, wherein the network comprises a delay and fault tolerant network to distribute data to a data consumer, wherein the network is delay tolerant by storing and prioritizing the data to reduce an impact of any delivery delays.

13. The system of claim 1, wherein the metadata catalogue comprises:

a metadata catalogue data store for storing the plurality of metadata tags; and

a metadata catalogue query services module to receive and respond to a query from a data consumer, wherein a response to the query includes metadata indicating the existence and location in the system of data corresponding to the query.

14. The system of claim 1, further comprising a metadata notification services module to listen for new metadata entering the metadata catalogue and to notify a data consumer if the new metadata corresponds to a query registered with the metadata notification services module by the data consumer, wherein the metadata notification services module is operable in a distributed manner to pass any query to another metadata notification services module that can best respond to the query.

15. The system of claim 1, further comprising a metadata entry/update services module to receive any new and updated metadata tags from the data collector.

16. The system of claim 1, further comprising a data retrieval service module to distribute data collected by the data collector to a data consumer based on a distribution/caching policy.

17. The system of claim 1, further comprising a data streamer to publish selected data collected by the data collector to a data consumer that has subscribed to receive the selected data based on a distribution/caching policy.

18. The system of claim 1, further comprising a mechanism to cache data collected by the data collector and to transport selected data across the network in response to a request, wherein the selected data is cached at a new system location to service subsequent requests.

19. The system of claim 1, further comprising a network resource manager to provide data about current loading and status of the network and for managing use of bandwidth on the network to transfer data from a data collection node to a data consumption node.

20. The system of claim 1, further comprising a network access mechanism for accessing the network and transmitting data across the network, wherein the network access mechanism is capable of being plugged-in between an application and the network to provide seamless network management.

21. The system of claim 1, further comprising a prioritizer to reorder data collected by the data collector and stored in a cache so that data with a higher priority is transmitted first.

22. The system of claim 1, further comprising a store and forward mechanism to hold data in a cache until the network for transmitting the data becomes available in response to a loss of network connectivity and to remove data from the cache that is beyond a predetermined age.

23. The system of claim 1, further comprising a mechanism to adapt to changing network bandwidths using a set of configuration parameters.

24. The system of claim 1, further comprising an event combiner to combine event updates on a single event for transmission and to remove older and any unnecessary events from a queue.

25. A system for collection, retrieval, and distribution of data, comprising:

a configurable distributed data collection and caching structure including a plurality of data collection nodes, each data collection node comprising a data collector to collect data from at least one data source and to tag each group of collected data with a metadata tag, wherein the metadata tag includes information about the collected data;

at least one metadata catalogue node comprising;

a metadata catalogue data store for storing a plurality of metadata tags; and

a metadata catalogue query services module to receive and respond to a query from a data consumer, wherein a response to the query includes metadata indicating the existence and location in the system of data corresponding to the query.

26. The system of claim 25, a collection/tagging policy function associated with the data collector for dynamically configuring a behavior of the data collector in collecting data, tagging the collected data and generating the metadata tags.

27. The system of claim 25, wherein each data collection node comprises a data store to store the collected data, wherein the collected data is only moved in accordance with a distribution/caching policy.

28. The system of claim 25, further comprising a metadata notification services module to listen for new metadata entering the metadata catalogue node and to notify a data consumer if the new metadata corresponds to a query registered with the metadata notification services module by the data consumer, wherein the metadata notification services module is operable in a distributed manner to pass any query to another metadata notification services module that can more appropriately respond to the query.

29. The system of claim 25, further comprising:

a data retrieval service module to distribute selected data collected by the data collector to a data consumer that has requested the selected data, wherein the selected data is retrieved based on a distribution/caching policy;

a data streamer to publish chosen data collected by the data collector to the data consumer or another data consumer that has subscribed to receive the chosen data based on the distribution/caching policy; and

a policy distribution manager to set the distribution/caching policy.

30. A method for collection, retrieval, and distribution of data, comprising:

collecting data from at least one data source;

tagging each group of collected data with a metadata tag that includes at least one of a location of the group of data associated with the metadata tag on the system, a description of the group of data associated with the metadata tag, and a summary of the group of data associated with the metadata tag;

forming a metadata catalogue containing a plurality of metadata tags to allow a search for existence and a location of any data corresponding to the search.

31. The method of claim 30, further comprising allowing dynamic configuration of a data collector, wherein allowing dynamic configuration of the data collector comprises:

allowing configuration of how often data is collected and stored;

allowing configuration of how much data is collected and stored;

allowing configuration of a format to be used for the metadata tags;

allowing configuration of how often metadata tags are generated;

allowing configuration of how specific fields in the metadata tag format will be filled in; and

allowing assignment of a significance or a priority to each group of data collected.

32. The method of claim 30, further comprising:

storing the collected data at a point of collection; and

moving the data when needed in accordance with a distributed caching policy.

33. The method of claim 30, further comprising reducing a flow of data to a plurality of data consumers based on a set of predetermined strategies and policies, wherein the policies include a set of dynamic, configurable data distribution policies.

34. The method of claim 33, wherein reducing the flow of data to the plurality of data consumers based on a set of predetermined strategies and policies comprise:

using cache collapsing to reduce an amount of data to be distributed to data consumers by only sending an updated portion of data rather than an entire data set; and

using dynamic flow control to reduce the amount of data to be distributed based on a group of factors including usage of the data source, priority of the data, age of the data, resource management based on data flow, smart caching, and small grained distribution of data.

35. The method of claim 30, further comprising:

listening for new metadata to enter the metadata catalogue;

notifying a data consumer if the new metadata corresponds to a query registered by the data consumer; and

passing the query to another metadata notification service if the other metadata notification service can more appropriately respond to the query.

36. A computer program product for collection, retrieval, and distribution of data, the computer program product comprising:

a computer usable medium having computer usable program code embodied therewith, the computer usable medium comprising;

computer usable program code configured to collect data from at least one data source;

computer usable program code configured to tag each group of collected data with a metadata tag that includes at least one of a location of the data associated with the metadata tag on the system, a description of the data associated with the metadata tag, and a summary of the data associated with the metadata tag;

computer usable program code configured to form a metadata catalogue containing a plurality of metadata tags to allow a search for existence and a location of any data corresponding to the search.

37. The computer program product of claim 36, further comprising computer usable program code configured to allow dynamic configuration of a data collector, wherein the computer usable program code configured to allow dynamic configuration of the data collector comprises:

computer usable program code configured to allow configuration of how often data is collected and stored;

computer usable program code configured to allow configuration of how much data is collected and stored;

computer usable program code configured to allow configuration of a format to be used for the metadata tags;

computer usable program code configured to allow configuration of how often metadata tags are generated;

computer usable program code configured to allow configuration of how specific fields in the metadata tag format will be filled in; and

computer usable program code configured to allow assignment of a significance or a priority to each group of data collected.

38. The computer program product of claim 36, further comprising computer usable program code configured to reduce a flow of data to a plurality of data consumers based on a set of predetermined strategies and policies, wherein the policies include a set of dynamic, configurable data distribution policies.

39. The computer program product of claim 38, wherein the computer usable program code configured to reduce a flow of data to a plurality of data consumers based on a set of predetermined strategies and policies comprises:

computer usable program code configured to use cache collapsing to reduce an amount of data to be distributed data consumers by only sending an updated portion of data rather than an entire data set; and

computer usable program code configured to use dynamic flow control to reduce the amount of data to be distributed based on a group of factors including usage of the data source, priority of the data, age of the data, resource management based on data flow, smart caching, and small grained distribution of data.

40. The computer program product of claim 36, further comprising:

computer usable program code configured to listen for new metadata to enter the metadata catalogue;

computer usable program code configured to notify a data consumer if the new metadata corresponds to a query registered by the data consumer; and

computer usable program code configured to pass the query to another metadata notification service the other metadata notification service can more appropriately respond to the query.