Automated data flow tracking

• US 9,734,195 B1
• Filed: 05/16/2013
• Issued: 08/15/2017
• Est. Priority Date: 05/16/2013
• Status: Active Grant

• Current Assignee: Veritas Technologies, LLC (Whitehouse Group Ltd.)
• Original Assignee: Veritas Technologies, LLC (Whitehouse Group Ltd.)
• Inventors: Pandit, Bhushan, Ilkal, Wasim, Agrawal, Namita S., Bajpai, Vishal
• Primary Examiner(s): Burke, Jeffrey A
• Assistant Examiner(s): Vu, Thong
• Application Number:
  US13/895,914
• Time in Patent Office: 1,552 Days
• Field of Search: 707737, 707758, 707100, 707692, 707693, 707741, 707724, 707749, 705 10, 705 26, 704209
• CPC Class Codes: G06F 16/245 Query processing
• CPC Class Codes: G06F 16/35 Clustering; Classification

Abstract

Various automated data flow tracking techniques can involve obtaining metadata identifying the data items from multiple data sources and using that information to identify the relationships among the data items. This information can then be provided to users. For example, a method can involve receiving a query; accessing metadata, which identifies data items generated by multiple data sources; identifying a set of responsive data items from among the available data items; identifying one or more relationships between the data items in the set of responsive data items, and responding to the query with information identifying the set of responsive data items and the one or more relationships.

Claims

1. A method comprising:

receiving a query comprising one or more search terms;

identifying, using the one or more search terms, a set of responsive text-based data items from a plurality of text-based data items, wherein

the plurality of text-based data items were generated by a plurality of data sources, and

the set of responsive text-based data items comprises a plurality of the text-based data items that are responsive to the query;

identifying one or more related text-based data items in the set of responsive text-based data items, wherein

each text-based data item comprises a plurality of text-based data pieces, and

the identifying the one or more related text-based data items comprises

parsing a first text-based data item of the plurality of text-based data items into at least two first text-based data pieces,

parsing a second text-based data item of the plurality of text-based data items into at least two second text-based data pieces,

generating a fingerprint for each of the first text-based data pieces and for each of the second text-based data pieces, and

comparing the fingerprints corresponding to the first text-based data pieces with the fingerprints corresponding to the second text-based data pieces to determine whether any of the first text-based data pieces are identical to any of the second text-based data pieces;

responding to the query with information identifying the set of responsive text-based data items and one or more relationships, wherein

the responding comprises generating a time-sorted list of information based on two or more of a plurality of sets of relationship-related metadata;

generating one or more clusters of related data items, wherein

each of the one or more clusters is generated using the information, and

each cluster is grouped based upon a respective search term of the one or more search terms; and

displaying the one or more clusters to a user.

2. The method of claim 1, further comprising

accessing metadata, wherein

the metadata identifies the plurality of text-based data items,

the metadata comprises the plurality of sets of relationship-related metadata, and

each of the plurality of sets of relationship-related metadata is specific to a respective one of the plurality of data sources.

3. The method of claim 2, wherein the time-sorted list of information identifies data in more than one of the data sources.

4. The method of claim 1, further comprising

accessing metadata associated with the plurality of text-based data items, wherein the metadata comprises

location information identifying a respective location of each of the plurality of text-based data items,

trigger information identifying a trigger for a transition or transformation, and

transition or transformation information identifying a relationship between a first one of the plurality of text-based data items and a second one of the plurality of text-based data items.

5. The method of claim 4, wherein the trigger information comprises an error indication if the trigger cannot be identified as a specific type of trigger.

6. The method of claim 2, wherein the accessing the metadata comprises querying a data governance tool, wherein the data governance tool performs data discovery for the plurality of data sources.

7. The method of claim 1, further comprising

accessing metadata associated with the plurality of text-based data items, wherein

the metadata comprises identity information identifying each of the plurality of text-based data items, and

the identity information comprises a fingerprint and indexing information.

8. A computer readable storage medium comprising program instructions executable to:

receive a query comprising one or more search terms;

identify, using the one or more search terms, a set of responsive text-based data items from a plurality of text-based data items, wherein

the plurality of text-based data items were generated by a plurality of data sources, and

the set of responsive text-based data items comprises a plurality of the text-based data items that are responsive to the query;

identify one or more related text-based data items in the set of responsive text-based data items, wherein

each text-based data item comprises a plurality of text-based data pieces, and

identifying the one or more related text-based data items comprises

parsing a first text-based data item of the plurality of text-based data items into at least two first text-based data pieces,

parsing a second text-based data item of the plurality of text-based data items into at least two second text-based data pieces,

generating a fingerprint for each of the first text-based data pieces and for each of the second text-based data pieces, and

comparing the fingerprints corresponding to the first text-based data pieces with the fingerprints corresponding to the second text-based data pieces to determine whether any of the first text-based data pieces are identical to any of the second text-based data pieces;

respond to the query with information identifying the set of responsive text-based data items and one or more relationships, wherein

responding to the query comprises generating a time-sorted list of information based on two or more of a plurality of sets of relationship-related metadata;

generate one or more clusters of related data items, wherein

each of the one or more clusters is generated using the information, and

each cluster is grouped based upon a respective search term of the one or more search terms; and

displaying the one or more clusters to a user.

9. The computer readable storage medium of claim 8, wherein the program instructions are further executable to access metadata, wherein

the metadata identifies the plurality of text-based data items,

the metadata comprises the plurality of sets of relationship-related metadata, and

each of the plurality of sets of relationship-related metadata is specific to a respective one of the plurality of data sources.

10. The computer readable storage medium of claim 9, wherein the time-sorted list of information identifies data in more than one of the data sources.

11. The computer readable storage medium of claim 8, wherein the program instructions are further executable to access metadata associated with the plurality of text-based data items, wherein

the metadata comprises location information identifying a respective location of each of the plurality of text-based data items,

trigger information identifying a trigger for a transition or transformation, and

transition or transformation information identifying a relationship between a first one of the plurality of text-based data items and a second one of the plurality of text-based data items.

12. The computer readable storage medium of claim 11, wherein the trigger information comprises an error indication if the trigger cannot be identified as a specific type of trigger.

13. The computer readable storage medium of claim of claim 9, wherein accessing the metadata comprises querying a data governance tool, wherein the data governance tool performs data discovery for the plurality of data sources.

14. The computer readable storage medium of claim 8, wherein the program instructions are further executable to:

access metadata associated with the plurality of text-based data items, wherein

the metadata comprises identity information identifying each of the plurality of text-based data items, and

the identity information comprises a fingerprint and indexing information.

15. A system comprising:

one or more processors; and

a memory coupled to the one or more processors, wherein the memory stores program instructions executable by the one or more processors to;

receive a query comprising one or more search terms;

identify, using the one or more search terms, a set of responsive text-based data items from a plurality of text-based data items, wherein

the plurality of text-based data items were generated by a plurality of data sources, and

the set of responsive text-based data items comprises a plurality of the text-based data items that are responsive to the query;

identify one or more related text-based data items in the set of responsive text-based data items, wherein

each text-based data item comprises a plurality of text-based data pieces, and

identifying the one or more related text-based data items comprises

parsing a first text-based data item of the plurality of text-based data items into at least two first text-based data pieces,

parsing a second text-based data item of the plurality of text-based data items into at least two second text-based data pieces,

generating a fingerprint for each of the first text-based data pieces and for each of the second text-based data pieces, and

comparing the fingerprints corresponding to the first text-based data pieces with the fingerprints corresponding to the second text-based data pieces to determine whether any of the first text-based data pieces are identical to any of the second text-based data pieces;

respond to the query with information identifying the set of responsive text-based data items and one or more relationships, wherein

responding to the query comprises generating a time-sorted list of information based on two or more of a plurality of sets of relationship-related metadata;

generate one or more clusters of related data items, wherein

each of the one or more clusters is generated using the information, and

each cluster is grouped based upon a respective search term of the one or more search terms; and

displaying the one or more clusters to a user.

16. The system of claim 15, wherein the program instructions are further executable to access metadata, wherein

the metadata identifies the plurality of text-based data items,

the metadata comprises the plurality of sets of relationship-related metadata, and

each of the plurality of sets of relationship-related metadata is specific to a respective one of the plurality of data sources.

17. The system of claim 16, wherein the time-sorted list of information identifies data in more than one of the data sources.

18. The system of claim 15, wherein the program instructions are further executable to access metadata associated with the plurality of text-based data items, wherein the metadata comprises

location information identifying a respective location of each of the plurality of text-based data items,

trigger information identifying a trigger for a transition or transformation, and

transition or transformation information identifying a relationship between a first one of the plurality of text-based data items and a second one of the plurality of text-based data items.

19. A system of claim 15, wherein the program instructions are further executable to:

access metadata associated with the plurality of text-based data items, wherein

the metadata comprises identity information identifying each of the plurality of text-based data items, and

the identity information comprises a fingerprint and indexing information.