APPARATUS AND METHODS OF IDENTIFYING POTENTIALLY SIMILAR CONTENT FOR DATA REDUCTION

• US 20090171990A1
• Filed: 12/28/2007
• Published: 07/02/2009
• Est. Priority Date: 12/28/2007
• Status: Active Grant

• Current Assignee: MidCap Financial Trust
• Original Assignee: Group Logic, Inc. (Acronis AG)
• Inventors: Naef, Frederick E. III
• Granted Patent: US 7,836,053 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06Q 10/06 Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling

Abstract

Apparatus and methods of identifying potentially similar content include utilizing workflow metadata to identify potential similarities in content to be processed, or between content to be processed and known content. As a result, a subset of potentially similar content is identified, and the subset can be used in data reduction operations to reduce data in the content to be processed.

Claims

1. A computer-implemented method of identifying potentially similar content for data reduction, comprising:

receiving content workflow metadata corresponding to content to be processed, wherein the content to be processed includes a data component, and wherein the content workflow metadata represents workflow processing information corresponding to the data component;

receiving known content workflow metadata corresponding to a first plurality of known content, wherein each known content includes a known data component, and wherein the known content workflow metadata represents workflow processing information corresponding to each respective known data component;

determining a potential similarity between the data component of the content to be processed and at least one known data component of at least one of the first plurality of known content based on a similarity between the respective content workflow metadata and the respective known content workflow metadata; and

outputting an identification of potentially similar content, based on the determined potential similarity, for use in reducing data in the content to be processed.

2. The method of claim 1, wherein receiving content workflow metadata further comprises obtaining the content workflow metadata from a system comprising a plurality of content workflow metadata corresponding to a plurality of workflow content, wherein the content to be processed comprises one of the plurality of workflow content.

3. The method of claim 1, wherein determining the potential similarity further comprises executing a workflow-specific similarity rule to determine the potential similarity, wherein the workflow-specific similarity rule used for the content to be processed depends on a type of the workflow associated with the content to be processed.

4. The method of claim 1, further comprising:

identifying a second plurality of known content that represent at least a portion of the identified potentially similar content, wherein the second plurality of known content is a subset of the first plurality of known content, and wherein the second plurality of known content is not equal to the first plurality of known content;

performing a data compression technique on the content to be processed and the second plurality of known content to identify a reduced data representation of the content to be processed; and

wherein outputting comprises outputting an identification of the reduced data representation.

5. The method of claim 4, further comprising transmitting or receiving the reduced data representation.

6. The method of claim 4, further comprising replacing a duplicate data component in the content to be processed with a token to form the reduced data representation.

7. The method of claim 4, wherein performing the data compression technique comprises identifying a same data component in both the content to be processed and the second plurality of known content.

8. The method of claim 1, further comprising:

determining a data component difference between the content to be processed and the first plurality of known content based on the determined potential similarity;

determining a network storage location of each of a plurality of network-based content having the data component difference;

determining a network destination location for receiving a transmission of the data component difference;

determining a delivery efficiency between each network storage location and the network destination location; and

causing transmission of the data component difference to the network destination location from the respective network storage location having a most efficient one of the determined delivery efficiencies.

9. The method of claim 1, further comprising:

wherein outputting an identification of potentially similar content further comprises outputting an identification of the respective ones of the first plurality of known content having the determined potential similarity;

identifying at least one of a same data component or a different data component between the content to be processed and one of the respective ones of the first plurality of known content having the determined potential similarity; and

wherein the outputting further comprises outputting an identification of at least one of the same data component or the different data component.

10. The method of claim 9, further comprising transmitting or receiving the different data component based on the identification of at least one of the same data component or the different data component.

11. The method of claim 9, further comprising replacing the same data component in the content to be processed with a token based on the identification of at least one of the same data component or the different data component.

12. The method of claim 1, further comprising:

transmitting the identification of the potentially similar content to a data reduction component;

receiving from the data reduction component an identification of at least one of a same data component or a different data component between the content to be processed and the first plurality of known content based on execution of a data compression protocol on the potentially similar content by the data reduction component; and

transmitting a reduced data representation of the content to be processed to a file transfer destination based on the identification of the at least one of a same data component or a different data component.

13. The method of claim 12, wherein the potentially similar content comprises a subset of the content to be processed and a corresponding subset of the first plurality of known content having a respective potential similarity, wherein the transmitting further comprises transmitting a respective data reduction signature of one or more portions of each of the subset of the content to be processed and transmitting an identification of the subset of the first plurality of known content.

14. The method of claim 13, wherein the receiving of the identification of at least one of a same data component or a different data component is further based on the data reduction component generating a respective data reduction signature of one or more portions of each of the subset of the first plurality of known content, and comparing the respective data reduction signatures to determine the same data component.

15. The method of claim 12, wherein transmitting the reduced data representation of the content to be processed further comprises transmitting one or more different data components and one or more tokens representing a respective one or more same data components.

16. The method of claim 1, further comprising:

obtaining a reduced data representation of the content to be processed based on the identification of potentially similar content;

processing the reduced data representation; and

updating the content workflow metadata corresponding to content to be processed with information describing the processing.

17. The method of claim 1, wherein receiving known content workflow metadata corresponding to a first plurality of known content further comprises receiving known content workflow metadata corresponding to at least one of:

a plurality of previously-transferred content;

or, a plurality of previously-received content;

or a plurality of content to be processed, wherein the plurality of contents to be processed includes the content to be processed.

18. The method of claim 1, further comprising:

wherein receiving content workflow metadata corresponding to a content to be processed further comprises receiving a plurality of content workflow metadata corresponding to a plurality of content to be processed, wherein each of the plurality of content to be processed includes a respective data component, and wherein each respective content workflow metadata represents workflow processing information corresponding to a respective data component;

identifying potentially similar ones of the plurality of content to be processed based on determining a potential similarity between respective data components based on the respective content workflow metadata;

identifying a proper subset of the plurality of content to be processed based on performing a data compression technique on the identified potentially similar ones of the plurality of content to be processed;

wherein determining the potential similarity with the first plurality of known content further comprise determining a potential similarity between a respective data component of a respective one of the proper subset of the plurality of content to be processed and a respective known data component of a respective one of the first plurality of known content based on a similarity between the respective content workflow metadata and the respective known content metadata;

identifying a second plurality of known content that represent content potentially similar to the proper subset of the plurality of content to be processed based on the determined potential similarity, wherein the second plurality of known content is a proper subset of the first plurality of known content;

performing a data compression technique on the proper subset of the plurality of content to be processed and the second plurality of known content to identify a reduced data representation of the plurality of content to be processed; and

wherein outputting comprises outputting the reduced data representation.

19. The method of claim 1, wherein receiving the content workflow metadata corresponding to the content to be processed further comprises receiving at a destination from a source located across a communication network, wherein determining the potential similarity further comprises determining at the destination, and wherein outputting the identification of the potentially similar content further comprises transmitting from the destination to the source.

20. A computer program product configured to identify potentially similar content for data reduction, comprising:

a computer-readable medium comprising;

at least one set of instructions operable to cause a computer to receive content workflow metadata corresponding to content to be processed, wherein the content to be processed includes a data component, and wherein the content workflow metadata represents workflow processing information corresponding to the data component;

at least one set of instructions operable to cause the computer to receive known content workflow metadata corresponding to a first plurality of known contents, wherein each known content includes a known data component, and wherein the known content workflow metadata represents workflow processing information corresponding to each respective known data component;

at least one set of instructions operable to cause the computer to determine a potential similarity between the data component of the content to be processed and at least one known data component of at least one of the first plurality of known contents based on a potential similarity between the respective content workflow metadata and the respective known content workflow metadata; and

at least one set of instructions operable to cause the computer to output an identification of potentially similar content, based on the determined potential similarity, for use in reducing data in the content to be processed.

21. At least one processor configured to identify potentially similar content for data reduction, comprising:

a first module for receiving content workflow metadata corresponding to content to be processed, wherein the content to be processed includes a data component, and wherein the content workflow metadata represents workflow processing information corresponding to the data component;

a second module for receiving known content workflow metadata corresponding to a first plurality of known contents, wherein each known content includes a known data component, and wherein the known content workflow metadata represents workflow processing information corresponding to each respective known data component;

a third module for determining a potential similarity between the data component of the content to be processed and at least one known data component of at least one of the first plurality of known contents based on a potential similarity between the respective content workflow metadata and the respective known content workflow metadata; and

a fourth module for outputting an identification of potentially similar content, based on the determined potential similarity, for use in reducing data in the content to be processed.

22. A computing device for identifying potentially similar content for data reduction, comprising:

means for receiving content workflow metadata corresponding to content to be processed, wherein the content to be processed includes a data component, and wherein the content workflow metadata represents workflow processing information corresponding to the data component;

means for receiving known content workflow metadata corresponding to a first plurality of known contents, wherein each known content includes a known data component, and wherein the known content workflow metadata represents workflow processing information corresponding to each respective known data component;

means for determining a potential similarity between the data component of the content to be processed and at least one known data component of at least one of the first plurality of known contents based on a potential similarity between the respective content workflow metadata and the respective known content workflow metadata; and

means for outputting an identification of potentially similar content, based on the determined potential similarity, for use in reducing data in the content to be processed.

23. A computing device for identifying potentially similar content for data reduction, comprising:

a communications module operable to receive content workflow metadata corresponding to content to be processed, wherein the content to be processed includes a data component, and wherein the content workflow metadata represents workflow processing information corresponding to the data component;

wherein the communications module is further operable to receive known content workflow metadata corresponding to a first plurality of known content, wherein each known content includes a known data component, and wherein the known content workflow metadata represents workflow processing information corresponding to each respective known data component;

a similarity identifier module having one or more similarity rules operable to determine a potential similarity between the data component of the content to be processed and at least one known data component of at least one of the first plurality of known content based on a potential similarity between the respective content workflow metadata and the respective known content workflow metadata; and

wherein the similarity identifier component is further operable to output an identification of potentially similar content, based on the determined potential similarity, for use in reducing data in the content to be processed.

24. The computing device of claim 23, wherein the communications module is further operable to obtain the content workflow metadata from a system comprising a plurality of content workflow metadata corresponding to a plurality of workflow content, wherein the content to be processed comprises one of the plurality of workflow content.

25. The computing device of claim 23, wherein the similarity rule comprises a workflow-specific similarity rule, wherein the workflow-specific similarity rule used for the content to be processed depends on a type of the workflow associated with the content to be processed.

26. The computing device of claim 23, further comprising:

wherein the similarity identifier component is further operable to identify a second plurality of known content that represent at least a portion of the identified potentially similar content, wherein the second plurality of known content is a subset of the first plurality of known content, and wherein the second plurality of known content is not equal to the first plurality of known content;

a data reduction component having a data compression protocol operable to compress the content to be processed and the second plurality of known content to identify a reduced data representation of the content to be processed; and

wherein outputting comprises outputting an identification of the reduced data representation.

27. The computing device of claim 26, wherein the communications module is further operable to transmit or receive the reduced data representation.

28. The computing device of claim 26, wherein the data reduction component is further operable to replace a duplicate data component in the content to be processed with a token to form the reduced data representation.

29. The computing device of claim 26, wherein the data reduction component is further operable to identify a same data component in both the content to be processed and the second plurality of known content.

30. The computing device of claim 23, further comprising:

a data reduction component having a data compression protocol operable to determine a data component difference between the content to be processed and the first plurality of known content based on the determined potential similarity;

a delivery management component having a data location identifier operable to determine a network storage location of each of a plurality of network-based content having the data component difference;

a content processing component operable to determine a network destination location for receiving a transmission of the data component difference;

wherein the delivery management component further comprises a delivery optimizer operable to determine a delivery efficiency between each network storage location and the network destination location; and

wherein the delivery management component is further operable to cause transmission of the data component difference to the network destination location from the respective network storage location having a most efficient one of the determined delivery efficiencies.

31. The computing device of claim 23, further comprising:

wherein the identification of potentially similar content further comprises an identification of the respective ones of the first plurality of known content having the determined potential similarity;

a data reduction component having a data compression protocol operable to identify at least one of a same data component or a different data component between the content to be processed and one of the respective ones of the first plurality of known content having the determined potential similarity; and

wherein the data reduction component is further operable to output an identification of at least one of the same data component or the different data component.

32. The computing device of claim 31, wherein the communications module is further operable to transmit or receive the different data component based on the identification of at least one of the same data component or the different data component.

33. The computing device of claim 31, wherein the data reduction component is further operable to replace the same data component in the content to be processed with a token based on the identification of at least one of the same data component or the different data component.

34. The computing device of claim 23, further comprising:

wherein the similarity identifier component is further operable to transmit the identification of the potentially similar content to a data reduction component;

a content processing component operable to receive from the data reduction component an identification of at least one of a same data component or a different data component between the content to be processed and the first plurality of known content based on execution of a data compression protocol on the potentially similar content by the data reduction component; and

wherein the content processing component is further operable to initiate transmission of a reduced data representation of the content to be processed to a file transfer destination based on the identification of the at least one of a same data component or a different data component.

35. The computing device of claim 34, wherein the potentially similar content comprises a subset of the content to be processed and a corresponding subset of the first plurality of known content having a respective potential similarity, wherein the content processing component is further operable to initiate transmission of a respective data reduction signature of one or more portions of each of the subset of the content to be processed and transmitting an identification of the subset of the first plurality of known content.

36. The computing device of claim 35, wherein the identification of at least one of a same data component or a different data component is further based on the data reduction component generating a respective data reduction signature of one or more portions of each of the subset of the first plurality of known content, and comparing the respective data reduction signatures to determine the same data component.

37. The computing device of claim 34, wherein the reduced data representation of the content to be processed further comprises one or more different data components and one or more tokens representing a respective one or more same data components.

38. The computing device of claim 23, further comprising:

wherein the communications module is further operable to obtain a reduced data representation of the content to be processed based on the identification of potentially similar content;

a content processing component operable to process the reduced data representation; and

wherein the content processing component is further operable to update the content workflow metadata corresponding to content to be processed with information describing the processing.

39. The computing device of claim 23, wherein the known content workflow metadata corresponding to the first plurality of known content further comprises known content workflow metadata corresponding to at least one of:

a plurality of previously-transferred content;

or, a plurality of previously-received content;

or a plurality of content to be processed, wherein the plurality of contents to be processed includes the content to be processed.

40. The computing device of claim 23, further comprising:

wherein the content workflow metadata corresponding to the content to be processed further comprises a plurality of content workflow metadata corresponding to a plurality of content to be processed, wherein each of the plurality of contents to be processed includes a respective data component, and wherein each respective content workflow metadata represents workflow processing information corresponding to a respective data component;

wherein the similarity identifier component is further operable to identify potentially similar ones of the plurality of content to be processed based on determining a potential similarity between respective data components based on the respective content workflow metadata;

a data reduction component having a data compression protocol operable to identify a proper subset of the plurality of content to be processed based on performing a data compression technique on the identified potentially similar ones of the plurality of content to be processed;

wherein the similarity identifier component is further operable to determine a potential similarity between a respective data component of a respective one of the proper subset of the plurality of content to be processed and a respective known data component of a respective one of the first plurality of known content based on a similarity between the respective content workflow metadata and the respective known content metadata;

wherein the similarity identifier component is further operable to identify a second plurality of known content that represent content potentially similar to the proper subset of the plurality of contents to be processed based on the determined potential similarity, wherein the second plurality of known content is a proper subset of the first plurality of known content;

wherein the data reduction component is further operable to execute the data compression protocol on the proper subset of the plurality of content to be processed and the second plurality of known content to identify a reduced data representation of the plurality of content to be processed; and

wherein the data reduction component is further operable to output the reduced data representation.

41. The computing device of claim 23, wherein the computing device is located at a destination on a communications network, wherein the communications module is further operable to receive the content workflow metadata corresponding to the content to be processed from a source located across the communications network from the destination, and wherein the similarity identifier component is further operable to transmit the identification of the potentially similar content from the destination to the source.