Grouping and differentiating files based on content

• US 9,053,120 B2
• Filed: 12/15/2009
• Issued: 06/09/2015
• Est. Priority Date: 07/16/2009
• Status: Active Grant

• Current Assignee: Micro Focus Software, Inc. (Open Text Corporation)
• Original Assignee: Novell Incorporated (Open Text Corporation)
• Inventors: Teerlink, Craig N.
• Primary Examiner(s): Truong, Dennis
• Application Number:
  US12/637,807
• Publication Number:
  US 20110016138A1
• Time in Patent Office: 2,002 Days
• Field of Search: 707/758, 707/750, 707/754, 707/736, 707/999.006, 707/737, 704/10, 704/500, 341/50
• US Class Current: 1/1
• CPC Class Codes: G06F 16/1744 using compression, e.g. sparse files
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06T 9/005 Statistical coding, e.g. Huffman, run length coding
• CPC Class Codes: H04L 2209/30 Compression, e.g. Merkle-Damgard construction
• CPC Class Codes: H04L 2209/34 Encoding or coding, e.g. Huffman coding or error correction
• CPC Class Codes: H04L 9/0894 Escrow, recovery or storing of secret information, e.g. secret key escrow or cryptographic key storage
• CPC Class Codes: H04N 19/00 Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
• CPC Class Codes: Y10S 707/972 Partitioning

Abstract

Methods and apparatus teach a digital spectrum of a file. The digital spectrum is used to map a file'"'"'s position. This position relative to another file'"'"'s position reveals distances between the files. Representatively, files have a plurality of symbols representing an underlying data stream of original bits of data. The number of occurrences of each symbol in each file is compared to like symbols in other files. This can occur via algorithms, mapping, or both. In certain instances, comparison reveals a difference in counts between the symbols of the files. This difference is then squared, added together, and a square root taken. Comparing “distance values” reveals file adjacency, grouping, or the like. Also, normalizing, weighting, filtering functions and/or other statistical computations are applied in certain instances.

Claims

1. In a computing system environment, a method of differentiating files stored on one or more computing devices, each file having a plurality of symbols derived from an underlying data stream of all original bits of raw data of said each file, comprising:

encoding said each file as a plurality of symbols representing an underlying data stream of all original bits of binary data of the file;

determining a number of occurrences of each said symbol in said each file; and

computing a distance between said each file and every other file based on the determined number of occurrences.

2. The method of claim 1, further including normalizing the determined number of occurrences for said each symbol by comparing to a total number of symbols in a source file.

3. The method of claim 1, further including weighting the determined number of occurrences for said each symbol according to a bit length of said original bits of data represented by the symbols.

4. The method of claim 3, wherein the weighting further includes multiplying the determined number of occurrences for said each symbol by a corresponding said bit length.

5. The method of claim 1, further including grouping together closest neighbors of the files.

6. The method of claim 1, further including sorting into an ordered list the computed said distances between said each file and every other file.

7. The method of claim 1, further including normalizing and weighting the determined number of occurrences for said each symbol.

8. The method of claim 7, wherein the normalizing further includes comparing the determined number of occurrences for said each symbol to a total number of symbols in a source file.

9. The method of claim 7, wherein the weighting further includes multiplying the determined number of occurrences for said each symbol by a bit length of said original bits of data represented by the symbols.

10. In a computing system environment, a method of differentiating files stored on one or more computing devices, each file having a plurality of symbols derived from an underlying data stream of all original bits of raw data of said each file, comprising encoding said each file as a plurality of symbols representing an underlying data stream of all original bits of binary data of the file, determining a number of occurrences of each said symbol in said each file, and computing a distance in an N-dimensional space between said each file and every other file based on said determined number of occurrences, said N-dimensional space being defined by a total number N of different symbols in the plurality of symbols.

11. The method of claim 10, wherein the computing the distance further includes determining for like symbols a difference in said determined number of occurrences.

12. The method of claim 11, further including squaring the determined difference for said like symbols.

13. The method of claim 12, further including adding together the squared determined differences.

14. The method of claim 13, further including taking a square root of the added together squared determined differences.

15. In a computing system environment, a method of determining closest files stored on one or more computing devices, each file having a plurality of symbols derived from an underlying data stream of all original bits of raw data of said each file, comprising:

encoding said each file as a plurality of symbols representing an underlying data stream of all original bits of binary data of the file;

determining a number of occurrences of each said symbol in said each file;

computing a distance value between said each file and every other file based on said number of occurrences; and

concluding a closest two files based on the computed distance value.

16. The method of claim 15, further including normalizing the determined number of occurrences for said each symbol.

17. The method of claim 15, further including weighting the determined number of occurrences for said each symbol.

18. The method of claim 15, further including considering only a portion of the determined number of occurrences when said computing the distance.

19. In a computing system environment, a method of differentiating files stored on one or more computing devices, each file having a plurality of symbols derived from an underlying data stream of all original bits of raw data of said each file, comprising:

encoding said each file as a plurality of symbols representing an underlying data stream of all original bits of binary data of the file;

defining an N-dimensional space wherein N is a total number of different symbols in the plurality of symbols;

determining a number of occurrences of each said symbol in said each file; and

computing a distance in the N-dimensional space between said each file and every other file of the stored files based on the determined number of occurrences.