Apparatus and Method for Efficient Identification of Code Similarity

• US 20160127398A1
• Filed: 10/28/2015
• Published: 05/05/2016
• Est. Priority Date: 10/30/2014
• Status: Active Grant

• Current Assignee: Johns Hopkins University
• Original Assignee: Johns Hopkins University
• Inventors: Cohen, Jonathan D.
• Granted Patent: US 9,805,099 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/2455 Query execution
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/955 using information identifiers, e.g. uniform resource locators [URL]
• CPC Class Codes: G06F 8/71 Version control security arrangements therefor G06F21/57; Configuration management
• CPC Class Codes: H04L 63/1416 Event detection, e.g. attack signature detection
• CPC Class Codes: H04L 63/145 the attack involving the propagation of malware through the network, e.g. viruses, trojans or worms
• CPC Class Codes: H04L 9/3231 Biological data, e.g. fingerprint, voice or retina network architectures or network communication protocols for supporting authentication of entities using biometrical features in a packet data network H04L63/0861

Abstract

A method for identifying similarity between query samples and stored samples in an efficiently maintained reference library may include receiving a first threshold and a second threshold, receiving a plurality of binary reference samples, and processing each reference sample of the plurality of reference samples. The processing may include operations of assigning each reference sample a respective unique identifier, producing a reference sample fingerprint for each reference sample, and registering each respective unique identifier to reference sample fingerprint pair in a reference library. The registering may include scoring the reference sample fingerprint with each previously stored fingerprint in the reference library to produce a first matching score, if the first matching score meets or exceeds the first threshold for a previously stored fingerprint, determining the reference sample fingerprint to be a duplicate of the previously stored fingerprint and recording only a unique identifier associated with the reference sample fingerprint in the reference library where the unique identifier is marked as a duplicate of the previously stored fingerprint, and otherwise, if the first matching score is less than the first threshold, storing a corresponding reference sample unique identifier to reference sample fingerprint pair in the reference library. The method may further include receiving a binary query sample and processing the binary query sample via operations including producing a query sample fingerprint from the binary query sample, scoring the query sample fingerprint with each previously stored fingerprint in the reference library to produce a second matching score, and for each previously stored fingerprint for which the second matching score meets or exceeds the second threshold, reporting a corresponding reference sample unique identifier associated with the previously stored fingerprint and the second matching score.

Claims

1. An apparatus comprising processing circuitry configured to execute instructions for:

receiving a first threshold and a second threshold;

receiving a plurality of binary reference samples;

processing each reference sample of the plurality of reference samples via operations including;

assigning each reference sample a respective unique identifier;

producing a reference sample fingerprint for each reference sample; and

registering each respective unique identifier to reference sample fingerprint pair in a reference library via operations including;

scoring the reference sample fingerprint with each previously stored fingerprint in the reference library to produce a first matching score;

if the first matching score meets or exceeds the first threshold for a previously stored fingerprint, determining the reference sample fingerprint to be a duplicate of the previously stored fingerprint, and recording only a unique identifier associated with the reference sample fingerprint in the reference library, the unique identifier being marked as a duplicate of the previously stored fingerprint; and

otherwise, if the first matching score for each previously stored fingerprint is less than the first threshold, storing a corresponding reference sample unique identifier to reference sample fingerprint pair in the reference library;

receiving a binary query sample;

processing the binary query sample via operations including;

producing a query sample fingerprint from the binary query sample;

scoring the query sample fingerprint with each previously stored fingerprint in the reference library to produce a second matching score;

for each previously stored fingerprint for which the second matching score meets or exceeds the second threshold, reporting a corresponding reference sample unique identifier associated with the previously stored fingerprint and the second matching score.

2. The apparatus of claim 1, wherein the processing circuitry is further configured for partitioning the reference library into a plurality of cells, each cell being associated with a corresponding size.

3. The apparatus of claim 2, wherein processing the binary query sample comprises determining a size of the binary query sample and scoring the binary query sample against selected previously stored fingerprints in selected ones of the cells based on the size of the binary query sample.

4. The apparatus of claim 3, wherein the selected ones of the cells comprise a cell corresponding to the size of the binary query sample and to at least one other adjacent cell.

5. The apparatus of claim 3, wherein the selected ones of the cells comprise a cell corresponding to the size of the binary query sample and no more than two other adjacent cells.

6. The apparatus of claim 2, wherein at least one of the cells includes sub-cells, the sub-cells being distributed over corresponding different CPUs.

7. The apparatus of claim 6, wherein a subsequent sub-cell is dynamically established on one of the corresponding different CPUs responsive to a prior sub-cell being filled.

8. The apparatus of claim 1, wherein each of the reference samples is a code binary.

9. The apparatus of claim 1, wherein each of the reference samples is a passage of text.

10. The apparatus of claim 1, wherein the first threshold is larger than the second threshold.

11. A method for identifying similarity between query samples and stored samples in an efficiently maintained reference library, the method comprising:

receiving a first threshold and a second threshold;

receiving a plurality of binary reference samples;

processing each reference sample of the plurality of reference samples via operations including;

assigning each reference sample a respective unique identifier;

producing a reference sample fingerprint for each reference sample; and

registering each respective unique identifier to reference sample fingerprint pair in a reference library via operations including;

scoring the reference sample fingerprint with each previously stored fingerprint in the reference library to produce a first matching score;

if the first matching score meets or exceeds the first threshold for a previously stored fingerprint, determining the reference sample fingerprint to be a duplicate of the previously stored fingerprint, and recording only a unique identifier associated with the reference sample fingerprint in the reference library, the unique identifier being marked as a duplicate of the previously stored fingerprint; and

otherwise, if the first matching score is less than the first threshold for each previously stored fingerprint, storing a corresponding reference sample unique identifier to reference sample fingerprint pair in the reference library;

the method further comprising;

receiving a binary query sample;

processing the binary query sample via operations including;

producing a query sample fingerprint from the binary query sample;

scoring the query sample fingerprint with each previously stored fingerprint in the reference library to produce a second matching score;

for each previously stored fingerprint for which the second matching score meets or exceeds the second threshold, reporting a corresponding reference sample unique identifier associated with the previously stored fingerprint and the second matching score.

12. The method of claim 11, further comprising partitioning the reference library into a plurality of cells, each cell being associated with a corresponding size.

13. The method of claim 12, wherein processing the binary query sample comprises determining a size of the binary query sample and scoring the binary query sample against selected previously stored fingerprints in selected ones of the cells based on the size of the binary query sample.

14. The method of claim 13, wherein the selected ones of the cells comprise a cell corresponding to the size of the binary query sample and to at least one other adjacent cell.

15. The method of claim 13, wherein the selected ones of the cells comprise a cell corresponding to the size of the binary query sample and no more than two other adjacent cells.

16. The method of claim 12, wherein at least one of the cells includes sub-cells, the sub-cells being distributed over corresponding different CPUs.

17. The method of claim 16, wherein a subsequent sub-cell is dynamically established on one of the corresponding different CPUs responsive to a prior sub-cell being filled.

18. The method of claim 11, wherein each of the reference samples is a code binary.

19. The method of claim 11, wherein each of the reference samples is a passage of text.

20. The method of claim 11, wherein the first threshold is larger than the second threshold.