Apparatus and method for efficient identification of code similarity

• US 10,152,518 B2
• Filed: 10/29/2015
• Issued: 12/11/2018
• Est. Priority Date: 10/30/2014
• Status: Active Grant

• Current Assignee: Johns Hopkins University
• Original Assignee: Johns Hopkins University
• Inventors: Cohen, Jonathan D.
• Primary Examiner(s): Cervetti, David Garcia
• Application Number:
  US14/926,274
• Publication Number:
  US 20160124966A1
• Time in Patent Office: 1,139 Days
• CPC Class Codes: G06F 16/2455 Query execution
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/955 using information identifiers, e.g. uniform resource locators [URL]
• CPC Class Codes: G06F 8/71 Version control security arrangements therefor G06F21/57; Configuration management
• CPC Class Codes: H04L 63/1416 Event detection, e.g. attack signature detection
• CPC Class Codes: H04L 63/145 the attack involving the propagation of malware through the network, e.g. viruses, trojans or worms
• CPC Class Codes: H04L 9/3231 Biological data, e.g. fingerprint, voice or retina network architectures or network communication protocols for supporting authentication of entities using biometrical features in a packet data network H04L63/0861

Abstract

A method for identifying similarity between query samples and stored samples in an efficiently maintained reference library may include receiving a binary query sample and processing the binary query sample via operations including producing a query sample fingerprint from the binary query sample, scoring the query sample fingerprint with each previously stored fingerprint in the reference library to produce a matching score, and for each previously stored fingerprint for which the matching score meets or exceeds a predetermined threshold, reporting a corresponding reference sample unique identifier associated with the previously stored fingerprint and the matching score. Each previously stored fingerprint in the reference library has been determined, prior to storage, as not being duplicative of another fingerprint in the reference library.

Claims

1. An apparatus comprising:

processing circuitry comprising at least one processor configured to execute instructions for;

receiving a first threshold;

receiving a plurality of binary reference samples;

processing each reference sample of the plurality of reference samples via operations including;

producing a reference sample fingerprint for each reference sample; and

initializing a reference library without including duplicate reference sample fingerprints and registering each respective unique identifier via operations including;

scoring the reference sample fingerprint with each previously stored fingerprint in the reference library to produce a first matching score;

if the first matching score meets or exceeds the first threshold for a previously stored fingerprint, determining the reference sample fingerprint to be a duplicate of the previously stored fingerprint, and recording, in a storage device, only a unique identifier associated with the reference sample fingerprint in the reference library and not the reference sample fingerprint, the unique identifier being marked as a duplicate of the previously stored fingerprint; and

otherwise, if the first matching score for each previously stored fingerprint is less than the first threshold, storing, in the storage device, the reference sample fingerprint and a corresponding reference sample unique identifier in the reference library.

2. The apparatus of claim 1, wherein the processing circuitry is further configured for partitioning the reference library into a plurality of cells, wherein each reference sample fingerprint is grouped into one of the plurality of cells based on a size of the respective reference sample fingerprint.

3. The apparatus of claim 2, wherein the processing circuitry is further configured for receiving a second threshold and processing a binary query sample to determine a second matching score.

4. The apparatus of claim 3, wherein processing the binary query sample comprises determining a size of the binary query sample and scoring the binary query sample against selected previously stored fingerprints in selected ones of the cells based on the size of the binary query sample.

5. The apparatus of claim 4, wherein the selected ones of the cells comprise a cell corresponding to the size of the binary query sample and to at least one other adjacent cell.

6. The apparatus of claim 4, wherein the selected ones of the cells comprise a cell corresponding to the size of the binary query sample and no more than two other adjacent cells.

7. The apparatus of claim 3, wherein at least one of the cells includes sub-cells, the sub-cells being distributed over corresponding different central processing units (CPUs).

8. The apparatus of claim 7, wherein a subsequent sub-cell is dynamically established on one of the corresponding different CPUs responsive to a prior sub-cell being filled.

9. The apparatus of claim 2, wherein the first threshold is larger than the second threshold.

10. The apparatus of claim 1, wherein each of the reference samples is a code binary.

11. The apparatus of claim 1, wherein each of the reference samples is a passage of text.

12. An apparatus comprising:

processing circuitry comprising at least one processor configured to execute instructions for;

receiving a binary query sample;

processing the binary query sample via operations including;

producing a query sample fingerprint from the binary query sample;

scoring the query sample fingerprint with each previously stored fingerprint in a reference library to produce a matching score, wherein the reference library has been initialized such that each previously stored fingerprint in the reference library has been determined, prior to storage in a storage device, as not being duplicative of another fingerprint in the reference library and duplicative fingerprints have not been stored in the reference library; and

determining whether the query sample fingerprint is a duplicate of any previously stored fingerprint in response to the matching score with a previously stored fingerprint meeting or exceeding a predetermined threshold, and reporting a corresponding reference sample unique identifier associated with the matched previously stored fingerprint and the matching score in response to determining that the query sample fingerprint is a duplicate.

13. The apparatus of claim 12, wherein the processing circuitry is further configured for partitioning the reference library into a plurality of cells, wherein each reference sample fingerprint is grouped into one of the plurality of cells based on a size of the respective reference sample fingerprint.

14. The apparatus of claim 13, wherein processing the binary query sample comprises determining a size of the binary query sample and scoring the binary query sample against selected previously stored fingerprints in selected ones of the cells based on the size of the binary query sample.

15. The apparatus of claim 14, wherein the selected ones of the cells comprise a cell corresponding to the size of the binary query sample and to at least one other adjacent cell.

16. The apparatus of claim 14, wherein the selected ones of the cells comprise a cell corresponding to the size of the binary query sample and no more than two other adjacent cells.

17. The apparatus of claim 13, wherein at least one of the cells includes sub-cells, the sub-cells being distributed over corresponding different central processing units (CPUs).

18. The apparatus of claim 17, wherein a subsequent sub-cell is dynamically established on one of the corresponding different CPUs responsive to a prior sub-cell being filled.

19. The apparatus of claim 12, wherein each of the binary query samples is a code binary.

20. The apparatus of claim 12, wherein each of the binary query samples is a passage of text.