System and method for the programmatic runtime de-obfuscation of obfuscated software utilizing virtual machine introspection and manipulation of virtual machine guest memory permissions

• US 9,459,901 B2
• Filed: 05/13/2013
• Issued: 10/04/2016
• Est. Priority Date: 12/28/2012
• Status: Active Grant

• Current Assignee: Musarubra US LLC (Musarubra US SellCo LLC)
• Original Assignee: FireEye, Inc. (Alphabet Inc.)
• Inventors: Jung, Robert, Saba, Antony
• Primary Examiner(s): Hirl, Joseph P
• Assistant Examiner(s): MURPHY, JOSEPH B
• Application Number:
  US13/892,800
• Publication Number:
  US 20140189882A1
• Time in Patent Office: 1,240 Days
• Field of Search: 726/24
• US Class Current: 1/1
• CPC Class Codes: G06F 2009/45583 Memory management, e.g. access or allocation
• CPC Class Codes: G06F 2009/45591 Monitoring or debugging support
• CPC Class Codes: G06F 21/14 against software analysis or reverse engineering, e.g. by obfuscation
• CPC Class Codes: G06F 21/53 by executing in a restricted environment, e.g. sandbox or secure virtual machine
• CPC Class Codes: G06F 21/566 Dynamic detection, i.e. detection performed at run-time, e.g. emulation, suspicious activities
• CPC Class Codes: G06F 2221/033 Test or assess software
• CPC Class Codes: G06F 9/45558 Hypervisor-specific management and integration aspects

Abstract

A system and method operable to programmatically perform runtime de-obfuscation of obfuscated software via virtual machine introspection and manipulation of virtual machine guest memory permissions.

Claims

1. A method to de-obfuscate obfuscated malicious software code in a virtual machine, the method comprising:

enumerating a first physical page associated with a virtual address space of a first piece of analyzed software code;

setting the first physical page to non-writable;

detecting a write to the first physical page; and

enumerating a second physical page following (i) a change in virtual address space allocation, and (ii) cessation of execution of the first piece of analyzed software,

wherein,

programmatic control of the virtual machine is provided via instrumentation.

2. The method according to claim 1, further comprising:

saving the write to a memory address of the first physical page.

3. The method of claim 1, further comprising:

changing the permission of the first physical page to not executable if the write is from the first piece of analyzed software code.

4. The method of claim 3, further comprising:

executing an instruction associated with the first physical page; and

updating a status of the first physical page.

5. The method of claim 4, further comprising:

storing the contents of the first physical page for review.

6. The method according to claim 3, further comprising:

reading memory allocated for use by the virtual machine via a virtual machine introspection tool to determine if the write to the first physical page is from the first piece of analyzed software code.

7. The method of claim 3, further comprising:

reading memory allocated for use by the virtual machine via a virtual machine introspection tool to determine a written to memory address of the first physical page.

8. The method of claim 7, further comprising:

storing the memory address.

9. The method of claim 8, wherein the memory address is stored in an analysis engine.

10. The method of claim 1, further comprising:

traversing a page table for the enumerating of the first physical page.

11. The method of claim 1, further comprising:

traversing a page table for the enumerating of the second physical page.

12. The method of claim 1, further comprising:

enumerating the first piece of analyzed software code by at least one of (i) reviewing process tracking structures and (ii) matching byte strings in binary executable.

13. The method of claim 1, wherein the enumerating of the first physical page, setting the first physical page to non-writable, the detecting of the write to the first physical page, and the enumerating of the second physical page are conducted by an analysis engine.

14. The method of claim 1, further comprising:

enumerating a second piece of analyzed software following at least one of (i) a change in virtual address space allocation, and (ii) cessation of execution of the first piece of analyzed software.

15. A system to de-obfuscate obfuscated malicious software in a virtual machine, the system comprising:

an analysis engine configured to update a physical page status and save the physical page status to memory upon an occurrence of at least one of (i) execution of an instruction associated with a first physical page, and (ii) setting of a physical page permission to not executable,

wherein,

the analysis engine is configured to enumerate a second physical page following (i) a change in virtual address space allocation, and (ii) cessation of execution of a first piece of analyzed software, and

programmatic control of the virtual machine is provided via instrumentation.

16. The system of claim 15, wherein the analysis engine is configured to set the physical page permission to not executable based upon data or an instruction received which indicate the potential presence of malware.

17. The system of claim 15, further comprising:

a virtual machine introspection tool configured to determine if a write to the first physical page is from the first piece of analyzed software code by reading memory allocated for use by the virtual machine.

18. The system of claim 15, further comprising:

a virtual machine introspection tool configured to determine a written to memory address of a write to the first physical page by reading memory allocated for use by the virtual machine.

19. The system of claim 18, wherein the memory address is stored in the analysis engine.

20. A system for de-obfuscating obfuscated malicious software in a virtual machine, the system comprising:

a hardware processor; and

a storage medium communicatively coupled to the processor, the storage medium comprises an analysis engine configured to (a) update a physical page status and save the physical page status upon at least either (i) execution of an instruction associated with a first physical page or (ii) setting of a physical page permission to not executable, and (b) enumerate a second physical page following (i) a change in virtual address space allocation, and (ii) cessation of execution of a first piece of analyzed software.

21. The system of claim 20, wherein the analysis engine is configured to set the physical page permission to not executable based upon data or an instruction received which indicates the potential presence of malware.

22. The system of claim 20, further comprising:

a virtual machine introspection tool configured to determine if a write to the first physical page is from the first piece of analyzed software code by reading memory allocated for use by the virtual machine.

23. The system of claim 20, further comprising:

a virtual machine introspection tool configured to determine a written to memory address of a write to the first physical page by reading memory allocated for use by the virtual machine.

24. The system of claim 23, wherein the memory address is stored in the analysis engine.