DETERMINING A FREQUENCY AT WHICH TO EXECUTE TRAP CODE IN AN EXECUTION PATH OF A PROCESS EXECUTING A PROGRAM TO GENERATE A TRAP ADDRESS RANGE TO DETECT POTENTIAL MALICIOUS CODE

• US 20190318091A1
• Filed: 04/16/2018
• Published: 10/17/2019
• Est. Priority Date: 04/16/2018
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Gupta, Lokesh M., Borlick, Matthew G., Nguyen, Trung N., Robison, Micah
• Granted Patent: US 11,003,777 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 21/566 Dynamic detection, i.e. detection performed at run-time, e.g. emulation, suspicious activities
• CPC Class Codes: G06F 21/567 using dedicated hardware
• CPC Class Codes: G06F 2221/033 Test or assess software

Abstract

Provided are a computer program product, system, and method for determining a frequency at which to execute trap code in an execution path of a process executing a program to generate a trap address range to detect potential malicious code. Trap code is executed in response to processing a specified type of command in application code to allocate a trap address range used to detect potentially malicious code. A determination is whether to modify a frequency of executing the trap code in response to processing a specified type of command. The frequency of executing the trap code is modified in response to processing the specified type of command in response to determining to determining to modify the frequency of executing the trap code.

Claims

1. A computer program product for detecting potentially malicious code accessing data from a storage, the computer program product comprising a computer readable storage medium having computer readable program code embodied therein that when executed performs operations, the operations comprising:

executing trap code in response to processing a specified type of command in application code to allocate a trap address range used to detect potentially malicious code;

executing the specified type of command in the application code;

determining whether to modify a frequency of executing the trap code in response to processing the specified type of command; and

modifying the frequency of executing the trap code in response to processing the specified type of command in response to determining to determining to modify the frequency of executing the trap code.

2. The computer program product of claim 1, wherein the trap address range is not accessed by the application code being executed, and wherein the specified type of command comprises a command to switch from a user mode to kernel mode to access kernel addresses, and wherein the trap address range is defined in a kernel address space.

3. The computer program product of claim 1, wherein the determining whether to modify the frequency of executing the trap code comprises:

determining that an application has not accessed a trap address range in a period of time, wherein a determination is made to modify the frequency of executing the trap code in response to determining that an application has not accessed the trap address range in the period of time.

4. The computer program product of claim 3, wherein the operations further comprise:

starting a timer in response to executing the trap code, wherein the determining that the application has not accessed the trap address range in the period of time is made in response to the timer expiring.

5. The computer program product of claim 4, wherein the operations further comprise:

determining whether the timer is active in response to executing the trap code, wherein the timer is active if the timer has started and not expired, and wherein the timer is started in response to determining that the timer is not active.

6. The computer program product of claim 1, wherein the modifying the frequency comprises stopping execution of the trap code in response to the processing of the specified type of command.

7. The computer program product of claim 1, wherein the modifying the frequency comprises executing the trap code in response to processing a plurality of instances of the specified type of command.

8. The computer program product of claim 1, wherein a frequency of executing the trap code prior to determining whether to modify the frequency of executing comprises a first frequency of executing the trap code, wherein the modifying the frequency of executing the trap code comprises changing the frequency of executing the trap code to a second frequency of executing trap code, wherein the operations further comprise:

performing a subsequent determination of whether to modify the frequency of executing the trap code after changing the frequency of executing the trap code to the second frequency of executing the trap code; and

modifying the frequency of executing the trap code to the first frequency of executing the trap code in response to the subsequent determination to modify the frequency of executing the trap code.

9. The computer program product of claim 8, wherein the first frequency of executing the trap code comprises executing the trap code in response to processing the specified type of command in the application code, and wherein the second frequency of executing the trap code comprises stopping execution of the trap code in response to the processing of the specified type of command.

10. A computer program product for detecting potentially malicious code accessing data from a storage, the computer program product comprising a computer readable storage medium having computer readable program code embodied therein that when executed by a processor performs operations, the operations comprising:

executing, by the processor, application code;

speculatively executing, by the processor, branches of conditional branches of the application code in advance of a location at which the application code is being executed, wherein a result of only one of the conditional branches is maintained depending on a condition used to determine which of the conditional branches to traverse;

detecting potentially malicious activity; and

in response to detecting the potentially malicious activity, disabling the speculatively executing of the application code.

11. The computer program product of claim 10, wherein the operations further comprise:

executing trap code in response to processing a specified type of command in application code to allocate a trap address range used to detect potentially malicious code;

executing the specified type of command in the application code,

wherein the detecting the potentially malicious activity comprises detecting that an application has accessed the trap address range.

12. The computer program product of claim 11, wherein the trap code is executed in a conditional branch of the conditional branches speculatively executed by the processor when speculative processing the conditional branches.

13. The computer program product of claim 11, wherein the operations further comprise:

detecting an absence of potentially malicious activity for a time period after disabling the speculatively executing the application code; and

restarting the speculatively executing of the application code in response to detecting the absence of potentially malicious activity.

14. A system for detecting potentially malicious code accessing data from a storage, comprising:

processor; and

a computer readable storage medium having computer readable program that when executed by the processor performs operations, the operations comprising;

executing trap code in response to processing a specified type of command in application code to allocate a trap address range used to detect potentially malicious code;

executing the specified type of command in the application code;

determining whether to modify a frequency of executing the trap code in response to processing the specified type of command; and

modifying the frequency of executing the trap code in response to processing the specified type of command in response to determining to determining to modify the frequency of executing the trap code.

15. The system of claim 14, wherein the determining whether to modify the frequency of executing the trap code comprises:

determining that an application has not accessed a trap address range in a period of time, wherein a determination is made to modify the frequency of executing the trap code in response to determining that an application has not accessed the trap address range in the period of time.

16. The system of claim 14, wherein the modifying the frequency comprises stopping execution of the trap code in response to the processing of the specified type of command.

17. The system of claim 14, wherein the modifying the frequency comprises executing the trap code in response to processing a plurality of instances of the specified type of command.

18. The system of claim 14, wherein a frequency of executing the trap code prior to determining whether to modify the frequency of executing comprises a first frequency of executing the trap code, wherein the modifying the frequency of executing the trap code comprises changing the frequency of executing the trap code to a second frequency of executing trap code, wherein the operations further comprise:

performing a subsequent determination of whether to modify the frequency of executing the trap code after changing the frequency of executing the trap code to the second frequency of executing the trap code; and

modifying the frequency of executing the trap code to the first frequency of executing the trap code in response to the subsequent determination to modify the frequency of executing the trap code.

19. A method for detecting potentially malicious code accessing data from a storage, comprising:

executing trap code in response to processing a specified type of command in application code to allocate a trap address range used to detect potentially malicious code;

executing the specified type of command in the application code;

determining whether to modify a frequency of executing the trap code in response to processing the specified type of command; and

modifying the frequency of executing the trap code in response to processing the specified type of command in response to determining to determining to modify the frequency of executing the trap code.

20. The method of claim 19, wherein the determining whether to modify the frequency of executing the trap code comprises:

determining that an application has not accessed a trap address range in a period of time, wherein a determination is made to modify the frequency of executing the trap code in response to determining that an application has not accessed the trap address range in the period of time.

21. The method of claim 19, wherein the modifying the frequency comprises stopping execution of the trap code in response to the processing of the specified type of command.

22. The method of claim 19, wherein the modifying the frequency comprises executing the trap code in response to processing a plurality of instances of the specified type of command.

23. The method of claim 19, wherein a frequency of executing the trap code prior to determining whether to modify the frequency of executing comprises a first frequency of executing the trap code, wherein the modifying the frequency of executing the trap code comprises changing the frequency of executing the trap code to a second frequency of executing trap code, wherein the operations further comprise:

performing a subsequent determination of whether to modify the frequency of executing the trap code after changing the frequency of executing the trap code to the second frequency of executing the trap code; and

modifying the frequency of executing the trap code to the first frequency of executing the trap code in response to the subsequent determination to modify the frequency of executing the trap code.

24. A system for detecting potentially malicious code accessing data from a storage, comprising:

processor; and

a computer readable storage medium having computer readable program that when executed by the processor performs operations, the operations comprising;

executing, by the processor, application code;

speculatively executing, by the processor, branches of conditional branches of the application code in advance of a location at which the application code is being executed, wherein a result of only one of the conditional branches is maintained depending on a condition used to determine which of the conditional branches to traverse;

detecting potentially malicious activity; and

in response to detecting the potentially malicious activity, disabling the speculatively executing of the application code.

25. The system of claim 24, wherein the operations further comprise:

executing trap code in response to processing a specified type of command in application code to allocate a trap address range used to detect potentially malicious code;

executing the specified type of command in the application code,

wherein the detecting the potentially malicious activity comprises detecting that an application has accessed the trap address range.

26. The system of claim 24, wherein the operations further comprise:

detecting an absence of potentially malicious activity for a time period after disabling the speculatively executing the application code; and

restarting the speculatively executing of the application code in response to detecting the absence of potentially malicious activity.