System and method for detecting a security compromise on a device

• US 9,619,653 B2
• Filed: 10/10/2014
• Issued: 04/11/2017
• Est. Priority Date: 07/31/2012
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Inventors: Swaminathan, Viswanathan, Wei, Sheng
• Primary Examiner(s): PYZOCHA, MICHAEL J
• Application Number:
  US14/512,346
• Publication Number:
  US 20150033031A1
• Time in Patent Office: 914 Days
• CPC Class Codes: G06F 21/10 Protecting distributed programs or content, e.g. vending or licensing of copyrighted material protection in video systems or pay television H04N7/16 ; Digital rights management [DRM]
• CPC Class Codes: G06F 21/554 involving event detection and direct action
• CPC Class Codes: G06F 21/577 Assessing vulnerabilities and evaluating computer system security

Abstract

Embodiments of a system and method for detecting a security compromise on a device are described. Embodiments may be implemented by a content consumption application configured to protect content decryption keys on a device, such as a computer system (e.g., a desktop or notebook computer) or a mobile device (e.g., a smartphone or tablet). For instance, the content consumption application may be configured to provide decryption keys for respective content to a media component (or another component of the operating system) if multiple conditions have been met. For instance, in various embodiments, the content consumption application may pass the key to the media component after ensuring that i) one or more security mechanisms of the device operating system have not been compromised and ii) one or more executable instructions of the content consumption application have not been tampered (e.g., instructions corresponding to a function that handles the decryption key(s)).

Claims

1. A computer-implemented method for verifying security mechanisms of an operating system, the method comprising:

receiving, via a security application executing on a computer, one or more decryption keys configured to enable a media component on the computer to decrypt encrypted content;

determining, via the security application executing on the computer, whether a replacement operating system or an altered operating system is present on the computer;

determining, via the security application executing on the computer, that one or more security mechanisms of the operating system of the computer are not compromised in response to a replacement operating system or an altered operating system not being present on the computer;

attempting, via the security application executing on the computer, to initiate execution of an insecure application through the operating system;

determining, via the security application executing on the computer, that the one or more security mechanisms of the operating system are not compromised in response to the operating system preventing execution of the insecure application; and

providing, via the security application executing on the computer, the one or more decryption keys to the media component on the computer in response to determining that the one or more security mechanisms of the operating system of the computer are not compromised.

2. The computer-implemented method of claim 1, wherein to determine whether a replacement operating system or an altered operating system is present on the computer, the security application executing on the computer determines whether a file indicative of a replacement operating system or an altered operating system is present on the computer.

3. The computer-implemented method of claim 2, wherein to determine whether the file is present on the computer, the security application executing on the computer determines that a file having a specific file name or content associated with a replacement operating system or an altered operating system is present on the computer.

4. The computer-implemented method of claim 1, wherein the insecure application does not have a valid digital signature associated with approval by an approval authority.

5. The computer-implemented method of claim 1, further comprising:

generating, via the security application executing on the computer, a hash based on one or more executable instructions of a function of the security application that handles the one or more decryption keys;

determining, via the security application executing on the computer, that the one or more instructions of the security application are not compromised in response to the generated hash matching a hash generated from an unaltered version of the function; and

providing, via the security application executing on the computer, the one or more decryption keys to the media component on the computer in response to determining that the one or more instructions of the security application are not compromised.

6. The computer-implemented method of claim 5, wherein to generate the hash the security application executing on the computer generates a cyclic redundancy check (CRC) code based on the one or more executable instructions of the function that handles the one or more decryption keys.

7. The computer-implemented method of claim 1, further comprising:

attempting, via the security application executing on the computer, to access a memory space assigned by the operating system of the computer to another application to test whether one or more security mechanisms of the operating system are compromised; and

determining, via the security application executing on the computer, that the one or more security mechanisms of the operating system are not compromised in response to the operating system not permitting the security application to access the memory space assigned to the other application.

8. The computer-implemented method of claim 1, further comprising:

causing, via the security application executing on the computer, another application executing on the computer to access a memory space assigned by the operating system to the security application; and

determining that the one or more security mechanisms of the operating system are not compromised in response to the operating system not permitting the other application to access the memory space assigned to the security application.

9. The computer-implemented method of claim 1, wherein the one or more decryption keys are received as part of an encrypted content license and the method further comprises:

receiving, via the security application executing on the computer, the encrypted content license from a license server; and

decrypting, via the security application executing on the computer, the encrypted content license with a public key associated with the computing device effective to enable the security application to access the one or more decryption keys.

10. A system, comprising:

one or more hardware-based processors;

one or more hardware-based computer-readable storage media comprising processor-executable instructions that, responsive to execution by the one or more hardware-based processors, implement a security application to perform operations comprising;

receiving, via the security application executing on the system, one or more decryption keys configured to enable a media component on the system to decrypt encrypted content;

determining, via the security application executing on the system, whether a file indicative of a replacement operating system or an altered operating system is present on the system;

determining, via the security application executing on the system, that one or more security mechanisms of an operating system of the system are not compromised in response to the file indicative of a replacement operating system or an altered operating system not being present on the system;

attempting, via the security application executing on the system, to initiate execution of an insecure application through the operating system of the system;

determining, via the security application executing on the system, that the one or more security mechanisms of the operating system are not compromised in response to the operating system preventing execution of the insecure application; and

providing, via the security application executing on the system, the one or more decryption keys to the media component on the system in response to determining that the one or more security mechanisms of the operating system of the system are not compromised.

11. The system of claim 10, wherein to determine whether the file is present on the system, the security application executing on the system determines that a file having a specific file name or content associated with a replacement operating system or an altered operating system is present on the system.

12. The system of claim 11, wherein the one or more decryption keys are received with, or as part of, a license associated with the encrypted content.

13. The system of claim 10, wherein the operations further comprise:

attempting, via the security application executing on the system, to access a memory space assigned by the operating system of the computer to another application to test whether one or more security mechanisms of the operating system are compromised; and

determining, via the security application executing on the system, that the one or more security mechanisms of the operating system are not compromised in response to the operating system not permitting the security application to access the memory space assigned to the other application.

14. The system of claim 10, wherein the operations further comprise:

causing, via the security application, another application being executed by the one or more hardware-based processors to attempt to access a memory space of the security application; and

determining, via the security application executing on the system, that the one or more that the one or more security mechanisms of the operating system are not compromised in response to the operating system not permitting the other application to access the memory space assigned to the security application.

15. The system of claim 10, wherein the insecure application comprises a test application that is associated with the security application and has an invalid digital signature.

16. A computer-readable storage device comprising processor-executable instructions that, responsive to execution by one or more processors, implement a security application to perform operations comprising:

receiving, via the security application executing on the one or more processors, one or more decryption keys configured to enable a media component of a computing device to decrypt encrypted content;

determining, via the security application executing on the one or more processors, whether a replacement operating system or an altered operating system is present on the computing device;

determining, via the security application executing on the one or more processors, that one or more security mechanisms of an operating system of the computing device are not compromised in response to a replacement operating system or an altered operating system not being present on the computing device;

attempting, via the security application executing on the one or more processors, to initiate execution of an insecure application through the operating system of the computing device;

determining, via the security application executing on the one or more processors, that the one or more security mechanisms of the operating system are not compromised in response to the insecure application not being able to run on the computing device; and

providing, via the security application executing on the one or more processors, the one or more decryption keys to the media component of the computing device in response to determining that the one or more security mechanisms of the operating system of the computing device are not compromised.

17. The computer-readable storage device system of claim 16, wherein the operations further comprise:

generating, via the security application executing on the one or more processors, a hash based on one or more processor-executable instructions of a function of the security application that handles the one or more decryption keys;

determining, via the security application executing on the one or more processors, that the processor-executable instructions of the security application are not compromised in response to the generated hash matching a hash generated from an unaltered version of the function; and

providing, via the security application executing on the one or more processors, the one or more decryption keys to the media component of the computing device in response to determining that the processor-executable instructions of the security application are not compromised.

18. The computer-readable storage device system of claim 16, wherein the operations further comprise:

initiating, via the security application executing on the one or more processors, execution of an anti-debugging function of the operating system, the anti-debugging function configured to prevent debugging applications from intercepting the decryption keys;

determining, via the security application executing on the one or more processors, that the processor-executable instructions of the security application are not altered in response to the anti-debugging function executing and preventing the debugging applications from intercepting the decryption keys; and

providing, via the security application executing on the one or more processors, the one or more decryption keys to the media component of the computing device in response to determining that the processor-executable instructions of the security application are not altered.

19. The computer-readable storage device system of claim 18, wherein execution of the anti-debugging function is effective to prevent the debugging applications from attaching to the security application.

20. The computer-readable storage device system of claim 16, wherein the one or more decryption keys are received as part of an encrypted content license and the operations further comprise:

receiving the encrypted content license from a license server; and

decrypting the encrypted content license with a public key associated with the computing device effective to enable the security application to access the one or more decryption keys.