Anomaly detection using sequences of system calls

• US 10,656,981 B2
• Filed: 03/25/2019
• Issued: 05/19/2020
• Est. Priority Date: 07/19/2016
• Status: Active Grant

• Current Assignee: Blackberry Limited
• Original Assignee: Blackberry Limited
• Inventors: Al Sharnouby, Mohamed
• Primary Examiner(s): Shayanfar, Ali
• Application Number:
  US16/363,150
• Publication Number:
  US 20190220334A1
• Time in Patent Office: 421 Days
• CPC Class Codes: G06F 21/52 during program execution, e.g. stack integrity ; Preventing unwanted data erasure; Buffer overflow
• CPC Class Codes: G06F 21/554 involving event detection and direct action
• CPC Class Codes: G06F 21/56 Computer malware detection or handling, e.g. anti-virus arrangements
• CPC Class Codes: G06F 21/566 Dynamic detection, i.e. detection performed at run-time, e.g. emulation, suspicious activities
• CPC Class Codes: G06F 2221/034 Test or assess a computer or a system
• CPC Class Codes: G06F 9/4812 by interrupt, e.g. masked
• CPC Class Codes: G06F 9/4843 by program, e.g. task dispatcher, supervisor, operating system
• CPC Class Codes: G06F 9/546 Message passing systems or structures, e.g. queues

Abstract

Systems and methods of detecting a call sequence anomaly in a message-based operating system are provided. A message may be received that indicates a programmatic procedure of an operating system was invoked. The message may include a programmatic procedure identifier, a sender process identifier, and a receiver process identifier. An invocation hash may be generated based on the message. The invocation hash may be translated to a smaller invocation identifier. The invocation identifier may be included in a translated call sequence that comprises invocation identifiers for a series of invocations. Depending on whether the translated call sequence is included in previously generated predetermined call sequences, the translated call sequence may be determined as an anomaly or not an anomaly.

Claims

1. A method of detecting a call sequence in a computing system, the computing system storing a set of call sequences, each call sequence containing a sequence of invocation identifiers that specify a valid sequence of programmatic procedures, wherein each invocation identifier corresponds to a message that invokes a corresponding programmatic procedure, the method comprising:

receiving a new message from a process, the new message including at least a programmatic procedure identifier and a receiver process identifier;

determining a corresponding invocation identifier based on at least the programmatic procedure identifier and the receiver process identifier;

appending the corresponding invocation identifier to a sequence of previously-determined invocation identifiers corresponding to a sequence of previously-received messages from the process to generate a translated call sequence;

identifying the translated call sequence as an anomaly based on selecting a portion of the translated call sequence and determining that the portion of the translated call sequence does not match at least one of the call sequences in the set of call sequences; and

causing an action based on identification of the translated call sequence as an anomaly.

2. The method of claim 1, wherein determining a corresponding invocation identifier includes:

generating a new invocation hash based on at least the programmatic procedure identifier and the receiver process identifier; and

determining the corresponding invocation identifier from the new invocation hash and stored associations between invocation hashes and invocation identifiers.

3. The method of claim 2, wherein the message further includes a channel identifier, a sender process identifier, a receiver node identifier and wherein the invocation hash is further based on at least the channel identifier or the sender process identifier.

4. The method of claim 1, wherein each of the previously-received messages contains a same sender process identifier, and the new message contains the same sender process identifier.

5. The method of claim 1, wherein selecting a portion is based on window size.

6. The method of claim 1, wherein appending the corresponding invocation identifier includes removing an oldest invocation identifier from the sequence of previously-determined invocation identifiers to maintain the translated call sequence at a fixed number of invocation identifiers.

7. A non-transitory computer readable storage medium comprising computer executable instructions, the computer executable instructions executable by a processor to detect a call sequence in a computing system, the computing system storing a set of call sequences, each call sequence containing a sequence of invocation identifiers that specify a valid sequence of programmatic procedures, wherein each invocation identifier corresponds to a message that invokes a corresponding programmatic procedure, the computer executable instructions comprising:

instructions executable to receive a new message from a process, the new message including at least a programmatic procedure identifier and a receiver process identifier;

instructions executable to determine a corresponding invocation identifier based on at least the programmatic procedure identifier and the receiver process identifier;

instructions executable to append the corresponding invocation identifier to a sequence of previously-determined invocation identifiers corresponding to a sequence of previously-received messages from the process to generate a translated call sequence;

instructions executable to identify the translated call sequence as an anomaly based on selecting a portion of the translated call sequence and determining that the portion of the translated call sequence does not match at least one of the call sequences in the set of call sequences; and

instructions executable to cause an action based on identification of the translated call sequence as an anomaly.

8. The non-transitory computer readable storage medium of claim 7, wherein the instructions executable to determine a corresponding invocation identifier include:

instructions executable to generate a new invocation hash based on at least the programmatic procedure identifier and the receiver process identifier; and

instructions executable to determine the corresponding invocation identifier from the new invocation hash and stored associations between invocation hashes and invocation identifiers.

9. The non-transitory computer readable storage medium of claim 8, wherein the message further includes a channel identifier, a sender process identifier, a receiver node identifier and wherein the invocation hash is further based on at least the channel identifier or the sender process identifier.

10. The non-transitory computer readable storage medium of claim 7, wherein each of the previously-received messages contains a same sender process identifier, and the new message contains the same sender process identifier.

11. The non-transitory computer readable storage medium of claim 7, wherein the instructions executable to append the corresponding invocation identifier include instructions executable to remove an oldest invocation identifier from the sequence of previously-determined invocation identifiers to maintain the translated call sequence at a fixed number of invocation identifiers.

12. A computing system to detect a call sequence anomaly in the computing system, the computing system comprising:

memory storing a set of call sequences, each call sequence containing a sequence of invocation identifiers that specify a valid sequence of programmatic procedures, wherein each invocation identifier corresponds to a message that invokes a corresponding programmatic procedure;

a processor;

processor-executable instructions that, when executed by the processor, are to cause the processor to;

receive a new message from a process, the new message including at least a programmatic procedure identifier and a receiver process identifier;

determine a corresponding invocation identifier based on at least the programmatic procedure identifier and the receiver process identifier;

append the corresponding invocation identifier to a sequence of previously-determined invocation identifiers corresponding to a sequence of previously-received messages from the process to generate a translated call sequence;

identify the translated call sequence as an anomaly based on selecting of a portion of the translated call sequence and a determination that the portion of the translated call sequence does not match at least one of the call sequences in the set of call sequences; and

cause an action based on identification of the translated call sequence as an anomaly.

13. The computing system of claim 12, wherein the instructions, when executed, are to cause the processor to determine a corresponding invocation identifier by:

generation of a new invocation hash based on at least the programmatic procedure identifier and the receiver process identifier; and

a determination of the corresponding invocation identifier from the new invocation hash and stored associations between invocation hashes and invocation identifiers.

14. The computing system of claim 13, wherein the message further includes a channel identifier, a sender process identifier, a receiver node identifier and wherein the invocation hash is further based on at least the channel identifier or the sender process identifier.

15. The computing system of claim 12, wherein each of the previously-received messages contains a same sender process identifier, and the new message contains the same sender process identifier.

16. The computing system of claim 12, wherein selection of a portion is based on window size.

17. The computing system of claim 12, wherein the instructions, when executed, are to cause the processor to append the corresponding invocation identifier by removal of an oldest invocation identifier from the sequence of previously-determined invocation identifiers to maintain the translated call sequence at a fixed number of invocation identifiers.