Detecting behavioral deviations by measuring eye movements

• US 8,345,935 B2
• Filed: 03/17/2008
• Issued: 01/01/2013
• Est. Priority Date: 03/17/2008
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Angell, Robert Lee, Kraemer, James R.
• Primary Examiner(s): Fitzpatrick, Atiba O
• Application Number:
  US12/049,818
• Publication Number:
  US 20090232357A1
• Time in Patent Office: 1,751 Days
• Field of Search: 382/100, 382/103, 382115-118, 351200-207, 600300-305, 340 582- 583, 128920-923
• US Class Current: 382/117
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/19 Sensors therefor

Abstract

Detecting behavioral deviations in members of a cohort group is provided. A member of a cohort group is identified. Each member of the cohort group shares a common characteristic. Ocular metadata associated with the member of the cohort group is generated in real-time. The ocular metadata describes movements of an eye of the member of the cohort group. The ocular metadata is analyzed to identify patterns of ocular movements. In response to the patterns of ocular movements indicating behavioral deviations in the member of the cohort group, the member of the cohort group is identified as a person of interest. A person of interest may be subjected to an increased level of monitoring and/or other security measures.

Claims

1. A computer implemented method for detecting behavioral deviations in members of a cohort group, the computer implemented method comprising:

generating ocular metadata associated with a member of a cohort group using images of eye movements captured by a set of cameras, wherein the ocular metadata describes movements of an eye of the member of the cohort group and describes changes in a size of a pupil of the eye due to dilation and constriction of the pupil, and wherein the ocular metadata is generated automatically in real time as the images of the eye movements are received;

analyzing the ocular metadata using a set of data models to identify patterns of ocular movements;

analyzing the ocular metadata to identify patterns in the changes in the size of the pupil of the eye, wherein the patterns in the changes in the size of the pupil comprises a rate of changes in the size of the pupil, a degree of change in the size of the pupil, a number of changes in the size of the pupil, and wherein the patterns in the changes in the size of the pupil indicates any external stimuli associated with the changes in the size of the pupil; and

responsive to the patterns of ocular movements and the changes in the size of the pupil of the eye indicating behavioral deviations in the member of the cohort group, identifying the member of the cohort group as a person of interest, wherein the person of interest is subject to an increased level of monitoring.

2. The computer implemented method of claim 1 further comprising:

analyzing the ocular metadata to identify a given direction of gaze and a duration the gaze remains fixed in the given direction; and

responsive to the ocular movements indicating a direction of gaze in the given direction for a duration of time that is greater than a threshold duration, indicating an occurrence of a fixed stare in the patterns of ocular movements.

3. The computer implemented method of claim 1 further comprising:

analyzing the ocular metadata to identify visual lines of gaze, wherein a visual line of gaze indicates where the eye is looking;

responsive to the visual lines of gaze indicating the member of the cohort group is watching a single object or a type of object to form a target object or target type of object, indicating the member of the cohort group is watching the target object or the target type of object in the patterns of ocular movements; and

analyzing the patterns of ocular movements to determine whether the target object or target type of object indicates a behavioral deviation.

4. The computer implemented method of claim 3 further comprising:

analyzing the ocular metadata to identify a number of times the visual line of gaze is redirected from a first direction to a second direction over a time interval to form a rate of redirection of the gaze; and

responsive to the rate of redirection of the gaze exceeding a threshold rate, identifying the ocular movements as a furtive glance; and

responsive to identifying a set of furtive glances occurring over a threshold interval of time, identifying the set of furtive glances as a pattern of ocular movements in the patterns of ocular movements.

5. The computer implemented method of claim 1 further comprising:

analyzing the ocular metadata to identify a rate of blinking over an interval of time; and

responsive to the rate of blinking falling below a threshold rate of blinking, identifying a pattern of ocular movement comprising an unblinking stare, wherein the unblinking stare is a pattern of ocular movements in the patterns of ocular movements.

6. The computer implemented method of claim 1 wherein the patterns of ocular movements indicating behavioral deviations comprises at least one of a set of furtive glances, an unblinking stare, a fixed stare, an atypical rate of blinking, an atypical target object or type of object watched by the person, a flat or unemotional affect, atypical patterns in pupil dilation, atypical narrowing of the eyes, and atypical widening of the eyes.

7. The computer implemented method of claim 1 wherein the cohort group is a group of people in a transportation facility.

8. The computer implemented method of claim 7 wherein the transportation facility is an airport.

9. The computer implemented method of claim 1 wherein increasing a level of monitoring associated with the person of interest comprises at least one of continuing capturing video images of the eye movements and behavior of the person of interest, questioning the person of interest by security personnel, checking an identification of the person of interest, searching baggage associated with the person of interest, searching the person of interest, monitoring the activities of the person of interest by security personnel, scanning the person of interest with a metal detector or metal detector wand, x-ray the person of interest, and checking the person of interest with a drug sniffing dog.

10. A computer program product for detecting behavioral deviations in members of a cohort group, the computer program product comprising:

a computer readable storage device;

program code stored on the computer readable storage device for generating ocular metadata associated with a member of a cohort group using images of eye movements captured by a set of cameras, wherein the ocular metadata describes movements of an eye of the member of the cohort group and describes changes in a size of a pupil of the eye due to dilation and constriction of the pupil, and wherein the ocular metadata is generated automatically in real time as the images of the eye movements are received;

program code stored on the computer readable storage device for analyzing the ocular metadata using a set of data models to identify patterns of ocular movements;

program code stored on the computer readable storage device for analyzing the ocular metadata to identify patterns in the changes in the size of the pupil of the eye, wherein the patterns in the changes in the size of the pupil comprises a rate of changes in the size of the pupil, a degree of change in the size of the pupil, a number of changes in the size of the pupil, and wherein the patterns in the changes in the size of the pupil indicates any external stimuli associated with the changes in the size of the pupil; and

program code stored on the computer readable storage device for identifying the member of the cohort group as a person of interest in response to the patterns of ocular movements and the changes in the size of the pupil of the eye indicating behavioral deviations in the member of the cohort group, wherein the person of interest is subject to an increased level of monitoring.

11. The computer program product of claim 10 further comprising:

program code stored on the computer readable storage device for analyzing the ocular movements to identify a given direction of gaze and a duration the gaze remains fixed in the given direction; and

program code stored on the computer readable storage device for indicating an occurrence of a fixed stare in the patterns of ocular movements in response to the ocular movements indicating a direction of gaze in the given direction for a duration of time that is greater than a threshold duration.

12. The computer program product of claim 10 further comprising:

program code stored on the computer readable storage device for analyzing the ocular metadata to identify visual lines of gaze, wherein a visual line of gaze indicates where the eye is looking;

program code stored on the computer readable storage device for indicating the member of the cohort group is watching a target object or a target type of object in the patterns of ocular movements in response to the visual lines of gaze indicating the member of the cohort group is watching a single object or a type of object to form the target object or the target type of object; and

program code stored on the computer readable storage device for analyzing the patterns of ocular movements to determine whether the target object or target type of object indicates a behavioral deviation.

13. The computer program product of claim 12 further comprising:

program code stored on the computer readable storage device for analyzing the ocular metadata to identify a number of times the visual line of gaze is redirected from a first direction to a second direction over a time interval to form a rate of redirection of the gaze; and

program code stored on the computer readable storage device for identifying the ocular movements as a furtive glance in response to the rate of redirection of the gaze exceeding a threshold rate; and

program code stored on the computer readable storage device for identifying a set of furtive glances as a pattern of ocular movements in the patterns of ocular movements in response to identifying the set of furtive glances occurring over a threshold interval of time.

14. The computer program product of claim 10 further comprising:

program code stored on the computer readable storage device for analyzing the ocular metadata to identify a rate of blinking over an interval of time; and

program code stored on the computer readable storage device for identifying a pattern of ocular movement comprising an unblinking stare in response to the rate of blinking falling below a threshold rate of blinking, wherein the unblinking stare is a pattern of ocular movements in the patterns of ocular movements.

15. An apparatus comprising:

a bus system;

a communications system coupled to the bus system;

a storage device connected to the bus system, wherein the storage device stores computer usable program code; and

a processor unit coupled to the bus system, wherein the processor unit executes the computer usable program code to generate ocular metadata associated with a member of a cohort group using images of eye movements captured by a set of cameras, wherein the ocular metadata describes movements of an eye of the member of the cohort group and describes changes in a size of a pupil of the eye due to dilation and constriction of the pupil, and wherein the ocular metadata is generated automatically in real time as the images of the eye movements are received;

analyze the ocular metadata using a set of data models to identify patterns of ocular movements;

analyze the ocular metadata to identify patterns in the changes in the size of the pupil of the eye, wherein the patterns in the changes in the size of the pupil comprises a rate of changes in the size of the pupil, a degree of change in the size of the pupil, a number of changes in the size of the pupil, and wherein the patterns in the changes in the size of the pupil indicates any external stimuli associated with the changes in the size of the pupil; and

identify the member of the cohort group as a person of interest in response to the patterns of ocular movements and the changes in the size of the pupil of the eye indicating behavioral deviations in the member of the cohort group, wherein the person of interest is subject to an increased level of monitoring.

16. The apparatus of claim 15 wherein the processor unit further executes the computer usable program code to analyze the patterns of ocular movements to analyze the ocular movements to identify a given direction of gaze and a duration the gaze remains fixed in the given direction;

and indicate an occurrence of a fixed stare in the patterns of ocular movements in response to the ocular movements indicating a direction of the gaze in the given direction for a duration of time is greater than a threshold duration.

17. The apparatus of claim 15 wherein the processor unit further executes the computer usable program code to analyze the ocular metadata to identify visual lines of gaze, wherein a visual line of gaze indicates where the eye is looking;

indicate the member of the cohort group is watching a target object or a target type of object in the patterns of ocular movements in response to the visual lines of gaze indicating the member of the cohort group is watching a single object or a type of object to form the target object or the target type of object; and

analyze the patterns of ocular movements to determine whether the target object or target type of object indicates a behavioral deviation.

18. The apparatus of claim 15 wherein the processor unit further executes the computer usable program code to analyze the ocular metadata to identify a rate of blinking over an interval of time;

and identify a pattern of ocular movement comprising an unblinking stare in response to the rate of blinking falling below a threshold rate of blinking, wherein the unblinking stare is a pattern of ocular movements in the patterns of ocular movements.

19. A data processing system for detecting behavioral deviations in members of a cohort group, the data processing system comprising:

a video analysis system, wherein the video analysis system generates ocular metadata associated with a member of a cohort group using images of eye movements associated with the member of the cohort group, wherein the images of the eye movements are captured by a set of cameras, wherein the ocular metadata describes movements of an eye of the member of the cohort group and describes changes in a size of a pupil of the eye due to dilation and constriction of the pupil, and wherein the ocular metadata is generated automatically in real time as the images of the eye movements are received; and

an analysis server, wherein the analysis server analyzes the ocular metadata using analytics and a set of data models to identify patterns of ocular movement and patterns in the changes in the size of the pupil of the eye, wherein the patterns in the changes in the size of the pupil comprises a rate of changes in the size of the pupil, a degree of change in the size of the pupil, a number of changes in the size of the pupil, and wherein the patterns in the changes in the size of the pupil indicates any external stimuli associated with the changes in the size of the pupil, and wherein the analysis server identifies the member of the cohort group as a person of interest in response to the patterns of ocular movements and the changes in the size of the pupil of the eye indicating behavioral deviations in the member of the cohort group.

20. The data processing system of claim 19 further comprising:

the set of cameras, wherein the set of cameras generates the images of the eye movements associated with a member of a cohort group, and wherein the set of cameras are located in a transportation facility.

21. The data processing system of claim 19 wherein the video analysis system further comprises:

an ocular movement analytics engine, wherein the ocular movement analytics engine analyzes the ocular metadata to identify a number of times a visual line of gaze is redirected from a first direction to a second direction over a time interval to form a rate of redirection of the gaze;

identify the ocular movements as a furtive glance in response to the rate of redirection of the gaze exceeding a threshold rate; and

identify a set of furtive glances as a pattern of ocular movements in the patterns of ocular movements in response to identifying the set of furtive glances occurring over a threshold interval of time.