SYSTEM AND METHOD FOR DETECTING INVISIBLE HUMAN EMOTION

• US 20160098592A1
• Filed: 09/29/2015
• Published: 04/07/2016
• Est. Priority Date: 10/01/2014
• Status: Abandoned Application

• Current Assignee: NuraLogix Corporation
• Original Assignee: The Governing Council of the University of Toronto (University of Toronto)
• Inventors: LEE, Kang, ZHENG, Pu
• Application Number:
  US14/868,601
• Publication Number:
  US 20160098592A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 18/24155 Bayesian classification
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20224 Image subtraction
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 2201/03 Recognition of patterns in medical or anatomical images
• CPC Class Codes: G06V 40/15 Biometric patterns based on physiological signals, e.g. heartbeat, blood flow
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/176 Dynamic expression
• CPC Class Codes: G09B 19/00 Teaching not covered by other main groups of this subclass teaching or practice apparatus for gun-aiming or gun-laying F41G3/26
• CPC Class Codes: G16H 15/00 ICT specially adapted for medical reports, e.g. generation or transmission thereof
• CPC Class Codes: G16H 30/40 for processing medical images, e.g. editing
• CPC Class Codes: G16H 50/20 for computer-aided diagnosis, e.g. based on medical expert systems

Abstract

A system and method for emotion detection and more specifically to an image-capture based system and method for detecting invisible and genuine emotions felt by an individual. The system provides a remote and non-invasive approach by which to detect invisible emotion with a high confidence. The system enables monitoring of hemoglobin concentration changes by optical imaging and related detection systems.

Claims

1. A system for detecting invisible human emotion expressed by a subject from a captured image sequence of the subject, the system comprising an image processing unit trained to determine a set of bitplanes of a plurality of images in the captured image sequence that represent the hemoglobin concentration (HC) changes of the subject, and to detect the subject'"'"'s invisible emotional states based on HC changes, the image processing unit being trained using a training set comprising a set of subjects for which emotional state is known.

2. The system of claim 1, wherein the image processing unit isolates the hemoglobin concentration in each image of the captured image sequence to obtain transdermal hemoglobin concentration changes.

3. The system of claim 2, wherein the training set comprises a plurality of captured image sequences obtained for a plurality of human subjects exhibiting various known emotions determinable from the transdermal blood changes.

4. The system of claim 3, wherein the training set is obtained by capturing image sequences from the human subjects being exposed to stimuli known to elicit specific emotional responses.

5. The system of claim 4, wherein the system further comprises a facial expression detection unit configured to determine whether each captured image shows a visible facial response to the stimuli and, upon making the determination that the visible facial response is shown, discard the respective image.

6. The system of claim 1, wherein the image processing unit further processes the captured image sequence to remove signals associated with cardiac, respiratory, and blood pressure activities.

7. The system of claim 6, wherein the system further comprises an EKG machine, a pneumatic respiration machine, and a continuous blood pressure measuring system and the removal comprises collecting EKG, pneumatic respiratory, and blood pressure data from the subject.

8. The system of claim 7, wherein the removal further comprises de-noising.

9. The system of claim 8, wherein the de-noising comprises one or more of Fast Fourier Transform (FFT), notch and band filtering, general linear modeling, and independent component analysis (ICA).

10. The system of claim 1, wherein the image processing unit determines HC changes on one or more regions of interest comprising the subject'"'"'s forehead, nose, cheeks, mouth, and chin.

11. The system of claim 10, wherein the image processing unit implements reiterative data-driven machine learning to identify the optimal compositions of the biplanes that maximize detection and differentiation of invisible emotional states.

12. The system of claim 11, wherein the machine learning comprises manipulating bitplane vectors using image subtraction and addition to maximize the signal differences in the regions of interest between different emotional states across the image sequence.

13. The system of claim 12, wherein the subtraction and addition are performed in a pixelwise manner.

14. The system of claim 1, wherein the training set is a subset of preloaded images, the remaining images comprising a validation set.

15. The system of claim 1, wherein the HC changes are obtained from any one or more of the subject'"'"'s face, wrist, hand, torso, or feet.

16. The system of claim 15, wherein the image processing unit is embedded in one of a wrist watch, wrist band, hand band, clothing, footwear, glasses or steering wheel.

17. The system of claim 1, wherein the image processing unit applies machine learning processes during training.

18. The system of claim 1, wherein the system further comprises an image capture device and an image display device, the image display device providing images viewable by the subject, and the subject viewing the images.

19. The system of claim 18, wherein the images are marketing images.

20. The system of claim 18, wherein the images are images relating to health care.

21. The system of claim 18, wherein the images are used to determine deceptiveness of the subject in screening or interrogation.

22. The system of claim 18, wherein the images are intended to elicit an emotion, stress or fatigue response.

23. The system of claim 18, wherein the images are intended to elicit a risk response.

24. The system of claim 1, wherein the system is implemented in robots.

25. The system of claim 4, wherein the stimuli comprises auditory stimuli.

26. A method for detecting invisible human emotion expressed by a subject, the method comprising:

capturing an image sequence of the subject, determining a set of bitplanes of a plurality of images in the captured image sequence that represent the hemoglobin concentration (HC) changes of the subject, and detecting the subject'"'"'s invisible emotional states based on HC changes using a model trained using a training set comprising a set of subjects for which emotional state is known.

27. The method of claim 26, wherein the image processing unit isolates the hemoglobin concentration in each image of the captured image sequence to obtain transdermal hemoglobin concentration changes.

28. The method of claim 27, wherein the training set comprises a plurality of captured image sequences obtained for a plurality of human subjects exhibiting various known emotions determinable from the transdermal blood changes.

29. The method of claim 28, wherein the training set is obtained by capturing image sequences from the human subjects being exposed to stimuli known to elicit specific emotional responses.

30. The method of claim 29, wherein the method further comprises determining whether each captured image shows a visible facial response to the stimuli and, upon making the determination that the visible facial response is shown, discarding the respective image.

31. The method of claim 26, wherein the method further comprises removing signals associated with cardiac, respiratory, and blood pressure activities.

32. The method of claim 31, wherein the removal comprises collecting EKG, pneumatic respiratory, and blood pressure data from the subject using an EKG machine, a pneumatic respiration machine, and a continuous blood pressure measuring system.

33. The method of claim 32, wherein the removal further comprises de-noising.

34. The method of claim 33, wherein the de-noising comprises one or more of Fast Fourier Transform (FFT), notch and band filtering, general linear modeling, and independent component analysis (ICA).

35. The method of claim 26, wherein the HC changes are on one or more regions of interest, comprising the subject'"'"'s forehead, nose, cheeks, mouth, and chin.

36. The method of claim 35, wherein the image processing unit implements reiterative data-driven machine learning to identify the optimal compositions of the biplanes that maximize detection and differentiation of invisible emotional states.

37. The method of claim 36, wherein the machine learning comprises manipulating bitplane vectors using image subtraction and addition to maximize the signal differences in the regions of interest between different emotional states across the image sequence.

38. The method of claim 37, wherein the subtraction and addition are performed in a pixelwise manner.

39. The method of claim 26, wherein the training set is a subset of preloaded images, the remaining images comprising a validation set.

40. The method of claim 26, wherein the HC changes are obtained from any one or more of the subject'"'"'s face, wrist, hand, torso or feet.

41. The method of claim 40, wherein the method is implemented by one of a wrist watch, wrist band, hand band, clothing, footwear, glasses or steering wheel.

42. The method of claim 26, wherein the image processing unit applies machine learning processes during training.

43. The method of claim 26, wherein the method further comprises providing images viewable by the subject, and the subject viewing the images.

44. The method of claim 43, wherein the images are marketing images.

45. The method of claim 43, wherein the images are images relating to health care.

46. The method of claim 43, wherein the images are used to determine deceptiveness of the subject in screening or interrogation

47. The method of claim 43, wherein the images are intended to elicit an emotion, stress or fatigue response.

48. The method of claim 43, wherein the images are intended to elicit a risk response.

49. The method of claim 26, wherein the method is implemented by robots.

50. The method of claim 29, wherein the stimuli comprises auditory stimuli.