Method, Apparatus and Computer Program Product for Emotion Detection

• US 20140025385A1
• Filed: 11/15/2011
• Published: 01/23/2014
• Est. Priority Date: 12/30/2010
• Status: Abandoned Application

• Current Assignee: Nokia Technologies Oy (Nokia Corporation)
• Original Assignee: Nokia Corporation
• Inventors: Atri, Rohit, Patil, Sidharth, S, Basavaraja V
• Application Number:
  US13/996,146
• Publication Number:
  US 20140025385A1
• Time in Patent Office: Days
• US Class Current: 704/270
• CPC Class Codes: G10L 25/57 for processing of video signals
• CPC Class Codes: G10L 25/63 for estimating an emotional state
• CPC Class Codes: H04N 21/42203 sound input device, e.g. microphone
• CPC Class Codes: H04N 21/4394 involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams arrangements characterised by components specially adapted for monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/44218 Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV program methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45
• CPC Class Codes: H04N 5/77 between a recording apparatus and a television camera

Abstract

In accordance with an example embodiment a method and apparatus is provided. The method comprises determining a value of at least one speech element associated with the audio stream. The value of the at least one speech element is compared with at least one threshold value of the speech element. Processing of a video stream is initiated based on the comparison of the value of the at least one speech element with the at least one threshold value. The video stream is associated with the audio stream. An emotional state is determined based on the processing of the video stream.

Claims

1-56. -56. (canceled)

57. A method comprising:

determining a value of at least one speech element associated with an audio stream;

comparing the value of the at least one speech element with at least one threshold value of the speech element;

initiating processing of a video stream associated with the audio stream based on the comparison; and

determining an emotional state based on the processing of the video stream.

58. The method of claim 57, wherein the at least one threshold value comprises:

at least one upper threshold limit representative of the value of the at least one speech element in at least one loudly expressed emotional state, and

at least one lower threshold limit representative of the value of the at least one speech element in at least one subtly expressed emotional state.

59. The method of claim 58, wherein the at least one upper threshold value is determined by:

performing for the at least one loudly expressed emotional state;

determining, for a plurality of audio streams, a plurality of values (X_li) of the at least one speech element associated with the at least one loudly expressed emotional state; and

determining a minimum value (X<sub>li—

min</sub>) of the plurality of values (X_li); and

calculating the at least one upper threshold limit (X_u) from the equation;

X_u=Σ

(X<sub>lin—

min</sub>)/n,

where n is the number of the at least one loudly expressed emotional states.

60. The method of claim 58, wherein the at least one lower threshold value is determined by:

performing for the at least one subtly expressed emotional state;

determining, for a plurality of audio streams, a plurality of values (X_si) of the at least one speech element associated with the at least one subtly expressed emotional state; and

determining a minimum value (X<sub>si—

min</sub>) of the plurality of values (X_si); and

calculating the at least one lower threshold limit (X₁) from the equation;

X₁=Σ

(X<sub>sin—

min</sub>)/n,

where n is the number of the at least one subtly expressed emotional states.

61. The method of claim 58, wherein the processing of the video stream is initiated if the value of the at least one speech element is determined to be higher than the at least one upper threshold limit;

or

if the value of the at least one speech element is determined to be less than the at least one lower threshold limit.

62. The method of claim 58, wherein the comparison of the value of the at least one speech element with the at least one threshold value is performed for a predetermined time period.

63. The method of claim 62 further comprising:

decrementing the at least one upper threshold limit if the value of the at least one speech element is determined to be less than the at least one upper value threshold limit for the predetermined time period;

or

incrementing the at least one lower threshold limit if the value of the at least one speech element is determined to be higher than the lower threshold limit for the predetermined time period.

64. The method of claim 62 further comprising:

incrementing the at least one upper threshold limit if the value of the at least one speech element is determined to be higher than the upper threshold limit at least a predetermined number of times during the predetermined time period;

or

decrementing the at least one lower threshold limit if the value of the at least one speech element is determined to be less than the one lower threshold limit at least a predetermined number of times during the predetermined time period.

65. The method of claim 57, wherein the at least one threshold value is determined by performing:

computing a percentage change in the value of at least one speech element associated with the audio stream from at least one emotional state to a neutral emotional state;

monitoring the video stream to determine value of the at least one speech element at a current emotional state; and

determining an initial value of the at least one threshold value based on the value of the at least one speech element at the current emotional state, and the computed percentage change in the value of at least one speech element.

66. An apparatus comprising:

at least one processor; and

at least one memory comprising computer program code, the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus at least to perform;

determine a value of at least one speech element associated with an audio stream;

compare the value of the at least one speech element with at least one threshold value of the speech element;

initiate processing of a video stream associated with the audio stream based on the comparison; and

determine an emotional state based on the processing of the video stream.

67. The apparatus of claim 66, wherein the at least one threshold value comprises:

at least one upper threshold limit representative of the value of the at least one speech element in at least one loudly expressed emotional state, and

at least one lower threshold limit representative of the value of the at least one speech element in at least one subtly expressed emotional state.

68. The apparatus of claim 67, wherein, to determine the at least one upper threshold value, the apparatus is further caused, for the at least one loudly expressed emotional states, at least in part, to perform:

determine for a plurality of audio streams, a plurality of values (X_li) of the at least one speech element associated with the at least one loudly expressed emotional state; and

determine a minimum value (X<sub>li—

min</sub>) of the plurality of values (X_li); and

calculate the at least one upper threshold limit (X_u) from the equation;

X_u=Σ

(X<sub>lin—

min</sub>)/n,

where n is the number of the at least one loudly expressed emotional states.

69. The apparatus of claim 67, wherein, to determine the at least one lower threshold value, the apparatus is further caused, for the at least one subtly expressed emotional state, at least in part, to perform:

determine for a plurality of audio streams, a plurality of values (X_si) of the at least one speech element associated with the at least one subtly expressed emotional state; and

determine a minimum value (X<sub>si—

min</sub>) of the plurality of values (X_si); and

calculate the at least one lower threshold limit (X_l) from the equation;

X_l=Σ

(X<sub>sin—

min</sub>)/n,

where n is the number of the at least one subtly expressed emotional states.

70. The apparatus of claim 67, wherein the apparatus is further caused, at least in part, to perform:

initiate the processing of the video stream if the value of the at least one speech element is determined to be higher than the at least one upper threshold limit;

or

if the value of the at least one speech element is determined to be less than the at least one lower threshold limit.

71. The apparatus of claim 67, wherein the apparatus is further caused, at least in part, to perform the comparison of the value of the at least one speech element with the at least one threshold value for a predetermined time period.

72. The apparatus of claim 71, wherein the apparatus is further caused, at least in part, to perform:

decrement the at least one upper threshold limit if the value of the at least one speech element is determined to be less than the at least one upper value threshold limit for the predetermined time period;

or

increment the at least one lower threshold limit upon determining the value of the at least one speech element being higher than the lower threshold limit for the predetermined time period.

73. The apparatus of claim 71, wherein the apparatus is further caused, at least in part, to perform:

increment the at least one upper threshold limit if the value of the at least one speech element is determined to be higher than the one upper threshold limit at least a predetermined number of times during the predetermined time period;

or

decrement the at least one lower threshold limit if the value of the at least one speech element is determined to be less than the one lower threshold limit at least a predetermined number of times during the predetermined time period.

74. The apparatus of claim 66, wherein, determine the at least one threshold value, the apparatus is further caused, at least in part, to perform:

compute a percentage change in the value of at least one speech element associated with the audio stream from at least one emotional state to a neutral emotional state;

monitor the video stream to determine value of the at least one speech element at a current emotional state; and

determine an initial value of the at least one threshold value based on the value of the at least one speech element at the current emotional state, and the computed percentage change in the value of at least one speech element.

75. A computer program product comprising at least one computer-readable storage medium, the computer-readable storage medium comprising a set of instructions, which, when executed by one or more processors, cause an apparatus at least to perform:

determine a value of at least one speech element associated with an audio stream;

compare the value of the at least one speech element with at least one threshold value of the speech element;

initiate processing of a video stream associated with the audio stream based on the comparison; and

determine an emotional state based on the processing of the video stream.

76. The computer program product of claim 75, wherein the at least one threshold value comprises:

at least one upper threshold limit representative of the value of the at least one speech element in at least one loudly expressed emotional state, and

at least one lower threshold limit representative of the value of the at least one speech element in at least one subtly expressed emotional state.