Audience state estimation system, audience state estimation method, and audience state estimation program

• US 8,244,537 B2
• Filed: 05/13/2008
• Issued: 08/14/2012
• Est. Priority Date: 06/26/2002
• Status: Expired due to Fees

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Kondo, Tetsujiro, Okumura, Yuji, Fujishima, Koichi, Ohtsuki, Tomoyuki
• Primary Examiner(s): He, Jialong
• Application Number:
  US12/119,844
• Publication Number:
  US 20080263580A1
• Time in Patent Office: 1,554 Days
• Field of Search: 704/270, 704/275, 704/200, 704/205, 704/211, 725/10, 725/12
• US Class Current: 704/270
• CPC Class Codes: G10L 17/26 Recognition of special voice characteristics, e.g. for use in lie detectors; Recognition of animal voices
• CPC Class Codes: H04H 60/33 Arrangements for monitoring the users' behaviour or opinions
• CPC Class Codes: H04N 21/41415 involving a public display, viewable by several users in a public space outside their home, e.g. movie theatre, information kiosk
• CPC Class Codes: H04N 21/42203 sound input device, e.g. microphone
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/4394 involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams arrangements characterised by components specially adapted for monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/44213 Monitoring of end-user related data arrangements for monitoring the users' behaviour or opinions in broadcast systems H04H60/33
• CPC Class Codes: H04N 21/44218 Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV program methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45
• CPC Class Codes: H04N 7/163 by receiver means only

Abstract

Video signal relative to an imaged audience and audio signal according to voices from the audience are generated in an input unit. A characteristic amount detection unit detects information on a movement amount, movement periodicity, a volume, voice periodicity of the audience, and a frequency component of voices from the audience based on the video signal or the audio signal. An estimation unit estimates an audience state based on the detected result. An output unit outputs the estimated result of the audience state. The audience state can be easily estimated without observing the audience state by a person.

Claims

1. An audience state estimation system comprising:

sound-obtaining device for obtaining sound from an audience and generating an audio signal according to the sound thus obtained;

volume detection device for detecting a volume of the sound from said audience based on said audio signal,

wherein the volume detection device calculates variance of volume of said audio signal for each frame, and averages the calculated variance of the volume in time direction to calculate a time macro volume, the number of frames averaged in calculating the time macro volume being set according to an audience state to be estimated; and

estimation device for estimating an audience state based on said volume.

2. The audience state estimation system according to claim 1, wherein said sound from the audience includes voice.

3. The audience state estimation system according to claim 1, wherein in said volume detection device, the variance of volume of said audio signal is set as said volume.

4. The audience state estimation system according to claim 1, wherein the time macro volume is set as said volume.

5. The audience state estimation system according to claim 1, wherein when said volume is larger than a predetermined level, said estimation device estimates said audience state to be in a state of laughing.

6. An audience state estimation system comprising:

sound-obtaining device for obtaining sound from an audience and generating an audio signal according to the sound thus obtained;

sound periodicity detection device for detecting sound periodicity from said audience based on said audio signal,

wherein the sound periodicity detection device calculates a ratio of a power of a low-frequency component to all powers a range of the low-frequency component being set according to the sound periodicity of a state to be detected; and

estimation device for estimating an audience state based on said sound periodicity.

7. The audience state estimation system according to claim 6, wherein said sound from the audience includes voice.

8. The audience state estimation system according to claim 6, wherein said sound periodicity detection device sequentially detects autocorrelation maximum positions from said audio signal, and wherein variance of the autocorrelation maximum position is set as said sound periodicity.

9. The audience state estimation system according to claim 6, further comprising volume detection device, wherein in said sound periodicity detection device, a ratio of low-frequency component to a time macro volume in a volume of the sound from said audience detected by said volume detection device based on said audio signal is set as said sound periodicity.

10. The audience state estimation system according to claim 9, wherein a frequency range of said low-frequency component is decided according to the sound periodicity to be detected.

11. The audience state estimation system according to claim 6, wherein when said sound periodicity is larger than a predetermined level, said estimation device estimates said audience state to be in a state of clapping.

12. An audience state estimation system comprising:

sound-obtaining device for obtaining sound from an audience and generating an audio signal according to the sound thus obtained;

spectrum detection device for performing Fourier transform on said audio signal and detecting a center of gravity of the Fourier transformed result,

wherein the spectrum detection device calculates the center of gravity for a time length as an average obtained by weighting power to frequency, the time length being set to include a frequency component corresponding to an audience state to be detected; and

estimation device for estimating an audience state based on said center of gravity.

13. The audience state estimation system according to claim 12, wherein said sound from the audience includes voice.

14. The audience state estimation system according to claim 12, wherein when said center of gravity is larger than a predetermined level, said estimation device estimates said audience state to be in any one of states of beating time with the hands and of clapping.

15. An audience state estimation system comprising:

sound-obtaining device for obtaining sound from an audience and generating an audio signal according to the sound thus obtained;

component ratio detection device for performing Fourier transform on said audio signal and detecting a high-frequency to low-frequency component ratio by calculating said ratio based on the Fourier transformed result,

wherein component ratio detection device sets a predetermined section according to an audience state to be estimated, and divides power for the predetermined section into a high-frequency component and a low-frequency component, and calculates the high-frequency to low-frequency component ratio; and

estimation device for estimating an audience state based on said high-frequency to low-frequency component ratio.

16. The audience state estimation system according to claim 15, wherein said sound from the audience includes voice.

17. The audience state estimation system according to claim 15, wherein when in said high-frequency to low-frequency component ratio, a ratio of said high-frequency component to said low-frequency component is larger than a predetermined level, said estimation device estimates said audience state to be in any one of states of beating time with the hands and of clapping.

18. An audience state estimation method comprising:

obtaining sound from an audience and generating an audio signal according to the sound thus obtained, utilizing a sound-obtaining device;

detecting a volume of the sound from said audience based on said audio signal and calculating variance of volume of said audio signal for each frame, and averaging the calculated variance of the volume in time direction to calculate a time macro volume, the number of frames averaged in calculating the time macro volume being set according to an audience state to be estimated, utilizing a volume detection device; and

estimating an audience state based on said volume, utilizing an estimation device.

19. The audience state estimation method according to claim 18, wherein said sound from the audience includes voice.

20. The audience state estimation method according to claim 18, wherein the variance of volume of said audio signal is set as said volume.

21. The audience state estimation method according to claim 18, wherein the time macro volume is set as said volume.

22. The audience state estimation method according to claim 18, wherein when said volume is larger than a predetermined level, said audience state is estimated to be in a state of laughing.

23. An audience state estimation method comprising:

obtaining sound from an audience and generating an audio signal according to the sound thus obtained, utilizing a sound-obtaining device;

detecting sound periodicity from said audience based on said audio signal and calculating a ratio of a power of a low-frequency component to all powers, a range of the low-frequency component being set according to the sound periodicity of a state to be detected; and

estimating an audience state based on said sound periodicity, utilizing estimation device.

24. The audience state estimation method according to claim 23, wherein said sound from the audience includes voice.

25. The audience state estimation method according to claim 23, wherein autocorrelation maximum positions are sequentially detected from said audio signal, and wherein variance of the autocorrelation maximum position is set as said sound periodicity.

26. The audience state estimation method according to claim 23, wherein a ratio of low-frequency component to the time macro volume in the volume of the sound from said audience detected based on said audio signal is set as said sound periodicity.

27. The audience state estimation method according to claim 23, wherein when said sound periodicity is larger than a predetermined level, said audience state is estimated to be in a state of clapping.

28. An audience state estimation method comprising:

obtaining sound from an audience and generating an audio signal according to the sound thus obtained, utilizing a sound-obtaining device;

performing Fourier transform on said audio signal and detecting a center of gravity of the Fourier transformed result, utilizing a spectrum detection device,

wherein the center of gravity is calculated for a time length as an average obtained by weighting power to frequency, the time length being set to include a frequency component corresponding to an audience state to be detected; and

estimating an audience state based on said center of gravity, utilizing estimation device.

29. The audience state estimation method according to claim 28, wherein said sound from the audience includes voice.

30. The audience state estimation method according to claim 29, wherein when said center of gravity is larger than a predetermined level, said audience state is estimated to be in any one of states of beating time with the hands and of clapping.

31. An audience state estimation method comprising:

obtaining sound from an audience and generating an audio signal according to the sound thus obtained, utilizing a sound-obtaining device;

performing Fourier transform on said audio signal, setting a predetermined section according to an audience state to be estimated, and dividing power for the predetermined section into a high-frequency component and a low-frequency component, and calculating the high-frequency to low-frequency component ratio based on the Fourier transformed result, utilizing a component ratio detection device; and

estimating an audience state based on said high-frequency to low-frequency component ratio, utilizing estimation device.

32. The audience state estimation method according to claim 31, wherein said sound from the audience includes voice.

33. The audience state estimation method according to claim 31, wherein when in said high-frequency to low-frequency component ratio, a ratio of said high-frequency component to said low-frequency component is larger than a predetermined level, said audience state is estimated to be in any one of states of beating time with the hands and of clapping.