Audience response determination

• US 7,555,766 B2
• Filed: 09/28/2001
• Issued: 06/30/2009
• Est. Priority Date: 09/29/2000
• Status: Expired due to Fees

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Kondo, Tetsujiro, Okumura, Yuji
• Primary Examiner(s): Saltarelli; Dominic D
• Application Number:
  US09/966,704
• Publication Number:
  US 20020073417A1
• Time in Patent Office: 2,832 Days
• Field of Search: 725 9- 21, 340/666, 340/667, 52/6, 521/36, 345/418, 345/156, 715/703, 715/704, 715/7, 200/85, 745/12
• US Class Current: 725/12
• CPC Class Codes: H04H 60/33 Arrangements for monitoring the users' behaviour or opinions
• CPC Class Codes: H04N 21/439 Processing of audio elementary streams
• CPC Class Codes: H04N 21/44218 Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV program methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45
• CPC Class Codes: H04N 21/466 Learning process for intelligent management, e.g. learning user preferences for recommending movies services using the results of monitoring in broadcast systems H04H60/61
• CPC Class Codes: H04N 7/002 Special television systems not provided for by H04N7/007 - H04N7/18 still pictures via a television channel H04N1/00098

Abstract

An audience response determination apparatus detects the state of an audience and the individual states of the members of the audience to determine the response of the audience to the content of entertainment and the like. Based on the thus determined audience response, a playback operation of the content such as a movie is controlled, so that the level of satisfaction felt by the audience to the content can be increased.

Claims

1. An audience response determination apparatus for determining an audience response to displayed content, comprising:

image input means for inputting image signals representing an image of members of said audience;

motion vector determining means for receiving said image signal and determining motion vectors in said image;

sound input means for inputting audio signals representing sounds generated by said members of such audience;

audio determining means responsive to said audio signals to determine a sound state of said audience;

individual state detection means for detecting individual load conditions of the members of the audience, including left and right stepping force detection means for detecting the stepping force of an individual member of the audience, thereby representing the individual response states of the members of the audience;

auxiliary information means for supplying auxiliary information indicating whether the content currently is displayed and, if so, whether said content is video or audio content and, if video content, said auxiliary information is indicative of a scene in said video content;

determination means for determining the audience response on the basis of said determined motion vectors, said determined sound state of said audience, the detected load conditions of said members, and said supplied auxiliary information, such that for given motion vectors, a given sound state and given load conditions, a different audience response is determined depending upon the auxiliary information; and

wherein said determination means for determining the audience response comprises an audience state determination unit for estimating if a majority of said audience is intently watching or listening to said content, for estimating if a majority of said audience is clapping or singing along with the content, for estimating if a majority of said audience is clapping or shouting, for estimating if a majority of said audience is applauding or cheering and for estimating if a majority of said audience is standing, and for selecting the audience response as a function of said estimating.

2. An audience response determination apparatus according to claim 1, wherein said overall state detection means takes an image of the entire audience and detects the overall bodily state of the audience based on the image taken.

3. An audience response determination apparatus according to claim 1, wherein said overall state detection means collects sounds uttered by the entire audience and detects the overall state of the audience based on the sounds collected.

4. An audience response determination apparatus according to claim 1, wherein the load condition detected by said individual state detection means includes a load applied to each of the audience'"'"'s seats.

5. An audience response determination apparatus according to claim 1, wherein said left and right stepping force detection means comprises:

a left stepping force detector for detecting the stepping force provided by the left foot of each member of said audience; and

a right stepping force detector for detecting the stepping force provided by the right foot of each member of said audience.

6. A playback output control system for controlling the output from playback means for the playback and output of data to be seen or heard by an audience, comprising:

image input means for inputting image signals representing an image of members of said audience;

motion vector determining means for receiving said image signal and determining motion vectors in said image;

sound input means for inputting audio signals representing sounds generated by said members of such audience;

audio determining means responsive to said audio signals to determine a sound state of said audience;

individual state detection means for detecting individual load conditions of the members of the audience, including left and right stepping force detection means for detecting the stepping force of an individual member of the audience, thereby representing individual response states of the members of said audience;

auxiliary information means for supplying auxiliary information indicating whether the content currently is displayed and, if so, whether said content is video or audio content and, if video content, said auxiliary information is indicative of a scene in said video content;

determination means for determining an audience response on the basis of said determined motion vectors, said determined sound state of said audience, the detected load conditions of said members, and said supplied auxiliary information, such that for given motion vectors, a given sound state and given load conditions, a different audience response is determined depending upon the auxiliary information;

wherein said determination means for determining an audience response comprises an audience state determination unit for estimating if a majority of said audience is intently watching or listening to said content, for estimating if a majority of said audience is clapping or singing along with the content, for estimating if a majority of said audience is clapping or shouting, for estimating if a majority of said audience is applauding or cheering and for estimating if a majority of said audience is standing, and for selecting the audience response as a function of said estimating; and

control means for controlling the operation of said playback means based on the audience response selected by said determination means.

7. A playback output control system according to claim 6, wherein said control means selects, on the basis of said audience response determined by said determination means, data to be played back by said playback means.

8. A playback output control system according to claim 6, wherein said control means controls, on the basis of said audience response determined by said determination means, signal processing on the data played back by said playback means.

9. A playback output control system according to claim 6, wherein said overall state detection means takes an image of said audience and detects the overall bodily state of said audience based on the image.

10. A playback output control system according to claim 9, further comprising reduction means for reducing the effect of video data played back by and output from said playback means, wherein said overall state detection means detects the overall bodily state of said audience by reducing the effect of said video data on said image of said audience.

11. A playback output control system according to claim 6, wherein said overall state detection means detects the overall state of said audience by collecting sounds emitted by the audience.

12. A playback output control system according to claim 11, further comprising reduction means for reducing the effect of sound data played back and output by said playback means, wherein overall state detection means detects the overall state of the audience by reducing the effect of said sound data on the collected sounds emitted by said audience.

13. A playback output control system according to claim 11, wherein said overall state detection means detects the overall state of the audience by comparing the collected sounds with a reference sound level.

14. A playback output control system according to claim 13, further comprising varying means for varying said reference sound level on the basis of the audience size.

15. A playback output control system according to claim 11, further comprising a filter which passes a predetermined audio band, wherein said overall state determination means detects the overall state of the audience based on the sound passed through said filter.

16. A playback output control system according to claim 6, wherein the load condition detected by said individual state detection means includes a load applied to each of the audience'"'"'s seats.

17. A playback output control system according to claim 6, wherein said left and right stepping force detection means comprises:

a left stepping force detector for detecting the stepping force provided by the left foot of each member of the audience; and

a right stepping force detector for detecting the stepping force provided by the right foot of each member of the audience.

18. An audience response determination method for determining an audience response to displayed content, comprising the steps of:

inputting image signals representing an image of members of said audience;

receiving said image signal and determining motion vectors in said image;

inputting audio signals representing sounds generated by said members of such audience;

determining a sound state of said audience in response to said inputted audio signals;

detecting individual load conditions of the members of the audience, including detecting a stepping force provided by a left foot of an individual member of the audience and detecting a stepping force provided by a right foot of the individual member, thereby representing individual response states of the members of said audience;

supplying auxiliary information indicating whether the content currently is displayed and, if so, whether said content is video or audio content and, if video content, said auxiliary information is indicative of a scene in said video content;

determining the audience response based on said determined motion vectors, said determined sound state of said audience, the detected load conditions of the members of said audience, and said supplied auxiliary information, such that for given motion vectors, a given sound state and given load conditions, a different audience response is determined depending upon the auxiliary information; and

wherein the audience response is determined by estimating if a majority of said audience is intently watching or listening to said content, by estimating if a majority of said audience is clapping or singing along with the content, by estimating if a majority of said audience is clapping or shouting, by estimating if a majority of said audience is applauding or cheering and for estimating if a majority of said audience is standing, and by selecting the audience response as a function of said estimating.

19. A playback output control method for controlling a playback output, comprising the steps of:

inputting image signals representing an image of members of said audience;

receiving said image signal and determining motion vectors in said image;

inputting audio signals representing sounds generated by said members of such audience;

determining a sound state of said audience in response to said inputted audio signals;

detecting individual load conditions of members of the audience, including detecting a stepping force provided by a left foot of an individual member of the audience and detecting a stepping force provided by a right foot of the individual member, thereby representing individual response states of the members of said audience;

supplying auxiliary information indicating whether content currently is being played back and, if so, whether said content is video or audio content and, if video content, said auxiliary information is indicative of a scene in said video content;

determining an audience response based on said determined motion vectors, said determined sound state of said audience, the detected load conditions of the members of said audience, and said supplied auxiliary information, such that for given motion vectors, a given sound state and given load conditions, a different audience response is determined depending upon the auxiliary information;

wherein the audience response is determined by estimating if a majority of said audience response is intently watching or listening to said content, by estimating if a majority of said audience is clapping or singing along with the content, by estimating if a majority of said audience is clapping or shouting, by estimating if a majority of said audience is applauding or cheering and for estimating if a majority of said audience is standing, and by selecting the audience response as a function of said estimating, and

controlling the playback operation of data to be seen or heard by said audience based on the audience response that is selected.

20. A computer readable recording medium storing a processing program to control of processor to perform the method comprising the steps of:

inputting image signals representing an image of members of said audience;

receiving said image signal and determining motion vectors in said image;

inputting audio signals representing sounds generated by said members of such audience;

determining a sound state of said audience in response to said inputted audio signals;

detecting individual load conditions of the members of the audience, including detecting a stepping force provided by a left foot of an individual member of the audience and detecting a stepping force provided by a right foot of the individual member, thereby representing individual response states of the members of said audience;

supplying auxiliary information indicating whether content currently is being played back and, if so, whether said content is video or audio content and, if video content, said auxiliary information is indicative of a scene in said video content;

determining an audience response based on said determined motion vectors, said determined sound state of said audience, the detected load conditions of the members of said audience, and said supplied auxiliary information, such that for given motion vectors, a given sound state and given load conditions, a different audience response is determined depending upon the auxiliary information; and

wherein the audience response is determined by estimating if a majority of said audience is intently watching or listening to said content, by estimating if a majority of said audience is clapping or singing along with the content, by estimating if a majority of said audience is clapping or shouting, by estimating if a majority of said audience is applauding or cheering and for estimating if a majority of said audience is standing, and by selecting the audience response as a function of said estimating.

21. A computer readable recording medium storing a processing program to control of processor to perform the method comprising the steps of:

inputting image signals representing an image of members of said audience;

receiving said image signal and determining motion vectors in said image;

inputting audio signals representing sounds generated by said members of such audience;

determining a sound state of said audience in response to said inputted audio signals;

detecting individual load conditions of members of the audience, including detecting a stepping force provided by a left foot of an individual member of the audience and detecting a stepping force provided by a right foot of the individual member, thereby representing individual response states of the members of said audience;

supplying auxiliary information indicating whether content is currently played back and, if so, whether said content is video or audio content and, if video content, said auxiliary information is indicative of a scene in said video content;

determining an audience response based on said determined motion vectors, said determined sound state of said audience, the detected load conditions of the members of said audience, and said supplied auxiliary information, such that for given motion vectors, a given sound state and given load conditions, a different audience response is determined depending upon the auxiliary information;

wherein the audience response is determined by estimating if a majority of said audience response is intently watching or listening to said content, by estimating if a majority of said audience is clapping or singing along with the content, by estimating if a majority of said audience is clapping or shouting, by estimating if a majority of said audience is applauding or cheering and for estimating if a majority of said audience is standing, and by selecting the audience response as a function of said estimating, and

controlling a playback operation of data to be seen or heard by said audience based on the audience response that is selected.

22. An audience response determination apparatus for determining an audience response to displayed content, comprising:

image input means for inputting image signals representing an image of members of said audience;

motion vector determining means for receiving said image signal and determining motion vectors in said image;

sound input means for inputting audio signals representing sounds generated by said members of such audience;

audio determining means responsive to said audio signals to determine a sound state of said audience;

individual state detection means for detecting individual load conditions of the members of the audience, including a left stepping force detector for detecting a stepping force provided by a left foot of an individual member of the audience and a right stepping force detector for detecting a stepping force provided by a right foot of the individual member, thereby representing the individual states of the members of the audience;

auxiliary information means for supplying auxiliary information indicating the content currently displayed;

determination means for determining the audience response on the basis of said determined motion vectors, said determined sound state of said audience, the detected load conditions of said members, and said supplied auxiliary information, such that for given motion vectors, a given sound state and given load conditions, a different audience response is determined depending upon the auxiliary information; and

wherein said determination means for determining the audience response comprises an audience state determination unit (i) for estimating whether most of the members of the audience are intently watching or listening to the content, (ii) for estimating whether most of the members of the audience are clapping their hands or singing along with the sound of the content, (iii) for estimating whether most of the members of the audience are clapping their hands or shouting, (iv) for estimating whether most of the members of the audience are applauding or cheering, (v) for estimating whether most of the members of the audience are standing and for selecting the audience response as a function of said estimating.