ADAPTIVE GROUP INTERACTIVE MOTION CONTROL SYSTEM AND METHOD FOR 2D AND 3D VIDEO

• US 20170182406A1
• Filed: 03/16/2015
• Published: 06/29/2017
• Est. Priority Date: 03/21/2014
• Status: Abandoned Application

• Current Assignee: Audience Entertainment, LLC (YDreams - Informatica S.A.)
• Original Assignee: Audience Entertainment, LLC (YDreams - Informatica S.A.)
• Inventors: CASTIGLIA, Clilly, ALMEIDA, Andre, CAULEY, Mac, FAURE, Xavier, MONTEIRO, Ruben Miguel Silva, COSTA, Manuel Ricardo Fonseca, da CRUZ, Filipe Manuel Miranda
• Application Number:
  US15/127,298
• Publication Number:
  US 20170182406A1
• Time in Patent Office: Days
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/42 by mapping the input signals into game commands, e.g. mapping the displacement of a stylus on a touch screen to the steering angle of a virtual vehicle
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/6045 for mapping control signals received from the input arrangement into game commands
• CPC Class Codes: G06T 7/269 using gradient-based methods
• CPC Class Codes: G06V 20/53 Recognition of crowd images, e.g. recognition of crowd congestion
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence

Abstract

Adaptive system and method for interacting with and navigating 2D and 3D audiovisual content by interpreting the side to side and/or up and down physical gestures and sounds of a multi-person audience. The system may be used in a movie theatre, stadium, music arena, or other venue where an audience views the same view screen. A camera pointed at the audience captures audience motion and a microphone captures sounds. Optical flow is determined by comparing successive video frames. A Motion Index between −100 and 100 is calculated using Optical flow, comparing more recent Optical Flow vectors with previous values. Methods are provided to give equal weight to audience members closer to and further from the camera. Motion Index Values are used to control and interact with content scenarios.

Claims

1. A method of presenting interactive 3D video content to an audience, the method comprising:

providing an audience area for accommodating a plurality of audience members, and a view screen which is visible from the audience area;

providing a video camera which is capable of recording digital video, the video camera being pointed at the audience area, the video camera being functionally connected to a computer for passing video data to the computer;

wherein the computer comprises a memory and a processor, and wherein the computer is functionally connected to the view screen for displaying 3D video content on the view screen;

the method further comprising;

displaying 3D video content on the view screen, the video content on the view screen being at least partially controlled by the computer, the video content comprising simulation of movement through a 3D scenario from a first person perspective, said first person perspective being a camera view;

the video camera capturing digital video of the audience area while the video content is presented on the view screen and while a plurality of audience members are in the audience area, the captured digital video comprising a chronological series of audience images, and the video camera passing the chronological series of audience images on to the computer;

determining Optical Flow vectors for a plurality of different locations in each of the audience images, wherein Optical Flow vectors are determined by a process comprising comparing a location in an audience image with the same location in a chronologically earlier audience image to determine motion at that location;

determining Motion Index values for a chronological series of audiences images, with Motion Index values being determined by a process comprising comparing a current Optical Flow vector with one or more previous Optical Flow vector;

controlling rotation of the camera view using successive Motion Index values;

wherein the 3D video content displayed on the view screen comprises at least one interaction item;

the video camera capturing digital video of lateral gestures by a plurality of audience members in the audience area, the computer determining a plurality of Optical Flow vectors and a plurality of Motion Index values based on said digital video of lateral gestures, and rotating the camera view to one side in response to said Motion Index values;

the interaction item moving from a side of the camera view to a center area of the camera view during said camera rotation, with the camera rotation stopping at a point where the interaction item is in the center area of the camera view in response to a plurality of audience members ending said lateral gesturing; and

after the camera rotation stops, and in response to the interaction item remaining substantially in the center of the camera view for a threshold period of time, displaying a 3D animation of the camera view moving toward the interaction item on the view screen.

2. The method of presenting interactive 3D video content to an audience of claim 1:

wherein the audience area comprises a plurality of seats in at least one of a theater, a cinema, a stadium, a grandstand, a set of bleachers, and an arena.

3. The method of presenting interactive 3D video content to an audience of claim 1:

wherein said lateral gestures by a plurality of audience members comprise lateral arm movements.

4. The method of presenting interactive 3D video content to an audience of claim 1:

wherein the Motion Index values for the audience images are values from −

100 to 100;

wherein negative Motion Index values cause the camera view to rotate in a first direction;

wherein positive Motion Index values cause the camera view to rotate in a second direction which is different from the first direction; and

wherein Motion Index values of zero do not cause the camera view to rotate.

5. The method of presenting interactive 3D video content to an audience of claim 1:

wherein the interaction item comprises at least one of a character, a living creature, an object, a vehicle, an intersection, and a building.

6. The method of presenting interactive 3D video content to an audience of claim 1:

wherein said simulation of movement through a 3D scenario from a first person perspective comprises simulation of movement through at least one of a city, a town, a maze, a building, an outdoor landscape, a racecourse, an underwater landscape, and outer space.

7. The method of presenting interactive 3D video content to an audience of claim 1:

wherein the audience area and corresponding audience images are divided into a plurality of control zones;

wherein Optical Flow vectors and Motion Index values are determined separately for each control zone, with said Optical Flow vectors and Motion Index values being determined at least partially based on gestures by different audience members in different respective control zones;

wherein a plurality of camera views are displayed on one or more view screens; and

wherein Motion Index values from different control zones are used to control different respective camera views.

8. The method of presenting interactive 3D video content to an audience of claim 1:

wherein the audience images are divided into a plurality of partition zones;

wherein Optical Flow vectors are separately determined for each partition zone;

wherein Motion Index values are separately calculated for each partition zone;

wherein Average Motion Index values are calculated for each of a plurality of time points, the Average Motion Index values each being an average of the Motion Index values for a plurality of individual partition zones; and

controlling rotation of the camera view using the Average Motion Index.

9. The method of presenting interactive 3D video content to an audience of claim 1:

wherein the audience area comprises areas which are closer to the video camera and areas which are further away from the video camera;

wherein the audience images are divided into a plurality of partition zones, with some partition zones being larger than other partition zones;

wherein a first partition zone corresponding to a first part of the audience area is larger than a second partition zone corresponding a second part of the audience area, and wherein the first part of the audience area is closer to the video camera than the second part of the audience area;

wherein Optical Flow vectors are separately determined for each partition zone;

wherein Motion Index values are separately calculated for each partition zone;

wherein Average Motion Index values are calculated for each of a plurality of time points, the Average Motion Index values each being an average of the Motion Index values for a plurality of individual partition zones at a particular time point; and

controlling rotation of the camera view using the Average Motion Index values.

10. The method of presenting interactive 3D video content to an audience of claim 1:

wherein, when the computer is processing the audience images, the audience images are divided into a plurality of partition zones, with some partition zones being larger than other partition zones;

wherein the plurality of partition zones are determined by a dynamic partitioning process, the dynamic partitioning process comprising;

a. providing at least a first partition zone;

b. dividing the first partition zone into a plurality of smaller subzones;

c. determining one or more comparison values for each of the subzones, wherein the comparison values are one of Optical Flow vectors and Motion Index values;

d. comparing the comparison values of the subzones;

e. if at least one of the subzones has a comparison value which differs from the comparison value of one or more of the other subzones at the same time point by at least a threshold amount, retaining all of the subzones as independent partition zones; and

f. if none of the subzones have a comparison value which differs from the comparison values of the other subzones at the same time point by at least the threshold amount, re-merging the subzones into the first partition zone;

wherein, during the presenting of 3D video content to a plural audience;

Optical Flow vectors are separately determined for each partition zone;

Motion Index values are separately calculated for each partition zone;

Average Motion Index values are calculated for each of a plurality of time points, the Average Motion Index values each being an average of the Motion Index values for a plurality of individual partition zones at a particular time point; and

controlling rotation of the camera view using the Average Motion Index values.

11. The method of presenting interactive 3D video content to an audience of claim 1:

further comprising applying a force feedback to the camera view which rotates the camera view towards the interaction item independent of the Motion Index values and independent of gestures by audience members.

12. A method of presenting interactive 3D video content to an audience, the method comprising:

providing an audience area for accommodating a plurality of audience members, and a view screen which is visible from the audience area;

providing a video camera which is capable of recording digital video, the video camera being pointed at the audience area, the video camera being functionally connected to a computer for passing video data to the computer;

wherein the computer comprises a memory and a processor, and wherein the computer is functionally connected to the view screen for displaying 3D video content on the view screen;

the method further comprising;

displaying 3D video content on the view screen, the video content on the view screen being at least partially controlled by the computer, the video content comprising simulation of movement through a 3D scenario along a linear path from a first person perspective, said first person perspective being a camera view;

the video camera capturing digital video of the audience area while the video content is presented on the view screen and while a plurality of audience members are in the audience area, the captured digital video comprising a chronological series of audience images, and the video camera passing the chronological series of audience images on to the computer;

determining Optical Flow vectors for a plurality of different locations in each of the audience images, wherein Optical Flow vectors are determined by a process comprising comparing a location in an audience image with the same location in a chronologically earlier audience image to determine motion at that location;

determining Motion Index values for a chronological series of audiences images, with Motion Index values being determined by a process comprising comparing a current Optical Flow vector with one or more previous Optical Flow vectors;

controlling rotation of the camera view using successive Motion Index values; and

the video camera capturing digital video of lateral gestures by a plurality of audience members in the audience area, the computer determining a plurality of Optical Flow vectors and a plurality of Motion Index values based on said digital video of lateral gestures, and rotating the camera view to one side in response to said Motion Index values.

13. The method of presenting interactive 3D video content to an audience of claim 12:

wherein the video content displayed on the view screen comprises simulation of movement through a 3D scenario along a linear path having a forward direction and a backwards direction;

wherein, when the camera view is oriented in the forward direction, the camera view displays movement through the 3D scenario in the forward direction along the path at a first speed; and

wherein, when the camera view is oriented 90°

away from the forward direction, the camera view displays one of (i) no movement along the path, and (ii) movement along the path in the forward direction at a speed which is less than the first speed.

14. The method of presenting interactive 3D video content to an audience of claim 12:

wherein the video content displayed on the view screen comprises simulation of movement through a 3D scenario along a linear path having a forward direction and a backwards direction;

wherein, when the camera view is oriented in the forward direction, the camera view displays movement through the 3D scenario in the forward direction along the path; and

wherein, when the camera view is oriented 90°

away from the forward direction, the camera view displays no movement along the path; and

wherein, when the camera view is oriented in the backwards direction, the camera view displays movement through the 3D scenario in the backwards direction along the path.

15. The method of presenting interactive 3D video content to an audience of claim 12:

wherein the video content displayed on the view screen comprises simulation of movement through a 3D scenario along a linear path having a forward direction and a backwards direction;

wherein, when the camera view is oriented in the forward direction, the camera view displays movement through the 3D scenario in the forward direction along the path at a first speed; and

wherein, when the camera view is oriented at a diagonal angle intermediate between the forward direction and 90°

away from the forward direction, the camera view displays movement along the path in the forward direction at a speed which is less than the first speed.

16. The method of presenting interactive 3D video content to an audience of claim 12:

wherein the video content displayed on the view screen comprises simulation of movement through a 3D scenario along a linear path having a forward direction and a backwards direction;

wherein direction and velocity of movement along the linear path (V_Camera) is determined by the angle of the camera view with respect to the forward direction (α

_Camera), wherein the camera view facing directly in the forward direction is an angle of 0° and

the camera view facing in the backwards direction is an angle of 180°

; and

wherein V_Camera is determined by a method comprising the relationship;

17. The method of presenting interactive 3D video content to an audience of claim 12:

wherein the 3D video content displayed on the view screen comprises at least one interaction item;

the video camera capturing digital video of lateral gestures by a plurality of audience members in the audience area, the computer determining a plurality of Optical Flow vectors and a plurality of Motion Index values based on said digital video of lateral gestures, and rotating the camera view to one side in response to said Motion Index values;

the camera rotation stopping at a point where the interaction item is in the center area of the camera view in response to a plurality of audience members ending said lateral gesturing; and

after the camera rotation stops, and in response to the interaction item remaining substantially in the center of the camera view for a threshold period of time, displaying a 3D animation of the camera view zooming in towards the interaction item on the view screen, and displaying an enlarged image of the interaction item.

18. The method of presenting interactive 3D video content to an audience of claim 12:

wherein the video content displayed on the view screen comprises simulation of movement through a 3D scenario along a linear path having a forward direction and a backwards direction;

wherein the linear path further comprises at least one intersection, the intersection being a branched location in the linear path where at least three different travel directions are available;

wherein when the 3D scenario reaches an intersection, the audience selects a travel direction by a plurality of audience members gesturing in a rotation direction, the camera view rotating in the rotation direction until the camera view is facing in a selected linear path, at which time a plurality of audience members stop gesturing which in turn causes the camera view rotation to stop; and

after the camera rotation stops, and in response to the camera view facing the selected linear path for at least a threshold period of time;

displaying a 3D animation of the camera view moving down the selected linear path on the view screen.

19. The method of presenting interactive 3D video content to an audience of claim 12:

wherein the 3D scenario displayed on the view screen comprises a virtual tour of a real or fictional physical location, with said linear path traversing the physical location.

20. A method of presenting interactive video content to an audience, the method comprising:

providing an audience area for accommodating a plurality of human audience members, and a view screen which is visible from the audience area;

providing a video camera which is capable of recording digital video, the video camera being pointed at the audience area, the video camera being functionally connected to a computer for passing video data to the computer;

wherein the computer is functionally connected to the view screen for displaying video content on the view screen;

the method further comprising;

displaying video content on the view screen, the video content comprising simulation of movement through a video scenario, with a field of vision being displayed on the view screen being a camera view;

while video content is presented on the view screen, the video camera capturing digital video of the audience area comprising a plurality of audience images, which audience images depict gestures by a plurality of audience members in the audience area; and

the method further comprising comparing a current audience image to one or more previous audience images, and rotating the camera view in response to said gestures by a plurality of audience members.

21. The method of presenting interactive video content to an audience of claim 20:

wherein the video content displayed on the view screen comprises at least one interaction item;

wherein a plurality of audience members gesture in a first direction, and the camera view rotates in said first direction in response to said gestures;

wherein said camera view rotation continues until the interaction item is in a center area of the camera view, wherein a plurality of audience members who were gesturing in the first direction stop gesturing and, in response to the stopping of said gesturing, camera rotation stops;

after the camera rotation stops, and in response to the interaction item being substantially in the center of the camera view;

displaying an animation of the camera view moving toward the interaction item on the view screen.

22. The method of presenting interactive video content to an audience of claim 20:

wherein the video content comprises 3D video content;

wherein the step of displaying 3D video content on the view screen comprises displaying a simulation of movement through a 3D scenario along a linear path from a first person perspective, with the linear path having a forward direction and a backwards direction; and

wherein a plurality of audience members in the audience area control movement of the camera view along the linear path by using gestures to rotate the camera view with respect to the forward direction.

23. The method of presenting interactive video content to an audience of claim 20:

wherein the audience area comprises areas which are closer to the video camera and areas which are further away from the video camera;

wherein, when the computer is processing the audience images, the audience images are divided into a plurality of partition zones, with some partition zones being larger than other partition zones;

wherein a first partition zone corresponding to a first part of the audience area is larger than a second partition zone corresponding a second part of the audience area, and wherein the first part of the audience area is closer to the video camera than the second part of the audience area;

wherein Optical Flow vectors are separately determined for each partition zone;

wherein Motion Index values are separately calculated for each partition zone;

wherein Average Motion Index values are calculated for a plurality of time points, the Average Motion Index values each being an average of the Motion Index values for a plurality of individual partition zones; and

controlling rotation of the camera view using the Average Motion Index values.

24. The method of presenting interactive video content to an audience of claim 20:

wherein the step of displaying video content on the view screen comprises displaying 2D content.

25. The method of presenting interactive video content to an audience of claim 20:

the method further comprising detecting at least one sound in the audience area, and controlling video content in response to detecting said sound.