Method and apparatus for real-time viewer interaction with a media presentation

• US 10,412,456 B2
• Filed: 02/28/2017
• Issued: 09/10/2019
• Est. Priority Date: 04/24/2008
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment LLC (Sony Group Corp.)
• Original Assignee: Sony Interactive Entertainment LLC (Sony Group Corp.)
• Inventors: Zalewski, Gary M.
• Primary Examiner(s): Idowu, Olugbenga O
• Application Number:
  US15/445,848
• Publication Number:
  US 20170171625A1
• Time in Patent Office: 924 Days
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/215 comprising means for detecting acoustic signals, e.g. using a microphone
• CPC Class Codes: A63F 13/338 using television networks
• CPC Class Codes: A63F 13/42 by mapping the input signals into game commands, e.g. mapping the displacement of a stylus on a touch screen to the steering angle of a virtual vehicle
• CPC Class Codes: A63F 13/63 by the player, e.g. authoring using a level editor
• CPC Class Codes: A63F 13/65 automatically by game devices or servers from real world data, e.g. measurement in live racing competition
• CPC Class Codes: A63F 2300/409 Data transfer via television network
• CPC Class Codes: A63F 2300/69 Involving elements of the real world in the game world, e.g. measurement in live races, real video
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/44218 Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV program methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45
• CPC Class Codes: H04N 21/4622 Retrieving content or additional data from different sources, e.g. from a broadcast channel and the Internet web site content organization and management for information retrieval from the Internet G06F16/958; transmission by internet of broadcast information H04H60/82; stock exchange data over packet-switching network H04L12/1804; push services including data channel over packet-switching network H04L12/1859
• CPC Class Codes: H04N 21/4725 using interactive regions of the image, e.g. hot spots details of information retrieval from the Internet by using URLs G06F16/955; processing chained hypermedia data for information retrieval G06F16/94
• CPC Class Codes: H04N 21/4781 Games
• CPC Class Codes: H04N 5/445 for displaying additional information H04N5/50 takes precedence

Abstract

Real-time participation within a media presentation is disclosed. A participant overlay may be generated by preprocessing a media presentation captured in real time with a camera to extract a time-variant metadata stream associated with one or more interactive regions depicted in the media presentation. The participant overlay includes simulated physics, which may include boundaries of the interactive regions defined in accordance with content in said media presentation. A participant'"'"'s interaction with said media presentation through the interactive regions may be captured and combined with the media presentation for display on a display. The interaction may comprise a user action upon one or more elements in the media presentation, a user action expressed in the media presentation through an avatar, or a user action in said media presentation in connection with a game element.

Claims

1. A system for real-time participation within a media presentation, comprising:

a camera; and

a compositing processor programmed to

a) generate a participant overlay by preprocessing said media presentation to extract a time-variant metadata stream associated with one or more interactive regions depicted in a media presentation captured in real time with the camera, wherein said participant overlay includes simulated physics, wherein said simulated physics include boundaries of said one or more interactive regions defined in accordance with one or more content elements in said media presentation, wherein at least one element of said media presentation is extracted therefrom and re-rendered into an interactive feature and simulated physics are applied to the interactive feature, wherein the simulated physics applied to the interactive feature include elastic and inelastic collisions;

b) capture a participant'"'"'s interaction with said media presentation through said one or more interactive regions of said overlay; and

c) combine said overlay with said media presentation for display to said participant on a display, wherein said overlay is tied temporally to action within the media presentation.

2. The system of claim 1, wherein said participant'"'"'s interaction comprises any of a user action upon said one or more content elements in said media presentation, a user action expressed in said media presentation through an avatar, or a user action in said media presentation in connection with a game element.

3. The system of claim 1, further comprising a display for simultaneously displaying said media presentation and said participant overlay.

4. The system of claim 3, further comprising a controller configured to facilitate the participant'"'"'s interaction within said media presentation in real time.

5. The system of claim 4, said controller comprising:

any of a game controller, joystick, mouse, touchpad, ultrasonic motion detector, microphone, and video camera.

6. The system of claim 1, further comprising:

a plurality of participant locations, each participant location supporting at least one participant; and

a network for interconnecting each participant location;

wherein said compositing processor is configured to simultaneously capture interaction of each participant at each of said plurality of participant locations and add the interaction of each participant to said overlay in real time for display at a display at each of said participant locations.

7. The system of claim 1, further comprising a participant location comprising a facility that supports a plurality of participants and a display configured to display said media presentation and said participant overlay simultaneously to each of said plurality of participants.

8. The system of claim 7, further comprising means for each participant in said plurality of participants to interact with said media presentation in real time.

9. The system of claim 1, wherein said compositing processor comprises:

a local processor at said participant location.

10. The system of claim 1, further comprising:

a game processor programmed to provide at least one game play element to said media presentation via said participant overlay.

11. The system of claim 10, wherein said game processor and said compositing processor comprise:

one or more local processor at said participant location.

12. The system of claim 1, further comprising:

a media processor programmed to provide said media presentation to said compositing processor.

13. The system of claim 1, wherein said compositing processor is further programmed to synchronize captured participant interaction with said media presentation.

14. The system of claim 1, wherein said compositing processor is further programmed to map said media presentation on a predetermined basis, to add interactive elements to said media presentation in accordance with said map, and to track participant interaction with said interactive elements based upon said map.

15. The system of claim 14, wherein said compositing processor is programmed to use said map to track participant movement of said interactive and to maintain accurate registration between said participant overlay and said media presentation.

16. The system of claim 1, wherein said participant overlay comprises:

an avatar representation of said participant.

17. The system of claim 1, wherein said participant overlay comprises:

interactions of said participant with any of said media presentation and interactive elements within said participant overlay.

18. A method for real-time participation within a media presentation, comprising:

a) capturing a media presentation in real time with a camera;

b) generating a participant overlay by preprocessing said media presentation to extract a time-variant metadata stream associated with one or more interactive regions depicted in said media presentation, wherein said participant overlay includes simulated physics, wherein said simulated physics include boundaries of said one or more interactive regions defined in accordance with one or more content elements in said media presentation, wherein at least one element of said media presentation is extracted therefrom and re-rendered into an interactive feature and simulated physics are applied to the interactive feature wherein the simulated physics applied to the interactive feature include elastic and inelastic collisions;

c) capturing a participant'"'"'s interaction with said media presentation through said one or more interactive regions of said overlay; and

d) combining said overlay with said media presentation for display to said participant on a display, said interaction comprising any of a user action upon said one or more content elements in said media presentation, a user action expressed in said media presentation through an avatar, or a user action in said media presentation in connection with a game element, wherein said overlay is tied temporally to action within the media presentation; and

e) simultaneously displaying said media presentation and said overlay on the display.

19. A game system comprising:

a camera;

a display; and

a processor coupled to the display and camera, wherein the processor is configured to combine video content captured in real time with the camera with an interactive overlay that permits user interaction with one or more virtual objects overlaid onto a scene of the pre-existing video content, wherein said overlay includes simulated physics, wherein said simulated physics include boundaries defined in accordance with one or more content elements in said video content, wherein at least one element of said media presentation is extracted therefrom and re-rendered into an interactive feature and simulated physics are applied to the interactive feature wherein the simulated physics applied to the interactive feature include elastic and inelastic collisions, wherein said overlay is tied temporally to action within the media presentation.