SYSTEMS AND METHODS TO FACILITATE INTERACTIONS WITH VIRTUAL CONTENT

• US 20120200667A1
• Filed: 11/09/2011
• Published: 08/09/2012
• Est. Priority Date: 02/08/2011
• Status: Abandoned Application

• Current Assignee: Disney Enterprises Incorporated (The Walt Disney Company)
• Original Assignee: Disney Enterprises Incorporated (The Walt Disney Company)
• Inventors: Gay, Michael F., Golding, Frank, Gefen, Smadar
• Application Number:
  US13/292,560
• Publication Number:
  US 20120200667A1
• Time in Patent Office: Days
• US Class Current: 348/43
• CPC Class Codes: A63F 13/20 Input arrangements for video game devices
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/40 Processing input control signals of video game devices, e.g. signals generated by the player or derived from the environment
• CPC Class Codes: A63F 13/52 involving aspects of the displayed game scene
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay

Abstract

According to some embodiments, a graphics platform may receive a video signal, including an image of a person, from a video camera. The graphics platform may then add a virtual object to the video signal to create a viewer or broadcast signal. 3D information associated with a spatial relationship between the person and the virtual object is determined. The graphics platform may then create a supplemental signal based on the 3D information, wherein the supplemental signal includes sufficient information to enable the person to interact with the virtual object as if such object was ‘seen’ or sensed from the person'"'"'s perspective. The supplemental signal may comprise video, audio and/or pressure all as necessary to enable the person to interact with the virtual object as if he/she were physically present with the virtual object.

Claims

1. A method comprising:

receiving, at a graphics platform, 3D information about a virtual object;

determining 3D information associated with a person in a scene; and

creating, by the graphics platform;

(i) a viewer signal of the scene and (ii) a supplemental signal, wherein the viewer signal and the supplemental signal are from different perspectives.

2. The method of claim 1, wherein the viewer signal is a viewer video signal generated from a perspective of a video camera filming the scene and wherein the viewer video signal includes the virtual object and a representation of the person.

3. The method of claim 1, wherein and the supplemental signal is a view of the virtual object from the perspective of the person.

4. The method of claim 1, wherein the supplemental signal is a sensory signal indicative of the location of the virtual object relative to the person.

5. The method of claim 4, wherein the sensory signal is associated with at least one of video information, audio information, pressure information, or thermal information.

6. The method of claim 1, wherein the 3D information associated with the person is received from one or more sensors.

7. The method of claim 1, wherein the 3D information associated with the person is determined by analyzing at least one of (i) video from at least one video camera or (ii) telemetry data from one or more sensors.

8. The method of claim 7, wherein a plurality of video signals are received and analyzed by the graphics platform to determine at least one of:

(i) a three dimensional location of the person, (ii) a distance between the person and virtual object, (iii) one or more angles associated with the person and virtual object, or (iv) an orientation of the person.

9. The method of claim 1, wherein the view of the virtual object within the supplemental signal is associated with at least one of:

(i) a marker, (ii) a lower or different resolution image as compared with the viewer signal, (iii) a lower or different frame rate image as compared with the viewer signal, or (iv) a dynamically generated occlusion zone.

10. The method of claim 9, further comprising:

transmitting the supplemental signal to a display device worn by the person.

11. The method of claim 10, wherein the display device comprises at least one of:

(i) an eyeglasses display, (ii) a retinal display, (iii) a contact lens display, or (iv) a hearing aid or other in-“

ear”

device.

12. The method of claim 10, wherein the supplemental signal is wirelessly transmitted to the display device.

13. The method of claim 1, wherein the supplemental signal is further based on an orientation of the person'"'"'s line of sight and/or body position.

14. The method of claim 1, wherein multiple people and virtual objects are able to interact, whether or not any or all such images are included in the final scene presented to the viewing audience, and further comprising:

creating a supplemental signal for each person, wherein each supplemental signal includes a view or a means for perceiving the location of the virtual objects as would be seen or perceived from that person'"'"'s perspective.

15. The method of claim 1, further comprising:

detecting a command from an entity; and

responsive to said detection, adjusting the virtual object.

16. The method of claim 15, wherein the command comprises at least one of:

(i) an audible command, (ii) a gesture command, or (iii) a third party or second virtual object command.

17. The method of claim 1, wherein the virtual object is associated with at least one of:

(i) a virtual human, (ii) a video game, or (iii) an avatar.

18. A system, comprising:

a device to capture a feed including an image or location data of a person;

a platform to receive the feed from the device and to render a supplemental feed in substantially real-time, based on 3D information relating to the person and 3D information relating to a virtual object, wherein the supplemental feed includes sufficient information to provide the person with a means of perceiving a location of the virtual object as would be perceived from the person'"'"'s perspective; and

a device worn by the person to receive and present to the person the supplemental feed.

19. The system of claim 18, wherein the platform includes at least one of:

(i) a computer generated image database, (ii) a control system, (iii) a video processor, (iv) an audio processor, (v) a three dimensional graphics engine, (vi) a mixer, or (vii) a location sensor to detect a location of the person.

20. The system of claim 18, wherein the device comprises a light detection and ranging camera.

21. The system of claim 18, wherein the platform is further to render a viewer feed including a information of the virtual object as would be necessary to perceive the virtual object from a camera'"'"'s perspective.

22. The system of claim 18, wherein the platform receives two feeds that include images of the human and performs stereoscopic matching of pixels within the feeds to determine a three dimensional location of the person.

23. The system of claim 18, wherein the platform uses a depth map to create a binary occlusion mask for the supplemental feed.

24. A non-transitory, computer-readable medium storing instructions adapted to be executed by a processor to perform a method, the method comprising:

receiving a signal from a camera, the signal including an image of a person;

inserting a virtual object into the signal to create a viewer signal, wherein the viewer signal includes a view of the virtual object as would be seen from the camera'"'"'s perspective; and

creating a supplemental video signal, wherein the supplemental video signal includes a view of the virtual object as would be seen from the person'"'"'s perspective.

25. The medium of claim 24, wherein the method further comprises:

outputting the supplemental feed to a device to be perceived by the person.

26. The medium of claim 24, wherein the method further comprises:

modeling the person based on the received video signal; and

using the model to adjust image information associated with the person in the viewer signal.