Virtual reality virtual theater system

• US 9,396,588 B1
• Filed: 09/11/2015
• Issued: 07/19/2016
• Est. Priority Date: 06/30/2015
• Status: Expired due to Fees

• Current Assignee: KAYA Dynamics LLC
• Original Assignee: Ariadne's Thread (Usa), Inc.
• Inventors: Li, Adam
• Primary Examiner(s): Du, Haixia
• Application Number:
  US14/852,202
• Time in Patent Office: 312 Days
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/0134 of stereoscopic type
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06T 15/00 3D [Three Dimensional] image rendering
• CPC Class Codes: G06T 19/003 Navigation within 3D models or images
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2210/61 Scene description
• CPC Class Codes: G06T 2215/12 Shadow map, environment map
• CPC Class Codes: G06T 3/18 Image warping, e.g. rearranging pixels individually
• CPC Class Codes: H04S 7/304 For headphones

Abstract

A virtual reality virtual theater system that generates or otherwise displays a virtual theater, for example in which to view videos, such as movies or television. Videos may be for example 2D or 3D movies or television programs. Embodiments create a virtual theater environment with elements that provide a theater-like viewing experience to the user. Virtual theaters may be for example indoor movie theaters, drive-in movie theaters, or home movie theaters. Virtual theater elements may include for example theater fixtures, décor, and audience members; these elements may be selectable or customizable by the user. One or more embodiments also render sound by placing virtual speakers in a virtual theater and projecting sounds from these virtual speakers onto virtual microphones corresponding to the position and orientation of the user'"'"'s ears. Embodiments may also employ minimal latency processing to improve the theater experience.

Claims

1. A virtual reality virtual theater system comprising:

at least one display viewable by a user;

at least one sensor that generates sensor data that measures one or more aspects of a pose of one or more body parts of said user;

a pose analyzer coupled to said at least one sensor, that calculates pose data of said pose of one or more body parts of said user, based on said sensor data generated by said at least one sensor;

a video input configured to accept a video, wherein said video comprises a plurality of frames;

a 3D model of a virtual theater, comprising

a virtual screen;

a virtual video projector; and

one or more theater environment elements; and

,

a scene renderer coupled to said at least one display, said pose analyzer, said video input, and

said 3D model of said virtual theater, wherein said scene renderer

projects each of said plurality of frames successively from said virtual video projector onto said virtual screen;

receives said pose data from said pose analyzer;

calculates one or more rendering virtual camera poses, based on said pose data;

calculates one or more 2D projections of said 3D model of said virtual theater, based on said one or more rendering virtual camera poses; and

transmits said one or more 2D projections to said at least one display; and

,

wherein said scene renderer is configured to modify a dimension of said virtual screen to correspond to a format of said video.

2. The system of claim 1 wherein said virtual theater is a virtual indoor movie theater.

3. The system of claim 2 wherein said one or more theater environment elements comprise one or more of

walls,

a floor,

a ceiling,

lights,

curtains,

seats,

aisles,

one or more decorative elements coupled to said walls or to said ceiling or to both of said walls and said ceiling, and,

one or more persons or characters.

4. The system of claim 1 wherein said virtual theater is a virtual outdoor movie theater.

5. The system of claim 1 wherein said virtual theater is a virtual home theater.

6. The system of claim 1 wherein one or more of said one or more theater environment elements are animated.

7. The system of claim 1 wherein

said virtual screen is curved;

said calculates one or more 2D projections of said 3D model of said virtual theater warps said plurality of frames based on a curvature of said virtual screen.

8. The system of claim 1, further comprising

a theater configuration subsystem coupled to said 3D model of said virtual theater;

wherein said theater configuration subsystem is configured to

accept a theater configuration input from said user or from a second user; and

,

modify said 3D model of said virtual theater based on said theater configuration input.

9. The system of claim 1, further comprising

a user location subsystem coupled to said pose analyzer;

wherein

said user location subsystem is configured to accept a user location in said 3D model of said virtual theater from said user or from a second user;

said calculates pose data of said pose of one or more body parts of said user comprises

calculate a location component of said pose data based on said user location and optionally on said sensor data; and

,

calculate an orientation component of said pose data based on said sensor data.

10. The system of claim 1 wherein

said video is a stereoscopic 3D video, wherein each frame of said plurality of frames comprises

a left eye frame, and

a right eye frame;

said pose data comprises

a left eye pose of said user; and

a right eye pose of said user;

said at least one display comprises

a left eye display region configured to be placed proximal to a left eye of said user; and

a right eye display region configured to be placed proximal to a right eye of said user; and

,

for each of said plurality of frames, said scene renderer is configured to

project said left eye frame from said virtual video projector onto said virtual screen;

calculate a left eye rendering virtual camera pose, based on said pose data;

calculate a left 2D projection of said 3D model of said virtual theater, based on said left eye rendering virtual camera pose;

transmit said left 2D projection to said left eye display region;

project said right eye frame from said virtual video projector onto said virtual screen;

calculate a right eye rendering virtual camera pose, based on said pose data;

calculate a right 2D projection of said 3D model of said virtual theater, based on said right eye rendering virtual camera pose; and

transmit said right 2D projection to said right eye display region.

11. The system of claim 1, further comprising:

stereo speakers comprising

a left speaker configured to be placed proximal to a left ear of said user; and

a right speaker configured to be placed proximal to a right ear of said user; and

,

an audio renderer coupled to said stereo speakers, said pose analyzer, said video input, and said 3D model of said virtual theater;

wherein

said video comprises one or more audio tracks;

said 3D model of said virtual theater further comprises

one or more virtual speakers located in said virtual theater, each having a virtual speaker location in said virtual theater; and

said audio renderer

associates each of said one or more audio tracks with one or more of said one or more virtual speakers;

sets a volume level for each of said one or more virtual speakers;

calculates an emitted sound for each of said one or more virtual speakers;

receives said pose data from said pose analyzer;

calculates a left received sound for said left ear of said user and a right received sound for said right ear of said user based on

said emitted sound for each of said one or more virtual speakers; and

said pose data;

sends said left received sound to said left speaker; and

sends said right received sound to said right speaker.

12. The system of claim 11 wherein

one or more of said one or more theater environment elements emit theater environment sounds;

and,

said audio renderer calculates said left received sound for said left ear of said user and said right received sound for said right ear of said user based on

said sounds emitted from said one or more virtual speakers,

said pose data, and

said theater environment sounds.

13. The system of claim 12 wherein said theater environment sounds comprise sounds from a virtual audience.

14. The system of claim 12 wherein

said one or more audio tracks comprise one or more audio commentary tracks;

said theater environment elements comprise one or more virtual commentator characters located in said 3D model of said virtual theater; and

,

said theater environment sounds comprise one or more of said one or more audio commentary tracks emitted from one or more of said one or more virtual commentator characters.

15. The system of claim 14 wherein said one or more virtual commentator characters comprise one or more of

an actor or actress or animated character in said video or in another video;

a director of said video or of another video;

a producer of said video or of another video;

a critic;

a journalist;

an author;

a historical figure; and

a historian.

16. The system of claim 1, further comprising:

an image warper coupled to said at least one display, said scene renderer, and said pose analyzer,

wherein said image warper

receives said one or more rendering virtual camera poses from said scene renderer;

receives said pose data from said pose analyzer;

calculates a change in pose between said one or more virtual camera poses and said pose data;

generates a rerendering approximation of said one or more 2D projections of said 3D model of said virtual theater on said at least one display based on said change in pose; and

modifies one or more pixels of said at least one display based on said rerendering approximation.

17. The system of claim 16, wherein said rerendering approximation comprises

calculating a pixel translation vector;

and,

translating one or more pixels of said one or more 2D projections by said pixel translation vector.

18. The system of claim 17, wherein said calculating said pixel translation vector comprises

approximating said change in pose as a rotation around a unit vector {circumflex over (ω

)} {circumflex over (ω

)}_y and {circumflex over (ω

)}_x comprising {circumflex over (ω

)}_y and {circumflex over (ω

)}_x by an angle Δ

θ

;

calculating a spatial translation vector (ω

_yΔ

θ

, ω

_xΔ

θ

);

calculating a scaling factor to convert spatial distances to pixels based on pixel dimensions and fields of view of said one or more 2D projections; and

calculating said pixel translation vector by scaling said spatial translation vector by said scaling factor.

19. A virtual reality virtual theater system comprising:

at least one display viewable by a user;

stereo speakers comprising

a left speaker configured to be placed proximal to a left ear of said user;

a right speaker configured to be placed proximal to a right ear of said user;

at least one sensor that generates sensor data that measures one or more aspects of a pose of one or more body parts of said user;

a pose analyzer coupled to said at least one sensor, that calculates pose data of said pose of one or more body parts of said user, based on said sensor data generated by said at least one sensor;

a video input configured to accept a video, wherein said video comprises

a plurality of frames; and

one or more audio tracks;

a 3D model of a virtual theater, comprising

a virtual screen;

a virtual video projector;

one or more virtual speakers located in said virtual theater, each having a virtual speaker location in said virtual theater; and

one or more theater environment elements;

a theater configuration subsystem coupled to said 3D model of said virtual theater, wherein said theater configuration subsystem is configured to

accept a theater configuration input from said user or from a second user; and

modify said 3D model of said virtual theater based on said theater configuration input;

a scene renderer coupled to said at least one display, said pose analyzer, said video input, and said 3D model of said virtual theater, wherein said scene renderer

projects each of said plurality of frames successively from said virtual video projector onto said virtual screen;

receives said pose data from said pose analyzer;

calculates one or more rendering virtual camera poses, based on said pose data;

calculates one or more 2D projections of said 3D model of said virtual theater, based on said one or more rendering virtual camera poses; and

transmits said one or more 2D projections to said at least one display; and

,

an audio renderer coupled to said stereo speakers, said pose analyzer, said video input, and said 3D model of said virtual theater, wherein said audio renderer

associates each of said one or more audio tracks with one or more of said one or more virtual speakers;

sets a volume level for each of said one or more virtual speakers;

calculates an emitted sound for each of said one or more virtual speakers;

receives said pose data from said pose analyzer;

calculates a left received sound for said left ear of said user and a right received sound for said right ear of said user based on

said emitted sound for each of said one or more virtual speakers; and

said pose data;

sends said left received sound to said left speaker; and

sends said right received sound to said right speaker.

20. A virtual reality virtual theater system comprising:

at least one display viewable by a user;

at least one sensor that generates sensor data that measures one or more aspects of a pose of one or more body parts of said user;

a pose analyzer coupled to said at least one sensor, that calculates pose data of said pose of one or more body parts of said user, based on said sensor data generated by said at least one sensor;

a video input configured to accept a video, wherein said video comprises a plurality of frames;

a 3D model of a virtual theater, comprising

a virtual screen;

a virtual video projector; and

one or more theater environment elements;

a scene renderer coupled to said at least one display, said pose analyzer, said video input, and

said 3D model of said virtual theater, wherein said scene renderer

projects each of said plurality of frames successively from said virtual video projector onto said virtual screen;

receives said pose data from said pose analyzer;

calculates one or more rendering virtual camera poses, based on said pose data;

calculates one or more 2D projections of said 3D model of said virtual theater, based on said one or more rendering virtual camera poses; and

transmits said one or more 2D projections to said at least one display; and

,

a theater configuration subsystem coupled to said 3D model of said virtual theater;

wherein said theater configuration subsystem is configured to

accept a theater configuration input from said user or from a second user; and

,

modify said 3D model of said virtual theater based on said theater configuration input.

21. A virtual reality virtual theater system comprising:

at least one display viewable by a user;

at least one sensor that generates sensor data that measures one or more aspects of a pose of one or more body parts of said user;

a pose analyzer coupled to said at least one sensor, that calculates pose data of said pose of one or more body parts of said user, based on said sensor data generated by said at least one sensor;

a video input configured to accept a video, wherein said video comprises a plurality of frames;

a 3D model of a virtual theater, comprising

a virtual screen;

a virtual video projector; and

one or more theater environment elements;

a scene renderer coupled to said at least one display, said pose analyzer, said video input, and

said 3D model of said virtual theater, wherein said scene renderer

projects each of said plurality of frames successively from said virtual video projector onto said virtual screen;

receives said pose data from said pose analyzer;

calculates one or more rendering virtual camera poses, based on said pose data;

calculates one or more 2D projections of said 3D model of said virtual theater, based on said one or more rendering virtual camera poses; and

transmits said one or more 2D projections to said at least one display;

stereo speakers comprising

a left speaker configured to be placed proximal to a left ear of said user; and

a right speaker configured to be placed proximal to a right ear of said user; and

,

an audio renderer coupled to said stereo speakers, said pose analyzer, said video input, and said 3D model of said virtual theater;

wherein

said video comprises one or more audio tracks;

said 3D model of said virtual theater further comprises

one or more virtual speakers located in said virtual theater, each having a virtual speaker location in said virtual theater; and

said audio renderer

associates each of said one or more audio tracks with one or more of said one or more virtual speakers;

sets a volume level for each of said one or more virtual speakers;

calculates an emitted sound for each of said one or more virtual speakers;

receives said pose data from said pose analyzer;

calculates a left received sound for said left ear of said user and a right received sound for said right ear of said user based on

 

said emitted sound for each of said one or more virtual speakers; and

 

said pose data;

sends said left received sound to said left speaker; and

sends said right received sound to said right speaker.

22. A virtual reality virtual theater system comprising:

at least one display viewable by a user;

at least one sensor that generates sensor data that measures one or more aspects of a pose of one or more body parts of said user;

a pose analyzer coupled to said at least one sensor, that calculates pose data of said pose of one or more body parts of said user, based on said sensor data generated by said at least one sensor;

a video input configured to accept a video, wherein said video comprises a plurality of frames;

a 3D model of a virtual theater, comprising

a virtual screen;

a virtual video projector; and

one or more theater environment elements;

a scene renderer coupled to said at least one display, said pose analyzer, said video input, and

said 3D model of said virtual theater, wherein said scene renderer

projects each of said plurality of frames successively from said virtual video projector onto said virtual screen;

receives said pose data from said pose analyzer;

calculates one or more rendering virtual camera poses, based on said pose data;

calculates one or more 2D projections of said 3D model of said virtual theater, based on said one or more rendering virtual camera poses; and

transmits said one or more 2D projections to said at least one display;

an image warper coupled to said at least one display, said scene renderer, and said pose analyzer,

wherein said image warper

receives said one or more rendering virtual camera poses from said scene renderer;

receives said pose data from said pose analyzer;

calculates a change in pose between said one or more virtual camera poses and said pose data;

generates a rerendering approximation of said one or more 2D projections of said 3D model of said virtual theater on said at least one display based on said change in pose; and

modifies one or more pixels of said at least one display based on said rerendering approximation.