Coherent presentation of multiple reality and interaction models

• US 8,963,916 B2
• Filed: 03/15/2013
• Issued: 02/24/2015
• Est. Priority Date: 08/26/2011
• Status: Expired due to Fees

• Current Assignee: ReinCloud Corp.
• Original Assignee: ReinCloud Corp.
• Inventors: Reitan, Dan
• Primary Examiner(s): Ma, Tize
• Assistant Examiner(s): He, Yingchun
• Application Number:
  US13/841,077
• Publication Number:
  US 20130335407A1
• Time in Patent Office: 711 Days
• Field of Search: 345/633
• US Class Current: 345/419
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 8/315 Object-oriented languages
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G09G 5/377 for mixing or overlaying two or more graphic patterns G09G5/02, G09G5/397 take precedence

Abstract

A method for navigating concurrently and from point-to-point through multiple reality models is described. The method includes: generating, at a processor, a first navigatable virtual view of a first location of interest, wherein the first location of interest is one of a first virtual location and a first non-virtual location; and concurrently with the generating the first navigatable virtual view of the first location of interest, generating, at the processor, a second navigatable virtual view corresponding to a current physical position of an object, such that real-time sight at the current physical position is enabled within the second navigatable virtual view.

Claims

1. A method for delivering immersive media for a device, said method comprising:

mapping, by a computer, an image from at least one input media stream to a virtual display surface, wherein said at least one input media stream comprises video images captured by lens arrays of a plurality of video cameras, wherein a first lens array of said lens arrays captures a first set of images comprising a first field of view and a second lens array of said lens arrays captures a second set of images comprising a second field of view, wherein said first field of view and said second field of view overlap and have at least one object in common between said first field of view and said second field of view;

receiving, by said computer, an indication of a virtual viewport orientation of a playback device;

determining a video media stream of said at least one input media stream that has a lowest normal angle from said at least one object to a viewpoint vector of said plurality of video cameras;

using, by said computer, said indication of said virtual viewport orientation and a determination of said video media stream that has said lowest normal angle from said at least one object to said viewpoint vector, to select a portion of said image for displaying; and

outputting, by said computer, said portion of said image which has been mapped to said virtual display surface.

2. The method of claim 1 further comprising:

determining a characteristic of said playback device; and

selecting a shape of said virtual display surface based upon said characteristic of said playback device.

3. The method of claim 2 further comprising:

determining that said playback device comprises a Graphics Processing Unit (GPU);

creating a polygonal virtual display surface using the GPU; and

mapping said image to said polygonal virtual display surface.

4. The method of claim 2 further comprising:

using at least one Central Processing Unit (CPU) of said playback device to create a planar virtual display surface; and

mapping said image to said planar virtual display surface.

5. The method of claim 1 further comprising:

selecting at least two images from two respective input media streams based upon said indication of said virtual viewport orientation;

mapping each of said at least two images to respective virtual display surfaces; and

outputting said selected portions of said at least two images which have been mapped to said respective virtual display surfaces to a three-dimensional (3-D) display device.

6. The method of claim 1 further comprising:

pre-blending at least two input media streams to create a blended input media stream;

mapping said blended input stream to a spherical virtual display surface; and

outputting said selected portion of said image which has been mapped to said spherical virtual display surface.

7. The method of claim 1 further comprising:

using said indication of said virtual viewport orientation to determine a position of said playback device relative to a virtual display space defined at least in part by said virtual display surface.

8. A non-transitory computer-readable storage medium comprising computer executable code for directing a processor to execute a method for delivering immersive media for a device, said method comprising:

mapping an image from at least one input media stream to a virtual display surface, wherein said at least one input media stream comprises video images captured by lens arrays of a plurality of video cameras camera, wherein a first lens array of said lens arrays captures a first set of images comprising a first field of view and a second lens array of said lens arrays captures a second set of images comprising a second field of view, wherein said first field of view and said second field of view overlap and have at least one object in common between said first field of view and said second field of view;

receiving an indication of a virtual viewport orientation of a playback device;

determining a video media stream of said at least one input media stream that has a lowest normal angle from said at least one object to a viewpoint vector of said plurality of video cameras;

using said indication of said virtual viewport orientation and a determination of said video media stream that has said lowest normal angle from said at least one object to said viewpoint vector, to select a portion of said image for displaying; and

outputting said portion of said image which has been mapped to said virtual display surface.

9. The non-transitory computer-readable storage medium of claim 8 wherein said method further comprises:

determining a characteristic of said playback device; and

selecting a shape of said virtual display surface based upon said characteristic of said playback device.

10. The non-transitory computer-readable storage medium of claim 9 wherein said method further comprises:

determining that said playback device comprises a Graphics Processing Unit (GPU);

creating a polygonal virtual display surface using the GPU; and

mapping said image to said polygonal virtual display surface.

11. The non-transitory computer-readable storage medium of claim 9 wherein said method further comprises:

using at least one Central Processing Unit (CPU) of the playback device to create a planar virtual display surface; and

mapping said image to said planar virtual display surface.

12. The non-transitory computer-readable storage medium of claim 8 wherein said method further comprises:

selecting at least two images from two respective input media streams based upon said indication of said virtual viewport orientation;

mapping each of said at least two images to respective virtual display surfaces; and

outputting said selected portions of said at least two images which have been mapped to said respective virtual display surfaces to a three-dimensional (3-D) display device.

13. The non-transitory computer-readable storage medium of claim 8 wherein said method further comprises:

pre-blending at least two input media streams to create a blended input media stream;

mapping said blended input stream to a spherical virtual display surface; and

outputting said selected portion of said image which has been mapped to said spherical virtual display surface.

14. The non-transitory computer-readable storage medium of claim 8 wherein said method further comprises:

using said indication of said virtual viewport orientation to determine a position of said playback device relative to a virtual display space defined at least in part by said virtual display surface.

15. A system for delivering immersive media for a device comprising;

a recording device for capturing a plurality of video data streams and a plurality of audio data streams; and

a rendering component, residing on a computer, for mapping an image from at least one input media stream to a virtual display surface, wherein said at least one input media stream comprises video images captured by lens arrays of a plurality of video cameras, wherein a first lens array of said lens arrays captures a first set of images comprising a first field of view and a second lens array of said lens arrays captures a second set of images comprising a second field of view, wherein said first field of view and said second field of view overlap and have at least one object in common between said first field of view and said second field of view, receive an indication of a virtual viewport orientation of a playback device, determine a video media stream of said at least one input media stream that has a lowest normal angle from said at least one object to a viewpoint vector of said plurality of video cameras, use said indication of said virtual viewport orientation and a determination of said video media stream that has said lowest normal angle from said at least one object to said viewpoint vector, to select a portion of said image for displaying, and to output said portion of said image which has been mapped to said virtual display surface.

16. The system of claim 15 wherein said rendering component further comprises:

a playback device characteristic determination component for determining a characteristic of said playback device, and wherein said rendering component selects a shape of said virtual display surface based upon said characteristic of said playback device.

17. The system of claim 16 wherein said rendering component further creates a polygonal virtual display surface and to map said image to said polygonal virtual display surface in response to determining that said playback device comprises a Graphics Processing Unit (GPU) and to create a planar virtual display surface using at least one Central Processing Unit (CPU) of said playback device and to map said image to said planar virtual display surface.

18. The system of claim 16 wherein said rendering component further selects at least two images from two respective input media streams based upon said indication of said virtual viewport orientation, maps each of said at least two images to respective virtual display surfaces, and outputs said selected portions of said at least two images which have been mapped to said respective virtual display surfaces to a three-dimensional (3-D) display device.

19. The system of claim 15 further comprising:

a pre-blending component, residing at said computer, for pre-blending at least two input media streams to create a blended input media stream, and wherein said rendering component maps said blended input stream to a spherical virtual display surface and outputs said selected portion of said image which has been mapped to said spherical virtual display surface.

20. The system of claim 15 wherein said rendering component uses said indication of said virtual viewport orientation to determine a position of said playback device relative to a virtual display space defined at least in part by said virtual display surface.