Techniques for capturing and displaying partial motion in virtual or augmented reality scenes

• US 10,977,864 B2
• Filed: 02/20/2015
• Issued: 04/13/2021
• Est. Priority Date: 02/21/2014
• Status: Active Grant

• Current Assignee: Dropbox, Inc.
• Original Assignee: Dropbox, Inc.
• Inventors: Newhouse, Benjamin Zeis, McArdle, Terrence Edward
• Primary Examiner(s): Liu, Gordon G
• Application Number:
  US14/628,113
• Publication Number:
  US 20150243085A1
• Time in Patent Office: 2,244 Days
• Field of Search: 345633
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2215/16 Using real world measurements to influence rendering
• CPC Class Codes: H04N 5/77 between a recording apparatus and a television camera

Abstract

The present disclosure relates to techniques for capturing and displaying partial motion in VAR scenes. VAR scenes can include a plurality of images combined and oriented over any suitable geometry. Although VAR scenes may provide an immersive view of a static scene, current systems do not generally support VAR scenes that include dynamic content (e.g., content that varies over time). Embodiments of the present invention can capture, generate, and/or share VAR scenes. This immersive, yet static, view of the VAR scene lacks dynamic content (e.g., content which varies over time). Embodiments of the present invention can efficiently add dynamic content to the VAR scene, allowing VAR scenes including dynamic content to be uploaded, shared, or otherwise transmitted without prohibitive resource requirements. Dynamic content can be captured by device and combined with a preexisting or simultaneously captured VAR scene, and the dynamic content may be played back upon selection.

Claims

1. A method comprising:

generating a virtual or augmented reality (VAR) scene based on a first plurality of images captured by a camera of a device;

generating dynamic content for a portion of the VAR scene, the dynamic content comprising a second plurality of images captured by the camera of the device;

detecting a first orientation of the device;

based on detecting the first orientation of the device, presenting a first view of the VAR scene on the device;

detecting a second orientation of the device; and

based on detecting the second orientation of the device, presenting a second view of the VAR scene comprising the dynamic content overlaid over the portion of the VAR scene by overlaying at least one image from the second plurality of images on at least one image from the first plurality of images and varying the at least one image from the second plurality of images.

2. The method of claim 1, wherein varying the at least one image from the second plurality of images comprises varying the at least one image over a spatial domain within the VAR scene.

3. The method of claim 1, further comprising, before generating the VAR scene based on the first plurality of images captured by the camera of the device, capturing the first plurality of images by:

detecting location and orientation data of the device; and

associating each image of the first plurality of images with the location and orientation data detected when that image is captured.

4. The method of claim 3, further comprising, before generating the dynamic content for the portion of the VAR scene, capturing the second plurality of images by:

receiving a selection to capture the second plurality of images with the device in a starting location and orientation;

detecting additional location and orientation data of the device; and

associating each image of the second plurality of images with the additional location and orientation data detected when that image is captured.

5. The method of claim 4, wherein:

generating the VAR scene based on the first plurality of images captured by the camera of the device comprises assembling the first plurality of images into a spherical VAR scene based on the detected location and orientation data associated with each image of the first plurality of images; and

overlaying the at least one image from the second plurality of images on the at least one image from the first plurality of images comprises overlaying the at least one image from the second plurality of images on the spherical VAR scene based on the location and orientation data associated with the at least one image from the second plurality of images.

6. The method of claim 5, wherein overlaying the at least one image from the second plurality of images on the spherical VAR scene comprises:

blending the at least one image from the second plurality of images with the spherical VAR scene.

7. The method of claim 6, wherein the blending comprises one or more of direct placement, blending using a Gaussian mask, Poisson blending, or seaming.

8. A device comprising:

at least one processor; and

at least one non-transitory computer readable storage medium storing instructions that, when executed by the at least one processor, cause the device to;

generate a virtual or augmented reality (VAR) scene based on a first plurality of images captured by a camera of the device;

generate dynamic content that varies over spatial domain of the VAR scene, the dynamic content comprising a second plurality of images captured by the camera of the device;

detect a first location of the device;

based on detecting the first location of the device, present a first view of the VAR scene comprising a first portion of the dynamic content overlaid over the VAR scene by overlaying at least an image from the second plurality of images on at least one image from the first plurality of images;

detect a second location of the device; and

based on detecting the second location of the device, present a second view of the VAR scene comprising a second portion of the dynamic content overlaid over the VAR scene by overlaying at least one an additional image from the second plurality of images on at least one image from the first plurality of images.

9. The device of claim 8, wherein the VAR scene comprises a static base VAR scene.

10. The device of claim 8, further comprising instructions that, when executed by the at least one processor, cause the device to, before generating the VAR scene based on the first plurality of images captured by the camera of the device, capture the first plurality of images by:

detecting location and orientation data of the device; and

associating each image of the first plurality of images with the location and orientation data detected when that image is captured.

11. The device of claim 10, further comprising instructions that, when executed by the at least one processor, cause the device to, before generating the dynamic content for a portion of the VAR scene, capture the second plurality of images by:

receiving a selection to capture the second plurality of images with the device in a starting location and orientation;

detecting additional location and orientation data of the device; and

associating each image of the second plurality of images with the additional location and orientation data detected when that image is captured.

12. The device of claim 11, further comprising instructions that, when executed by the at least one processor, cause the device to:

generate the VAR scene based on the first plurality of images captured by the camera of the device by assembling the first plurality of images into a spherical VAR scene based on the detected location and orientation data associated with each image of the first plurality of images; and

overlay at least the image from the second plurality of images on the at least one image from the first plurality of images by, based on detecting the first location of the device, overlaying at least the image from the second plurality of images on the spherical VAR scene based on the location and orientation data associated with the image from the second plurality of images.

13. The device of claim 12, wherein overlaying at least the image from the second plurality of images on the spherical VAR scene comprises:

blending the image from the second plurality of images with the spherical VAR scene.

14. The device of claim 13, wherein the blending comprises one or more of direct placement, blending using a Gaussian mask, Poisson blending, or seaming.

15. A non-transitory computer readable storage medium storing instructions thereon that, when executed by at least one processor, cause a device to:

generate a virtual or augmented reality (VAR) scene based on a first plurality of images captured by a camera of the device;

generate dynamic content for a portion of the VAR scene, the dynamic content comprising a second plurality of images captured by the camera of the device;

detect a first orientation of the device;

based on detecting the first orientation of the device, present a first view of the VAR scene on the device;

detect a second orientation of the device; and

based on detecting the second orientation of the device, present a second view of the VAR scene comprising the dynamic content overlaid over the VAR scene by overlaying at least one image from the second plurality of images on at least one image from the first plurality of images and varying over time the at least one image from the second plurality of images.

16. The non-transitory computer readable storage medium of claim 15, further comprising instructions that, when executed by the at least one processor, cause the device to vary the at least one image from the second plurality of images by varying the at least one image further over time within the VAR scene.

17. The non-transitory computer readable storage medium of claim 15, further comprising instructions that, when executed by the at least one processor, cause the device to, before generating the VAR scene based on the first plurality of images captured by the camera of the device, capture the first plurality of images by:

detecting location and orientation data of the device; and

associating each image of the first plurality of images with the location and orientation data detected when that image is captured.

18. The non-transitory computer readable storage medium of claim 17, further comprising instructions that, when executed by the at least one processor, cause the device to, before generating the dynamic content for the portion of the VAR scene, capture the second plurality of images by:

receiving a selection to capture the second plurality of images with the device in a starting location and orientation;

detecting additional location and orientation data of the device; and

associating each image of the second plurality of images with the additional location and orientation data detected when that image is captured.

19. The non-transitory computer readable storage medium of claim 18, further comprising instructions that, when executed by the at least one processor, cause the device to:

generate the VAR scene based on the first plurality of images captured by the camera of the device by assembling the first plurality of images into a spherical VAR scene based on the detected location and orientation data associated with each image of the first plurality of images; and

overlay the at least one image from the second plurality of images on the at least one image from the first plurality of images by overlaying the at least one image from the second plurality of images on the spherical VAR scene based on the location and orientation data associated with the at least one image from the second plurality of images.

20. The non-transitory computer readable storage medium of claim 19, further comprising instructions that, when executed by the at least one processor, cause the device to overlay the at least one image from the second plurality of images on the spherical VAR scene by:

blending the at least one image from the second plurality of images with the spherical VAR scene, wherein the blending comprises one or more of direct placement, blending using a Gaussian mask, Poisson blending, or seaming.