Virtual reality environment based manipulation of multi-layered multi-view interactive digital media representations

• US 10,725,609 B2
• Filed: 10/03/2017
• Issued: 07/28/2020
• Est. Priority Date: 07/15/2015
• Status: Active Grant

• Current Assignee: Fyusion, Inc. (Cox Enterprises Incorporated)
• Original Assignee: Fyusion, Inc. (Cox Enterprises Incorporated)
• Inventors: Holzer, Stefan Johannes Josef, Miller, Stephen David, Rusu, Radu Bogdan, Trevor, Alexander Jay Bruen, Chande, Krunal Ketan
• Primary Examiner(s): Pringle-Parker, Jason A
• Application Number:
  US15/724,087
• Publication Number:
  US 20180046357A1
• Time in Patent Office: 1,029 Days
• Field of Search: None
• CPC Class Codes: A63F 2300/303 for displaying additional data, e.g. simulating a Head Up Display
• CPC Class Codes: A63F 2300/8082 Virtual reality
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 19/003 Navigation within 3D models or images
• CPC Class Codes: G06V 10/768 using context analysis, e.g. recognition aided by known co-occurring patterns
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: H04L 67/131 Protocols for games, networked simulations or virtual reality
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/261 with monoscopic-to-stereoscopic image conversion
• CPC Class Codes: H04N 2013/0085 Motion estimation from stereoscopic image signals
• CPC Class Codes: H04N 21/00 Selective content distribution, e.g. interactive television or video on demand [VOD] real-time bi-directional transmission of motion video data H04N7/14 ; broadcast or conference over packet switching networks H04L12/18

Abstract

Various embodiments of the present disclosure relate generally to systems and methods for generating multi-view interactive digital media representations in a virtual reality environment. According to particular embodiments, a plurality of images is fused into a first content model and a first context model, both of which include multi-view interactive digital media representations of s. Next, a virtual reality environment is generated using the first content model and the first context model. The virtual reality environment includes a first layer and a second layer. The user can navigate through and within the virtual reality environment to switch between multiple viewpoints of the content model via corresponding physical movements. The first layer includes the first content model and the second layer includes a second content model and wherein selection of the first layer provides access to the second layer with the second content model.

Claims

1. A method for generating multi-view interactive digital media representations in a virtual reality environment comprising:

obtaining a plurality of images having location information and an object, wherein the plurality of images include at least a portion of overlapping subject matter;

fusing, via a processor, the plurality of images to generate a content model and a context model, wherein the content model includes a multi-view interactive digital media representation of the object and the context model includes a multi-view interactive digital media representation of scenery surrounding the object, wherein the multi-view interactive digital media representations are generated via stitching together two-dimensional images such that the stitched two-dimensional images are experienced as a three dimensional view that is also 360 degree rotatable without actually rendering or generating an actual three-dimensional model, wherein the content model is generated using surround view based object segmentation such that the content model is treated as a separate entity, wherein the content model occupies a three-dimensional space such that a user can circle around the object, wherein the content model is generated using a first set of images with overlapping subject matter and the context model is generated using a second set of images with overlapping subject matter, wherein the first set and the second sets are generated independently and do not have overlapping subject matter between the sets; and

generating, via the processor, a virtual reality environment using the content model and the context model;

wherein the multi-view interactive digital media representations are configured such that the user can manipulate the content model separately within the virtual reality environment without moving or manipulating the context model and wherein the user can also maneuver within the virtual reality environment causing both the context model and the content model to change views simultaneously.

2. The method of claim 1, wherein the content model is configured such that the user can control 360 degrees of the angle and view of the content model within the virtual reality environment.

3. The method of claim 1, wherein the virtual reality environment includes action tabs for assisting in manipulation of the content model.

4. The method of claim 1, wherein the content model includes embedded objects that can be selectable by the user.

5. The method of claim 4, wherein the embedded objects are fixed onto a specific location on the content model such that manipulation of the content model also manipulates the embedded objects.

6. The method of claim 4, wherein after the user selects an embedded object on the content model, a pop-up window appears in the virtual reality environment, the pop-up window configured to display a multi-view interactive digital media representation of an embedded object.

7. The method of claim 6, wherein the multi-view interactive digital media representation of the embedded object is a separate embedded content model configured to be manipulated by the user.

8. A system for generating multi-view interactive digital media representations in a virtual reality environment comprising:

a processor; and

memory, the memory comprising instructions for;

obtaining a plurality of images having location information and an object, wherein the plurality of images include at least a portion of overlapping subject matter;

fusing, via a processor, the plurality of images to generate a content model and a context model, wherein the content model includes a multi-view interactive digital media representation of the object and the context model includes a multi-view interactive digital media representation of scenery surrounding the object, wherein the multi-view interactive digital media representations are generated via stitching together two-dimensional images such that the stitched two-dimensional images are experienced as a three dimensional view that is also 360 degree rotatable without actually rendering or generating an actual three- dimensional model, wherein the content model is generated using surround view based object segmentation such that the content model is treated as a separate entity, wherein the content model occupies a three-dimensional space such that a user can circle around the object, wherein the content model is generated using a first set of images with overlapping subject matter and the context model is generated using a second set of images with overlapping subject matter, wherein the first set and the second sets are generated independently and do not have overlapping subject matter between the sets; and

generating, via the processor, a virtual reality environment using the content model and the context model;

wherein the multi-view interactive digital media representations are configured such that the user can manipulate the content model separately within the virtual reality environment without moving or manipulating the context model and wherein the user can also maneuver within the virtual reality environment causing both the context model and the content model to change views simultaneously.

9. The system of claim 8, wherein the content model is configured such that the user can control 360 degrees of the angle and view of the content model within the virtual reality environment.

10. The system of claim 8, wherein the virtual reality environment includes action tabs for assisting in manipulation of the content model.

11. The system of claim 8, wherein the content model includes embedded objects that can be selectable by the user.

12. The system of claim 11, wherein the embedded objects are fixed onto a specific location on the content model such that manipulation of the content model also manipulates the embedded objects.

13. The system of claim 11, wherein after the user selects an embedded object on the content model, a pop-up window appears in the virtual reality environment, the pop-up window configured to display a multi-view interactive digital media representation of an embedded object.

14. The system of claim 13, wherein the multi-view interactive digital media representation of the embedded object is a separate embedded content model configured to be manipulated by the user.

15. A non-transitory computer readable medium comprising instructions to execute a method for generating multi-view interactive digital media representations in a virtual reality environment, the method comprising:

obtaining a plurality of images having location information and an object, wherein the plurality of images include at least a portion of overlapping subject matter;

fusing, via a processor, the plurality of images to generate a content model and a context model, wherein the content model includes a multi-view interactive digital media representation of the object and the context model includes a multi-view interactive digital media representation of scenery surrounding the object, wherein the multi-view interactive digital media representations are generated via stitching together two-dimensional images such that the stitched two-dimensional images are experienced as a three dimensional view that is also 360 degree rotatable without actually rendering or generating an actual three-dimensional model, wherein the content model is generated using surround view based object segmentation such that the content model is treated as a separate entity, wherein the content model occupies a three-dimensional space such that a user can circle around the object, wherein the content model is generated using a first set of images with overlapping subject matter and the context model is generated using a second set of images with overlapping subject matter, wherein the first set and the second sets are generated independently and do not have overlapping subject matter between the sets; and

generating, via the processor, a virtual reality environment using the content model and the context model;

wherein the multi-view interactive digital media representations are configured such that the user can manipulate the content model separately within the virtual reality environment without moving or manipulating the context model and wherein the user can also maneuver within the virtual reality environment causing both the context model and the content model to change views simultaneously.

16. The non-transitory computer readable medium of claim 15, wherein the content model is configured such that the user can control 360 degrees of the angle and view of the content model within the virtual reality environment.

17. The non-transitory computer readable medium of claim 15, wherein the virtual reality environment includes action tabs for assisting in manipulation of the content model.

18. The non-transitory computer readable medium of claim 15, wherein the content model includes embedded objects that can be selectable by the user.

19. The non-transitory computer readable medium of claim 18, wherein the embedded objects are fixed onto a specific location on the content model such that manipulation of the content model also manipulates the embedded objects.

20. The non-transitory computer readable medium of claim 18, wherein after the user selects an embedded object on the content model, a pop-up window appears in the virtual reality environment, the pop-up window configured to display a multi-view interactive digital media representation of an embedded object, the multi-view interactive digital media representation of the embedded object being a separate embedded content model configured to be manipulated by the user.