Analysis and manipulation of panoramic surround views

• US 10,026,219 B2
• Filed: 11/02/2017
• Issued: 07/17/2018
• Est. Priority Date: 11/12/2013
• Status: Active Grant

• Current Assignee: Fyusion, Inc. (Cox Enterprises Incorporated)
• Original Assignee: Fyusion, Inc. (Cox Enterprises Incorporated)
• Inventors: Holzer, Stefan Johannes Josef, Miller, Stephen David, Rusu, Radu Bogdan
• Primary Examiner(s): WILLIAMS, JEFFERY A
• Application Number:
  US15/802,437
• Publication Number:
  US 20180068485A1
• Time in Patent Office: 257 Days
• Field of Search: 348 36
• CPC Class Codes: G06F 16/532 Query formulation, e.g. graphical querying
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/738 Presentation of query results
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 2215/12 Shadow map, environment map
• CPC Class Codes: G06T 2215/16 Using real world measurements to influence rendering
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: H04N 13/279 the virtual viewpoint locations being selected by the viewers or determined by tracking
• CPC Class Codes: H04N 13/282 for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 5/265 Mixing

Abstract

Various embodiments of the present invention relate generally to systems and methods for analyzing and manipulating images and video. According to particular embodiments, the spatial relationship between multiple images and video is analyzed together with location information data, for purposes of creating a representation referred to herein as a surround view. In particular embodiments, a surround view can be generated by combining a panoramic view of an object with a panoramic view of a distant scene, such that the object panorama is placed in a foreground position relative to the distant scene panorama. Such combined panoramas can enhance the interactive and immersive viewing experience of the surround view.

Claims

1. A method comprising:

generating a combined surround view including a first surround view placed in a foreground position relative to a second surround view, wherein the first and second surround views are obtained using different capture motions, and wherein non-linear surround views are separated into discrete components and complex motions are broken down into locally convex and linear portions such that a more complex surround view corresponding to a more complex capture motion can be generated using separate smaller surround views corresponding to separate smaller capture motions.

2. The method of claim 1, wherein the first surround view is an object panorama.

3. The method of claim 1, wherein the first surround view includes a panoramic multi-view interactive digital media representation of an object.

4. The method of claim 1, wherein the second surround view includes a panoramic view of a distant scene.

5. The method of claim 1, wherein the combined surround view includes dynamic content that is animated during user navigation.

6. The method of claim 1, wherein an object in the combined surround view appears three-dimensional when the combined surround view is navigated.

7. The method of claim 1, wherein an object is displayed on a device, wherein a viewing angle of the object is manipulated by tilting and rotating the device.

8. A system comprising:

a processor; and

memory, wherein the memory includes instructions for;

generating a combined surround view including a first surround view placed in a foreground position relative to a second surround view, wherein the first and second surround views are obtained using different capture motions, and wherein non-linear surround views are separated into discrete components and complex motions are broken down into locally convex and linear portions such that a more complex surround view corresponding to a more complex capture motion can be generated using separate smaller surround views corresponding to separate smaller capture motions.

9. The system of claim 8, wherein the first surround view is an object panorama.

10. The system of claim 8, wherein the first surround view includes a panoramic multi-view interactive digital media representation of an object.

11. The system of claim 8, wherein the second surround view includes a panoramic view of a distant scene.

12. The system of claim 8, wherein the combined surround view includes dynamic content that is animated during user navigation.

13. The system of claim 8, wherein an object in the combined surround view appears three-dimensional when the combined surround view is navigated.

14. The system of claim 8, wherein an object is displayed on a device, wherein a viewing angle of the object is manipulated by tilting and rotating the device.

15. A non-transitory computer readable medium comprising instructions for:

generating a combined surround view including a first surround view placed in a foreground position relative to a second surround view, wherein the first and second surround views are obtained using different capture motions, and wherein non-linear surround views are separated into discrete components and complex motions are broken down into locally convex and linear portions such that a more complex surround view corresponding to a more complex capture motion can be generated using separate smaller surround views corresponding to separate smaller capture motions.

16. The non-transitory computer readable medium of claim 15, wherein the first surround view is an object panorama.

17. The non-transitory computer readable medium of claim 15, wherein the first surround view includes a panoramic multi-view interactive digital media representation of an object.

18. The non-transitory computer readable medium of claim 15, wherein the second surround view includes a panoramic view of a distant scene.

19. The non-transitory computer readable medium of claim 15, wherein the combined surround view includes dynamic content that is animated during user navigation.

20. The non-transitory computer readable medium of claim 15, wherein an object in the combined surround view appears three-dimensional when the combined surround view is navigated.