Analysis and manipulation of panoramic surround views

• US 10,169,911 B2
• Filed: 07/12/2018
• Issued: 01/01/2019
• Est. Priority Date: 11/12/2013
• Status: Active Grant

• Current Assignee: Fyusion, Inc. (Cox Enterprises Incorporated)
• Original Assignee: Fyusion, Inc. (Cox Enterprises Incorporated)
• Inventors: Holzer, Stefan Johannes Josef, Miller, Stephen David, Rusu, Radu Bogdan
• Primary Examiner(s): Williams, Jeffery A
• Application Number:
  US16/033,752
• Publication Number:
  US 20180330537A1
• Time in Patent Office: 173 Days
• Field of Search: 348 36
• CPC Class Codes: G06F 16/532 Query formulation, e.g. graphical querying
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/738 Presentation of query results
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 2215/12 Shadow map, environment map
• CPC Class Codes: G06T 2215/16 Using real world measurements to influence rendering
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: H04N 13/279 the virtual viewpoint locations being selected by the viewers or determined by tracking
• CPC Class Codes: H04N 13/282 for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 5/265 Mixing

Abstract

Various embodiments of the present invention relate generally to systems and methods for analyzing and manipulating images and video. According to particular embodiments, the spatial relationship between multiple images and video is analyzed together with location information data, for purposes of creating a representation referred to herein as a surround view. In particular embodiments, a surround view can be generated by combining a panoramic view of an object with a panoramic view of a distant scene, such that the object panorama is placed in a foreground position relative to the distant scene panorama. Such combined panoramas can enhance the interactive and immersive viewing experience of the surround view.

Claims

1. A method comprising:

obtaining a first surround view, wherein the first surround view includes a panoramic multi-view interactive digital media representation of an object, wherein the first surround view is an object panorama;

obtaining a second surround view, wherein the second surround view includes a panoramic view of a distant scene; and

generating a third surround view including the first surround view placed in a foreground position relative to the second surround view, wherein the first and second surround views are obtained using different capture motions, wherein the third surround view is generated using IMU data, keypoint tracks, and view interpolation in order to reduce the amount of data that must be transferred during upload or download of the third surround view.

2. The method of claim 1, wherein the first surround view is an object panorama.

3. The method of claim 1, wherein the object is dynamic, and wherein the third surround view is a dynamic panorama.

4. The method of claim 1, wherein the distant scene is dynamic, and wherein the third surround view is a dynamic panorama.

5. The method of claim 1, wherein the third surround view includes dynamic content that is animated during user navigation.

6. The method of claim 1, wherein the object is made of a group of items.

7. The method of claim 1, wherein the distant scene moves less than the object when the third surround view is navigated.

8. The method of claim 7, wherein the object appears three-dimensional when the third surround view is navigated.

9. The method of claim 1, wherein the object is display on a device, wherein the viewing angle of the object is manipulated by tilting and rotating the device.

10. The method of claim 1, wherein the viewing angle of the object is manipulated by rotating the device along an axis corresponding to a same axis of the object.

11. A non-transitory computer readable medium storing instructions for executing a method, the method comprising:

obtaining a first surround view, wherein the first surround view includes a panoramic multi-view interactive digital media representation of an object, wherein the first surround view is an object panorama;

obtaining a second surround view, wherein the second surround view includes a panoramic view of a distant scene; and

generating a third surround view including the first surround view placed in a foreground position relative to the second surround view, wherein the first and second surround views are obtained using different capture motions, wherein the third surround view is generated using IMU data, keypoint tracks, and view interpolation in order to reduce the amount of data that must be transferred during upload or download of the third surround view.

12. The non-transitory computer readable medium of claim 11, wherein the first surround view is an object panorama.

13. The non-transitory computer readable medium of claim 11, wherein the object is dynamic, and wherein the third surround view is a dynamic panorama.

14. The non-transitory computer readable medium of claim 11, wherein the distant scene is dynamic, and wherein the third surround view is a dynamic panorama.

15. The non-transitory computer readable medium of claim 11, wherein the third surround view includes dynamic content that is animated during user navigation.

16. The non-transitory computer readable medium of claim 11, wherein the object is made of a group of items.

17. The non-transitory computer readable medium of claim 11, wherein the distant scene moves less than the object when the third surround view is navigated.

18. The non-transitory computer readable medium of claim 17, wherein the object appears three-dimensional when the third surround view is navigated.

19. A system comprising:

a processor; and

memory, wherein the memory includes instructions for;

obtaining a first surround view, wherein the first surround view includes a panoramic multi-view interactive digital media representation of an object, wherein the first surround view is an object panorama;

obtaining a second surround view, wherein the second surround view includes a panoramic view of a distant scene; and

generating a third surround view including the first surround view placed in a foreground position relative to the second surround view, wherein the first and second surround views are obtained using different capture motions, wherein the third surround view is generated using IMU data, keypoint tracks, and view interpolation in order to reduce the amount of data that must be transferred during upload or download of the third surround view.

20. The system of claim 19, wherein the first surround view is an object panorama.