Efficient canvas view generation from intermediate views

• US 10,257,501 B2
• Filed: 04/11/2016
• Issued: 04/09/2019
• Est. Priority Date: 04/06/2016
• Status: Active Grant

• Current Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Original Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Inventors: Cabral, Brian Keith, Briggs, Forrest Samuel, Pozo, Albert Parra, Vajda, Peter
• Primary Examiner(s): Pringle-Parker, Jason A
• Application Number:
  US15/096,169
• Publication Number:
  US 20170293997A1
• Time in Patent Office: 1,093 Days
• Field of Search: None
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2200/32 involving image mosaicing
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 7/207 for motion estimation over a hierarchy of resolutions multi-resolution motion estimation or hierarchical motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/53
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/269 using gradient-based methods
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/156 Mixing image signals
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/257 Colour aspects
• CPC Class Codes: H04N 13/282 for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Abstract

A canvas generation system generates a canvas view of a scene based on a set of original camera views depicting the scene, for example to recreate a scene in virtual reality. Canvas views can be generated based on a set of synthetic views generated from a set of original camera views. Synthetic views can be generated, for example, by shifting and blending relevant original camera views based on an optical flow across multiple original camera views. An optical flow can be generated using an iterative method which individually optimizes the optical flow vector for each pixel of a camera view and propagates changes in the optical flow to neighboring optical flow vectors.

Claims

1. A method comprising:

receiving, at a canvas view generation system, a set of camera views depicting a scene as captured by a plurality of cameras, each camera view associated with a camera view location from which that camera view was captured;

identifying a set of canvas view regions for a canvas view of the scene depicting a range of angles of the scene, each canvas view region in the set of regions associated with an angle in the range of angles;

generating the canvas view by, for each canvas view region in the set of regions;

determining a synthetic camera location for the canvas view region based on the angle;

generating a first mapping associating the canvas view region with a synthetic view region of a synthetic view associated with the synthetic camera location;

generating a second mapping associating regions of a plurality of camera views of the set of camera views with the synthetic view region;

combining the first mapping and the second mapping to generate a combined mapping associating the canvas view region of the canvas view with regions of one or more camera views of the set of camera views; and

applying the combined mapping to generate the canvas view for the canvas view region.

2. The method of claim 1, wherein the second mapping is generated based on an optical flow vector field associating points in the set of camera views.

3. The method of claim 2, further comprising calculating a set of optical flow vector fields based on the synthetic camera locations and the set of camera views.

4. The method of claim 1, wherein the canvas view is a 360 degree panoramic or spherical panoramic image of the scene.

5. The method of claim 1, wherein the canvas view is output in cubemap, equirectangular, or cylindrical format.

6. The method of claim 1, further comprising determining a canvas viewpoint for each canvas view region in the set of regions and wherein the synthetic camera location for a region is based on the canvas viewpoint for the region.

7. The method of claim 6, wherein determining the synthetic camera location for a region is based on a line of sight from the canvas viewpoint of the region to a zero parallax distance in the scene.

8. The method of claim 6, wherein each canvas view region approximates the light information at the canvas viewpoint of the canvas view region.

9. The method of claim 1, wherein each camera view of the set camera views overlaps with at least one other camera view of the set of camera views.

10. The method of claim 1, further comprising sending the canvas view to a client virtual reality device for display.

11. The method of claim 1, wherein each canvas view region is a vertical column of pixels.

12. A system comprising:

a processor; and

a non-transitory computer readable storage medium comprising instructions that, when executed by the processor, cause the processor to;

receive a set of camera views depicting a scene as captured by a plurality of cameras, each camera view associated with a camera view location from which that camera view was captured;

identify a set of canvas view regions for a canvas view of the scene depicting a range of angles of the scene, each canvas view region in the set of regions associated with an angle in the range of angles; and

generate the canvas view by, for each canvas view region in the set of regions;

determine a synthetic camera location for the canvas view region based on the angle;

generate a first mapping associating the canvas view region with a synthetic view region of a synthetic view associated with the synthetic camera location;

generate a second mapping associating regions of a plurality of camera views of the set of camera views with the synthetic view region;

combine the first mapping and the second mapping to generate a combined mapping associating the canvas view region of the canvas view with regions of one or more camera views of the set of camera views; and

apply the combined mapping to generate the canvas view for the canvas view region.

13. The system of claim 12, wherein the second mapping is generated based on an optical flow vector field associating points in the set of camera views.

14. The system of claim 12, wherein the instructions, when executed by the processor, further causes the processor to calculate a set of optical flow vector fields based on the synthetic camera locations and the set of camera views.

15. The system of claim 12, wherein the canvas view is a 360 degree panoramic or spherical panoramic image of the scene.

16. The system of claim 12, wherein the instructions, when executed by the processor, further causes the processor to determine a canvas viewpoint for each canvas view region in the set of regions and wherein the synthetic camera location for a region is based on the canvas viewpoint for the region.

17. The system of claim 16, wherein determining the synthetic camera location for a region is based on a line of sight from the canvas viewpoint of the region to a zero parallax distance in the scene.

18. The system of claim 16, wherein each canvas view region approximates the light information at the canvas viewpoint of the canvas view region.

19. The system of claim 12, wherein each camera view of the set camera views overlaps with at least one other camera view of the set of camera views.

20. The system of claim 12, further comprising sending the canvas view to a client virtual reality device for display.