Efficient determination of optical flow between images

• US 10,165,258 B2
• Filed: 04/11/2016
• Issued: 12/25/2018
• Est. Priority Date: 04/06/2016
• Status: Active Grant

• Current Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Original Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Inventors: Cabral, Brian Keith, Briggs, Forrest Samuel, Pozo, Albert Parra, Vajda, Peter
• Primary Examiner(s): Patel, Jayanti K
• Assistant Examiner(s): Habib, Irfan
• Application Number:
  US15/096,167
• Publication Number:
  US 20170295354A1
• Time in Patent Office: 988 Days
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2200/32 involving image mosaicing
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 7/207 for motion estimation over a hierarchy of resolutions multi-resolution motion estimation or hierarchical motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/53
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/269 using gradient-based methods
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/156 Mixing image signals
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/257 Colour aspects
• CPC Class Codes: H04N 13/282 for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Abstract

A canvas generation system generates a canvas view of a scene based on a set of original camera views depicting the scene, for example to recreate a scene in virtual reality. Canvas views can be generated based on a set of synthetic views generated from a set of original camera views. Synthetic views can be generated, for example, by shifting and blending relevant original camera views based on an optical flow across multiple original camera views. An optical flow can be generated using an iterative method which individually optimizes the optical flow vector for each pixel of a camera view and propagates changes in the optical flow to neighboring optical flow vectors.

Claims

1. A method for generating an optical flow, the method comprising:

receiving a first camera view and a second camera view each comprised of a set of pixels, each camera view representing an image captured by a camera and wherein each camera view is associated with a location from which that camera view was captured;

initializing an optical flow that maps pixels of the first camera view with corresponding pixels of the second camera view; and

optimizing the optical flow for each pixel of the first camera view according to an ordered sequence of the pixels of the first camera view, wherein optimizing the optical flow for each pixel comprises;

generating a set of optical flow proposals for the pixel, each optical flow proposal mapping the pixel to a pixel of the second camera view, the set of optical flow proposals comprising an optical flow proposal associated with one or more previous pixels in the ordered sequence;

analyzing each optical flow proposal of the set of optical flow proposals; and

updating the optical flow for the pixel based on a selected optical flow proposal of the set of optical flow proposals.

2. The method of claim 1, wherein an optical flow is a vector field associating each pixel of a camera view with an optical flow vector giving a displacement between the pixel and a corresponding pixel of another camera view.

3. The method of claim 2, wherein a set of optical flow proposals for a pixel comprises the optical flow vector associated with an upper neighbor of the pixel, the optical flow vector associated with the left neighbor of the pixel, and a random perturbation of the current optical flow vector.

4. The method of claim 1, further comprising establishing a variational optimization problem including data and regularization terms.

5. The method of claim 4, wherein updating the optical flow for a pixel comprises selecting the optical flow proposal that moves the variational optimization problem closest to a solution.

6. The method of claim 1, further comprising applying a median filter to the optical flow.

7. The method of claim 1, wherein receiving a first camera view and a second camera view includes receiving a set of camera views including the first and second camera views and further comprising initializing and optimizing an optical flow for each other camera view of the set of camera views.

8. The method of claim 1, wherein the optical flow for the pixels of the first camera view are optimized in an ordered sequence starting from the top left pixel and proceeding to the lower right pixel of the first camera view.

9. The method of claim 1, further comprising optimizing the optical flow for each pixel of the first camera view until a measurement of the optical flow quality reaches a threshold score.

10. The method of claim 1, wherein initializing the optical flow comprises:

scaling up a lower resolution optical flow; and

proportionally increasing the magnitude of optical flow vectors within the lower resolution optical flow.

11. A system comprising:

an input module, the input module configured to receive a first camera view and a second camera view each comprised of a set of pixels, each camera view representing an image captured by a camera and wherein each camera view is associated with a location from which that camera view was captured;

an optical flow calculation module, the optical flow calculation module configured to;

initialize an optical flow that maps pixels of the first camera view with corresponding pixels of the second camera view; and

optimize the optical flow for each pixel of the first camera view according to an ordered sequence of the pixels of the first camera view, wherein optimizing the optical flow for each pixel comprises;

generating a set of optical flow proposals for the pixel, each optical flow proposal mapping the pixel to a pixel of the second camera view, the set of optical flow proposals comprising an optical flow proposal associated with one or more previous pixels in the ordered sequence;

analyzing each optical flow proposal of the set of optical flow proposals; and

updating the optical flow for the pixel based on a selected optical flow proposal of the set of optical flow proposals.

12. The system of claim 11, wherein an optical flow is a vector field associating each pixel of a camera view with an optical flow vector giving a displacement between the pixel and a corresponding pixel of another camera view.

13. The system of claim 12, wherein a set of optical flow proposals for a pixel comprises the optical flow vector associated with an upper neighbor of the pixel, the optical flow vector associated with the left neighbor of the pixel, and a random perturbation of the current optical flow vector.

14. The system of claim 11, wherein the optical flow calculation module is further configured to establish a variational optimization problem including data and regularization terms.

15. The system of claim 14, wherein updating the optical flow for a pixel comprises determining the optical flow proposal that moves the variational optimization problem closest to a solution.

16. The system of claim 11, further comprising applying a median filter to the optical flow.

17. The system of claim 11, wherein receiving a first camera view and a second camera view includes receiving a set of camera views including the first can second camera views and wherein the optical flow calculation module is further configured to initialize and optimize an optical flow for each other camera view of the set of camera views.

18. The system of claim 11, wherein the optical flow for the pixels of the first camera view are optimized in an ordered sequence starting from the top left pixel and proceeding to the lower right pixel of the first camera view.

19. The system of claim 11, wherein the optical flow calculation module is further configured to optimize the optical flow for each pixel of the first camera view until a measurement of the optical flow quality reaches a threshold score.

20. The system of claim 11, wherein initializing the optical flow comprises:

scaling up a lower resolution optical flow; and

proportionally increasing the magnitude of optical flow vectors within the lower resolution optical flow.