Removing occlusion in camera views

• US 10,210,660 B2
• Filed: 04/11/2016
• Issued: 02/19/2019
• Est. Priority Date: 04/06/2016
• Status: Active Grant

• Current Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Original Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Inventors: Cabral, Brian Keith, Briggs, Forrest Samuel
• Primary Examiner(s): Patel, Jayanti K
• Assistant Examiner(s): Habib, Irfan
• Application Number:
  US15/096,157
• Publication Number:
  US 20170294006A1
• Time in Patent Office: 1,044 Days
• CPC Class Codes: G06T 15/40 Hidden part removal
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 2207/20228 Disparity calculation for image-based rendering
• CPC Class Codes: G06T 2219/028 Multiple view windows (top-side-front-sagittal-orthogonal)
• CPC Class Codes: G06T 3/18 Image warping, e.g. rearranging pixels individually
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06V 10/147 Details of sensors, e.g. sensor lenses fingerprint or palmprint sensors G06V40/13; vascular sensors G06V40/145; eye sensors G06V40/19
• CPC Class Codes: G06V 10/16 using multiple overlapping images; Image stitching
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 2013/0088 Synthesising a monoscopic image signal from stereoscopic images, e.g. synthesising a panoramic or high resolution monoscopic image
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture

Abstract

An image processing system is designed to generate a canvas view that has smooth transition between binocular views and monocular views. Initially, the image processing system receives top/bottom images and side images of a scene and calculates offsets to generate synthetic side images for left and right view of a user. To realize smooth transition between binocular views and monocular views, the image processing system first warps top/bottom images onto corresponding synthetic side images to generate warped top/bottom images, which realizes the smooth transition in terms of shape. The image processing system then morphs the warped top/bottom images onto the corresponding synthetic side images to generate blended images for left and right eye views with the blended images. The image processing system creates the canvas view which has smooth transition between binocular views and monocular views in terms of image shape and color based on the blended images.

Claims

1. A method comprising:

receiving a primary image associated with a primary camera, the primary image occluded by an object in a first field of view of the primary camera, wherein the object is a support post of a camera system attached to the primary camera;

receiving a secondary image associated with a secondary camera, the secondary image occluded by the object in a second field of view of the secondary camera;

identifying a portion of the primary image occluded by the object;

filtering the primary image to exclude the portion of the primary image occluded by the object;

determining an optical flow from the secondary image to the filtered primary image; and

generating a combined image by filling the portion of the primary image occluded by the object with image data from the secondary image based on the optical flow, wherein the combined image simulates an image captured by the primary camera without the object occluding the first field of view.

2. The method of claim 1, further comprising:

identifying a portion of the secondary image occluded by the object;

filtering the secondary image to exclude the portion of the secondary image occluded by the object; and

wherein the optical flow from the secondary image to the primary image excludes the filtered portion of the secondary image.

3. The method of claim 1, further comprising:

identifying a blend area in the primary image adjacent to the filtered area excluding the portion of the primary image occluded by the object; and

wherein

generating the combined image comprises filling the blend area with color components from the primary image and color components from the secondary image according to a transition function.

4. The method of claim 3, wherein the transition function is a linear function.

5. The method of claim 3, wherein the transition function is based on the similarity of color components in the primary image and the secondary image in the blend area.

6. The method of claim 3, wherein the transition function is based on the optical flow.

7. The method of claim 1, wherein the object is in a different portion of the second field of view of the second camera relative to the first field of view of the primary camera.

8. A non-transitory computer-readable storage medium storing computer program instructions, the computer program instructions when executed by a computer processor causes the processor to perform steps of:

receiving a primary image associated with a primary camera, the primary image occluded by an object in a first field of view of the primary camera, wherein the object is a support post of a camera system attached to the primary camera;

receiving a secondary image associated with a secondary camera, the secondary image occluded by the object in a second field of view of the secondary camera;

identifying a portion of the primary image occluded by the object;

filtering the primary image to exclude the portion of the primary image occluded by the object;

determining an optical flow from the secondary image to the filtered primary image; and

generating a combined image by filling the portion of the primary image occluded by the object with image data from the secondary image based on the optical flow, wherein the combined image simulates an image captured by the primary camera without the object occluding the first field of view.

9. The non-transitory computer-readable storage medium of claim 8, further comprising:

identifying a portion of the secondary image occluded by the object;

filtering the secondary image to exclude the portion of the secondary image occluded by the object; and

wherein the optical flow from the secondary image to the primary image excludes the filtered portion of the secondary image.

10. The non-transitory computer-readable storage medium of claim 8, further comprising:

identifying a blend area in the primary image adjacent to the filtered area excluding the portion of the primary image occluded by the object; and

wherein

generating the combined image comprises filling the blend area with color components from the primary image and color components from the secondary image according to a transition function.

11. The non-transitory computer-readable storage medium of claim 10, wherein the transition function is a linear function.

12. The non-transitory computer-readable storage medium of claim 10, wherein the transition function is based on the similarity of color components in the primary image and the secondary image in the blend area.

13. The non-transitory computer-readable storage medium of claim 10, wherein the transition function is based on the optical flow.

14. The non-transitory computer-readable storage medium of claim 8, wherein the object is in a different portion of the second field of view of the second camera relative to the first field of view of the primary camera.