Transition between binocular and monocular views

• US 10,460,521 B2
• Filed: 02/21/2018
• Issued: 10/29/2019
• Est. Priority Date: 04/06/2016
• Status: Active Grant

• Current Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Original Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Inventors: Cabral, Brian Keith, Briggs, Forrest Samuel
• Primary Examiner(s): Pringle-Parker, Jason A
• Application Number:
  US15/900,970
• Publication Number:
  US 20180182175A1
• Time in Patent Office: 615 Days
• Field of Search: None
• CPC Class Codes: G06T 15/40 Hidden part removal
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 2207/20228 Disparity calculation for image-based rendering
• CPC Class Codes: G06T 2219/028 Multiple view windows (top-side-front-sagittal-orthogonal)
• CPC Class Codes: G06T 3/18 Image warping, e.g. rearranging pixels individually
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06V 10/147 Details of sensors, e.g. sensor lenses fingerprint or palmprint sensors G06V40/13; vascular sensors G06V40/145; eye sensors G06V40/19
• CPC Class Codes: G06V 10/16 using multiple overlapping images; Image stitching
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 2013/0088 Synthesising a monoscopic image signal from stereoscopic images, e.g. synthesising a panoramic or high resolution monoscopic image
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture

Abstract

An image processing system is designed to generate a canvas view that transitions between binocular views and monocular views. Initially, the image processing system receives top/bottom images and side images of a scene and calculates offsets to generate synthetic side images for left and right view of a user. To transition between binocular views and monocular views, the image processing system first warps top/bottom images onto corresponding synthetic side images to generate warped top/bottom images, which realizes the transition in terms of shape. The image processing system then morphs the warped top/bottom images onto the corresponding synthetic side images to generate blended images for left and right eye views with the blended images. The image processing system creates the canvas view which transitions between binocular views and monocular views in terms of image shape and color based on the blended images.

Claims

1. A method comprising:

identifying a horizontal angle and vertical angle of view for a portion of a display for an eye, the horizontal angle and vertical angle representing a portion of a canvas for view by the eye;

identifying an overlapping portion of a top image and an image corresponding to a view for the eye, the images being combined to generate the canvas for view by the eye;

determining an optical flow for the overlapping portion; and

applying the optical flow to the top image as a function of the vertical angle, wherein a portion of the optical flow applied to the top image increases from a vertical angle when the overlapping portion begins adjacent to the top image to a pivotal row in the overlapping portion.

2. The method of claim 1, further comprising:

for each of a left eye image and a right eye image;

morphing a color of the top image with a corresponding image between the pivotal row and a side-only portion of the corresponding image where the overlapping portion is adjacent to the side-only portion.

3. The method of claim 2, wherein the color morphing ranges from zero to complete color morphing of a side-only image color between the pivotal row and the overlapping portion adjacent to the side-only portion.

4. The method of claim 1, wherein a left eye view and a right eye view are separated by a distance as a function of an interpupillary distance (IPD) between the left eye view and the right eye view.

5. The method of claim 4, wherein the IPD is a function of the vertical angle of the scene.

6. The method of claim 5, wherein at a vertical angle of π

/2, the IPD is a maximum IPD, and at a vertical angle of π

, the IPD is zero.

7. The method of claim 1, further comprising calculating a shift of the image to generate a portion of the image that simulates the eye view at a point where a camera is located when the eye is looking at a specific image in a scene.

8. The method of claim 1, wherein the optical flow is calculated from a secondary view to a primary view and wherein the secondary view is combined with the primary view using the optical flow.

9. A non-transitory computer-readable storage medium storing computer program instructions, the computer program instructions when executed by a computer processor causes the processor to perform steps of:

identifying a horizontal angle and vertical angle of view for a portion of a display for an eye, the horizontal angle and vertical angle representing a portion of a canvas for view by the eye;

identifying an overlapping portion of a top image and an image corresponding to a view for the eye, the images being combined to generate the canvas for view by the eye;

determining an optical flow for the overlapping portion; and

applying the optical flow to the top image as a function of the vertical angle, wherein a portion of the optical flow applied to the top image increases from a vertical angle when the overlapping portion begins adjacent to the top image to a pivotal row in the overlapping portion.

10. The non-transitory computer-readable storage medium of claim 9, the instructions, when executed, further causing the processor to:

for each of a left and a right eye images;

morph a color of the top image with a corresponding image between the pivotal row and a side-only portion of the corresponding image where the overlapping portion is adjacent to the side-only portion.

11. The non-transitory computer-readable storage medium of claim 10, wherein the color morphing ranges from zero to complete color morphing of a side-only image color between the pivotal row and the overlapping portion adjacent to the side-only portion.

12. The non-transitory computer-readable storage medium of claim 9, wherein a left eye view and a right eye view are separated by a distance as a function of an interpupillary distance (IPD) between the left eye view and the right eye view.

13. The non-transitory computer-readable storage medium of claim 12, wherein the IPD is a function of the vertical angle of the scene.

14. The non-transitory computer-readable storage medium of claim 13, wherein at a vertical angle of π

/2, the IPD is a maximum IPD, and at a vertical angle of π

, the IPD is zero.

15. The non-transitory computer-readable storage medium of claim 9, the instructions, when executed, further causing the processor to calculate a shift of the image to generate a portion of the image that simulates the eye view at a point where a camera is located when the eye is looking at a specific image in a scene.

16. The non-transitory computer-readable storage medium of claim 9, wherein the optical flow is calculated from a secondary view to a primary view and wherein the secondary view is combined with the primary view using the optical flow.