GENERATING STEREOSCOPIC PAIRS OF IMAGES FROM A SINGLE LENS CAMERA

• US 20170126988A1
• Filed: 01/17/2017
• Published: 05/04/2017
• Est. Priority Date: 10/31/2014
• Status: Active Grant

• Current Assignee: Fyusion, Inc. (Cox Enterprises Incorporated)
• Original Assignee: Fyusion, Inc. (Cox Enterprises Incorporated)
• Inventors: HOLZER, Stefan Johannes Josef, Rusu, Radu Bogdan, Ren, Yuheng
• Granted Patent: US 10,650,574 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/532 Query formulation, e.g. graphical querying
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/738 Presentation of query results
• CPC Class Codes: G06F 16/783 using metadata automatically derived from the content
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 2215/12 Shadow map, environment map
• CPC Class Codes: G06T 2215/16 Using real world measurements to influence rendering
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06T 7/337 involving reference images or patches
• CPC Class Codes: G06T 7/579 from motion
• CPC Class Codes: H04N 13/279 the virtual viewpoint locations being selected by the viewers or determined by tracking
• CPC Class Codes: H04N 13/282 for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 5/265 Mixing

Abstract

Various embodiments of the present disclosure relate generally to systems and processes for generating stereo pairs for virtual reality. According to particular embodiments, a method comprises obtaining a monocular sequence of images using the single lens camera during a capture mode. The sequence of images is captured along a camera translation. Each image in the sequence of images contains at least a portion of overlapping subject matter, which includes an object. The method further comprises generating stereo pairs, for one or more points along the camera translation, for virtual reality using the sequence of images. Generating the stereo pairs may include: selecting frames for each stereo pair based on a spatial baseline; interpolating virtual images in between captured images in the sequence of images; correcting selected frames by rotating the images; and rendering the selected frames by assigning each image in the selected frames to left and right eyes.

Claims

1. A method for generating stereo pairs for virtual reality using a single lens camera, the method including:

obtaining a monocular sequence of images using the single lens camera during a capture mode, the sequence of images being capture along a camera translation, wherein each image in the sequence of images contains at least a portion of overlapping subject matter, the subject matter including an object; and

generating stereo pairs for virtual reality using the sequence of images, the stereo pairs being generated for one or more points along the camera translation, wherein generating stereo pairs includes;

selecting frames for each stereo pair based on a spatial baseline;

interpolating virtual images in between captured images in the sequence of images;

correcting selected frames by rotating the images; and

rendering the selected frames by assigning each image in the selected frames to left and right eyes.

2. The method of claim 1, wherein each image in the sequence of images is captured at a predetermined distance along the camera translation.

3. The method of claim 1, wherein the camera captures images while traveling along an arc path with points t₁ through t₁₀₀ evenly distributed along the arc path.

4. The method of claim 3,

wherein the sequence of images are obtained as the camera travels along a uniform camera translation, and

wherein each stereo pair includes a first image and a second image located at a predetermined point from the first image.

5. The method of claim 1, wherein a stereo pair includes an interpolated virtual image as one of the selected frames.

6. The method of claim 1, wherein correcting the selected frames includes rotating the image of a selected frame such that the image is angled toward the object at a desired angle.

7. The method of claim 1, wherein the sequence of images are fused to generate a surround view of the object, the surround view depicting a three-dimensional view of the object without actually generating a three-dimensional model of the object.

8. A system for generating stereo pairs for virtual reality, the system comprising:

a single lens camera,

a processor, and

memory storing one or more programs configured for execution by the processor, the one or more programs comprising instructions for;

obtaining a monocular sequence of images using the single lens camera during a capture mode, the sequence of images being capture along a camera translation, wherein each image in the sequence of images contains at least a portion of overlapping subject matter, the subject matter including an object; and

generating stereo pairs for virtual reality using the sequence of images, the stereo pairs being generated for one or more points along the camera translation, wherein generating stereo pairs includes;

selecting frames for each stereo pair based on a spatial baseline;

interpolating virtual images in between captured images in the sequence of images;

correcting selected frames by rotating the images; and

rendering the selected frames by assigning each image in the selected frames to left and right eyes.

9. The system of claim 8, wherein each image in the sequence of images is captured at a predetermined distance along the camera translation.

10. The system of claim 8, wherein the camera captures images while traveling along an arc path with points t₁ through t₁₀₀ evenly distributed along the arc path.

11. The system of claim 10,

wherein the sequence of images are obtained as the camera travels along a uniform camera translation, and

wherein each stereo pair includes a first image and a second image located at a predetermined point from the first image.

12. The system of claim 8, wherein a stereo pair includes an interpolated virtual image as one of the selected frames.

13. The system of claim 8, wherein correcting the selected frames includes rotating the image of a selected frame such that the image is angled toward the object at a desired angle.

14. The system of claim 8, wherein the sequence of images are fused to generate a surround view of the object, the surround view depicting a three-dimensional view of the object without actually generating a three-dimensional model of the object.

15. A non-transitory computer readable medium storing one or more programs configured for execution by a computer, the one or more programs comprising instructions for:

obtaining a monocular sequence of images using the single lens camera during a capture mode, the sequence of images being capture along a camera translation, wherein each image in the sequence of images contains at least a portion of overlapping subject matter, the subject matter including an object; and

generating stereo pairs for virtual reality using the sequence of images, the stereo pairs being generated for one or more points along the camera translation, wherein generating stereo pairs includes;

selecting frames for each stereo pair based on a spatial baseline;

interpolating virtual images in between captured images in the sequence of images;

correcting selected frames by rotating the images; and

rendering the selected frames by assigning each image in the selected frames to left and right eyes.

16. The non-transitory computer readable medium of claim 15, wherein each image in the sequence of images is captured at a predetermined distance along the camera translation.

17. The non-transitory computer readable medium of claim 15, wherein the camera captures images while traveling along an arc path with points t₁ through t₁₀₀ evenly distributed along the arc path.

18. The non-transitory computer readable medium of claim 15, wherein a stereo pair includes an interpolated virtual image as one of the selected frames.

19. The non-transitory computer readable medium of claim 15, wherein correcting the selected frames includes rotating the image of a selected frame such that the image is angled toward the object at a desired angle.

20. The non-transitory computer readable medium of claim 15, wherein the sequence of images are fused to generate a surround view of the object, the surround view depicting a three-dimensional view of the object without actually generating a three-dimensional model of the object.