REAL TIME CALIBRATION FOR MULTI-CAMERA WIRELESS DEVICE

• US 20160277650A1
• Filed: 03/16/2015
• Published: 09/22/2016
• Est. Priority Date: 03/16/2015
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Nagaraja, Adarsh Hosaagrahara, Liu, Weiliang, Liu, Shizhong
• Granted Patent: US 9,955,056 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: G06T 7/85 Stereo camera calibration
• CPC Class Codes: H04N 17/002 for television cameras
• CPC Class Codes: H04N 2101/00 Still video cameras
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/54 Mounting of pick-up tubes, electronic image sensors, deviation or focusing coils
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04N 23/81 for suppressing or minimising disturbance in the image signal generation

Abstract

A method for generating image calibration data at an electronic device having multiple cameras includes capturing a first image of a first scene at a first time using a first camera at the electronic device. The method also includes capturing a second image of a second scene at the first time using a second camera at the electronic device. A portion of the first scene overlaps a portion of the second scene. The method further includes generating the image calibration data based on the first image and based on the second image.

Claims

1. A method for generating image calibration data at an electronic device having multiple cameras, the method comprising:

capturing a first image of a first scene at a first time using a first camera at the electronic device;

capturing a second image of a second scene at the first time using a second camera at the electronic device, at least a portion of the first scene overlapping a portion of the second scene;

generating image calibration data based on the first image and based on the second image; and

regenerating image calibration data at the electronic device based on a third image of a third scene captured by the first camera at a second time and based on a fourth image of a fourth scene captured by the second camera at the second time, at least a portion of the third scene overlapping a portion of the fourth scene.

2. The method of claim 1, wherein the image calibration data is usable for an image modification operation applied to an output of the second camera.

3. The method of claim 2, wherein the image modification operation includes at least one of an image rectification operation, an image alignment operation, or an image warping operation to enable additional image processing.

4. The method of claim 3, wherein the additional image processing includes depth map estimation for three-dimensional image processing or image combination.

5. The method of claim 1, wherein the image calibration data includes image alignment data.

6. The method of claim 1, wherein image calibration data is regenerated upon user initiation of an image capture at the electronic device.

7. The method of claim 1, wherein image calibration data is regenerated after a particular period of time.

8. The method of claim 1, further comprising detecting a change in orientation of the first camera or a change in orientation of the second camera, wherein image calibration data is regenerated in response to detecting the change in orientation of the first camera or in response to detecting the change in orientation of the second camera.

9. The method of claim 1, wherein regenerating image calibration data compensates for a change in orientation of the first camera, a change in orientation of the second camera, or any combination thereof.

10. The method of claim 9, wherein gravitational forces applied to the electronic device, temperature changes at the electronic device, physical damage to the electronic device, or any combination thereof, causes the change in orientation of the first camera or the change in orientation of the second camera.

11. The method of claim 1, wherein generating the image calibration data includes estimating a homography between the first image and the second image, the homography used to align the first image with the second image.

12. The method of claim 11, wherein the homography is estimated using a Random Sample Consensus (RANSAC) algorithm.

13. The method of claim 1, wherein generating the image calibration data comprises:

down-sampling the first image to generate a first down-sampled image;

down-sampling the second image to generate a second down-sampled image;

determining a homography to map a first feature point in the first down-sampled image to a second feature point in the second down-sampled image, wherein the first feature point and the second feature point correspond to the same location in the scene, and wherein the image calibration data includes the homography.

14. The method of claim 13, wherein the first feature point and the second feature point are detected using a Harris Corner detection algorithm, a Features from Accelerated Segment Test (FAST) Corner detection algorithm, a Scale-Invariant Feature Transform (SIFT) detection algorithm, a Speeded Up Robust Features (SURF) detection algorithm, or a FAST-9 detection algorithm.

15. The method of claim 13, wherein the first feature point and the second feature point are tracked using a Lucas-Kanade Optical Flow algorithm, a Pyramidal Lucas-Kanade algorithm, a Horn-Schunck algorithm, or a Black-Jepson algorithm.

16. The method of claim 13, further comprising:

adjusting the second down-sampled image based on the homography to generate a warped image; and

combining the first down-sampled image with the warped image to generate an aligned image.

17. The method of claim 16, wherein adjusting the second down-sampled image includes rotating the second down-sampled image, translating the second down-sampled image, or any combination thereof.

18. The method of claim 1, wherein the portion of the first scene and the portion of the second scene depict a common physical area.

19. An electronic device comprising:

a first camera configured to capture a first image of a first scene at a first time;

a second camera configured to capture a second image of a second scene at the first time, at least a portion of the first scene overlapping a portion of the second scene; and

a processor configured to;

generate image calibration data based on the first image and based on the second image; and

regenerate image calibration data based on a third image of a third scene captured by the first camera at a second time and based on a fourth image of a fourth scene captured by the second camera at the second time, at least a portion of the third scene overlapping a portion of the fourth scene.

20. The electronic device of claim 19, wherein the image calibration data is usable for an image modification operation applied to an output of the second camera.

21. The electronic device of claim 20, wherein the image modification operation includes at least one of an image rectification operation, an image alignment operation, or an image warping operation to enable additional image processing.

22. The electronic device of claim 19, wherein image calibration data is regenerated after a particular period of time.

23. The electronic device of claim 19, wherein the processor is further configured to estimate a homography between the first image and the second image.

24. The electronic device of claim 19, wherein the processor is configured to:

down-sample the first image to generate a first down-sampled image;

down-sample the second image to generate a second down-sampled image;

determine a homography to map a first feature point in the first down-sampled image to a second feature point in the second down-sampled image, wherein the first feature point and the second feature point correspond to the same location in the scene, and wherein the image calibration data includes the homography.

25. The electronic device of claim 24, wherein the processor is configured to:

adjust the second down-sampled image based on the homography to generate a warped image; and

combine the first down-sampled image with the warped image to generate an aligned image.

26. A non-transitory computer readable medium comprising instructions for generating image calibration data at an electronic device having multiple cameras, the instructions, when executed by a processor, cause the processor to:

receive a first image of a first scene that is captured by a first camera at a first time;

receive a second image of a second scene that is captured by a second camera at the first time, at least a portion of the first scene overlapping a portion of the second scene;

generate the image calibration data based on the first image and based on the second image; and

regenerate image calibration data at the electronic device based on a third image of a third scene captured by the first camera at a second time and based on a fourth image of a fourth scene captured by the second camera at the second time, at least a portion of the third scene overlapping a portion of the fourth scene.

27. The non-transitory computer readable medium of claim 26, wherein the image calibration data is usable for an image modification operation applied to an output of the second camera.

28. The non-transitory computer readable medium of claim 26, wherein image calibration data is regenerated upon user initiation of an image capture at the electronic device.

29. An electronic device comprising:

means for capturing a first image of a first scene at a first time;

means for capturing a second image of a scene at the first time, at least a portion of the first scene overlapping a portion of the second scene; and

means for generating image calibration data based on the first image and based on the second image.

30. The electronic device of claim 29, wherein the image calibration data is usable for an image modification operation applied to an output of the second camera.