Real time calibration for multi-camera wireless device

• US 9,955,056 B2
• Filed: 03/16/2015
• Issued: 04/24/2018
• Est. Priority Date: 03/16/2015
• Status: Expired due to Fees

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Nagaraja, Adarsh Hosaagrahara, Liu, Weiliang, Liu, Shizhong
• Primary Examiner(s): Entezari, Michelle M
• Application Number:
  US14/659,547
• Publication Number:
  US 20160277650A1
• Time in Patent Office: 1,135 Days
• Field of Search: None
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: G06T 7/85 Stereo camera calibration
• CPC Class Codes: H04N 17/002 for television cameras
• CPC Class Codes: H04N 2101/00 Still video cameras
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/54 Mounting of pick-up tubes, electronic image sensors, deviation or focusing coils
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04N 23/81 for suppressing or minimising disturbance in the image signal generation

Abstract

A method for generating image calibration data at an electronic device having multiple cameras includes capturing a first image of a first scene at a first time using a first camera at the electronic device. The method also includes capturing a second image of a second scene at the first time using a second camera at the electronic device. A portion of the first scene overlaps a portion of the second scene. The method further includes generating the image calibration data based on the first image and based on the second image.

Claims

1. A method for generating image calibration data at an electronic device having multiple cameras, the method comprising:

capturing a first image of a first scene at a first time using a first camera at the electronic device;

capturing a second image of a second scene at the first time using a second camera at the electronic device, at least a portion of the first scene overlapping a portion of the second scene;

generating image calibration data based on the first image and based on the second image, the image calibration data used to align images captured by the first camera with images captured by the second camera;

detecting a first change in orientation of the first camera or a second change in orientation of the second camera, the first change in orientation detected in response to a temperature change and a detected force applied to the first camera, and the second change detected in response to the temperature change and a detected force applied to the second camera; and

in response to detecting the first change in orientation or in response to detecting the second change in orientation, regenerating image calibration data at the electronic device based on a third image of a third scene captured by the first camera at a second time and based on a fourth image of a fourth scene captured by the second camera at the second time, at least a portion of the third scene overlapping a portion of the fourth scene, the regenerated image calibration data used to align images captured by the first camera with images captured by the second camera.

2. The method of claim 1, wherein the image calibration data includes image alignment data.

3. The method of claim 1, wherein image calibration data is further regenerated upon user initiation of an image capture at the electronic device.

4. The method of claim 1, wherein image calibration data is further regenerated after a particular period of time.

5. The method of claim 1, wherein regenerating the image calibration data compensates for the change in orientation of the first camera.

6. The method of claim 1, wherein regenerating image calibration data compensates for the change in orientation of the second camera.

7. The method of claim 1, wherein generating the image calibration data includes estimating a homography between the first image and the second image, the homography used to align the first image with the second image.

8. The method of claim 7, wherein the homography is estimated using a Random Sample Consensus (RANSAC) algorithm.

9. The method of claim 1, wherein generating the image calibration data comprises:

down-sampling the first image to generate a first down-sampled image;

down-sampling the second image to generate a second down-sampled image; and

determining a homography to map a first feature point in the first down-sampled image to a second feature point in the second down-sampled image, wherein the first feature point and the second feature point correspond to the same location in the scene, and wherein the image calibration data includes the homography.

10. The method of claim 9, wherein the first feature point and the second feature point are detected using a Harris Corner detection algorithm, a Features from Accelerated Segment Test (FAST) Corner detection algorithm, a Scale-Invariant Feature Transform (SIFT) detection algorithm, a Speeded Up Robust Features (SURF) detection algorithm, or a FAST-9 detection algorithm.

11. The method of claim 9, wherein the first feature point and the second feature point are tracked using a Lucas-Kanade Optical Flow algorithm, a Pyramidal Lucas-Kanade algorithm, a Horn-Schunck algorithm, or a Black-Jepson algorithm.

12. The method of claim 9, further comprising:

adjusting the second down-sampled image based on the homography to generate a warped image; and

combining the first down-sampled image with the warped image to generate an aligned image.

13. The method of claim 12, wherein adjusting the second down-sampled image includes rotating the second down-sampled image, translating the second down-sampled image, or any combination thereof.

14. The method of claim 1, wherein the portion of the first scene and the portion of the second scene depict a common physical area.

15. An electronic device comprising:

a first camera configured to capture a first image of a first scene at a first time;

a second camera configured to capture a second image of a second scene at the first time, at least a portion of the first scene overlapping a portion of the second scene; and

a processor configured to;

generate image calibration data based on the first image and based on the second image, the image calibration data used to align images captured by the first camera with images captured by the second camera;

detect a first change in orientation of the first camera or a second change in orientation of the second camera, the first change in orientation detected in response to a temperature change and a detected force applied to the first camera, and the second change detected in response to the temperature change and a detected force applied to the second camera; and

in response to detecting the first change in orientation or in response to detecting the second change in orientation, regenerate image calibration data based on a third image of a third scene captured by the first camera at a second time and based on a fourth image of a fourth scene captured by the second camera at the second time, at least a portion of the third scene overlapping a portion of the fourth scene, the regenerated image calibration data used to align images captured by the first camera with images captured by the second camera.

16. The electronic device of claim 15, wherein the image calibration data includes image alignment data.

17. The electronic device of claim 15, wherein image calibration data is further regenerated upon user initiation of an image capture at the electronic device.

18. The electronic device of claim 15, wherein image calibration data is further regenerated after a particular period of time.

19. The electronic device of claim 15, wherein the processor is further configured to estimate a homography between the first image and the second image.

20. The electronic device of claim 19, wherein the homography is estimated using a Random Sample Consensus (RANSAC) algorithm.

21. The electronic device of claim 15, wherein the processor is configured to:

down-sample the first image to generate a first down-sampled image;

down-sample the second image to generate a second down-sampled image;

determine a homography to map a first feature point in the first down-sampled image to a second feature point in the second down-sampled image, wherein the first feature point and the second feature point correspond to the same location in the scene, and wherein the image calibration data includes the homography.

22. The electronic device of claim 21, wherein the first feature point and the second feature point are detected using a Harris Corner detection algorithm, a Features from Accelerated Segment Test (FAST) Corner detection algorithm, a Scale-Invariant Feature Transform (SIFT) detection algorithm, a Speeded Up Robust Features (SURF) detection algorithm, or a FAST-9 detection algorithm.

23. The electronic device of claim 21, wherein the first feature point and the second feature point are tracked using a Lucas-Kanade Optical Flow algorithm, a Pyramidal Lucas-Kanade algorithm, a Horn-Schunck algorithm, or a Black-Jepson algorithm.

24. The electronic device of claim 21, wherein the processor is configured to:

adjust the second down-sampled image based on the homography to generate a warped image; and

combine the first down-sampled image with the warped image to generate an aligned image.

25. The electronic device of claim 24, wherein adjusting the second down-sampled image includes rotating the second down-sampled image, translating the second down-sampled image, or any combination thereof.

26. A non-transitory computer readable medium comprising instructions for generating image calibration data at an electronic device having multiple cameras, the instructions, when executed by a processor, cause the processor to:

receive a first image of a first scene that is captured by a first camera at a first time;

receive a second image of a second scene that is captured by a second camera at the first time, at least a portion of the first scene overlapping a portion of the second scene;

generate the image calibration data based on the first image and based on the second image, the image calibration data used to align images captured by the first camera with images captured by the second camera;

detect a first change in orientation of the first camera or a second change in orientation of the second camera, the first change in orientation detected in response to a temperature change and a detected force applied to the first camera, and the second change detected in response to the temperature change and a detected force applied to the second camera; and

in response to detecting the first change in orientation or in response to detecting the second change in orientation, regenerate image calibration data at the electronic device based on a third image of a third scene captured by the first camera at a second time and based on a fourth image of a fourth scene captured by the second camera at the second time, at least a portion of the third scene overlapping a portion of the fourth scene, the regenerated image calibration data used to align images captured by the first camera with images captured by the second camera.

27. The non-transitory computer readable medium of claim 26, wherein image calibration data is further regenerated upon user initiation of an image capture at the electronic device.

28. An electronic device comprising:

means for capturing a first image of a first scene at a first time;

means for capturing a second image of a scene at the first time, at least a portion of the first scene overlapping a portion of the second scene;

means for detecting a first change in orientation of the means for capturing the first image or a second change in orientation of the means for capturing the second image, the first change in orientation detected in response to a temperature change and a detected force applied to the means for capturing the first image, and the second change detected in response to the temperature change and a detected force applied to the means for capturing the second image; and

means for generating image calibration data based on the first image and based on the second image, the image calibration data used to align images captured by the means for capturing the first image with images captured by the means for capturing the second image, wherein calibration data is regenerated in response to detecting the first change in orientation or in response to detecting the second change in orientation.

29. The electronic device of claim 28, wherein image calibration data is further regenerated upon user initiation of an image capture at the electronic device.