SENSOR ARRAY MODULE WITH WIDE ANGLE, AND IMAGE CALIBRATION METHOD, OPERATION METHOD AND APPLICATION FOR THE SAME

• US 20100020201A1
• Filed: 07/21/2009
• Published: 01/28/2010
• Est. Priority Date: 07/23/2008
• Status: Active Grant

• Current Assignee: PixArt Imaging Incorporated
• Original Assignee: PixArt Imaging Incorporated
• Inventors: LIN, Chih Hsin, GU, Ren Hau, TSAI, Chang Che, CHAO, Tzu Yi, CHEN, Hsin Chia
• Granted Patent: US 8,384,789 B2
• Time in Patent Office: Days
• US Class Current: 348/239
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: H04N 17/002 for television cameras
• CPC Class Codes: H04N 23/13 with multiple sensors
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 5/2624 for obtaining an image which is composed of whole input images, e.g. splitscreen

Abstract

A sensor array module with wide angle for creating a composite image includes a first image sensor, a second image sensor, a storage unit and a processor. The first image sensor is for capturing a first image. The second image sensor is for capturing a second image and has a relative spatial relationship with respective to the first image sensor. The storage unit stores at least one transformation matrix obtained according to the relative spatial relationship between the first and the second image sensors. The processor combines the first image and the second image by using the transformation matrix to create the composite image. The present invention also provides an image calibration method, an operation method and an application for a sensor array module with wide angle.

Claims

1. A sensor array module with wide angle for creating a composite image, the sensor array module comprising:

a first image sensor for capturing a first image;

a second image sensor for capturing a second image and having a relative spatial relationship with respect to the first image sensor;

a storage unit storing at least one transformation matrix, wherein the transformation matrix is obtained according to the relative spatial relationship between the first image sensor and the second image sensor; and

a processor combining the first image and the second image by using the transformation matrix to create the composite image.

2. The sensor array module as claimed in claim 1, wherein the relative spatial relationship is a transverse angle between the first image sensor and the second image sensor.

3. The sensor array module as claimed in claim 2, wherein the transverse angle is 2Φ

and the transformation matrix is

$\left[\begin{array}{ccc}\cos <br><br>\left(-\mathrm{\Phi <br><br>}\right)& 0& -\sin <br><br>\left(-\mathrm{\Phi <br><br>}\right)\\ 0& 1& 0\\ \sin <br><br>\left(-\mathrm{\Phi <br><br>}\right)& 0& \cos <br><br>\left(-\mathrm{\Phi <br><br>}\right)\end{array}\right]<br><br><br><br>\mathrm{and}/\mathrm{or}<br><br>\left[\begin{array}{ccc}\cos <br><br>\left(\mathrm{\Phi <br><br>}\right)& 0& -\sin <br><br>\left(\mathrm{\Phi <br><br>}\right)\\ 0& 1& 0\\ \sin <br><br>\left(\mathrm{\Phi <br><br>}\right)& 0& \cos <br><br>\left(\mathrm{\Phi <br><br>}\right)\end{array}\right].$

4. The sensor array module as claimed in claim 1, wherein the relative spatial relationship is a transverse angle and a longitudinal angle between the first image sensor and the second image sensor.

5. The sensor array module as claimed in claim 4, wherein the transverse angle is 2Φ

, the longitudinal angle is θ

, and the transformation matrix is

$\left[\begin{array}{ccc}\cos <br><br>\left(-\mathrm{\Phi <br><br>}\right)& -\sin <br><br>\left(-\mathrm{\Phi <br><br>}\right)<br><br>\sin <br><br>\left(\mathrm{\theta <br><br>}\right)& -\sin <br><br>\left(-\mathrm{\Phi <br><br>}\right)<br><br>\cos <br><br><br><br>\left(\mathrm{\theta <br><br>}\right)\\ 0& \cos <br><br>\left(\mathrm{\theta <br><br>}\right)& -\sin <br><br>\left(\mathrm{\theta <br><br>}\right)\\ \sin <br><br>\left(-\mathrm{\Phi <br><br>}\right)& \cos <br><br>\left(-\mathrm{\Phi <br><br>}\right)<br><br>\sin <br><br>\left(\mathrm{\theta <br><br>}\right)& \cos <br><br>\left(-\mathrm{\Phi <br><br>}\right)<br><br>\cos <br><br>\left(\mathrm{\theta <br><br>}\right)\end{array}\right]<br><br><br><br>\mathrm{and}/\mathrm{or}<br><br>\left[\begin{array}{ccc}\cos <br><br>\left(\mathrm{\Phi <br><br>}\right)& -\sin <br><br>\left(\mathrm{\Phi <br><br>}\right)<br><br>\sin <br><br>\left(\mathrm{\theta <br><br>}\right)& -\sin <br><br>\left(\mathrm{\Phi <br><br>}\right)<br><br>\cos <br><br>\left(\mathrm{\theta <br><br>}\right)\\ 0& \cos <br><br>\left(\mathrm{\theta <br><br>}\right)& -\sin <br><br>\left(\mathrm{\theta <br><br>}\right)\\ \sin <br><br>\left(\mathrm{\Phi <br><br>}\right)& \cos <br><br>\left(\mathrm{\Phi <br><br>}\right)<br><br>\sin <br><br>\left(\mathrm{\theta <br><br>}\right)& \cos <br><br>\left(\mathrm{\Phi <br><br>}\right)<br><br>\cos <br><br>\left(\mathrm{\theta <br><br>}\right)\end{array}\right].$

6. The sensor array module as claimed in claim 1, further comprising a frame for mounting the first and the second image sensors, such that the relative spatial relationship is defined between the first image sensor and the second image sensor by the frame.

7. The sensor array module as claimed in claim 1, further comprising a third image sensor having a first relative spatial relationship with respect to the first image sensor and a fourth image sensor having a second relative spatial relationship with respect to the first image sensor, wherein a first transformation matrix obtained according to the first relative spatial relationship and a second transformation matrix obtained according to the second relative spatial relationship are stored in the storage unit and the processor combines images captured by the first, second, third and fourth image sensors by using the transformation matrix, the first transformation matrix and second transformation matrix to create the composite image.

8. An image calibration method for a sensor array module with wide angle, the sensor array module comprising a first image sensor and a second image sensor, the image calibration method comprising:

providing at least three reference points, wherein the reference points define a reference coordinate system and each reference point has a reference coordinate;

capturing a first image including at least three reference points with the first image sensor, wherein the first image defines a first image coordinate system and each reference point included in the first image has a first image coordinate;

obtaining a first transformation matrix according to the reference coordinate system and the first image coordinate system;

capturing a second image including at least three reference points with the second image sensor, wherein the second image defines a second image coordinate system and each reference point included in the second image has a second image coordinate;

obtaining a second transformation matrix according to the reference coordinate system and the second image coordinate system; and

transforming the first image and/or the second image to a virtual plane by using the first transformation matrix and/or the second transformation matrix.

9. The image calibration method as claimed in claim 8, wherein the reference coordinate system, the first image coordinate system and the second image coordinate system are three-dimensional Cartesian coordinate systems.

10. The image calibration method as claimed in claim 8, wherein the virtual plane is based on one of the reference coordinate system, the first image coordinate system and the second image coordinate system.

11. The image calibration method as claimed in claim 10, wherein when the virtual plane is based on the reference coordinate system, transform the first image to the virtual plane by using the first transformation matrix and transform the second image to the virtual plane by using the second transformation matrix.

12. The image calibration method as claimed in claim 10, wherein when the virtual plane is based on the first image coordinate system, transform the second image to the virtual plane by using the first and the second transformation matrices.

13. The image calibration method as claimed in claim 10, wherein when the virtual plane is based on the second image coordinate system, transform the first image to the virtual plane by using the first and the second transformation matrices.

14. An image calibration method for a sensor array module with wide angle, the sensor array module comprising a first image sensor and a second image sensor, the image calibration method comprising:

determining a relative spatial relationship between the first image sensor and the second image sensor;

obtaining at least one transformation matrix according to the relative spatial relationship; and

transforming images captured by the first image sensor and/or images captured by the second image sensor to a virtual plane by using the transformation matrix.

15. The image calibration method as claimed in claim 14, wherein the relative spatial relationship is a transverse angle and/or a longitudinal angle between the first image sensor and the second image sensor.

16. The image calibration method as claimed in claim 14, wherein the virtual plane is based on a coordinate system defined by the first image sensor or a coordinate system defined by the second image sensor.

17. The image calibration method as claimed in claim 14, wherein a field of view of the first image sensor is overlapped or non-overlapped with a field of view of the second image sensor.

18. An operation method for a sensor array module with wide angle, the sensor array module comprising a first image sensor, a second image sensor and a storage unit storing at least one transformation matrix obtained according to a relative spatial relationship between the first image sensor and the second image sensor, the operation method comprising:

capturing a third image with the first image sensor;

capturing a fourth image with the second image sensor;

transforming the third image and/or the fourth image to a virtual plane by using the transformation matrix; and

capturing at least one characteristic parameter of object images in the virtual plane.

19. The operation method as claimed in claim 18, wherein the characteristic parameter is selected from a group consisting of coordinates, areas, colors, directivities, boundaries, numbers of vertices, and aspect ratios of the object images.

20. The operation method as claimed in claim 18, wherein the relative spatial relationship between the first image sensor and the second image sensor is a transverse angle and/or a longitudinal angle between the first image sensor and the second image sensor.

21. An operation method for a sensor array module with wide angle, the sensor array module comprising a first image sensor, a second image sensor and a storage unit storing at least one transformation matrix obtained according to a relative spatial relationship between the first image sensor and the second image sensor, the operation method comprising:

capturing a third image with the first image sensor;

generating a first characteristic parameter according to the third image;

capturing a fourth image with the second image sensor;

generating a second characteristic parameter according to the fourth image; and

transforming the first characteristic parameter and the second characteristic parameter to a virtual plane by using the transformation matrix whereby combining the first characteristic parameter and the second characteristic parameter.

22. The operation method as claimed in claim 21, wherein the first and the second characteristic parameters are selected from a group consisting of coordinates, areas, colors, directivities, boundaries, numbers of vertices, and aspect ratios of the object images.

23. The operation method as claimed in claim 21, wherein the relative spatial relationship between the first image sensor and the second sensor is a transverse angle and/or a longitudinal angle between the first image sensor and the second image sensor.

24. A pointer positioning system, comprising:

at least one light source;

a sensor array module with wide angle for creating a composite image with a composite field of view, the sensor array module comprising;

a first image sensor for capturing a first image;

a second image sensor for capturing a second image and having a relative spatial relationship with respect to the first image sensor; and

an image processing unit storing a transformation matrix and combining the first image and the second image to create the composite image by using the transformation matrix, wherein the first image and/or the second image include an image of the light source and the transformation matrix is obtained according to the relative spatial relationship between the first image sensor and the second image sensor, and the image processing unit generating a characteristic parameter based on the composite image and the image of the light source; and

an image display device coupled to the sensor array module for displaying a picture generated from an application, and controlling the application and updating the picture based on the characteristic parameter.

25. The pointer positioning system as claimed in claim 24, wherein the light source is a sight of a light gun or a pointing device.

26. A pointer positioning system, comprising:

at least one light source;

a sensor array module with wide angle for creating composite characteristic parameters with a composite field of view, the sensor array module comprising;

a first image sensor for capturing a third image;

a second image sensor for capturing a fourth image and having a relative spatial relationship with respect to the first image sensor; and

an image processing unit generating a first characteristic parameter from the third image and generating a second characteristic parameter from the fourth image, and creating the composite characteristic parameters from the first characteristic parameter and the second characteristic parameter by using a transformation matrix, wherein at least one of the first characteristic parameter and the second characteristic parameter is determined from an image of the light source in the third image or the fourth image, and wherein the transformation matrix is obtained according to the relative spatial relationship between the first image sensor and the second image sensor; and

an image display device coupled to the sensor array module for displaying a picture generated from an application, and controlling the application and updating the picture based on the composite characteristic parameters.

27. The pointer positioning system as claimed in claim 26, wherein the light source is a sight of a light gun or a pointing device.