IMAGING SYSTEMS AND METHODS

• US 20160119541A1
• Filed: 10/23/2015
• Published: 04/28/2016
• Est. Priority Date: 10/24/2014
• Status: Active Grant

• Current Assignee: Bounce Imaging Inc.
• Original Assignee: Bounce Imaging Inc.
• Inventors: Alvarado-Moya, Pablo, Aguilar, Francisco, Cabrera, Laura, Dijkstra, Sietse
• Granted Patent: US 10,091,418 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06T 3/047 Fisheye or wide-angle transformations
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/261 with monoscopic-to-stereoscopic image conversion
• CPC Class Codes: H04N 21/43637 involving a wireless protocol, e.g. Bluetooth, RF or wireless LAN [IEEE 802.11] arrangements for wireless networking or broadcasting of information in indoor or near-field type systems H04B10/114
• CPC Class Codes: H04N 23/23 from thermal infrared radiation
• CPC Class Codes: H04N 23/51 Housings
• CPC Class Codes: H04N 23/56 provided with illuminating means
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Abstract

At least one combined image may be created from a plurality of images captured by a plurality of cameras. A sensor unit may receive the plurality of images from the plurality of cameras. At least one processor in communication with the sensor unit may correlate each received image with calibration data for the camera from which the image was received. The calibration data may comprise camera position data and characteristic data. The processor may combine at least two of the received images from at least two of the cameras into the at least one combined image by orienting the at least two images relative to one another based on the calibration data for the at least two cameras from which the images were received and merging the at least two aligned images into the at least one combined image.

Claims

1. A system for creating at least one combined image from a plurality of images captured by a plurality of cameras, comprising:

a sensor unit configured to receive the plurality of images from the plurality of cameras; and

at least one processor in communication with the sensor unit and configured to;

correlate each received image with predetermined calibration data for the camera from which the image was received, the calibration data comprising camera position data and characteristic data; and

combine at least two of the received images from at least two of the cameras into the at least one combined image by;

orienting the at least two images relative to one another based on the calibration data for the at least two cameras from which the images were received; and

merging the at least two aligned images into the at least one combined image.

2. The system of claim 1, wherein orienting the at least two images comprises:

correlating a spatial location of at least one pixel in each of the at least two images to a spatial location relative to the camera from which the image was received, based on the calibration data; and

orienting at least one pixel in one of the at least two images to at least one pixel in at least one other of the at least two images.

3. The system of claim 1, wherein the sensor unit comprises a throwable ball.

4. The system of claim 1, wherein at least one of the cameras comprises a fisheye lens.

5. The system of claim 1, wherein at least one of the cameras comprises a sensor and a lens.

6. The system of claim 1, wherein the characteristic data for each camera comprises data defining optical characteristics of the camera.

7. The system of claim 1, wherein the camera position data for each camera comprises data defining a relative position of the camera in relation to the other cameras.

8. The system of claim 1, further comprising a receiver unit in communication with the processor and configured to display the at least one combined image.

9. The system of claim 8, wherein the receiver unit comprises a smartphone, a tablet, a personal computer, a server, or a combination thereof.

10. The system of claim 8, wherein the receiver unit is wirelessly coupled to the sensor unit.

11. The system of claim 1, wherein the processor is wirelessly coupled to the sensor unit.

12. The system of claim 1, wherein the at least one combined image is a video frame.

13. The system of claim 1, wherein orienting the at least two images comprises correcting for a parallax effect in at least one of the images, determining a scaling factor for each of the at least two images, relating pixel coordinate systems between the at least two images, estimating a global alignment between the at least two images, detecting a common distinctive feature in the at least two images, computing a globally consistent set of alignments for the at least two images, selecting a final compositing surface and a parameterization for the at least two images, or a combination thereof.

14. The system of claim 1, further comprising a calibration unit in communication with the sensor unit and configured to generate the calibration data.

15. The system of claim 14, wherein generating the calibration data comprises:

performing an intrinsic calibration for each of the plurality of cameras; and

performing an extrinsic calibration for the plurality of cameras.

16. The system of claim 15, wherein the calibration unit is configured to perform the intrinsic calibration for each of the plurality of cameras by:

positioning the camera in a plurality of locations relative to a calibration object;

receiving an image generated by the camera at each of the plurality of locations; and

varying at least one intrinsic parameter based on the images generated by the camera at each of the plurality of locations until an optimum is found.

17. The system of claim 15, wherein the calibration unit is configured to perform the intrinsic calibration for each of the plurality of cameras by:

creating a projection model for the camera;

determining a ray originating at a coordinate system of the camera that contains a corresponding point in space;

performing a great circle fitting;

defining an objective function for the camera;

performing an optimization;

or

a combination thereof.

18. The system of claim 15, wherein the calibration unit is configured to perform the extrinsic calibration for the plurality of cameras by:

positioning each camera in a plurality of locations relative to a calibration object;

receiving an image generated by each camera at each of the plurality of locations;

for each camera, finding a rotation and translation between a coordinate system of the camera and a reference coordinate system based on the images generated by the camera at each of the plurality of locations; and

deriving camera positions and orientations relative to each other based on the rotations and translations.

19. The system of claim 15, wherein the calibration unit is configured to perform the extrinsic calibration for each of the plurality of cameras by:

performing an optimization;

calibrating the camera using a plurality of images of a calibration object;

creating a translation vector for the camera;

relating the camera to each of the other cameras;

or

a combination thereof.

20. The system of claim 1, wherein the calibration data is generated prior to the sensor unit receiving the plurality of images.

21. A method for creating at least one combined image from a plurality of images captured by a plurality of cameras, comprising:

receiving, at a sensor unit, the plurality of images from the plurality of cameras;

correlating, with at least one processor in communication with the sensor unit, each received image with predetermined calibration data for the camera from which the image was received, the calibration data comprising camera position data and characteristic data; and

combining, with the at least one processor, at least two of the received images from at least two of the cameras into the at least one combined image by;

orienting the at least two images relative to one another based on the calibration data for the at least two cameras from which the images were received; and

merging the at least two aligned images into the at least one combined image.

22. The method of claim 21, wherein orienting the at least two images comprises:

correlating a spatial location of at least one pixel in each of the at least two images to a spatial location relative to the camera from which the image was received, based on the calibration data; and

orienting at least one pixel in one of the at least two images to at least one pixel in at least one other of the at least two images.

23. The method of claim 21, wherein the sensor unit comprises a throwable ball.

24. The method of claim 21, wherein at least one of the cameras comprises a fisheye lens.

25. The method of claim 21, wherein at least one of the cameras comprises a sensor and a lens.

26. The method of claim 21, wherein the characteristic data for each camera comprises data defining optical characteristics of the camera.

27. The method of claim 21, wherein the camera position data for each camera comprises data defining a relative position of the camera in relation to the other cameras.

28. The method of claim 21, further comprising displaying, with a receiver unit in communication with the processor, the at least one combined image.

29. The method of claim 28, wherein the receiver unit comprises a smartphone, a tablet, a personal computer, a server, or a combination thereof.

30. The method of claim 21, wherein the receiver unit is wirelessly coupled to the sensor unit.

31. The method of claim 21, wherein the processor is wirelessly coupled to the sensor unit.

32. The method of claim 21, wherein the at least one combined image is a video frame.

33. The method of claim 21, wherein orienting the at least two images comprises correcting for a parallax effect in at least one of the images, determining a scaling factor for each of the at least two images, relating pixel coordinate systems between the at least two images, estimating a global alignment between the at least two images, detecting a common distinctive feature in the at least two images, computing a globally consistent set of alignments for the at least two images, selecting a final compositing surface and a parameterization for the at least two images, or a combination thereof.

34. The method of claim 21, further comprising generating, with a calibration unit in communication with the sensor unit, the calibration data.

35. The method of claim 34, wherein generating the calibration data comprises:

performing an intrinsic calibration for each of the plurality of cameras; and

performing an extrinsic calibration for the plurality of cameras.

36. The method of claim 35, wherein performing the intrinsic calibration for each of the plurality of cameras comprises:

positioning the camera in a plurality of locations relative to a calibration object;

receiving an image generated by the camera at each of the plurality of locations; and

varying at least one intrinsic parameter based on the images generated by the camera at each of the plurality of locations until an optimum is found.

37. The method of claim 35, wherein performing the intrinsic calibration for each of the plurality of cameras comprises:

creating a projection model for the camera;

determining a ray originating at a coordinate system of the camera that contains a corresponding point in space;

performing a great circle fitting;

defining an objective function for the camera;

performing an optimization;

or

a combination thereof.

38. The method of claim 35, wherein performing the extrinsic calibration for the plurality of cameras comprises:

positioning each camera in a plurality of locations relative to a calibration object;

receiving an image generated by each camera at each of the plurality of locations;

for each camera, finding a rotation and translation between a coordinate system of the camera and a reference coordinate system based on the images generated by the camera at each of the plurality of locations; and

deriving camera positions and orientations relative to each other based on the rotations and translations.

39. The method of claim 35, wherein performing the extrinsic calibration for the plurality of cameras comprises:

performing an optimization;

calibrating the camera using a plurality of images of a calibration object;

creating a translation vector for the camera;

relating the camera to each of the other cameras;

or

a combination thereof.

40. The method of claim 21, wherein the calibration data is generated prior to the sensor unit receiving the plurality of images.