IMAGE STITCHING AND AUTOMATIC-COLOR CORRECTION

• US 20170070674A1
• Filed: 02/25/2015
• Published: 03/09/2017
• Est. Priority Date: 02/26/2014
• Status: Active Grant

• Current Assignee: Searidge Technologies Inc.
• Original Assignee: Searidge Technologies Inc.
• Inventors: THUROW, Christian Tim, CHEIKH, Moodie, SAURIOL, Alexander
• Granted Patent: US 10,666,862 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06T 3/12 Panospheric to cylindrical image transformations
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06T 5/00 Image enhancement or restoration
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06T 7/90 Determination of colour characteristics
• CPC Class Codes: H04N 13/15 for colour aspects of image signals
• CPC Class Codes: H04N 13/156 Mixing image signals
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 17/002 for television cameras
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding
• CPC Class Codes: H04N 19/85 using pre-processing or post-processing specially adapted for video compression
• CPC Class Codes: H04N 2013/0077 Colour aspects
• CPC Class Codes: H04N 2013/0088 Synthesising a monoscopic image signal from stereoscopic images, e.g. synthesising a panoramic or high resolution monoscopic image
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 5/2622 Signal amplitude transition in the zone between image portions, e.g. soft edges
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

Panoramic videos are generated from multiple video feeds in real time received from multiple video cameras, such as an array of video cameras having overlapping fields of view. Texture mapping techniques are employed to correct lens distortion or other defects or deficiencies in the video frames in each of the video feeds caused by optical properties of the corresponding video camera. Video frames of related video feeds, such as the video feeds of cameras in an array of cameras having adjacent and overlapping fields of view, are seamlessly stitched automatically based on an initial manual configuration, again employing texture mapping techniques. A colour profile of the different video frames is normalized to provide a uniform and seamless colour profile of the video panoramic view.

Claims

1. A method of generating a panoramic video from a plurality of video feeds, the method comprising:

using a processor to define in computer memory, for each video feed, a three-dimensional (3D) surface modelling optical properties of the video camera absent lens distortion;

for each video feed, using the processor to project corresponding calibration images onto the corresponding 3D surface to generate corrected calibration images corrected of lens distortion;

using the processor to stitch the 3D surfaces based on the corrected calibration images; and

using the processor to project video frames received in the video feeds onto the respective 3D surfaces to generate the panoramic video.

2. The method according to claim 1, wherein each 3D surface comprises a corresponding panel having a vertex mesh, and the processor modifies the vertex mesh based on intrinsic camera parameters distortion coefficients to generate a warped panel, wherein the processor projects the corresponding calibration images onto the corresponding warped panels to generate the corrected calibration images, and projects the video frames onto the corresponding warped panels to generate the panoramic video.

3. The method according to claim 1, wherein using the processor to stitch the corrected video frames comprises:

for each 3D surface, using a display and a user interface to modify in the computer memory a position, orientation, and/or size of the 3D surface with reference to a predefined 3D frame of reference to overlap one or more neighbouring corrected calibration images in respective boundary areas.

4. The method according to claim 3 further comprising using the processor to blend the corrected calibration images in the boundary areas by modifying, in each boundary area, a transparency of a corresponding upper panel in the boundary area to blend the corresponding corrected calibration images in the boundary area.

5. The method according to claim 4, wherein the processor modifies the transparency of the corresponding upper panel in the boundary area by modifying an alpha channel of the corresponding upper panel in the boundary area.

6. The method according to claim 1, wherein the processor further normalizes a colour profile of the video frames to generate the panoramic video.

7. The method according to claim 6, wherein the processor normalizes the colour profile of the video frames by, for each neighbouring pair of the corrected calibration images comprising a first image and a second image overlapping in a boundary area:

a) defining a reference region in the first image in the boundary area;

b) defining a check region in the second image overlapping the reference region in the boundary area;

c) calculating one or more reference regions median in the reference region;

d) calculating one or more check region medians in the check region;

e) calculating median differences associated with the second image based on the reference region medians and the check region medians; and

f) based on the median differences, generating a difference texture associated with the 3D surface corresponding to the second image.

8. The method according to claim 7, wherein the processor normalizes the colour profile of the video frames by further, for each 3D surface, projecting the corresponding difference texture onto the 3D surface in addition to the video frames of the corresponding video feed.

9. The method according to claim 7, wherein for at least one corrected calibration image calculating the reference region medians comprises subtracting median differences associated with the corrected calibration image.

10. The method according to claim 7, wherein the reference region medians comprise respective medians of RGB colour channel values in the reference region, the check region medians comprise respective medians of RGB colour channel values in the check region, and e) comprises subtracting the medians of RGB colour channel values in the reference region from the respective medians of RGB colour channel values in check region.

11. The method according to claim 10, wherein the reference region medians comprise a median luminance value in the reference region, the check region medians comprise a median luminance value in the check region, and e) comprises subtracting the median luminance value in the reference region from the median luminance value in the check region.

12. The method according to claim 7, wherein a) to f) are performed iteratively on successive neighbouring pairs of the video frames beginning in a first iteration wherein the first image corresponds to a left-most or right-most video camera in an array of the video cameras.

13. A system for generating a panoramic video from a plurality of video feeds, the system comprising a processor and a memory storing instructions executable by the processor:

to define in computer memory, for each video feed, a three-dimensional (3D) surface modelling optical properties of the video camera absent lens distortion;

for each video feed, to project corresponding calibration images onto the corresponding 3D surface to generate corrected calibration images corrected of lens distortion;

to stitch the 3D surfaces based on the corrected calibration images; and

to project video frames received in the video feeds onto the respective 3D surfaces to generate the panoramic video.

14. The system according to claim 13, wherein the system further comprises a display and a user interface connected to the processor and memory to stitch the 3D surfaces, wherein the display and the user interface are used to modify in the computer memory a position, orientation, and/or size of each 3D surface with reference to a predefined 3D frame of reference to overlap one or more neighbouring corrected calibration images in respective boundary areas.

15. The system according to claim 14, wherein the instructions are further executable by the processor:

to blend the corrected calibration images in the boundary areas by modifying, in each boundary area, a transparency of a corresponding upper panel in the boundary area to blend the corresponding corrected calibration images in the boundary area.

16. The system according to claim 15, wherein the processor modifies the transparency of the corresponding upper panel in the boundary area by modifying an alpha channel of the corresponding upper panel in the boundary area.

17. The system according to claim 13, wherein the instructions are further executable by the processor to normalize colour profiles of the video frames, to generate the panoramic video, by, for each neighbouring pair of the corrected calibration images comprising a first image and a second image overlapping in a boundary area:

a) defining a reference region in the first image in the boundary area;

b) defining a check region in the second image overlapping the reference region in the boundary area;

c) calculating one or more reference regions median in the reference region;

d) calculating one or more check region medians in the check region;

e) calculating median differences associated with the second image based on the reference region medians and the check region medians; and

f) based on the median differences, generating a difference texture associated with the 3D surface corresponding to the second image.

18. The system according to claim 17, wherein the processor normalizes the colour profile of the video frames by further, for each 3D surface, projecting the corresponding difference texture onto the 3D surface in addition to the video frames of the corresponding video feed.

19. The system according to claim 17, wherein the reference region medians comprise respective medians of RGB colour channel values in the reference region, the check region medians comprise respective medians of RGB colour channel values in the check region, and e) comprises subtracting the medians of RGB colour channel values in the reference region from the respective medians of RGB colour channel values in check region.

20. The system according to claim 17, wherein the processor performs a) to f) iteratively on successive neighbouring pairs of the video frames beginning in a first iteration wherein the first image corresponds to a left-most or right-most video camera in an array of the video cameras.