SURROUND CAMERA SYSTEM WITH SEAMLESS STITCHING FOR ARBITRARY VIEWPOINT SELECTION

• US 20190325580A1
• Filed: 06/26/2019
• Published: 10/24/2019
• Est. Priority Date: 06/26/2019
• Status: Active Grant

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Lukac, Rastislav, Basavaraj, Ravi, Ali, Gazi Naser, Madaparambil, Krishna Kumar
• Granted Patent: US 11,158,056 B2
• Time in Patent Office: Days
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06T 7/12 Edge-based segmentation
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture

Abstract

An apparatus comprising a memory to store a first image captured by a first camera and a second image captured by a second camera; and a processor comprising circuitry, the processor to identify viewpoint information defining a view for a stitched image, the stitched image to be generated from a combination of a plurality of images comprising the first image and the second image; and based on the viewpoint information and a projection type, determine transformation parameters for at least one reference region associated with overlapping regions of the first image and the second image in order to: localize overlapping regions of the first image and the second image to determine stitching parameters; combine the first image and the second image using the stitching parameters; and assign data of the combined first image and the second image to the stitched image.

Claims

1. An apparatus comprising:

a memory to store a first image captured by a first camera and a second image captured by a second camera; and

a processor coupled to the memory, the processor comprising circuitry, the processor to;

identify viewpoint information defining a view for a stitched image, the stitched image to be generated from a combination of a plurality of images comprising the first image and the second image; and

based on the viewpoint information and a projection type, determine at least one transformation parameter for at least one reference region associated with overlapping regions of the first image and the second image in order to;

localize overlapping regions of the first image and the second image to determine stitching parameters;

combine the first image and the second image using the stitching parameters; and

assign data of the combined first image and the second image to the stitched image.

2. The apparatus of claim 1, wherein the at least one transformation parameter comprises one or more rotation parameters and one or more translation parameters.

3. The apparatus of claim 1, wherein the processor is to generate the stitched image based on the viewpoint information using at least one additional pair of images of the plurality of images by:

localizing overlapping regions of at least one additional pair of images to determine additional stitching parameters;

combining the at least one additional pair of images using the additional stitching parameters; and

assigning data of the combined at least one additional pair of images to the stitched image.

4. The apparatus of claim 1, wherein the stitching parameters comprise a configurable blending mask based on a projection type of the first image and the second image and the processor is further to combine the first image with the second image based at least in part on the configurable blending mask.

5. The apparatus of claim 4, wherein the processor is to generate the configurable blending mask by transforming a reference blending mask associated with one or more of the at least one reference region.

6. The apparatus of claim 1, wherein the stitching parameters comprise at least one adjustment factor based on statistics associated with the overlapping regions and the processor is further to combine the first image with the second image based at least in part on the at least one adjustment factor.

7. The apparatus of claim 6, wherein the processor is to perform at least one of:

generate the at least one adjustment factor, wherein the at least one adjustment factor specifies an adjustment of one of brightness and color of pixels of the first image and the second image; and

modify one or more of the at least one adjustment factor to prevent at least one of undershoots and overshoots of pixel values in the stitched image.

8. The apparatus of claim 6, wherein the processor is to determine the stitching parameters based on a combination of the at least one adjustment factor and a configurable blending mask based on a projection type of the first image and the second image.

9. The apparatus of claim 1, wherein the processor is to:

detect at least one of common features and points in the first image and the second image through at least one of image analysis, feature extraction and matching, corner detection, template matching, semantic segmentation, object detection, image registration, and trained classifiers; and

use the detected at least one of common features or points to refine a determined location of the overlapping regions of the first image and the second image or adjust at least one of the first image and the second image through geometric transformation prior to localizing the overlapping regions of the first image and the second image.

10. The apparatus of claim 1, wherein the processor is further to perform at least one of:

suppress errors in the combination of the first image and second image caused by one or more of data alignment, image registration, feature extraction, matching, and repetition, ghosting due to motion, and parallax; and

utilize at least one of object segmentation, semantic processing, depth information, and multiresolution analysis in the combination of the first image and second image.

11. The apparatus of claim 1, wherein the processor comprises:

a first multi-camera stitching module to combine the first image and the second image using the stitching parameters; and

a second multi-camera stitching module to, based on at least one of second viewpoint information different from the viewpoint information and a second projection type different from the projection type;

localize overlapping regions of two images of the plurality of images to determine second stitching parameters;

combine the two images of the plurality of images using the second stitching parameters; and

assign the data of the combined two images of the plurality of images to a second stitched image.

12. The apparatus of claim 11, wherein the two images include the first image and the second image.

13. The apparatus of claim 1, wherein the processor comprises:

a first multi-camera stitching module to generate intermediate image data using the plurality of images based on at least one of the viewpoint information and the projection type, the first multi-camera stitching module to further produce a first view from the generated intermediate image data; and

a second multi-camera stitching module to produce a second view from the generated intermediate image data based on at least one of second viewpoint information different from the viewpoint information and a second projection type different from the first projection type.

14. The apparatus of claim 1, wherein the processor is to specify at least one of the viewpoint information defining a view for a stitched image for at least one of the viewpoint and the projection type based on a driving situation and image or scene analysis.

15. The apparatus of claim 1, wherein the at least one transformation parameter is determined based on at least one other transformation parameter associated with one or more tuning viewpoints.

16. The apparatus of claim 15, wherein the processor is to perform at least one of:

select the one or more tuning viewpoints based on at least one of a difference and similarity between the one or more tuning viewpoints and a viewpoint specified by the viewpoint information; and

adjust the at least one other transformation parameter associated with the one or more tuning viewpoints with respect to a reference viewpoint.

17. The apparatus of claim 1, wherein the at least one transformation parameter is determined using at least one of linear transformation and perspective transformation based on a change in at least one of the viewpoint information and the projection type.

18. A method comprising:

identifying viewpoint information defining a view for a stitched image, the stitched image to be generated from a combination of a plurality of images comprising a first image and a second image; and

based on the viewpoint information and a projection type, determine transformation parameters for at least one reference region associated with overlapping regions of the first image and the second image in order to;

localizing overlapping regions of the first image and the second image to determine stitching parameters;

combining the first image and the second image using the stitching parameters; and

assigning data of the combined first image and the second image to the stitched image.

19. The method of claim 18, wherein the at least one transformation parameter comprises one or more rotation parameters and one or more translation parameters.

20. The method of claim 18, wherein the stitching parameters comprise a configurable blending mask based on a projection type of the first image and the second image and the processor is further to combine the first image with the second image based at least in part on the configurable blending mask.

21. The method of claim 18, wherein the stitching parameters comprise at least one adjustment factor based on statistics associated with the overlapping regions and the processor is further to combine the first image with the second image based at least in part on the at least one adjustment factor.

22. At least one non-transitory machine readable storage medium having instructions stored thereon, the instructions when executed by a machine to cause the machine to:

identify viewpoint information defining a view for a stitched image, the stitched image to be generated from a combination of a plurality of images comprising a first image and a second image; and

based on the viewpoint information and a projection type, determine transformation parameters for at least one reference region associated with overlapping regions of the first image and the second image in order to;

localize overlapping regions of the first image and the second image to determine stitching parameters;

combine the first image and the second image using the stitching parameters; and

assign data of the combined first image and the second image to the stitched image.

23. The at least one non-transitory machine readable storage medium of claim 22, wherein the stitching parameters comprise a configurable blending mask based on a projection type of the first image and the second image and the processor is further to combine the first image with the second image based at least in part on the configurable blending mask.

24. The at least one non-transitory machine readable storage medium of claim 22, wherein the stitching parameters comprise at least one adjustment factor based on statistics associated with the overlapping regions and the processor is further to combine the first image with the second image based at least in part on the at least one adjustment factor.