Rear-stitched view panorama for rear-view visualization

• US 10,674,079 B2
• Filed: 04/18/2019
• Issued: 06/02/2020
• Est. Priority Date: 01/04/2017
• Status: Active Grant

• Current Assignee: Texas Instruments, Inc.
• Original Assignee: Texas Instruments, Inc.
• Inventors: Pan, Janice Shuay-Ann, Appia, Vikram VijayanBabu
• Primary Examiner(s): Schnurr, John R
• Application Number:
  US16/387,817
• Publication Number:
  US 20190253625A1
• Time in Patent Office: 411 Days
• CPC Class Codes: B60R 1/00 Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
• CPC Class Codes: B60R 1/26 to the rear of the vehicle
• CPC Class Codes: B60R 1/31 providing stereoscopic vision
• CPC Class Codes: B60R 2300/105 using multiple cameras
• CPC Class Codes: B60R 2300/107 using stereoscopic cameras
• CPC Class Codes: B60R 2300/303 using joined images, e.g. multiple camera images
• CPC Class Codes: B60R 2300/8046 for replacing a rear-view mirror system
• CPC Class Codes: B60R 2300/8066 for monitoring rearward traffic
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/246 Calibration of cameras
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 5/44504 Circuit details of the additional information generator, e.g. details of the character or graphics signal generator, overlay mixing circuits
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

A rear-stitched view panorama (RSVP) system is provided that includes at least one processor and a memory storing software instructions that, when executed by the least one processor, cause the RSVP system to compute a disparity map for a left center rear image and a right center rear image captured by a stereo camera mounted on a rear of a vehicle, transform a right rear image, a left rear image, a reference center rear image, and the disparity map to a virtual world view, the right rear image and left rear image captured by respective right and left cameras mounted on the vehicle, compute an optimal left seam and an optimal right seam based on the transformed disparity map, and stitch the transformed images based on respective optimal seams to generate a panorama.

Claims

1. A method comprising:

receiving, by at least one processor, a first image and a second image;

computing, by the at least one processor, a disparity map for the first image and the second image;

receiving, by the at least one processor, a third image;

receiving, by the at least one processor, a fourth image;

transforming, by the at least one processor, the third image, the fourth image, a reference image of the first image or the second image, and the disparity map, to generate a transformed third image, a transformed fourth image, a transformed reference image, and a transformed disparity map;

computing, by the at least one processor, a left seam between the transformed third image and the transformed fourth image in accordance with the transformed disparity map;

computing, by the at least one processor, a right seam between the transformed third image and the transformed reference image in accordance with the transformed disparity map; and

generating, by the at least one processor, a panorama in accordance with the transformed fourth image, the transformed reference image, the left seam, the transformed third image, and the right seam.

2. The method of claim 1, wherein generating the panorama comprises:

stitching the transformed fourth image and the transformed reference image in accordance with the left seam; and

stitching the transformed third image and the transformed reference image based on the right seam to generate the panorama.

3. The method of claim 1, wherein receiving the first image and the second image comprises receiving a left center image and a right center rear image from a stereo camera mounted on a rear of a vehicle, wherein receiving the third image comprises receiving a right rear image from a right camera mounted on a right side of the vehicle, and wherein receiving the fourth image further comprises receiving a left rear image from a left camera mounted on a left side of the vehicle.

4. The method of claim 1, wherein computing the left seam further comprises:

computing a world disparity map;

computing a world disparity difference map in accordance with the world disparity map and the transformed disparity map;

computing costs of candidate seams in an overlap region between the transformed fourth image and the transformed reference image, in accordance with the world disparity difference map; and

searching the candidate seams for a first low cost seam.

5. The method of claim 4, wherein computing the costs of the candidate seams is performed as a sum of disparity values in the world disparity difference map corresponding to pixels in the candidate seams normalized by a number of valid corresponding disparity values in the transformed disparity map.

6. The method of claim 4, wherein computing the left seam further comprises:

computing a ground plane disparity difference map in accordance with a ground plane disparity map and the transformed disparity map;

computing second costs of candidate seams in an overlap region between the transformed fourth image and the transformed reference image;

searching the second costs of candidate seams for a second low cost seam; and

setting the left seam to the lesser of the first low cost seam and the second low cost seam.

7. The method of claim 1, further comprising applying temporal smoothing to the left seam and to the right seam, wherein applying temporal smoothing further comprises:

setting the left seam to a previous left seam upon determining that all pixels in the left seam are within a minimum distance threshold of the previous left seam;

setting the right seam to a previous right seam upon determining that all pixels in the right seam are within the minimum distance threshold of the previous right seam;

scaling the left seam such that a largest pixel displacement between the left seam and the previous left seam is no larger than a maximum distance threshold upon determining that any pixel in the left seam is greater than the maximum distance threshold away from the previous left seam; and

scaling the right seam such that a largest pixel displacement between the left seam and the left seam is no larger than the maximum distance threshold upon determining that any pixel in the right seam is greater than the maximum distance threshold away from the previous right seam.

8. The method of claim 1, wherein stitching the transformed fourth image further comprises:

computing a blending look-up table (LUT) comprising alpha-blending coefficients for combining the transformed fourth image and the transformed reference image; and

using the blending LUT to stitch the transformed fourth image and the transformed reference image.

9. The method of claim 1, further comprising biasing the left seam or the right seam upon determining a presence of an object in a blind zone.

10. The method of claim 1, further comprising displaying, by a display device, the panorama.

11. A vehicle, comprising:

a stereo camera configured to obtain a first image and a second image;

a first camera configured to obtain a third image;

a second camera configured to obtain a fourth image; and

at least one processor coupled to the stereo camera, to the first camera, and to the second camera, the at least one processor configured to;

receive, from the first camera, the first image and the second image;

compute a disparity map for the first image and the second image;

receive, from the first camera, the third image;

receive, from the second camera, the fourth image;

transform the third image, the fourth image, a reference image of the first image or the second image, and the disparity map, to generate a transformed third image, a transformed fourth image, a transformed reference image, and a transformed disparity map;

compute a left seam between the transformed third image and the transformed fourth image in accordance with the transformed disparity map;

compute a right seam between the transformed third image and the transformed reference image in accordance with the transformed disparity map; and

generate a panorama in accordance with the transformed fourth image, the transformed reference image, the left seam, the transformed third image, and the right seam.

12. The vehicle of claim 11, further comprising a display device coupled to the at least one processor, the display device configured to display the panorama.

13. The vehicle of claim 11, wherein generating the panorama comprises:

stitching the transformed fourth image and the transformed reference image in accordance with the left seam; and

stitching the transformed third image and the transformed reference image based on the right seam to generate the panorama.

14. The vehicle of claim 11, wherein the stereo camera is mounted on a rear of the vehicle, the first camera is mounted on a right side of the vehicle, and the second camera is mounted on a left side of the vehicle.

15. The vehicle of claim 11, wherein computing the left seam further comprises:

computing a world disparity map;

computing a world disparity difference map in accordance with the world disparity map and the transformed disparity map;

computing costs of candidate seams in an overlap region between the transformed fourth image and the transformed reference image, in accordance with the world disparity difference map; and

searching the candidate seams for a first low cost seam.

16. The vehicle of claim 15, wherein computing the costs of the candidate seams is performed as a sum of disparity values in the world disparity difference map corresponding to pixels in the candidate seams normalized by a number of valid corresponding disparity values in the transformed disparity map.

17. The vehicle of claim 15, wherein computing the left seam further comprises:

computing a ground plane disparity difference map in accordance with a ground plane disparity map and the transformed disparity map;

computing second costs of candidate seams in an overlap region between the transformed fourth image and the transformed reference image;

searching the second costs of candidate seams for a second low cost seam; and

setting the left seam to the lesser of the first low cost seam and the second low cost seam.

18. The vehicle of claim 11, wherein the at least one processor is further configured to apply temporal smoothing to the left seam and to the right seam, comprising:

setting the left seam to a previous left seam upon determining that all pixels in the left seam are within a minimum distance threshold of the previous left seam;

setting the right seam to a previous right seam upon determining that all pixels in the right seam are within the minimum distance threshold of the previous right seam;

scaling the left seam such that a largest pixel displacement between the left seam and the previous left seam is no larger than a maximum distance threshold upon determining that any pixel in the left seam is greater than the maximum distance threshold away from the previous left seam; and

scaling the right seam such that a largest pixel displacement between the left seam and the left seam is no larger than the maximum distance threshold upon determining that any pixel in the right seam is greater than the maximum distance threshold away from the previous right seam.

19. The vehicle of claim 11, wherein stitching the transformed fourth image further comprises:

computing a blending look-up table (LUT) comprising alpha-blending coefficients for combining the transformed fourth image and the transformed reference image; and

using the blending LUT to stitch the transformed fourth image and the transformed reference image.

20. The vehicle of claim 11, further comprising biasing the left seam or the right seam upon determining a presence of an object in a blind zone.