Rear-stitched view panorama for rear-view visualization

• US 10,313,584 B2
• Filed: 07/27/2017
• Issued: 06/04/2019
• Est. Priority Date: 01/04/2017
• Status: Active Grant

• Current Assignee: Texas Instruments, Inc.
• Original Assignee: Texas Instruments, Inc.
• Inventors: Pan, Janice Shuay-Ann, Appia, Vikram VijayanBabu
• Primary Examiner(s): Schnurr, John R
• Application Number:
  US15/661,844
• Publication Number:
  US 20180191954A1
• Time in Patent Office: 677 Days
• CPC Class Codes: B60R 1/26 to the rear of the vehicle
• CPC Class Codes: B60R 1/31 providing stereoscopic vision
• CPC Class Codes: B60R 2300/105 using multiple cameras
• CPC Class Codes: B60R 2300/107 using stereoscopic cameras
• CPC Class Codes: B60R 2300/303 using joined images, e.g. multiple camera images
• CPC Class Codes: B60R 2300/8046 for replacing a rear-view mirror system
• CPC Class Codes: B60R 2300/8066 for monitoring rearward traffic
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/246 Calibration of cameras
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 5/44504 Circuit details of the additional information generator, e.g. details of the character or graphics signal generator, overlay mixing circuits
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

A rear-stitched view panorama (RSVP) system is provided that includes at least one processor and a memory storing software instructions that, when executed by the least one processor, cause the RSVP system to compute a disparity map for a left center rear image and a right center rear image captured by a stereo camera mounted on a rear of a vehicle, transform a right rear image, a left rear image, a reference center rear image, and the disparity map to a virtual world view, the right rear image and left rear image captured by respective right and left cameras mounted on the vehicle, compute an optimal left seam and an optimal right seam based on the transformed disparity map, and stitch the transformed images based on respective optimal seams to generate a panorama.

Claims

1. A rear-stitched view panorama (RSVP) system comprising:

at least one processor; and

a memory storing software instructions that, when executed by the at least one processor, cause the RSVP system to

compute a disparity map for a left center rear image and a right center rear image, the left center rear image and the right center rear image captured by a stereo camera mounted on a rear of a vehicle;

transform a right rear image, a left rear image, a reference center rear image of the left center rear image and the right center rear image, and the disparity map to a virtual world view using virtual camera parameters, the right rear image captured by a right camera mounted on a right side of the vehicle and the left rear image captured by a left camera mounted on a left side of the vehicle;

compute an optimal left seam between the transformed left rear image and the transformed reference center rear image based on the transformed disparity map;

compute an optimal right seam between the transformed right rear image and the transformed reference center rear image based on the transformed disparity map; and

stitch the transformed left rear image and the transformed reference center rear image based on the optimal left seam and the transformed right rear image and the transformed reference center rear image based on the optimal right seam to generate a panorama.

2. The RSVP system of claim 1, wherein to compute an optimal left seam, the software instructions further cause the RSVP system to:

compute a world disparity map;

compute a world disparity difference map based on the world disparity map and the transformed disparity map; and

search an overlap region between the transformed left rear image and the transformed reference center rear image for a first minimum cost seam, wherein costs of candidate seams are computed based on the world disparity difference map.

3. The RSVP system of claim 2, wherein the software instructions further cause the RSVP system to compute a cost of a candidate seam as a sum of disparity values in the world disparity difference map corresponding to pixels in the candidate seam normalized by a number of valid corresponding disparity values in the transformed disparity map.

4. The RSVP system of claim 3, wherein pixels with a disparity value in the transformed disparity map greater than a threshold are not included in the sum.

5. The RSVP system of claim 2, wherein to search an overlap region, the software instructions further cause the RSVP system to:

search the overlap region for a minimum cost vertical seam; and

search the overlap region for the first minimum cost seam in a neighborhood around the minimum cost vertical seam.

6. The RSVP system of claim 2, wherein to compute an optimal left seam, the software instructions further cause the RSVP system to:

compute a ground plane disparity difference map based on a ground plane disparity map and the transformed disparity map; and

search the overlap region between the transformed left rear image and the transformed reference center rear image for a second minimum cost seam, wherein costs of candidate seams are computed based on the ground plane disparity difference map;

wherein the optimal left seam is a lowest cost seam of the first minimum cost seam and the second minimum cost seam.

7. The RSVP system of claim 1, wherein the software instructions further cause the RSVP system to:

apply temporal smoothing to the optimal left seam and the optimal right seam.

8. The RSVP system of claim 7, wherein to apply temporal smoothing, the software instructions further cause the RSVP system to:

set the optimal left seam to a previous optimal left seam if all pixels in the optimal left seam are within a minimum distance threshold of the previous optimal left seam;

set the optimal right seam to a previous optimal right seam if all pixels in the optimal right seam are within the minimum distance threshold of the previous optimal right seam;

scale the optimal left seam such that a largest pixel displacement between the optimal left seam and the previous optimal left seam is no larger than a maximum distance threshold if any pixel in the optimal left seam is greater than the maximum distance threshold number of pixels away from the previous optimal left seam; and

scale the optimal right seam such that a largest pixel displacement between the optimal left seam and the previous optimal left seam is no larger than the maximum distance threshold if any pixel in the optimal right seam is greater than the maximum distance threshold away from the previous optimal right seam.

9. The RSVP system of claim 1, wherein to stitch the transformed left rear image, the software instructions further cause the RSVP system to:

compute a blending look-up table (LUT) comprising alpha-blending coefficients for combining the transformed left rear image and the transformed reference center rear image; and

use the blending LUT to stitch the transformed left rear image and the transformed reference center rear image.

10. The RSVP system of claim 1, wherein the software instructions further cause the RSVP system to:

bias the optimal left seam when an object detection modality coupled to the RSVP system indicates presence of an object in a left blind zone, the bias allowing at least some of the object to be seen in the panorama.

11. The RSVP system of claim 1, wherein the software instructions further cause the RSVP system to:

bias the optimal right seam when an object detection modality coupled to the RSVP system indicates presence of an object in a right blind zone, the bias allowing at least some of the object to be seen in the panorama.

12. A method for generating a rear-stitched view panorama, the method comprising:

computing a disparity map for a left center rear image and a right center rear image, the left center rear image captured by a stereo camera mounted on a rear of a vehicle;

transforming a right rear image, a left rear image, a reference center rear image of one of the left center rear image and the right center rear image, and the disparity map to a virtual world view using virtual camera parameters, the right rear image captured by a right camera mounted on a right side of the vehicle and the left rear image captured by a left camera mounted on a left side of the vehicle;

computing an optimal left seam between the transformed left rear image and the transformed reference center rear image based on the transformed disparity map;

computing an optimal right seam between the transformed right rear image and the transformed reference center rear image based on the transformed disparity map; and

stitching the transformed left rear image and the transformed reference center rear image based on the optimal left seam and the transformed right rear image and the transformed reference center rear image based on the optimal right seam to generate a panorama.

13. The method of claim 12, wherein computing an optimal left seam further comprises:

computing a world disparity map;

computing a world disparity difference map based on the world disparity map and the transformed disparity map; and

searching an overlap region between the transformed left rear image and the transformed reference center rear image for a first minimum cost seam, wherein costs of candidate seams are computed based on the world disparity difference map.

14. The method of claim 13, wherein a cost of a candidate seam is computed as a sum of disparity values in the world disparity difference map corresponding to pixels in the candidate seam normalized by a number of valid corresponding disparity values in the transformed disparity map.

15. The method of claim 14, wherein pixels with a disparity value in the transformed disparity map greater than a threshold are not included in the sum.

16. The method of claim 13, wherein searching an overlap region further comprises:

searching the overlap region for a minimum cost vertical seam; and

searching the overlap region for the first minimum cost seam in a neighborhood around the minimum cost vertical seam.

17. The method of claim 13, wherein computing an optimal left seam further comprises:

computing a ground plane disparity difference map based on a ground plane disparity map and the transformed disparity map; and

searching the overlap region between the transformed left rear image and the transformed reference center rear image for a second minimum cost seam, wherein costs of candidate seams are computed based on the ground plane disparity difference map;

wherein the optimal left seam is a lowest cost seam of the first minimum cost seam and the second minimum cost seam.

18. The method of claim 12, further comprising:

applying temporal smoothing to the optimal left seam and the optimal right seam.

19. The method of claim 18, wherein applying temporal smoothing further comprises:

setting the optimal left seam to a previous optimal left seam if all pixels in the optimal left seam are within a minimum distance threshold of the previous optimal left seam;

setting the optimal right seam to a previous optimal right seam if all pixels in the optimal right seam are within the minimum distance threshold of the previous optimal right seam;

scaling the optimal left seam such that a largest pixel displacement between the optimal left seam and the previous optimal left seam is no larger than a maximum distance threshold if any pixel in the optimal left seam is greater than the maximum distance threshold away from the previous optimal left seam; and

scaling the optimal right seam such that a largest pixel displacement between the optimal left seam and the previous optimal left seam is no larger than the maximum distance threshold if any pixel in the optimal right seam is greater than the maximum distance threshold away from the previous optimal right seam.

20. The method of claim 12, wherein stitching the transformed left rear image further comprises:

computing a blending look-up table (LUT) comprising alpha-blending coefficients for combining the transformed left rear image and the transformed reference center rear image; and

using the blending LUT to stitch the transformed left rear image and the transformed reference center rear image.

21. The method of claim 12, further comprising:

biasing the optimal left seam when an object detection modality indicates presence of an object in a left blind zone, the biasing allowing at least some of the object to be seen in the panorama.

22. The method of claim 12, further comprising

biasing the optimal right seam when an object detection modality indicates presence of an object in a right blind zone, the biasing allowing at least some of the object to be seen in the panorama.