ADVANCED BLENDING OF STITCHED IMAGES FOR 3D OBJECT REPRODUCTION

• US 20160086033A1
• Filed: 09/19/2014
• Published: 03/24/2016
• Est. Priority Date: 09/19/2014
• Status: Active Grant

• Current Assignee: Bendix Commercial Vehicle Systems LLC (Estate of Heinz Hermann Thiele)
• Original Assignee: Bendix Commercial Vehicle Systems LLC (Estate of Heinz Hermann Thiele)
• Inventors: Molin, Hans M., Kuehnle, Andreas U., Gyori, Marton
• Granted Patent: US 10,055,643 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: B60R 1/27 providing all-round vision, e.g. using omnidirectional cameras
• CPC Class Codes: B60R 1/31 providing stereoscopic vision
• CPC Class Codes: B60R 2300/105 using multiple cameras
• CPC Class Codes: B60R 2300/304 using merged images, e.g. merging camera image with stored images
• CPC Class Codes: B60R 2300/802 for monitoring and displaying vehicle exterior blind spot views
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/174 involving the use of two or more images
• CPC Class Codes: G06T 7/90 Determination of colour characteristics
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 23/00 Cameras or camera modules comprising electronic image sensors; Control thereof
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

When blending individual vehicle-mounted camera images into a coherent composite image of the vehicle surroundings, objects in the respective images are analyzed to determine the presence of a pedestrian. When the individual images are blended to form the composite image, multiple pedestrian shadow views of the pedestrian are replaced with background pixels to erase the pedestrian, and a semi-symbolic graphical representation of the pedestrian is superimposed at the location of the actual pedestrian in the composite image. The graphical representation is proportional to the pedestrian in the composite image and includes features unique to the pedestrian it represents. The composite image with the graphical representation is then presented to a viewer.

Claims

1. A system that facilitates detecting an object in a surround-view image of a vehicle and superimposing a semi-symbolic graphical representation of the object in the surround-view image, comprising:

a first camera that captures a first image of an object from a first vantage point;

a second camera that captures a second image of the object from a second vantage point;

a processor configured to;

generate a composite image by merging the first and second images;

analyze the composite image to detect one or more features indicative of the object in the composite image;

replace object pixels with corresponding background pixels;

generate a semi-symbolic graphical representation of the object that comprises one or more object features detected in the first and second images;

insert the semi-symbolic graphical representation of the object into the composite image at a calculated position of the object relative to the vehicle; and

output the composite image with the semi-symbolic representation to a display.

2. The system according to claim 1, wherein the processor is further configured to replace object pixels corresponding to the object in the first image with background pixels from the second image, and replace object pixels corresponding to the object in the second image with background pixels from the first image.

3. The system according to claim 1, wherein the object is a pedestrian and the one or more object features comprise one or more of:

pedestrian clothing color;

and

pedestrian height in pixels.

4. The system according to claim 1, wherein the object is a pedestrian and the processor is further configured to superimpose an image of the pedestrian'"'"'s face, or a similar face, on the semi-symbolic graphical representation.

5. The system according to claim 1, further comprising a disagreement calculation module configured to analyze the composite image to identify an area of disagreement D between the first and second images.

6. The system according to claim 5, further comprising a ray calculation module configured to calculate at least one ray emanating from each camera through its field of view to a boundaries of a more distant half, relative to each camera, of the area of disagreement D.

7. The system according to claim 6, wherein the object is a pedestrian and further comprising a ray intersection identification module configured to identify an intersection R of camera rays, wherein the intersection corresponds to a highest point of the pedestrian, and wherein the processor is configured to execute a tracking module configured to track the highest point of the pedestrian and to adjust the position of the semi-symbolic graphical representation to correspond to the tracked point.

8. The system according to claim 1, wherein the object is a pedestrian and the processor is further configured to execute an object shape identification module that identifies a shape formed by each of a front view representation and a rear view representation of the pedestrian in the composite image, wherein the position of the pedestrian relative at the vehicle is determined to be at the point of intersection of the front and rear view representations of the pedestrian in the composite image.

9. The system according to claim 8, wherein the object is a pedestrian and the processor is further configured to determine a height of the head of the pedestrian above the ground in the composite image, and wherein the processor is configured to execute a tracking filter that tracks the location of the head of the pedestrian and to adjust the position of the semi-symbolic graphical representation to correspond to the location of the head of the pedestrian.

10. A method of detecting an object in a surround-view image of a vehicle and superimposing a semi-symbolic graphical representation of the object in the surround-view image, comprising:

capturing a first image, from a first vantage point, of an object;

capturing a second image, from a second vantage point, of the object;

generating a composite image by merging the first and second images;

analyzing the composite image to detect one or more features indicative of the object in the composite image;

replacing object pixels with corresponding background pixels;

generating a semi-symbolic graphical representation of the object that comprises one or more object features detected in the first and second images;

inserting the semi-symbolic graphical representation of the object into the composite image at a calculated position of the object relative to the vehicle; and

outputting the composite image with the semi-symbolic representation to a display.

11. The method according to claim 10, further comprising replacing replace object pixels corresponding to the object in the first image with background pixels from the second image, and replace object pixels corresponding to the object in the second image with background pixels from the first image.

12. The method according to claim 10, wherein the object is a pedestrian and the one or more object features comprise one or more of:

pedestrian clothing color; and

pedestrian height in pixels.

13. The method according to claim 10, the object is a pedestrian and further comprising superimposing an image of the pedestrian'"'"'s face on the semi-symbolic graphical representation.

14. The method according to claim 10, wherein the object is a pedestrian and further comprising:

analyzing the composite image to identify an area of disagreement D between the first and second images;

calculating at least one ray emanating from each camera through its field of view to a boundaries of a more distant half, relative to each camera, of the area of disagreement D; and

identifying an intersection R of camera rays, wherein the intersection corresponds to a highest point of the pedestrian, and tracking the highest point of the pedestrian as it moves past the vehicle.

15. The method according to claim 10, wherein the object is a pedestrian and further comprising identifying a shape formed by each of a front view representation and a rear view representation of the object in the composite image, wherein the pedestrian is determined to be positioned where the front and rear view representations intersect in the composite image.

16. The method according to claim 15, further comprising determining a height of the head of the pedestrian above the ground in the composite image, and tracking the location of the head of the pedestrian as it moves past the vehicle.

17. A processor configured to execute computer-executable instructions for detecting an object in a surround-view image of a vehicle and superimposing a semi-symbolic graphical representation of the object in the surround-view image, the instructions comprising:

capturing a first image, from a first vantage point, of an object;

capturing a second image, from a second vantage point, of the object;

generating a composite image by merging the first and second images;

analyzing the composite image to detect one or more features indicative of the object in the composite image;

replacing object pixels with corresponding background pixels;

generating a semi-symbolic graphical representation of the object that comprises one or more object features detected in the first and second images;

inserting the semi-symbolic graphical representation of the object into the composite image at a calculated position of the object relative to the vehicle; and

outputting the composite image with the semi-symbolic representation to a display.

18. The processor according to claim 17, the instructions further comprising replacing object pixels corresponding to the object in the first image with background pixels from the second image, and replace object pixels corresponding to the object in the second image with background pixels from the first image.

19. The processor according to claim 17, wherein the object is a pedestrian and the one or more object features comprise one or more of:

pedestrian clothing color;

and

pedestrian height in pixels.

20. The processor according to claim 17, wherein the object is a pedestrian, the instructions further comprising superimposing an image of the pedestrian'"'"'s face on the semi-symbolic graphical representation.

21. The processor according to claim 17, wherein the object is a pedestrian, the instructions further comprising:

analyzing the composite image to identify an area of disagreement D between the first and second images;

calculating at least one ray emanating from each camera through its field of view to a boundaries of a more distant half, relative to each camera, of the area of disagreement D; and

identifying an intersection R of camera rays, wherein the intersection corresponds to a highest point of the pedestrian, and tracking the highest point of the pedestrian as it moves past the vehicle.

22. The processor according to claim 17, wherein the object is a pedestrian, the instructions further comprising identifying a shape formed by each of a front view representation and a rear view representation of the object in the composite image, wherein the pedestrian is determined to be positioned where the front and rear view representations intersect in the composite image.

23. The processor according to claim 22, wherein the object is a pedestrian, the instructions further comprising determining a height of the head of the pedestrian above the ground in the composite image, and tracking the location of the head of the pedestrian as it moves past the vehicle.

24. An apparatus that facilitates detecting an object in a surround-view image of a vehicle and superimposing a semi-symbolic graphical representation of the object in the surround-view image, comprising:

means for capturing a first image, from a first vantage point, of an object;

means for capturing a second image, from a second vantage point, of the object;

means for generating a composite image by merging the first and second images;

means for analyzing the composite image to detect one or more features indicative of the object in the composite image;

means for replacing object pixels with corresponding background pixels;

means for generating a semi-symbolic graphical representation of the object that comprises one or more object features detected in the first and second images;

means for inserting the semi-symbolic graphical representation of the object into the composite image at a calculated position of the object relative to the vehicle; and

means for outputting the composite image with the semi-symbolic representation to a display.