Object detection system and method using a camera and a multiple zone temperature sensor

• US 8,659,408 B2
• Filed: 05/22/2012
• Issued: 02/25/2014
• Est. Priority Date: 05/22/2012
• Status: Active Grant

• Current Assignee: Aptiv Technologies AG
• Original Assignee: Delphi Technologies, Inc. (BorgWarner Incorporated)
• Inventors: Taylor, Ronald M., Greene, Jeremy S.
• Primary Examiner(s): Nguyen, Tai T
• Application Number:
  US13/477,122
• Publication Number:
  US 20130314221A1
• Time in Patent Office: 644 Days
• Field of Search: 340/345, 340/346, 340/903, 340/905, 340/932.2, 340/942
• US Class Current: 340/435
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 2207/30261 Obstacle
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/174 involving the use of two or more images

Abstract

A system and method to provide a notification to an operator of a vehicle that an object is proximate to the vehicle. Data from a visible light camera and a multiple zone temperature sensor are combined or fused to determine which regions of a display should be highlighted in order to help the vehicle operator better notice or discern an object shown on a display. The region or area of the display highlighted is determined by displaying a highlighted area or icon on the display corresponding to an area where objects detected on image data maps of hue and/or saturation and/or intensity data from the camera intersect with objects detected on a temperature data map from the temperature sensor. Misalignment between the camera and the temperature sensor may be tolerated by expanding the object detected on the various maps in order to increase the probability of an intersection occurring.

Claims

1. A system to provide a notification to an operator of a vehicle that an object is proximate to the vehicle, said system comprising

a visible light camera mounted on the vehicle effective to have a camera field of view, and configured to output image data indicative of visible light color and visible light intensity detected by pixels in the camera;

a multiple zone temperature sensor mounted on the vehicle effective to have a sensor field of view similar to the camera field of view, and configured to output temperature data indicative of a zone temperature for each of the multiple zones; and

a controller configured to

a) receive visible light image data from the camera, said image data indicative of visible light color and visible light intensity detected by pixels in the camera;

b) determine a hue map characterized as an array of hue data cells, wherein each hue data cell has a hue value based on a hue characteristic of the image data detected by of one or more of the pixels;

c) designate a hue object on an area of the hue map where a cluster of hue data cells are present that have notable hue values relative to a hue threshold;

d) determine a saturation map characterized as an array of saturation data cells, wherein each saturation data cell has a saturation value based on a saturation characteristic of the image data detected by of one or more of the pixels;

e) designate a saturation object on an area of the saturation map where a cluster of saturation data cells are present that have notable saturation values relative to a saturation threshold;

f) determine an intensity map characterized as an array of intensity data cells, wherein each intensity data cell has an intensity value based on an intensity characteristic of the image data detected by of one or more of the pixels;

g) designate an intensity object on an area of the intensity map where a cluster of intensity data cells are present that have notable intensity values relative to an intensity threshold;

h) receive temperature data from the multiple zone temperature sensor, wherein said temperature data includes a zone temperature value for each of the multiple zones;

i) determine a temperature map characterized as an array of temperature data cells, wherein each temperature data cell has a temperature value based on a temperature characteristic of one or more of the multiple zones;

j) designate a temperature object on an area of the temperature map where a cluster of temperature data cells are present that have notable temperature values relative to a temperature threshold;

k) designate a detected object on an area of an object map where the hue object, the intensity object, and the temperature object intersect, or where the saturation object, the intensity object, and the temperature object intersect; and

l) output a display signal corresponding to an image from the camera that includes a highlighted area selected based on the detected object.

2. The system in accordance with claim 1, wherein the camera has more than 100,000 pixels, and the multiple zone heat sensor has fewer than 1000 zones.

3. The system in accordance with claim 1, said system further comprising a warning indicator coupled to the controller effective to generate a warning signal when the detected object is detected.

4. A method to provide a notification to an operator of a vehicle that an object is proximate to the vehicle, said notification based on image data from a visible light camera mounted on the vehicle effective to have a camera field of view, and a multiple zone temperature sensor mounted on the vehicle effective to have a sensor field of view similar to the camera field of view, said method comprising:

a) receiving visible light image data from the camera, said image data indicative of visible light color and visible light intensity detected by pixels in the camera;

b) determining a hue map characterized as an array of hue data cells, wherein each hue data cell has a hue value based on a hue characteristic of the image data detected by of one or more of the pixels;

c) designating a hue object on an area of the hue map where a cluster of hue data cells are present that have notable hue values relative to a hue threshold;

d) determining a saturation map characterized as an array of saturation data cells, wherein each saturation data cell has a saturation value based on a saturation characteristic of the image data detected by of one or more of the pixels;

e) designating a saturation object on an area of the saturation map where a cluster of saturation data cells are present that have notable saturation values relative to a saturation threshold;

f) determining an intensity map characterized as an array of intensity data cells, wherein each intensity data cell has an intensity value based on an intensity characteristic of the image data detected by of one or more of the pixels;

g) designating an intensity object on an area of the intensity map where a cluster of intensity data cells are present that have notable intensity values relative to an intensity threshold;

h) receiving temperature data from the multiple zone temperature sensor, wherein said temperature data includes a zone temperature value for each of the multiple zones;

i) determining a temperature map characterized as an array of temperature data cells, wherein each temperature data cell has a temperature value based on a temperature characteristic of one or more of the multiple zones;

j) designating a temperature object on an area of the temperature map where a cluster of temperature data cells are present that have notable temperature values relative to a temperature threshold;

k) designating a detected object on an area of an object map where the hue object, the intensity object, and the temperature object intersect, or where the saturation object, the intensity object, and the temperature object intersect; and

l) displaying on a display an image from the camera that includes a highlighted area selected based on the detected object.

5. The method in accordance with claim 4, wherein said method further comprises generating a warning signal when the detected object is detected.

6. The method in accordance with claim 4, wherein said method further comprises increasing the number of hue data cells designated as the hue object based on one or more of a hue object shape and a hue object size.

7. The method in accordance with claim 4, wherein said method further comprises increasing the number of saturation data cells designated as the saturation object based on one or more of a saturation object shape and a saturation object size.

8. The method in accordance with claim 4, wherein said method further comprises increasing the number of intensity data cells designated as the intensity object based on one or more of an intensity object shape and an intensity object size.

9. The method in accordance with claim 4, wherein said method further comprises increasing the number of temperature data cells designated as the temperature object based on one or more of a temperature object shape and a temperature object size.

10. The method in accordance with claim 4, wherein the step of designating a detected object based on an object map includes determining that the detected object persists for a time longer than a persistence time threshold.

11. The method in accordance with claim 4, wherein the step of designating as a detected object an area on an object map includes determining that the detected object moves at a speed less than a speed threshold.

12. The method in accordance with claim 4, wherein determining a temperature map includes

determining a background temperature value based on the zone temperature value of at least one zone;

determining a differential temperature value for each zone based on a difference between the background temperature value and the corresponding zone temperature value; and

generating a thermal binary map based on a comparison of the differential temperature value to a differential temperature threshold value.

13. A controller for a system configured to provide a notification to an operator of a vehicle that an object is proximate to the vehicle, said system comprising a visible light camera mounted on the vehicle effective to have a camera field of view, and configured to output image data indicative of visible light color and visible light intensity detected by pixels in the camera, and a multiple zone temperature sensor mounted on the vehicle effective to have a sensor field of view similar to the camera field of view, and configured to output temperature data indicative of a zone temperature for each of the multiple zones, said controller configured to

a) receive visible light image data from the camera, said image data indicative of visible light color and visible light intensity detected by pixels in the camera;

b) determine a hue map characterized as an array of hue data cells, wherein each hue data cell has a hue value based on a hue characteristic of the image data detected by of one or more of the pixels;

c) designate a hue object on an area of the hue map where a cluster of hue data cells are present that have notable hue values relative to a hue threshold;

d) determine a saturation map characterized as an array of saturation data cells, wherein each saturation data cell has a saturation value based on a saturation characteristic of the image data detected by of one or more of the pixels;

e) designate a saturation object on an area of the saturation map where a cluster of saturation data cells are present that have notable saturation values relative to a saturation threshold;

f) determine an intensity map characterized as an array of intensity data cells, wherein each intensity data cell has an intensity value based on an intensity characteristic of the image data detected by of one or more of the pixels;

g) designate an intensity object on an area of the intensity map where a cluster of intensity data cells are present that have notable intensity values relative to an intensity threshold;

h) receive temperature data from the multiple zone temperature sensor, wherein said temperature data includes a zone temperature value for each of the multiple zones;

i) determine a temperature map characterized as an array of temperature data cells, wherein each temperature data cell has a temperature value based on a temperature characteristic of one or more of the multiple zones;

j) designate a temperature object on an area of the temperature map where a cluster of temperature data cells are present that have notable temperature values relative to a temperature threshold;

k) designate a detected object on an area of an object map where the hue object, the intensity object, and the temperature object intersect, or where the saturation object, the intensity object, and the temperature object intersect; and

l) output a display signal corresponding to an image from the camera that includes a highlighted area selected based on the detected object.