AUTOMATIC VEHICLE EQUIPMENT MONITORING, WARNING, AND CONTROL SYSTEM

• US 20120050074A1
• Filed: 02/25/2011
• Published: 03/01/2012
• Est. Priority Date: 02/26/2010
• Status: Active Grant

• Current Assignee: HL Klemove Corporation
• Original Assignee: Gentex Corporation
• Inventors: Bechtel, Jon H., Bush, Gregory S., Sherman, Tom B., Andrus, Jeremy C., Wright, David J., Schneider, Jim L.
• Granted Patent: US 8,456,327 B2
• Time in Patent Office: Days
• US Class Current: 340/988
• CPC Class Codes: B60R 1/04 mounted inside vehicle B60R1/025, B60R1/08 take precedence
• CPC Class Codes: B60R 2001/1215 with information displays
• CPC Class Codes: B60R 2001/1223 with sensors or transducers
• CPC Class Codes: B60R 2001/1253 with cameras, video cameras or video screens
• CPC Class Codes: B60R 2001/1284 with communication systems other than radio-receivers, e.g. keyless entry systems, navigation systems; with anti-collision systems
• CPC Class Codes: B60W 30/12 Lane keeping
• CPC Class Codes: B60W 30/14 Adaptive cruise control
• CPC Class Codes: G01C 21/3602 Input other than that of destination using image analysis, e.g. detection of road signs, lanes, buildings, real preceding vehicles using a camera
• CPC Class Codes: G06V 10/147 Details of sensors, e.g. sensor lenses fingerprint or palmprint sensors G06V40/13; vascular sensors G06V40/145; eye sensors G06V40/19
• CPC Class Codes: G06V 20/584 of vehicle lights or traffic lights
• CPC Class Codes: G06V 20/588 Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
• CPC Class Codes: G08G 1/166 for active traffic, e.g. moving vehicles, pedestrians, bikes

Abstract

An automatic vehicle equipment control system and methods thereof are provided, the system includes at least one imager configured to acquire a continuous sequence of high dynamic range single frame images, a processor, a color spectral filter array including a plurality of color filters, at least a portion of which are different colors, and pixels of an imager pixel array being in optical communication with substantially one spectral color filter, and a lens, wherein the imager is configured to capture a non-saturated image of nearby oncoming headlamps and at least one of a diffuse lane marking and a distant tail lamp in one image frame of the continuous sequence of high dynamic range single frame images, and the system configured to detect at least one of said highway markings and said tail lamps, and quantify light from the oncoming headlamp from data in the one image frame.

Claims

1. An automatic vehicle equipment control system comprising:

at least one imager configured to acquire a continuous sequence of high dynamic range single frame images, and said at least one imager comprising a pixel array;

a processor in communication with said at least one imager, and configured to process said continuous sequence of said high dynamic range images;

a color spectral filter array in optical communication with said at least one imager, said color spectral filter array comprising a plurality of color filters, at least a portion of which are different colors, and pixels of said pixel array being in optical communication with substantially one said spectral color filter; and

a lens in optical communication with said at least one imager and said color spectral filter array;

wherein said imager is configured to capture a non-saturated image of nearby oncoming headlamps and at least one of a diffuse lane marking and a distant tail lamp in one image frame of said continuous sequence of high dynamic range single frame images, and the automatic vehicle equipment control system configured to detect at least one of said highway markings and said tail lamps, and quantify light from said oncoming headlamp from data in said one image frame.

2. The automatic vehicle equipment control system of claim 1, wherein said oncoming headlamp and at least one said diffuse lane marking and said distant tail lamp are directly viewed by said imager.

3. The automatic vehicle equipment control system of claim 1, wherein said imager has an acquisition sequence for acquiring said continuous sequence of high dynamic range single frame images, and said acquisition sequence remains unchanged for an extended period of time when multiple different types of light sources are detected.

4. The automatic vehicle equipment control system of claim 1, wherein said lens is configured so said at least one imager has a reduced sensitivity to said light sources having a large area and low luminance, and said lens is further configured such that a majority of a light from a small light area light source is substantially focused onto one pixel of said pixel array.

5. The automatic vehicle equipment control system of claim 1, wherein said processor is further configured to process a mosaiced color image by interpolating at least one color component for each pixel that is missing said color data, and generate a grayscale image as a weighted average of said color components at each pixel location.

6. The automatic vehicle equipment control system of claim 1, wherein said processor is further configured to enhance a detection of yellow in said image, such that yellow lane markers in said image are detected.

7. The automatic vehicle equipment control system of claim 1, wherein an interpolation equation is based upon at least one of a position of a pixel relative to a mosaiced color filter array pattern, a color of a missing component that is being calculated, and an edge pattern detected for said pixel location where said interpolation is being performed.

8. The automatic vehicle equipment control system of claim 1, wherein said processor is further configured to use an edge pattern detection with a larger pixel base to select an interpolation equation based upon pixel values from a smaller pixel base.

9. The automatic vehicle equipment control system of claim 8 further configured to select interpolation equations that calculate an interpolated value based primarily on values of pixels that are adjacent to the interpolation site.

10. The automatic vehicle equipment control system of claim 1, wherein said processor is further configured to modify grayscale values for at least one pixel associated with a selected said color spectral filter by increasing their values relative to a non-yellow pixel.

11. The automatic vehicle equipment control system of claim 1, wherein said processor is further configured to locate a position of imaged light sources in successive image, statistically calculate light levels for each said spectral filter array colors, and compare said statistically calculated values for each of said color components to differentiate types of lamps.

12. The automatic vehicle equipment control system of claim 11, wherein said processor is further configured to take readings over time as the position of the projected image changes over time.

13. The automatic vehicle equipment control system of claim 11, wherein said processor is further configured, such that said image data from successive frames for a given light source is further utilized to provide classifications for position and trajectory of said light source.

14. The automatic vehicle equipment control system of claim 1 further comprising a memory device in communication with said processor, said memory device configured to store data obtained from captured said images, wherein said data comprises at least one of locations of bright pixels in said pixel array and of each said spectral color filter that covers each said pixel.

15. The automatic vehicle equipment control system of claim 14, wherein said processor is further configured to locate and track an imaged light source over a plurality of images, such that said data is analyzed to determine a plurality of properties of said light source.

16. The automatic vehicle equipment control system of claim 15, wherein said plurality of properties comprise at least one of color, intensity, a profile of intensity with respect to time, a position, a trajectory, and a velocity.

17. The automatic vehicle equipment control system of claim 1, wherein said color spectral filter array is arranged in a pattern, wherein a plurality of first colored filters are approximately twice the amount of a plurality of second colored filters and said plurality of first colored filters are approximately twice the amount of a plurality of third colored filters, and said processor is further configured to separately accumulate and compare a sum of first colored pixel values in a row with second colored pixel values and a sum of said first colored pixel values in a row with third colored pixel values to provide an estimate of a statistical quantity of measurements based on a time averaged response of said pixels with said spectral filters of a given color to a given light source adjusted for density of said pixels in said pixel array.

18. The automatic vehicle equipment control system of claim 1, wherein said processor is further configured to scan said image to locate local said pixels with a brightness peak reading value that exceeds a brightness peak threshold value set for a given spectral color filter.

19. The automatic vehicle equipment control system of claim 18, wherein said brightness peak threshold value is determined for each given said spectral color filter based upon a density of bright spots in an adjoining area of said image.

20. The automatic vehicle equipment control system of claim 1 wherein said processor is further configured to perform edge preserving color interpolation to provide pixel color and luminance information for objects that are large enough to span several contiguous said pixels in said image.

21. The automatic vehicle equipment control system of claim 1, wherein said processor is further configured to acquire pixel data for said pixels in a neighborhood of a targeted pixel location and to share such data to perform separate color interpolation and brightness peak detection functions.

22. The automatic vehicle equipment control system of claim 1, wherein said processor is further configured to detect at least one brightness peak in said image.

23. The automatic vehicle equipment control system of claim 22, wherein at least two different spectral filter colored pixels are in communication with at least a portion of the same compare circuits, said compare circuits configured for compare operations in two different pixel array configurations.

24. The automatic vehicle equipment control system of claim 22, wherein said processor is further configured to screen out at least a portion of said brightness peaks based upon said brightness peaks being an imaged reflection of a light source.

25. The automatic vehicle equipment control system of claim 22, wherein said processor is further configured to indicate at least a portion of said brightness peaks to be further processed based upon said bright spot being a local peak or plateau in intensity.

26. The automatic vehicle equipment control system of claim 22, wherein said brightness peak is where a pixel value of a given said colored pixel is greater than or equal to pixel values of proximate like colored pixels in assigned directions.

27. The automatic vehicle equipment control system of claim 26, wherein said proximate like colored pixels comprises approximately eight neighboring like colored pixels in at least one of vertical, horizontal, and diagonal directions.

28. The automatic vehicle equipment control system of claim 22, wherein a brightness peak threshold value used to screen brightness peaks is variable.

29. The automatic vehicle equipment control system of claim 28, wherein said variable brightness peak threshold value is increased based upon increased density of bright spots proximate a bright spot being screened.

30. The automatic vehicle equipment control system of claim 22, wherein said processor is further configured to group a plurality of brightness peaks.

31. The automatic vehicle equipment control system of claim 30, wherein said brightness peaks are grouped based upon a determination that said brightness peaks are members of a connected set of pixels in said pixel array that exceed said brightness peak threshold value.

32. The automatic vehicle equipment control system of claim 1, wherein said processor is further configured to map at least a portion of said pixels to reshape an image by spatial transformation.

33. The automatic vehicle equipment control system of claim 32, wherein said reshaping of said image is utilized to at least one of provide a top-down view, correcting lens distortion, and stitching multiple images obtained from a plurality of imagers.

34. The automatic vehicle equipment control system of claim 1, wherein said processor is further configured to detect and screen red brightness peaks according to a variable threshold value that is adjusted according to a position of said red brightness peak in said image and a frequency of occurrence of said red brightness peak in zones of said image.

35. The automatic vehicle equipment control system of claim 1, wherein said processor is further configured to detect and prioritize brightness peaks in said image, such that at least a portion of said brightness peaks having a higher prioritization are processed prior to at least a portion of said brightness peaks having a lower prioritization.

36. The automatic vehicle equipment control system of claim 35, wherein said processor is further configured, for each brightness peak that is detected, a direction of a brightest adjacent pixel of each of the other said colored pixels is determined.

37. The automatic vehicle equipment control system of claim 36, wherein said processor is further configured to generate an importance weighting of pixels having peak readings that are reported based upon a sum of individual weighting factors that include at least one of weightings factors for a row position of a peak in an image array, a column position of said peak in said image array, an approximate logarithm of a brightness of said peak, and a magnitude of said peak relative to that of adjacent red and blue pixels.

38. The automatic vehicle equipment control system of claim 1, wherein said frame rate of said imager is selectable.

39. The automatic vehicle equipment control system of claim 1, wherein said imager is configured to have said frame rate of approximately 20 to 25 frames per second.

40. The automatic vehicle equipment control system of claim 39, wherein said frame rate is such that an alternating current (AC) light source operating at both approximately 60 Hz and 50 Hz is detectable.

41. The automatic vehicle equipment control system of claim 1, wherein said imager is configured to have a non-harmonic frequency frame rate with respect to an alternating current (AC) light source.

42. The automatic vehicle equipment control system of claim 41, wherein said non-harmonic frequency frame rate is a beat pattern in which a signal for a sub-sampled flicker frequency oscillates at a rate approximately equal to one half of said frame rate and a phase of said signal is inverted during each beat frequency half cycle phase reversal period.

43. The automatic vehicle equipment control system of claim 1, wherein said imager is configured to have a beat frequency frame rate with respect to an alternating current (AC) light source, such that a repetition rate of image acquisition for said beat frequency is approximately half a cycle period that is approximately two image acquisition periods so that during said beat frequency half cycle period, a phase of an alternating acquisition pattern shifts by approximately 180 degrees relative to a flicker cycle.

44. The automatic vehicle equipment control system of claim 1, wherein said processor is further configured to locate brightness peaks in said image, grouping peaks to said light source, tracking a history of said light source over a plurality of frames, calculating an intensity based on a sum of intensities of brighter pixels related to said light source in each said image, classifying said light source, and generating a control signal based upon said classified light source.

45. The automatic vehicle equipment control system of claim 1 further comprising at least one light sensor having a different field of view from said at least one imager, wherein said imager has an integration period that is at least partly a function of light detected by said light sensor.

46. The automatic vehicle equipment control system of claim 1, wherein said at least one imager comprises a first imager having a field of view approximately forward of a controlled vehicle and a second imager having a field of view approximately rearward of said controlled vehicle.

47. The automatic vehicle equipment control system of claim 1, wherein said at least one imager is configured for use in at least one system comprising lane departure warning, adaptive cruise control, collision avoidance, traffic sign recognition, automatic headlamp control, pedestrian detection, occupant identification, drowsiness detection, smart airbag deployment, vehicle theft identification, drive-by wire, and blind spot detection.

48. The automatic vehicle equipment control system of claim 1, wherein said processor is further configured to detect a plurality of lane markers in said image, and determine a change ratio of lane marker curvature to detect additional lane markers.

49. The automatic vehicle equipment control system of claim 1, wherein said processor is further configured to detect lane markers, such that said colored spectral filter array comprising at least one blue filter, at least one green filter, and at least one red filter, wherein a low blue pixel value relative to a green pixel value and a red pixel value indicates a detection of a yellow lane marker.

50. The automatic vehicle equipment control system of claim 1, wherein said processor is further configured to perform color interpolation that is based upon an analysis of said image data to detect edge patterns, wherein interpolated values are a function of said detected edge patterns.

51. The automatic vehicle equipment control system of claim 1, wherein said processor is further configured to perform an interpolation equation to calculate a missing color component based upon an edge pattern detection using pixel values from a neighborhood of an interpolation site that extends beyond a three by three neighborhood of said interpolation site, and said interpolation equation further based upon values of pixels with said three by three neighborhood of a pixel for which a missing color component is being calculated.

52. The automatic vehicle equipment control system of claim 1, wherein said processor is further configured to calculate color component values for at least a portion of said pixels in said pixel array.

53. The automatic vehicle equipment control system of claim 52, wherein said processor is further configured to calculate grayscale values related to a luminance for at least a portion of said pixels in said pixel array.

54. The automatic vehicle equipment control system of claim 53, wherein said processor is further configured to modify grayscale values for at least a portion of pixels of a selected color by increasing their values relative to non-yellow pixels in order to facilitate their detection as lane markings.

55. The automatic vehicle equipment control system of claim 1, wherein said processor is further configured to double a grayscale value of at least a portion of yellow pixels in said image that are analyzed to detect lane markers.

56. The automatic vehicle equipment control system of claim 1, wherein said processor is further configured to adjust a color detection criteria based upon light conditions.

57. The automatic vehicle equipment control system of claim 56, wherein said processor is further configured to determine that a selected pixel is yellow when color ratios of various color component values for said pixel with respect to other selected color component values for the same said pixel are within a predetermined range.

58. The automatic vehicle equipment control system of claim 56, wherein said light conditions comprises one of direct sunlight, shaded area, and a headlamp.

59. The automatic vehicle equipment control system of claim 56, wherein said processor is further configured to detect a cloud type based upon said image of the sky as a function of at least one of color, pixel intensity, and spatial variation in pixel intensity.

60. The automatic vehicle equipment control system of claim 56, wherein said processor is further configured to utilize color of pixels from images of lane markings to adjust color detection thresholds to increase inclusion of such colors.

61. The automatic vehicle equipment control system of claim 1, wherein said processor is further configured to replace a denominator term that requires a repetitious division by a calculated adjustment to an exponent that is part of a calculation to provide an approximation to an original equation in order to eliminate said repetitious division.

62. The automatic vehicle equipment control system of claim 59, wherein said processor is further configured to utilize a scaling factor that reduces the number of pixel values that are saturated after tone mapping.

63. The automatic vehicle equipment control system of claim 1, wherein said processor is further configured to format pixel data to provide a luminance component and two compact color components that are substantially independent of luminance.

64. The automatic vehicle equipment control system of claim 1, wherein said processor is further configured to automatically select a color conversion algorithm based upon an external bit of information that comprises an amount of detected ambient light.

65. The automatic vehicle equipment control system of claim 1, wherein said processor is further configured to automatically select a color conversion algorithm based upon at least one of whether image data is being displayed, automatic vehicle exterior light control, lane detection, lane tracking, lane keeping, and blind spot warning.

66. The automatic vehicle equipment control system of claim 1, wherein said processor is further configured to alter a level of image enhancement applied as a result of a detected color based upon at least one of an assessment of a quality of color detection and a benefit of applying a color enhancement.

67. An automatic vehicle equipment control system comprising:

at least one imager configured to acquire a continuous sequence of high dynamic range single frame images, and said at least one imager comprising a pixel array;

a color spectral filter array in optical communication with said at least one imager, said color spectral filter array comprising a plurality of color filters, at least a portion of which are different colors, and pixels of said pixel array being in optical communication with substantially one said spectral color filter;

a lens in optical communication with said at least one imager and said color spectral filter array; and

a processor in communication with said at least one imager, and configured to process said continuous sequence of high dynamic range images, wherein said processor is further configured to use color information from pixels of said pixel array to enhance yellow colored features in said high dynamic range images to detect yellow lane markers.

68. An automatic vehicle equipment control system comprising:

at least one imager configured to acquire a continuous sequence of high dynamic range single frame images, and said at least one imager comprising a pixel array;

a color spectral filter array in optical communication with said at least one imager, said color spectral filter array comprising a plurality of color filters, at least a portion of which are different colors, and pixels of said pixel array being in optical communication with substantially one said spectral color filter;

a lens in optical communication with said at least one imager and said color spectral filter array; and

a processor in communication with said at least one imager, wherein said processor is further configured to aggregate a plurality of bright pixels into a single light source object in at least one image of said continuous sequence of high dynamic range images.

69. An automatic vehicle equipment control system comprising:

at least one imager configured to acquire a continuous sequence of high dynamic range single frame images, and said at least one imager comprising a pixel array;

a color spectral filter array in optical communication with said at least one imager, said color spectral filter array comprising a plurality of color filters, at least a portion of which are different colors, and each said pixel being in optical communication with substantially one said spectral color filter;

a lens in optical communication with said at least one imager and said color spectral filter array; and

a processor in communication with said at least one imager, said processor being configured to detect an alternating current (AC) light source in said continuous sequence of high dynamic range images, wherein said at least one imager is configured to have a sampling image repetition period that is longer than that of a flicker cycle period of said AC light source.

70. A method of generating a lane departure warning, said method comprising the steps of:

populating a world coordinate system grid for incoming pixel data;

scoring marker points;

processing said scored marker points;

determining a type of lane lines based upon said processed and scored marker points;

computing a vehicle departure from said lane lines; and

determining if a warning should be emitted.

71. The method of claim 70, wherein said step of populating a world coordinate system for incoming pixel data further comprises utilizing luminance information.

72. The method of claim 70, wherein said step of populating a world coordinate system for incoming pixel data further comprises bilinear interpolating when sample grid points translate to fractional coordinates in an image plane.

73. The method of claim 70, wherein said step of scoring marker points comprises calculating a cross correlation error for a series of grid sample points.

74. The method of claim 70, wherein said step of processing said scored marker points comprises classifying each lane line as one of an unknown type, a single type, a single-dashed type, a single-unknown type, a double type, a double-dashed-left type, a double-dashed-right type, and a double-unknown type.

75. The method of claim 70, wherein said step of processing said scored marker points comprises classifying each line based upon a number of points found along an edge of a lane marking and a displacement of points along said edge of said lane markings.

76. The method of claim 70, wherein said step of determining a type of lane line further comprises determining if said lane line is one of a double line, a solid line, and a dashed line.

77. The method of claim 70 further comprising the step of coping a history of said line detection from a left side into a right side when a controlled vehicle approximately completely crosses said line from right to left, and coping a history of said line detection from said right side into said left side when said controlled vehicle approximately completely crosses said line from left to right.

78. The method of claim 70, wherein said step of determining if a warning should be emitted further comprises evaluating at least one of a controlled vehicle'"'"'s speed, a turn signal switch position, a yaw rate of said controlled vehicle, and an output a lane departure warning application.

79. The method of claim 70 further comprising the step of creating a lookup table utilized to translate an image coordinate system to a world coordinate system;

80. A method for generating a control signal for vehicle headlamp control, said method comprising the steps of:

extracting a list of light objects;

combining single-peak light sources into multi-peak light objects;

connecting current light sources to light sources contained in previous images;

calculating at least one light source attribute; and

determining an appropriate state of a vehicle'"'"'s headlamp.

81. The method of claim 80, wherein said step of extracting a list of light objects further comprises a possible points of interest (POPI) harvest and a POPI blossom refinement.

82. The method of claim 81, wherein said POPI harvest comprises parsing said pre-processed encoded output, and creating a list of POPI blossom objects.

83. The method of claim 82, wherein said POPI blossom refinement comprises passing said list of POPI blossom objects to at least one registered refining factory.

84. The method of claim 83, wherein said at least one refining factory allocates light source objects, and populates a single peak light list that contains only a single pre-processed image located peak.

85. The method of claim 84, wherein said at least one registered refining factory determines at least one property of said light source, said at least one property comprising at least one of a red/white ratio, a peak pixel within said light source, a slope within a 5×

5 array of pixels surrounding and including said peak pixel with said light source, a total grey scale value, past frame light source position, present frame light source position, motion, width, height, area, age, and growth.

86. The method of claim 85 further comprising the step of calculating at least one scene attribute for an image frame based upon at least one of said light source attribute and light source history.