Vehicular monitoring systems using image processing

• US 6,772,057 B2
• Filed: 11/22/2002
• Issued: 08/03/2004
• Est. Priority Date: 06/07/1995
• Status: Expired due to Fees

• Current Assignee: American Vehicular Sciences LLC (Acacia Research Corporation)
• Original Assignee: Automotive Technologies International Incorporated
• Inventors: Johnson, Wendell C., Breed, David S., DuVall, Wilbur E.
• Primary Examiner(s): Black, Thomas G.
• Assistant Examiner(s): To, Tuan C
• Application Number:
  US10/302,105
• Publication Number:
  US 20030125855A1
• Time in Patent Office: 620 Days
• Field of Search: 701/45, 701/36, 701/301, 340/461, 340/435, 340/436, 340/438, 340/815.4, 340/573.1, 348/148, 348/77, 348/154, 180/271, 280/735, 280/728.1, 382/181, 382/115, 382/190, 382/224, 382/100, 382/104
• US Class Current: 701/45
• CPC Class Codes: B60N 2/002 Seats provided with an occupancy detection means mounted therein or thereon detection means responsive to presence or absence of children B60N2/266
• CPC Class Codes: B60N 2/02246 Electric motors therefor
• CPC Class Codes: B60N 2/0244 with logic circuits
• CPC Class Codes: B60N 2/0268 using sensors or detectors for adapting the seat or seat part, e.g. to the position of an occupant
• CPC Class Codes: B60N 2/0272 using sensors or detectors for detecting the position of seat parts
• CPC Class Codes: B60N 2/0276 reaction to emergency situations, e.g. crash
• CPC Class Codes: B60N 2/28 Seats readily mountable on, and dismountable from, existing seats or other parts of the vehicle
• CPC Class Codes: B60N 2/2806 for securing the child seat to the vehicle
• CPC Class Codes: B60N 2/2863 backward facing
• CPC Class Codes: B60Q 1/143 combined with another condition, e.g. using vehicle recognition from camera images or activation of wipers
• CPC Class Codes: B60Q 2300/41 preceding vehicle
• CPC Class Codes: B60Q 2300/42 oncoming vehicle
• CPC Class Codes: B60R 2022/208 by automatic or remote control means
• CPC Class Codes: B60R 21/013 including means for detecting collisions, impending collisions or roll-over
• CPC Class Codes: B60R 21/0134 responsive to imminent contact with an obstacle , e.g. using radar systems
• CPC Class Codes: B60R 21/0152 using strain gauges
• CPC Class Codes: B60R 21/01534 using electromagneticwaves, e.g. infrared
• CPC Class Codes: B60R 21/01536 using ultrasonic waves
• CPC Class Codes: B60R 21/01538 for image processing, e.g. cameras or sensor arrays
• CPC Class Codes: B60R 21/0154 in combination with seat heating
• CPC Class Codes: B60R 25/255 : Eye recognition
• CPC Class Codes: G01S 15/87 : Combinations of sonar systems
• CPC Class Codes: G01S 15/88 : Sonar systems specially adapted for specific applications seismic or acoustic prospecting or detecting G01V1/00
• CPC Class Codes: G01S 17/88 : Lidar systems specially adapted for specific applications
• CPC Class Codes: G06V 20/59 : inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
• CPC Class Codes: G06V 40/10 : Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands

Abstract

Vehicular monitoring arrangement for monitoring an environment of the vehicle including at least one active pixel camera for obtaining images of the environment of the vehicle and a processor coupled to the active pixel camera(s) for determining at least one characteristic of an object in the environment based on the images obtained by the active pixel camera(s). The active pixel camera can be arranged in a headliner, roof or ceiling of the vehicle to obtain images of an interior environment of the vehicle, in an A-pillar or B-pillar of the vehicle to obtain images of an interior environment of the vehicle, or in a roof, ceiling, B-pillar or C-pillar of the vehicle to obtain images of an interior environment of the vehicle behind a front seat of the vehicle. The determined characteristic can be used to enable optimal control of a reactive component, system or subsystem coupled to the processor. When the reactive component is an airbag assembly including at least one airbag, the processor can be designed to control at least one deployment parameter of the airbag(s).

Claims

1. A monitoring arrangement for monitoring an environment exterior of a vehicle, comprising:

at least one receiver arranged to receive waves from the environment exterior of the vehicle which contain information on any objects in the environment and generate a signal characteristic of the received waves; and

a processor coupled to said at least one receiver and comprising trained pattern recognition means for processing the signal to provide a classification, identification or location of the exterior object, said trained pattern recognition means being structured and arranged to apply a trained pattern recognition algorithm generated from data of possible exterior objects and patterns of received waves from the possible exterior objects to provide the classification, identification or location of the exterior object;

whereby a system in the vehicle is coupled to said processor such that the operation of the system is affected in response to the classification, identification or location of the exterior object.

2. The arrangement of claim 1, wherein said at least one receiver comprises a pair of receivers spaced apart from one another.

3. The arrangement of claim 1, wherein said at least one receiver is arranged to receive infrared waves.

4. The arrangement of claim 1, wherein the monitoring arrangement further comprises a transmitter for transmitting waves into the environment exterior of the vehicle whereby said at least one receiver is arranged to receive waves transmitted by said transmitter and reflected by any exterior objects.

5. The arrangement of claim 1, wherein said trained pattern recognition means comprise a neural computer.

6. The arrangement of claim 1, wherein said trained pattern recognition means comprise a neural network.

7. The arrangement of claim 1, wherein the another system is a display viewable by the driver and arranged to show an image or icon of the exterior object.

8. The arrangement of claim 1, wherein said at least one receiver is a CCD array.

9. The arrangement of claim 1, further comprising measurement means for measuring a distance between the exterior object and the vehicle.

10. The arrangement of claim 9, wherein said measurement means comprise a radar or laser radar system.

11. The arrangement of claim 1, wherein the another system is an airbag, said pattern recognition means being structured and arranged to apply the pattern recognition algorithm to provide the identification of the exterior object, deployment of said airbag being controlled based on the identification of the exterior object.

12. The arrangement of claim 11, wherein the airbag is an externally deployed airbag.

13. The arrangement of claim 1, wherein said processor is arranged to provide the classification of the exterior object.

14. The arrangement of claim 1, wherein said processor is arranged to provide the identification of the exterior object.

15. The arrangement of claim 1, wherein said processor is arranged to provide the location of the exterior object.

16. A monitoring arrangement for monitoring an environment exterior of a vehicle, comprising:

at least one CCD array positioned to obtain images of the environment exterior of the vehicle; and

a processor coupled to said at least one CCD array and comprising trained pattern recognition means for processing the images obtained by said at least one CCD array to provide a classification, identification or location of the exterior object,

whereby a system in the vehicle is coupled to said processor such that the operation of the system is affected in response to the classification, identification or location of the exterior object.

17. The arrangement of claim 16, wherein said pattern recognition means are structured and arranged to apply a pattern recognition algorithm generated from data of possible exterior objects and images from said at least one CCD array of the possible exterior objects.

18. The arrangement of claim 16, wherein said at least one CCD array comprises a pair of CCD arrays spaced apart from one another.

19. The arrangement of claim 16, wherein the monitoring arrangement further comprises a transmitter for transmitting infrared waves into the environment exterior of the vehicle.

20. The arrangement of claim 16, wherein said pattern recognition means comprise a neural computer.

21. The arrangement of claim 16, wherein said pattern recognition means comprise a neural network.

22. The arrangement of claim 16, wherein the another system is a display viewable by the driver and arranged to show an image or icon of the exterior object.

23. The arrangement of claim 16, further comprising measurement means for measuring a distance between the exterior object and the vehicle.

24. The arrangement of claim 23, wherein said measurement means comprise a radar or laser radar system.

25. The arrangement of claim 16, wherein the another system is an airbag, said trained pattern recognition means being structured and arranged to process the images obtained by said at least one CCD array to provide the identification of the exterior object, deployment of said airbag being controlled based on the identification of the exterior object.

26. The arrangement of claim 25, wherein the airbag is an externally deployed airbag.

27. The arrangement of claim 16, wherein said processor is arranged to provide the classification of the exterior object.

28. The arrangement of claim 16, wherein said processor is arranged to provide the identification of the exterior object.

29. The arrangement of claim 16, wherein said processor is arranged to provide the location of the exterior object.

30. A vehicle including a monitoring arrangement for monitoring an environment exterior of the vehicle, the monitoring arrangement comprising:

at least one receiver arranged on a rear view mirror of the vehicle to receive waves from the environment exterior of the vehicle which contain information on any objects in the environment and generate a signal characteristic of the received waves; and

a processor coupled to said at least one receiver and arranged to classify or identify the exterior object based on the signal and thereby provide the classification or identification of the exterior object;

whereby a system in the vehicle is coupled to said processor such that the operation of the system is affected in response to the classification or identification of the exterior object.

31. The vehicle of claim 30, wherein said processor comprises trained pattern recognition means for processing the signal to provide the classification or identification of the exterior object, said trained pattern recognition means being structured and arranged to apply a pattern recognition algorithm generated from data of possible exterior objects and patterns of received waves from the possible exterior objects.

32. The vehicle of claim 30, wherein said at least one receiver comprises a pair of receivers spaced apart from one another.

33. The vehicle of claim 30, wherein said at least one receiver is arranged to receive infrared waves.

34. The vehicle of claim 30, wherein the monitoring arrangement further comprises a transmitter for transmitting waves into the environment exterior of the vehicle whereby said at least one receiver is arranged to receive waves transmitted by said transmitter and reflected by any exterior objects.

35. The vehicle of claim 31, wherein said trained pattern recognition means comprise a neural computer.

36. The vehicle of claim 31, wherein said trained pattern recognition means comprise a neural network.

37. The vehicle of claim 30, wherein said at least one receiver is a CCD array.

38. The vehicle of claim 30, further comprising measurement means for measuring a distance between the exterior object and the vehicle.

39. The vehicle of claim 38, wherein said measurement means comprise a radar or laser radar system.

40. A monitoring arrangement for monitoring an environment exterior of a vehicle, comprising:

a plurality of receivers arranged apart from one another and to receive waves from different parts of the environment exterior of the vehicle which contain information on any objects in the environment and generate a signal characteristic of the received waves; and

a processor coupled to said receivers and arranged to classify, identify or locate the exterior object based on the signals generated by said receivers and thereby provide the classification identification or location of the exterior object,

whereby a system in the vehicle is coupled to said processor such that the operation of the system is affected in response to the classification, identification or location of the exterior object.

41. The arrangement of claim 40, wherein said processor comprises trained pattern recognition means for processing the signal to provide the classification, identification or location of the exterior object, said trained pattern recognition means being structured and arranged to apply a trained pattern recognition algorithm generated from data of possible exterior objects and patterns of received waves from the possible exterior objects.

42. The arrangement of claim 40, wherein said receivers are arranged to receive infrared waves.

43. The arrangement of claim 40, wherein the monitoring arrangement further comprises a transmitter for transmitting waves into the environment exterior of the vehicle whereby said receivers are arranged to receive waves transmitted by said transmitter and reflected by any exterior objects.

44. The arrangement of claim 41, wherein said trained pattern recognition means comprise a neural computer.

45. The arrangement of claim 41, wherein said trained pattern recognition means comprise a neural network.

46. The arrangement of claim 40, wherein the another system is a display viewable by the driver and arranged to show an image or icon of the exterior object.

47. The arrangement of claim 40, wherein said receivers are CCD arrays.

48. The arrangement of claim 40, further comprising measurement means for measuring a distance between the exterior object and the vehicle.

49. The arrangement of claim 48, wherein said measurement means comprise a radar or laser radar system.

50. The arrangement of claim 40, wherein the another system is an airbag, said processor being arranged to identify the exterior object to provide an identification of the exterior object, said output means being arranged to control deployment of said airbag based on the identification of the exterior object.

51. The arrangement of claim 50, wherein the airbag is an externally deployed airbag.

52. The monitoring arrangement of claim 40, wherein said processor is arranged to classify or identify the exterior object based on the signals generated by said receivers and thereby provide the classification or identification of the exterior object.

53. A vehicle including a monitoring arrangement for monitoring an environment exterior of the vehicle, the monitoring arrangement comprising:

at least one receiver arranged on a B-pillar of the vehicle to receive waves from the environment exterior of and on the side of the vehicle which contain information on any objects in the environment and generate a signal characteristic of the received waves; and

a processor coupled to said at least one receiver and arranged to classify, identify or locate the exterior object based on the signal and thereby provide the classification, identification or location of the exterior object;

whereby a system in the vehicle is coupled to said processor such that the operation of the system is affected in response to the classification, identification or location of the exterior object.

54. A vehicle including a monitoring arrangement for monitoring an environment exterior of the vehicle, the monitoring arrangement comprising:

at least one receiver arranged in a door window trim panel of the vehicle to receive waves from the environment exterior of the vehicle which contain information on any objects in the environment and generate a signal characteristic of the received waves; and

a processor coupled to said at least one receiver and arranged to classify, identify or locate the exterior object based on the signal and thereby provide the classification, identification or location of the exterior object;

whereby a system in the vehicle is coupled to said processor such that the operation of the system is affected in response to the classification, identification or location of the exterior object.

55. A vehicle including a monitoring arrangement for monitoring an environment exterior of the vehicle, the monitoring arrangement comprising:

at least one receiver arranged on a rear view mirror of the vehicle to receive waves from the environment exterior of the vehicle which contain information on any objects in the environment and generate a signal characteristic of the received waves;

a processor coupled to said at least one receiver and arranged to at least one of classify, identify and locate the exterior object based on the signal and thereby provide at least one of the classification, identification and location of the exterior object; and

a display viewable by the driver;

said display being coupled to and controlled by said processor to cause the display the presence and at least one of the classification, identification and location of the exterior object on said display.

56. A vehicle including a monitoring arrangement for monitoring an environment exterior of the vehicle, the monitoring arrangement comprising:

at least one receiver arranged to receive waves from the environment exterior of the vehicle which contain information on any objects in the environment and generate a signal characteristic of the received waves; and

a processor coupled to said at least one receiver and comprising trained pattern recognition means for processing the signal to provide a classification, identification or location of the exterior object, said trained pattern recognition means being structured and arranged to apply a trained pattern recognition algorithm generated from data of possible exterior objects and patterns of received waves from the possible exterior objects to provide the classification, identification or location of the exterior object;

whereby a system in the vehicle is coupled to said processor such that the operation of the system is affected in response to the classification, identification or location of the exterior object.

57. The vehicle of claim 56, wherein said at least one receiver is arranged on a B-pillar of the vehicle to receive waves from a side of the vehicle.

58. The vehicle of claim 56, wherein said at least one receiver is arranged in a door window trim panel.

59. The vehicle of claim 56, wherein the monitoring arrangement further comprises a transmitter for transmitting waves into the environment exterior of the vehicle whereby said at least one receiver is arranged to receive waves transmitted by said transmitter and reflected by any exterior objects.

60. The vehicle of claim 56, wherein said at least one receiver is arranged to receive waves from a blind spot of the vehicle.

61. The vehicle of claim 56, wherein the another system is a display viewable by the driver and arranged to show an image or icon of the exterior object.

62. The vehicle of claim 56, wherein said at least one receiver is mounted on a rear view mirror or in a rear window.

63. The vehicle of claim 56, wherein said at least one receiver is mounted in a C-pillar of the vehicle.

64. The vehicle of claim 56, further comprising measurement means for measuring a distance between the exterior object and the vehicle.

65. The vehicle of claim 56, wherein the another system is an airbag, said trained pattern recognition means being structured and arranged to apply the pattern recognition algorithm to provide the identification of the exterior object, deployment of said airbag being controlled based on the identification of the exterior object.

66. A vehicle including a monitoring arrangement for monitoring an environment exterior of the vehicle, the monitoring arrangement comprising:

at least one CCD array positioned to obtain images of the environment exterior of the vehicle; and

a processor coupled to said at least one CCD array and comprising trained pattern recognition means for processing the images obtained by said at least one CCD array to provide a classification, identification or location of the exterior object, said trained pattern recognition means being structured and arranged to apply a trained pattern recognition algorithm generated from data of possible exterior objects and images obtained by said at least one CCD array from the possible exterior objects to provide the classification, identification or location of the exterior object;

whereby a system in the vehicle is coupled to said processor such that the operation of the system is affected in response to the classification, identification or location of the exterior object.

67. The vehicle of claim 66, wherein said at least one CCD array is arranged on a B-pillar of the vehicle to obtain images from a side of the vehicle.

68. The vehicle of claim 66, wherein said at least one CCD array is arranged in a door window trim panel.

69. The vehicle of claim 66, wherein said at least one CCD array comprises a pair of CCD arrays spaced apart from one another.

70. The vehicle of claim 66, wherein the monitoring arrangement further comprises a transmitter for transmitting infrared waves into the environment exterior of the vehicle.

71. The vehicle of claim 66, wherein said at least one CCD array is arranged to obtain images from a blind spot of the vehicle.

72. The vehicle of claim 66, wherein the another system is a display viewable by the driver and arranged to show an image or icon of the exterior object.

73. The vehicle of claim 66, wherein said at least one CCD array is mounted on a rear view mirror or in a rear window.

74. The vehicle of claim 66, wherein said at least one CCD array is mounted in a C-pillar of the vehicle.

75. The vehicle of claim 66, further comprising measurement means for measuring a distance between the exterior object and the vehicle.

76. The vehicle of claim 66, wherein the another system is an airbag, said trained pattern recognition means being arranged to process the images obtained by said at least one CCD array to provide the identification of the exterior object, deployment of said airbag being controlled based on the identification of the exterior object.

77. A vehicle including a monitoring arrangement for monitoring an environment exterior of the vehicle, the monitoring arrangement comprising:

a plurality of receivers arranged apart from one another and to receive waves from different parts of the environment exterior of the vehicle which contain information on any objects in the environment and generate a signal characteristic of the received waves; and

a processor coupled to said receivers and arranged to classify, identify or locate the exterior object based on the signals generated by said receivers to provide the classification, identification or location of the exterior object,

whereby a system in the vehicle is coupled to said processor such that the operation of the system is affected in response to the classification, identification or location of the exterior object.

78. The vehicle of claim 77, wherein said processor comprises trained pattern recognition means for processing the signal to provide the classification, identification or location of the exterior object, said pattern recognition means being structured and arranged to apply a pattern recognition algorithm generated.

79. The vehicle of claim 77, wherein said receivers are arranged on a B-pillar of the vehicle to receive waves from a side of the vehicle.

80. The vehicle of claim 77, wherein said receivers are arranged to receive infrared waves.

81. The vehicle of claim 77, wherein the monitoring arrangement further comprises a transmitter for transmitting waves into the environment exterior of the vehicle whereby said receivers are arranged to receive waves transmitted by said transmitter and reflected by any exterior objects.

82. The vehicle of claim 77, wherein said receivers are arranged to receive waves from a blind spot of the vehicle.

83. The vehicle of claim 77, wherein the another system is a display viewable by the driver and arranged to show an image or icon of the exterior object.

84. The vehicle of claim 77, further comprising measurement means for measuring a distance between the exterior object and the vehicle.

85. The vehicle of claim 77, wherein the another system is an airbag, said processor being arranged to identify the exterior object to provide the identification of the exterior object, said output means being arranged to control deployment of said airbag based on the identification of the exterior object.

86. The vehicle of claim 77, wherein said processor is arranged to classify or identify the exterior object based on the signals generated by said receivers and thereby provide the classification or identification of the exterior object.