REAL TIME ENVIRONMENT MODEL GENERATION SYSTEM

• US 20090268947A1
• Filed: 03/31/2009
• Published: 10/29/2009
• Est. Priority Date: 03/31/2008
• Status: Active Grant

• Current Assignee: Harman Becker Automotive Systems GmbH (Samsung Electronics Co. Ltd.)
• Original Assignee: Harman Becker Automotive Systems GmbH (Samsung Electronics Co. Ltd.)
• Inventors: Schaufler, Roland
• Granted Patent: US 8,379,924 B2
• Time in Patent Office: Days
• US Class Current: 382/104
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads

Abstract

A vehicle environment monitoring system is provided that is based on a three-dimensional vector model. The three-dimensional vector model of the vehicle'"'"'s environment is generated on the basis of the image data captured by at least one three-dimensional camera. Out of the image data, particular data are extracted for generating the three-dimensional vector model in order to reduce the data volume. For data extraction, a data extraction algorithm is applied that is determined in accordance with at least one parameter that relates to the situation of the vehicle. Therefore, targeted data extraction is performed for generating a three-dimensional model that is particularly adapted for an application that is desired in the current vehicle situation. The applications of the vector model include driver assistance, external monitoring and vehicle control, as well as recording in an event data recorder. In one implementation, a sequence of three-dimensional vector models, representing a three-dimensional space-and-time model, is generated.

Claims

1. A method of monitoring the environment of a camera-equipped vehicle, comprising the steps of:

capturing a 3D-image of the environment of the vehicle, the 3D-image representing a predetermined area of the vehicle environment;

reducing the amount of information acquired in said image capturing step by extracting data from said 3D-image employing a data extraction algorithm; and

generating a 3D-vector model of the vehicle environment from said data extracted from said 3D-image characterized by the step of;

determining the data extraction algorithm to be employed in said reducing step based on at least one parameter characterizing the vehicle situation.

2. The method of claim 1, where said extraction algorithm determining step determines the data extraction algorithm to be adapted according to the processing time available for generating the 3D-vector model.

3. The method of claim 1, where said extraction algorithm determining step determines the data extraction algorithm to be adapted according to information reflected in the 3D-vector model.

4. The method of claim 1, where said data extraction algorithm includes object recognition.

6. The method of claim 1, where said data extraction algorithm extracts data only from a part of the 3D-image corresponding to a certain visual field.

7. The method of claim 1, further comprising the step of further processing said 3D-vector model for assisting a driver in controlling the vehicle based on the processing result.

8. The method of claim 7, where said extraction algorithm determining step includes the step of determining a maximum available amount of time for performing said steps of generating and further processing said 3D-vector model based on the vehicle situation and further takes into account said maximum available amount of time to determine said data extraction algorithm.

12. The method of claim 1, where said extraction algorithm determining step further includes the step of analyzing the vehicle situation based on the captured 3D-image, where a value of said at least one parameter is obtained from said analyzing step.

13. The method of claim 1, further comprising the step of detecting vehicle status information, where the detected vehicle status information includes a value of said at least one parameter.

14. The method of claim 13, where said vehicle status information includes velocity information of said vehicle, and where the size of the visual field from which said data extraction algorithm extracts data decreases as the velocity of the vehicle increases.

17. The method of claim 1, where said extraction algorithm determining step includes the step of selecting a particular data extraction algorithm out of a plurality of data extraction algorithms based on a parameter describing the vehicle situation.

18. The method of claim 1, where said 3D-vector model represents objects in the environment of the vehicle by vectors and object classification information, said object classification information associates an object with standard object classes of an object model.

19. The method of claim 18, where said object model is pre-stored in a database containing standard information of such objects that are most likely to occur and most relevant to the vehicle'"'"'s environment.

21. The method of claim 1, further comprising the step of:

generating a 3D-space-and-time model, where said 3D-space-and-time model includes a sequence of 3D-vector models.

23. The method of claim 21, further comprising the step of generating a response to the information of the vehicle environment represented by said 3D-space-and-time model.

24. The method of claim 23, where said response beg is directed towards avoiding an emergency situation to be expected by a driver.

27. The method of claim 24, where said response includes an automatic interference with the vehicle control.

28. The method of claim 23, where said response is directed towards assisting a driver in parking the vehicle.

29. The method of claim 23, where said response includes taking countermeasures to minimize the consequences of a threatening accident.

31. The method of claim 1, where said image capturing step captures a plurality of 3D-images from various camera positions, where said plurality of 3D-images re-present an environment area including a predetermined range around the vehice.

33. The method of claim 1, where said 3D-vector model generating step further employs data from sensors.

34. The method of claim 33, further comprising the step of sensor fusion to transform sensor data into a unique format and remove inconsistent and/or redundant data.

36. The method of claim 1, further comprising the step of transmitting the 3D vector model to an external location with respect to said vehicle.

38. The method of claim 36, where said 3D-vector model is transmitted via a wireless communication.

39. The method of claim 36, where said 3D-vector model is transmitted to and visualized on an external computer.

40. The method of claim 36, further comprising the step of evaluating said 3D-vector model at an external computer.

41. The method of claim 40, further comprising the step of re-transmitting results of said evaluation back to the vehicle.

42. The method of claim 40, further comprising the step of remotely controlling the vehicle by employing the evaluation results.

44. The method of claim 36, where said transmitting step transmits a sequence of 3D-vector models representing a 3D-space-and-time model.

45. The method of claim 1, further comprising the step of saving the 3D-vector model in an event data recorder.

46. The method of claim 45, further comprising the steps of:

generating a sequence of 3D-vector models during a predetermined period of time; and

saving said sequence of 3D-vector models into the event data recorder.

47. The method of claim 46, further comprising the step of associating the 3D-vector models with time stamps before saving into the event data recorder.

48. The method of claim 47, where the 3D-vector models are saved for a predetermined storing time;

the method further comprising the step of detecting an impact caused by an extraordinary event, where the 3D-vector models remain permanently saved upon said detection of an impact.

49. The method of claim 48, where said 3D-vector models are stored in a cyclic storage having a predetermined storage capacity;

the method further comprising the step of cyclically overwriting stored 3D-vector models having the earliest time stamp, until said detection of au impact.

5. (canceled)

9. (canceled)

10. The method of claim 9, where said extraction algorithm determining step includes the step of determining a maximum amount of data to be extracted in correspondence with said maximum amount of time.

11. The method of claim 10, where said extraction algorithm determining step further includes the step of determining priorities for different kinds of data that can be extracted from said 3D-image based on the particular relevance of different kinds of data in the vehicle situation characterized by said at least one parameter.

15. (canceled)

16. (canceled)

20. (canceled)

22. (canceled)

25. (canceled)

26. (canceled)

30. (canceled)

32. (canceled)

35. (canceled)

37. (canceled)

43. (canceled)

50. An environment monitoring device for monitoring the environment of a vehicle, comprising:

a 3D-camera for capturing an image of the environment of the vehicle, the image representing a predetermined area of the vehicle environment; and

a first processing unit for processing the information acquired by said 3D-camera, said first processing unit including;

a data extracting unit for extracting data from said image by employing a data extraction algorithm; and

a model generation unit for generating a 3D-vector model of the vehicle environment from said data extracted from said image;

characterized in that said first processing unit further includes a determining unit for determining the data extraction algorithm to be employed by said data extracting unit based on at least one parameter characterizing the vehicle situation.

51. The environment monitoring device of claim 50, where said determining unit determines the data extraction algorithm to be adapted according to the processing time available for generating the 3D-vector model.

52. The environment monitoring device of claim 50, where said determining unit determines the data extraction algorithm to be adapted according to information reflected in the 3D-vector model.

53. The environment monitoring device of claim 50, where said data extraction algorithm includes object recognition.

55. The environment monitoring device of claim 50, where said data extraction algorithm extract data only from a part of the image corresponding to a certain visual field.

56. The environment monitoring device of claim 50, further comprising a second processing unit for further processing said 3D-vector model for assisting a driver in controlling the vehicle based on the processing result.

57. The environment monitoring device of claim 56, where said determining units determines a maximum available amount of time for generating and further processing said 3D-vector model based on the vehicle situation, and further takes into account said maximum available amount of time for determining said data extraction algorithm.

61. The environment monitoring device of claim 50, where said determining unit further analyzes the vehicle situation based on the captured image for obtaining a value of said at least one parameter.

62. The environment monitoring device of claim 50, further comprising a detector for detecting vehicle status information, where the detected vehicle status information includes a value of said at least one parameter.

63. The environment monitoring device of claim 62, where said vehicle status information includes velocity information of said vehicle, and where the size of the visual field from which said data extraction algorithm extracts data decreases as the velocity of the vehicle is increases.

66. The environment monitoring device of claim 50, where said determining unit selects a particular data extraction algorithm out of a plurality of data extraction algorithms based on a parameter describing the vehicle situation.

67. The environment monitoring device of claim 50, where said 3D-vector model represents objects in the environment of the vehicle by vectors and object classification information, said object classification information associates an object with standard object classes of an object model.

68. The environment monitoring device of claim 67, further comprising a database for pre-storing said object model.

69. The environment monitoring device of claim 67, where said extraction algorithm obtains object classification information from the image.

70. The environment monitoring device of claim 50, where said model generation unit generates a 3D-space-and-time model that includes a sequence of 3D-vector models.

72. The environment monitoring device of claim 70, further comprising a second processing unit for generating a response to the information of the vehicle environment represented by said 3D-space-and-time model.

73. The environment monitoring device of claim 72, where said response is directed towards avoiding an emergency situation to be expected by a driver.

74. The environment monitoring device of claim 73, further comprising a device for notifying a driver of an expected emergency situation based on the response generated by said second processing unit.

75. The environment monitoring device of claim 73, further comprising a device for assisting a driver to avoid an emergency situation based on the response generated by said second processing unit.

76. The environment monitoring device of claim 75, where said device is a controller for automatically interfering with the vehicle control based on the response generated by said second processing unit.

77. The environment monitoring device of claim 72, where said response is directed towards assisting a driver in parking the vehicle.

78. The environment monitoring device of claim 72, further comprising a controller for taking countermeasures to minimize the consequences of a threatening accident based on the response generated by said second processing unit.

80. The environment monitoring device of claim 50, comprising a plurality of 3D cameras for capturing images from various camera positions, where the images captured by said plurality of 3D-cameras representing an environment area include a predetermined range around the vehicle.

82. The environment monitoring device of claim 50, further comprising sensors, where said model generation unit further employs data from said sensors for generating said 3D-vector model.

83. The environment monitoring device of claim 82, further comprising a sensor fusion unit for transforming sensor data into a unique format and removing inconsistent and/or redundant data.

85. The environment monitoring device of claim 50, further comprising a transmitter for transmitting the 3D-vector model to an external location with respect to said vehicle.

87. The environment monitoring device of claim 85, where said transmitter transmits said 3D-vector model via wireless communication.

88. The environment monitoring device of claim 85, where said transmitter transmits said 3D-vector model for visualizing on an external computer.

89. The environment monitoring device of claim 85, where said transmitter transmits said 3D-vector model for evaluating at an external computer.

90. The environment monitoring device of claim 89, further comprising receiving unit for receiving results of said evaluation transmitted back to the vehicle.

91. The environment monitoring device of claim 89, further comprising a controller for controlling the vehicle by employing the evaluation results received from the external computer.

93. The environment monitoring device of claim 85, where said transmitter transmits a sequence of 3D-vector models representing a 3D-space-and-time model.

94. The environment monitoring device of claim 50, further comprising an event data recorder for saving the 3D-vector model.

95. The environment monitoring device of claim 94, where said model generation unit generates a sequence of 3D-vector models during a predetermined period of time, and where said sequence of 3D-vector models is saved into the event data recorder.

96. The environment monitoring device of claim 95, where the 3D-vector models are associated with time stamps before saving.

97. The environment monitoring device of claim 96, further comprising an impact detector for detecting an impact caused by an extraordinary event, where the 3D-vector models are saved in the event data recorder for a predetermined storing time and the 3D-vector models remain permanently saved when an impact is detected by said impact detector.

54. (canceled)

58. (canceled)

59. The environment monitoring device of claim 58, where said determining unit determines a maximum amount of data to be extracted in correspondence with said maximum amount of time.

60. The environment monitoring device of claim 59, where said determining unit further determines priorities for different kinds of data that can be extracted from said image based on the particular relevance of different kinds of data in the vehicle situation characterized by said at least one parameter.

64. (canceled)

65. (canceled)

71. (canceled)

79. (canceled)

81. (canceled)

84. (canceled)

86. (canceled)

92. (canceled)

98. (canceled)