Mapping techniques using probe vehicles

• US 9,528,834 B2
• Filed: 12/15/2014
• Issued: 12/27/2016
• Est. Priority Date: 11/01/2013
• Status: Active Grant

• Current Assignee: Intelligent Technologies International, Inc.
• Original Assignee: Intelligent Technologies International, Inc.
• Inventors: Breed, David S, Breed, Ryan
• Primary Examiner(s): Sweeney, Brian P
• Application Number:
  US14/570,638
• Publication Number:
  US 20150127239A1
• Time in Patent Office: 743 Days
• Field of Search: 701/45
• US Class Current: 1/1
• CPC Class Codes: B60R 21/0132 responsive to vehicle motion parameters , e.g. to vehicle longitudinal or transversal deceleration or speed value
• CPC Class Codes: B60W 30/00 Purposes of road vehicle drive control systems not related to the control of a particular sub-unit, e.g. of systems using conjoint control of vehicle sub-units
• CPC Class Codes: G01C 21/1652 with ranging devices, e.g. LIDAR or RADAR
• CPC Class Codes: G01C 21/1656 with passive imaging devices, e.g. cameras
• CPC Class Codes: G01S 19/13 Receivers
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/73 using feature-based methods

Abstract

Vehicle-mounted device includes an inertial measurement unit (IMU) having at least one accelerometer or gyroscope, a GPS receiver, a camera positioned to obtain unobstructed images of an area exterior of the vehicle and a control system coupled to these components. The control system re-calibrates each accelerometer or gyroscope using signals obtained by the GPS receiver, and derives information about objects in the images obtained by the camera and location of the objects based on data from the IMU and GPS receiver. A communication system communicates the information derived by the control system to a location separate and apart from the vehicle. The control system includes a processor that provides a location of the camera and a direction in which the camera is imaging based on data from the IMU corrected based on data from the GPS receiver, for use in creating the map database.

Claims

1. A vehicle-mounted device, comprising:

an inertial measurement unit (IMU) comprising six acceleration or velocity measuring devices;

a receiver configured to receive satellite-sourced positioning signals;

at least one camera each positioned to obtain images of an area exterior of the vehicle;

a control system coupled to said IMU, said receiver and said at least one camera, said control system being configured to re-calibrate said six acceleration or velocity measuring devices using signals obtained by said receiver; and

a communication system coupled to said control system and that communicates the images obtained by said at least one camera or information about objects in the images obtained by said at least one camera to a location separate and apart from the vehicle,

wherein said control system includes a processor and is configured to provide a location of said at least one camera based on data from said IMU corrected based on data from said receiver and positioning correction data derived from satellite-sourced positioning signals and to provide a direction in which said at least one camera is imaging based on data from said IMU corrected based on data from said receiver.

2. The device of claim 1, wherein said IMU is manufactured using mass-production MEMS technology.

3. The device of claim 1, wherein said control system is further configured to re-calibrate said six acceleration or velocity measuring devices using a zero lateral and vertical speed of the vehicle and speedometer readings of the vehicle.

4. The device of claim 1, wherein said six acceleration or velocity measuring devices comprise three accelerometers that measure acceleration in a respective direction in a three-dimensional coordinate system and three gyroscopes that measure angular velocity about a respective axis in the three-dimensional coordinate system, said control system is further configured to correct projections of gravitational acceleration in readings from said three accelerometers, when the vehicle tilts, and projections of centrifugal accelerations in readings from said three gyroscopes during turning maneuvers.

5. The device of claim 1, wherein said IMU, said receiver, and said at least one camera are mounted in a single housing, said housing being positioned on the vehicle such that said at least one camera images a portion of a road in front of the vehicle and terrain on both sides of the vehicle, said at least one camera having a horizontal field of view of from about 45 to about 180 degrees.

6. The device of claim 1, further comprising a speed limiting apparatus that notifies a driver of the vehicle of a maximum speed of travel of the vehicle or automatically limits speed of the vehicle to the maximum speed of travel of the vehicle at the location at which the vehicle is travelling, said control system being coupled to said speed limiting apparatus and generating the maximum speed of travel of the vehicle based on speed and accuracy of processing of images obtained by said at least one camera by said control system.

7. The device of claim 1, further comprising a map database resident on the vehicle, said map database including location of objects in the images obtained by said at least one camera.

8. The device of claim 7, wherein said control system is further configured to identify objects in images obtained by said at least one camera and determine their location and to determine whether the identified objects and their location are present in said map database, said processor being configured to communicate only the images obtained by said at least one camera that include objects not present in said map database, only objects whose location in said map database is inaccurate and only information about an object derived by said control system that is not included in said map database to the location separate and apart from the vehicle.

9. The device of claim 7, wherein said control system is configured to correct data from said IMU using said map database and said at least one camera and without use of data from said receiver by comparing expected location of an object in an image obtained by said at least one camera when the vehicle is at a specific location using said map database to actual location of the same object in an image obtained by said at least one camera as determined by said processor when the vehicle is at the specific location, and correcting the data from said IMU when the actual vehicle location differs from the expected vehicle position.

10. The device of claim 7, wherein said control system is further configured to determine the location of objects in the images obtained by said at least one camera from multiple images using displacement of the vehicle between the times when the multiple images are obtained and a known orientation of said at least one camera relative to the vehicle when each of the multiple images is obtained, the determined location of the objects in the images obtained by said at least one camera being included in said map database.

11. The device of claim 7, wherein said communication system communicates location of the vehicle to the remote location, a determination being made at the remote station whether images of the area exterior of the vehicle at the vehicle'"'"'s location communicated to the remote station using said communication system are needed to obtain information about objects in the area to include in said map database, and when it is determined that images of the area exterior of the vehicle at the vehicle'"'"'s location are needed to obtain information about objects in the area to include in said map database, said at least one camera being configured to be directed to obtain images of the area exterior of the vehicle at the vehicle'"'"'s location.

12. The device of claim 1, wherein said communication system communicates the location of the vehicle to the remote location, and said at least one camera is configured to be remotely controlled to obtain images based on its location communicated by said communication system.

13. The device of claim 1, wherein said six acceleration or velocity measuring devices comprise three accelerometers that measure acceleration in a respective direction in a three-dimensional coordinate system and three gyroscopes that measure angular velocity about a respective axis in the three-dimensional coordinate system, and said control system is further configured to compare output from said three accelerometers and said three gyroscopes with the derived location and angular orientation of the vehicle, and based on the comparison, modify acceleration output from said three accelerometers and modify angular velocity output from said three gyroscopes.

14. The device of claim 13, wherein said control system is further configured to determine location and angular orientation of the vehicle from the modified acceleration output from said three accelerometers and modified angular velocity output from said three gyroscopes, said control system being further configured to determine a direction in which said at least one camera is pointing when images are obtained by said at least one camera based on the determined angular orientation of the vehicle, and said control system deriving information about objects in the images obtained by said at least one camera and location of the objects based on the determined location of the vehicle and the determined direction in which said at least one camera is pointing when images including the objects are obtained by said at least one camera.

15. The device of claim 1, wherein said at least one camera consists of a single camera.

16. The device of claim 1, wherein said at least one camera comprises a plurality of cameras.

17. The device of claim 1, wherein said control system is further configured to identify objects in images obtained by said at least one camera and determine their location based on data from the IMU and the receiver and data about the position of each of the least one camera relative to the receiver which constitutes information about objects in the images obtained by said at least one camera, said communication system being configured to communicate the information about objects in the images derived by said control system to the location separate and apart from the vehicle, the information including a location of the objects.

18. The device of claim 1, wherein said communication system is configured to communicate the images obtained by said at least one camera, and a location of the vehicle and pointing direction of said at least one camera when each of the images was obtained by said at least one camera to the location separate and apart from the vehicle.

19. A method for mapping terrain using a vehicle, comprising:

obtaining information about objects using one or more devices each comprising an inertial measurement unit (IMU) including six acceleration or velocity measuring devices, a receiver configured to receive satellite-sourced positioning signals, at least one camera each positioned to obtain images of an area exterior of the device, and a control system coupled to the IMU, the receiver and the at least one camera;

re-calibrating the six acceleration or velocity measuring devices using signals obtained by the receiver;

communicating the images obtained by the at least one camera or the information about objects in the images obtained by the at least one camera to a location separate and apart from the vehicle using a communications system; and

correcting data from the IMU using data from the receiver, using a processor of the control system, to provide a location of the at least one camera of each device when each image is obtained and a pointing direction of the at least one camera when each image is obtained,

the step of correcting the data from the IMU using data from the receiver to provide a location of the at least one camera of each device when each image is obtained comprising correcting the data based on positioning correction data derived from satellite-sourced positioning signals.

20. The method of claim 19, further comprising maintaining a map database on the vehicle by adding to the map database, objects in images obtained by the at least one camera, the map database maintaining step comprising:

identifying the same object in two or more images obtained from different locations using the processor; and

positioning, using the processor, the object in the map database based on the data about the locations of the at least one camera on the device from which the two or more images were obtained and the pointing direction of the at least one camera on the device when the images were obtained.

21. The method of claim 20, further comprising:

identifying objects in images obtained by the at least one camera and their location using the processor;

determining whether the identified objects are present in the map database using the processor; and

controlling, using the processor, communication of the images obtained by the at least one camera or of the derived information about objects in the images from the device to the location separate and apart from the vehicle using the communications system such that images or derived information about an object is transmitted to the location separate and apart from the vehicle only when the object is not present in the map database or only when the location of the object in the map database is not accurate.

22. The method of claim 20, further comprising:

correcting, using the processor, data from the IMU using the map database and the at least one camera and without use of data from the receiver by comparing expected location of an object in an image obtained by the at least one camera when the vehicle is at a specific location known from the map database to actual location of the same object in an image obtained by the at least one camera as determined by the processor when the vehicle when at the specific location, and correcting the data from the IMU when the actual vehicle location differs from the expected vehicle location.

23. The method of claim 19, wherein the six acceleration or velocity measuring devices comprise three accelerometers that measure acceleration in a respective direction in a three-dimensional coordinate system and three gyroscopes that measure angular velocity about a respective axis in the three-dimensional coordinate system, the method further comprising:

comparing output from the three accelerometers with the derived location and angular orientation of the vehicle and based on the comparison, modifying acceleration output from the three accelerometers and modifying angular velocity output from the three gyroscopes;

determining location and angular orientation of the vehicle from the modified acceleration output from the three accelerometers and modified angular velocity output from the three gyroscopes; and

determining a direction in which the at least one camera is pointing when images are obtained by the at least one camera based on the determined angular orientation of the vehicle;

information about objects in the images obtained by the at least one camera and location of the objects being derived based on the determined location of the vehicle and the determined direction in which the at least one camera is pointing when images including the objects are obtained by the at least one camera.

24. The method of claim 19, wherein each of the devices further includes an additional receiver configured to receive satellite-sourced positioning signals, the additional receiver being coupled to said control system, the six acceleration or velocity measuring devices being re-calibrated using signals obtained by the additional receiver, information about objects in the images obtained by the at least one camera and location of the objects being derived based further on data from the additional receiver and data about the position of each of the least one camera relative to the additional receiver.

25. The method of claim 19, further comprising determining location of objects in the images obtained by the at least one camera based on data from the IMU and the receiver and data about the position of each of the least one camera relative to the receiver which constitutes information about objects in the images obtained by the at least one camera, the step of communicating the images obtained by the at least one camera or information about objects in the images obtained by the at least one camera to the location separate and apart from the vehicle using the communications system comprising communicating the determined object location to the location separate and apart from the vehicle.

26. The method of claim 19, further comprising determining a location of the vehicle and pointing direction of the at least one camera when each of the images was obtained by the at least one camera, the step of communicating the images obtained by the at least one camera or information about objects in the images obtained by the at least one camera to the location separate and apart from the vehicle using the communications system comprising communicating the images obtained by the at least one camera, and the location of the vehicle and the pointing direction of the aid at least one camera when each of the images was obtained by the at least one camera to the location separate and apart from the vehicle.

27. A method for mapping terrain using vehicles, comprising:

receiving from each vehicle at a location separate and apart from the vehicles, information about objects obtained using one or more devices on each vehicle, each device comprising an inertial measurement unit (IMU) including six acceleration or velocity measuring devices, a receiver configured to receive satellite-sourced positioning signals, at least one camera each positioned to obtain images of an area exterior of the device, and a control system coupled to the IMU, the receiver and the at least one camera, the six acceleration or velocity measuring devices in each device being re-calibrated using signals obtained by the receiver in each device, information about objects in the images obtained by the at least one camera in each device being derived based on data from the IMU and the receiver and data about the position of each of the least one camera relative to the receiver, the images obtained by the at least one camera or the derived information about objects in the images obtained by the at least one camera being communicated to the location separate and apart from the vehicles using a communications system on each vehicle, the data from the IMU being corrected using data from the receiver, using a processor of the control system, to provide a location of the at least one camera of each device when each image is obtained and a pointing direction of the at least one camera when each image is obtained; and

creating a map database at the remote location from the received information by adding to the map database, objects in images obtained by the at least one camera of the device in the vehicles and the location of these objects.

28. The method of claim 27, wherein the step of creating the map database further comprises:

identifying the same object in two or more images obtained from different locations using the processor; and

positioning, using the processor, the object in the map database based on the data about the locations of the at least one camera on the device from which the two or more images were obtained and the pointing direction of the at least one camera on the device when the images were obtained.