Estimating friction based on image data

• US 10,275,662 B1
• Filed: 09/30/2016
• Issued: 04/30/2019
• Est. Priority Date: 09/30/2016
• Status: Active Grant

• Current Assignee: Zoox Inc. (Amazon.com, Inc.)
• Original Assignee: Zoox Inc. (Amazon.com, Inc.)
• Inventors: Askeland, Jacob Lee
• Primary Examiner(s): Desire, Gregory M
• Application Number:
  US15/281,416
• Time in Patent Office: 942 Days
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/30256 Lane; Road marking
• CPC Class Codes: G06T 7/0004 Industrial image inspection
• CPC Class Codes: G06T 7/40 Analysis of texture depth or shape recovery from texture G06T7/529
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 10/449 Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

A friction estimation system for estimating friction-related data associated with a surface on which a vehicle travels, may include a camera array including a plurality of imagers configured to capture image data associated with a surface on which a vehicle travels. The image data may include light data associated with the surface. The friction estimation system may also include an image interpreter in communication with the camera array and configured to receive the image data from the camera array and determine friction-related data associated with the surface based, at least in part, on the image data. The image interpreter may be configured to be in communication with a vehicle control system and provide the friction-related data to the vehicle control system.

Claims

1. A friction estimation system for estimating friction-related data associated with a surface on which a vehicle travels, the friction estimation system comprising:

a camera array comprising a plurality of imagers configured to capture image data associated with a surface on which a vehicle travels;

an image interpreter in communication with the camera array and configured to receive the image data from the camera array and determine friction-related data associated with the surface based, at least in part, on the image data, the image interpreter being configured to be in communication with a vehicle control system and provide the friction-related data to the vehicle control system;

a location sensor, an output of which comprises a location of the vehicle; and

one or more processors configured to associate, based at least in part on the location, the friction-related data with a map accessible to the vehicle.

2. The friction estimation system of claim 1, wherein the image data comprises at least one of image depth profile data associated with the surface or texture data associated with the surface.

3. The friction estimation system of claim 1, wherein the friction-related data comprises a friction coefficient between the surface and one or more tires of the vehicle.

4. The friction estimation system of claim 1, wherein the one or more processors are configured to provide a correlation between the image data and the friction-related data, and wherein the correlation between image data and friction-related data comprises at least one of a correlation table correlating image data and friction-related data, or a mathematical relationship between image data and friction-related data.

5. The friction estimation system of claim 1, wherein the one or more processors are configured to provide a correlation between the image data and the friction-related data, and wherein the one or more processors are configured to update the correlation between the image data and the friction-related data based on measured correlations between the image data and sensed traction events between the vehicle tires and surfaces on which the vehicle has travelled.

6. The friction estimation system of claim 1, wherein the one or more processors are configured to provide a correlation between the image data and the friction-related data, and wherein the correlation comprises a correlation between the image data and a material from which the surface is made, and a correlation between the material and the friction-related data.

7. The friction estimation system of claim 1, wherein the one or more processors are configured to provide a correlation between the image data and the friction-related data, and wherein the one or more processors are configured to receive the location of the vehicle and update the correlation between the image data and the friction-related data based, at least in part, on the location of the vehicle.

8. The friction estimation system of claim 1, further comprising a transmitter in communication with a network and configured to communicate data correlating the location of the vehicle and the friction-related data via the network.

9. The friction estimation system of claim 1, further comprising a receiver in communication with a network and configured to receive updates relating to a correlation between the image data and friction-related data.

10. A vehicle control system for controlling a vehicle, the vehicle control system comprising:

a friction estimation system for estimating friction-related data associated with a surface on which a vehicle travels, the friction estimation system comprising;

a camera array configured to capture image data associated with the surface on which the vehicle travels; and

an image interpreter in communication with the camera array and configured to receive the image data from the camera array and to determine friction-related data associated with the surface based, at least in part, on the image data;

a transmitter in communication with a network; and

a drive system in communication with the friction estimation system and configured to control maneuvering of the vehicle, the drive system comprising at least one of;

a vehicle propulsion system configured to propel the vehicle on the surface,

a vehicle steering system configured to alter a direction of travel of the vehicle, or

a vehicle braking system configured to reduce a speed of travel of the vehicle,

wherein the drive system is configured to receive the friction-related data from one or more of the friction estimation system or the network and control maneuvering of the vehicle based, at least in part, on the friction-related data.

11. The vehicle control system of claim 10, wherein the image data comprises at least one of image depth profile data associated with the surface or texture data associated with the surface.

12. The vehicle control system of claim 10, wherein the friction-related data comprises a friction coefficient between the surface and one or more tires of the vehicle.

13. The vehicle control system of claim 10, wherein the drive system comprises a vehicle propulsion system configured to propel the vehicle on the surface, and wherein the drive system is configured to receive the friction-related data from the friction estimation system and alter propulsion of the vehicle based, at least in part, on the friction-related data.

14. The vehicle control system of claim 10, wherein the drive system comprises a vehicle steering system configured to alter a direction of travel of the vehicle, and wherein the drive system is configured to receive the friction-related data from the friction estimation system and alter a steering angle of the steering system based, at least in part, on the friction-related data.

15. The vehicle control system of claim 10, wherein the drive system comprises a vehicle braking system configured to reduce a speed of travel of the vehicle, and wherein the drive system is configured to receive the friction-related data from the friction estimation system and alter a rate of deceleration of the vehicle based, at least in part, on the friction-related data.

16. The vehicle control system of claim 10, wherein the drive system further comprises a ride control system configured to control compliance of suspension of the vehicle, and wherein the ride control system is configured to control the compliance based, at least in part, on at least one of the image data or friction-related data.

17. The vehicle control system of claim 10, wherein the friction estimation system further comprises one or more processors in communication with the image interpreter and configured to provide a correlation between image data and friction-related data.

18. The vehicle control system of claim 17, wherein the drive system further comprises at least one of:

a traction control system configured to alter output of the propulsion system based, at least in part, on sensing slip between one or more tires of the vehicle and the surface;

a stability control system configured to selectively control rotation of one or more tires of the vehicle based, at least in part, on sensing slip between one or more of the tires during turning of the vehicle;

or

an anti-lock braking control system configured to prevent one or more tires of the vehicle from ceasing rotation during braking of the vehicle,

wherein the one or more processors are configured to;

update the correlation between the image data and the friction-related data based on correlations between the image data and traction events sensed by at least one of the traction control system, the stability control system, or the anti-lock braking control system; and

transmit, via the network, the updated correlation data to one or more of a second vehicle or a server.

19. The vehicle control system of claim 10, further comprising a safety system configured to protect at least one of objects outside the vehicle from collisions with the vehicle, or occupants inside the vehicle during collisions, and wherein the vehicle control system is configured to alter operation of the safety system based, at least in part, on the friction-related data.

20. A method for estimating friction-related data associated with a surface on which a vehicle travels, the method comprising:

obtaining image data associated with the surface via a camera array comprising a plurality of imagers;

correlating the image data with material data associated with a material of the surface;

correlating the material data with friction-related data;

estimating the friction-related data based, at least in part, on the correlation between the material data and the friction-related data; and

transmitting, via a network, data correlating a location of the vehicle and the friction-related data to one or more of a second vehicle or a server.

21. The method of claim 20, wherein the image data comprises at least one of image depth profile data associated with the surface or texture data associated with the surface, and wherein the method further comprises estimating the friction-related data by at least one of analyzing the image depth profile data using machine learning procedures or analyzing the texture data using multi-channel filtering techniques.

22. The method of claim 20, wherein the friction-related data comprises a friction coefficient between the surface and one or more tires of the vehicle.

23. The method of claim 20, wherein correlating the image data with the material data associated with the material of the surface comprises obtaining the correlation from at least one of a correlation table correlating the image data and the material data, or a mathematical relationship between the image data and the material data.

24. The method of claim 20, further comprising updating the correlation between the image data and the friction-related data based on measured correlations between the image data and sensed traction events between the vehicle tires and surfaces on which the vehicle has travelled.

25. The method of claim 20, further comprising updating the correlation between the image data and the friction-related data based, at least in part, on location data associated with a location of the vehicle.

26. The method of claim 20, further comprising receiving updates relating to correlations between the image data and the friction-related data via the network.

27. The method of claim 20, further comprising:

receiving, via the network, second friction-related data associated with a second location; and

controlling maneuvering of the vehicle at the second location based, at least in part, on the second friction-related data, wherein controlling maneuvering of the vehicle comprises at least one of;

altering propulsion of the vehicle based, at least in part, on the second friction-related data,

altering a steering angle of a steering system associated with the vehicle based, at least in part, on the second friction-related data,

altering a rate of deceleration of the vehicle based, at least in part, on the second friction-related data,

or

altering operation of a safety system based, at least in part, on the second friction-related data, wherein the safety system is configured to protect at least one of objects outside the vehicle from collisions with the vehicle, or occupants inside the vehicle during collisions.

28. The method of claim 20, further comprising updating the correlation between the image data and the friction-related data based on correlations between the image data and traction events sensed by at least one of a traction control system associated with the vehicle, a stability control system associated with the vehicle, or an anti-lock braking control system associated with the vehicle.