Systems and Methods for Using Machine Learning to Determine Passenger Ride Experience

• US 20190329790A1
• Filed: 05/09/2018
• Published: 10/31/2019
• Est. Priority Date: 04/25/2018
• Status: Abandoned Application

• Current Assignee: Uber Technologies, Inc.
• Original Assignee: Uber Technologies, Inc.
• Inventors: Nandakumar, Anand
• Application Number:
  US15/975,265
• Publication Number:
  US 20190329790A1
• Time in Patent Office: Days
• CPC Class Codes: B60W 2050/0014 Adaptive controllers
• CPC Class Codes: B60W 2420/403 Image sensing, e.g. optical camera
• CPC Class Codes: B60W 2420/54 Audio sensitive means, e.g. ultrasound
• CPC Class Codes: B60W 2540/22 Psychological state; Stress level or workload
• CPC Class Codes: B60W 50/0098 Details of control systems ensuring comfort, safety or stability not otherwise provided for
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G05D 1/0212 with means for defining a desired trajectory involving a plurality of land vehicles G05D1/0287
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 20/10 using kernel methods, e.g. support vector machines [SVM]
• CPC Class Codes: G06N 20/20 Ensemble learning
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/082 modifying the architecture, e.g. adding, deleting or silencing nodes or connections
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06N 3/088 Non-supervised learning, e.g. competitive learning
• CPC Class Codes: G06N 5/01 Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks

Abstract

Systems and methods are directed to using machine learning to determine passenger ride experience. In one example, a computer-implemented method includes receiving, by a computing system comprising one or more computing devices, sensor data from one or more sensors positioned within a cabin of a vehicle, the sensor data being descriptive of one or more passengers located within the cabin of the vehicle. The method further includes inputting, by the computing system, the sensor data to a machine-learned ride experience model and receiving, as an output of the machine-learned ride experience model, ride experience data including ride experience events detected from the sensor data and a classification for each detected ride experience event according to a ride experience rating. The method further includes determining, by the computing system and based on the ride experience rating for each detected ride experience event, a ride experience control signal associated with operation of the vehicle.

Claims

1. A computer-implemented method, comprising:

receiving, by a computing system comprising one or more computing devices, sensor data from one or more sensors positioned within a cabin of a vehicle, the sensor data being descriptive of one or more passengers located within the cabin of the vehicle;

inputting, by the computing system, the sensor data to a machine-learned ride experience model;

receiving, by the computing system as an output of the machine-learned ride experience model, ride experience data including ride experience events detected from the sensor data and ride experience ratings classifying each detected ride experience event; and

determining, by the computing system and based on the ride experience rating for each detected ride experience event, a ride experience control signal associated with operation of the vehicle.

2. The computer-implemented method of claim 1, wherein the machine-learned ride experience model is configured to implement at least one of facial expression analysis and body pose analysis of the sensor data.

3. The computer-implemented method of claim 2, wherein the sensor data comprises image data from one or more image sensors positioned within the cabin of the vehicle and audio data from one or more audio sensors positioned within the cabin of the vehicle;

and wherein the machine-learned ride experience model is configured to implement sound analysis of the audio data.

4. The computer-implemented method of claim 1, wherein the sensor data comprises image data from one or more image sensors positioned within the cabin of the vehicle;

and

wherein the machine-learned ride experience model is configured to detect within the image data one or more passengers and one or more body parts associated with the one or more passengers located within the cabin of the vehicle; and

the machine-learned ride experience model is configured to implement facial expression analysis and body pose analysis of the sensor data relative to the one or more body parts detected within the image data.

5. The computer-implemented method of claim 4, wherein the machine-learned ride experience model is further configured to determine a body connection framework connecting multiple of the one or more body parts together and to implement body pose analysis by measuring relative movement of the body connection framework.

6. The computer-implemented method of claim 1, wherein the ride experience rating classifying each detected ride experience event is selected from a predetermined class comprising a good passenger experience rating and a bad passenger experience rating.

7. The computer-implemented method of claim 1, wherein the ride experience rating classifying each detected ride experience event is dynamically determined on a gradient scale within a range of possible values.

8. The computer-implemented method of claim 1, wherein the machine-learned ride experience model comprises a plurality of shared layers that are used at least in part for both determining facial expressions and determining body pose as part of determining ride experience data.

9. The computer-implemented method of claim 1, wherein the ride experience control signal comprises a vehicle control signal, wherein the vehicle control signal provides data that can be used for adjusting a motion plan of the vehicle based in part on the ride experience data.

10. The computer-implemented method of claim 1, wherein the ride experience control signal comprises a driving data log signal, wherein the driving data log signal triggers storage of data associated with a detected ride experience event that can be used to determine metrics associated with overall passenger ride experience.

11. The computer-implemented method of claim 1, wherein the ride experience control signal comprises a trip assistance signal, wherein the trip assistance signal includes a request to initiate two-way conversation with the vehicle for use in a determination of subsequent assistance steps.

12. A computing system, comprising:

one or more image sensors positioned within a cabin of a vehicle and configured to obtain image data being descriptive of an appearance of one or more passengers located within the cabin of the vehicle;

one or more processors;

a machine-learned ride experience model that has been trained to analyze the image data by implementing at least one of facial expression analysis and body pose analysis of the image data and to generate ride experience data in response to receipt of the image data; and

at least one tangible, non-transitory computer readable medium that stores instructions that, when executed by the one or more processors, cause the one or more processors to perform operations, the operations comprising;

providing real-time samples of the image data to the machine-learned ride experience model; and

receiving as an output of the machine-learned ride experience model, ride experience data including ride experience events detected from the image data and a classification for each detected ride experience event according to a ride experience rating.

13. The computing system of claim 12, further comprising one or more audio sensors positioned within a cabin of a vehicle and configured to obtain audio data descriptive of sound associated with one or more passengers located within the cabin of the vehicle;

and wherein the machine-learned ride experience model has been trained to analyze the audio data as part of generating the ride experience data.

14. The computing system of claim 12, wherein the operations further comprise determining, based on the ride experience rating for each detected ride experience event, a ride experience control signal associated with operation of the vehicle.

15. The computing system of claim 12, wherein:

the machine-learned ride experience model is configured to detect one or more objects of interest within the image data, the one or more objects of interest including one or more body parts associated with the one or more passengers located within the cabin of the vehicle; and

the machine-learned ride experience model is configured to implement facial expression analysis and body pose analysis of the image data relative to the one or more body parts detected within the image data.

16. The computing system of claim 15, wherein the machine-learned ride experience model is further configured to determine a body connection framework connecting multiple of the one or more body parts together and to implement body pose analysis by measuring relative movement of the body connection framework.

17. An autonomous vehicle, comprising:

a sensor system comprising one or more image sensors and one or more audio sensors for obtaining respective image data and audio data associated with one or more passengers of an autonomous vehicle;

a vehicle computing system comprising;

one or more processors; and

at least one tangible, non-transitory computer readable medium that stores instructions that, when executed by the one or more processors, cause the one or more processors to perform operations, the operations comprising;

inputting the image data and audio data to a machine-learned ride experience model;

receiving, as an output of the machine-learned ride experience model, ride experience data including ride experience events detected from the image data and audio data and a classification for each detected ride experience event according to a ride experience rating; and

determining, based on the ride experience rating for each detected ride experience event, a ride experience control signal associated with operation of the vehicle.

18. The autonomous vehicle of claim 17, wherein the machine-learned ride experience model is configured to implement one or more of facial expression analysis of the image data, body pose analysis of the image data, and sound analysis of the audio data.

19. The autonomous vehicle of claim 17, wherein:

the machine-learned ride experience model is configured to detect one or more objects of interest within the image data, the one or more objects of interest including one or more body parts associated with the one or more passengers located within a cabin of the vehicle; and

the machine-learned ride experience model is configured to implement facial expression analysis and body pose analysis of the image data relative to the one or more body parts detected within the image data.

20. The autonomous vehicle of claim 19, wherein the machine-learned ride experience model is further configured to determine a body connection framework connecting multiple of the one or more body parts together and to implement body pose analysis by measuring relative movement of the body connection framework.