Method and system for providing behavior of vehicle operator using virtuous cycle

• US 10,460,183 B2
• Filed: 06/13/2017
• Issued: 10/29/2019
• Est. Priority Date: 06/13/2016
• Status: Expired due to Fees

• Current Assignee: Xevo, Inc. (Lear Corporation)
• Original Assignee: Xevo, Inc. (Lear Corporation)
• Inventors: Welland, Robert Victor, McKelvie, Samuel James, Tong, Richard Chia-Tsing, Fradin, Noah Harrison, Sadovsky, Vladimir
• Primary Examiner(s): Desire, Gregory M
• Application Number:
  US15/621,835
• Publication Number:
  US 20170357866A1
• Time in Patent Office: 868 Days
• CPC Class Codes: B60R 2300/105 using multiple cameras
• CPC Class Codes: B60R 2300/30 characterised by the type of image processing
• CPC Class Codes: B60R 2300/50 characterised by the display information being shared, e.g. external display, data transfer to other traffic participants or centralised traffic controller
• CPC Class Codes: G06F 18/256 of results relating to different input data, e.g. multimodal recognition
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06V 10/772 Determining representative reference patterns, e.g. averaging or distorting patterns; Generating dictionaries
• CPC Class Codes: G06V 10/774 Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06V 10/811 the classifiers operating on different input data, e.g. multi-modal recognition
• CPC Class Codes: G06V 10/95 structured as a network, e.g. client-server architectures
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/586 of parking space
• CPC Class Codes: G06V 20/597 Recognising the driver's state or behaviour, e.g. attention or drowsiness
• CPC Class Codes: H04L 67/12 specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks

Abstract

A method or system is capable of detecting operator behavior (“OB”) utilizing a virtuous cycle containing sensors, machine learning center (“MLC”), and cloud based network (“CBN”). In one aspect, the process monitors operator body language captured by interior sensors and captures surrounding information observed by exterior sensors onboard a vehicle as the vehicle is in motion. After selectively recording the captured data in accordance with an OB model generated by MLC, an abnormal OB (“AOB”) is detected in accordance with vehicular status signals received by the OB model. Upon rewinding recorded operator body language and the surrounding information leading up to detection of AOB, labeled data associated with AOB is generated. The labeled data is subsequently uploaded to CBN for facilitating OB model training at MLC via a virtuous cycle.

Claims

1. A method configured to detecting operator behavior (“

OB”

) utilizing a plurality of sensors, machine learning center, and cloud based network, comprising;

monitoring operator body language of an operator captured by a set of interior sensors and capturing surrounding information observed by a set of exterior sensors onboard a vehicle as the vehicle is in motion;

selectively recording data relating to the operator body language and the surrounding information in accordance with a containerized OB model generated by a machine learning center (“

MLC”

);

detecting an abnormal OB (“

AOB”

) in accordance with vehicular status signals received by the OB model while the vehicle is in operating;

rewinding recorded operator body language and the surrounding information leading up to detection of the AOB and generating labeled data associated with the AOB;

uploading the labeled data to the cloud based network for facilitating OB model training at the MLC via a virtuous cycle; and

correlating the labeled data with location information, time stamp, and vicinity traffic condition obtained from the cloud based network to update correlated labeled data relating to the AOB.

2. The method of claim 1, further comprising correlating the labeled data with local events, additional sampling data, and weather conditions obtained from the cloud based network to update the correlated labeled data relating to the AOB.

3. The method of claim 1, further comprising correlating the labeled data with historical body language samples relating to the operator body language of OB samples obtained from the cloud based network for update the correlated labeled data relating to the AOB.

4. The method of claim 3, wherein correlating the labeled data with historical body language samples includes revising labeled data in response to one of historical samples relating to facial expression, hand movement, body temperature, and audio recording retrieved from the cloud based network.

5. The method of claim 4, further comprising training the containerized OB model in accordance with the correlated labeled data forwarded from the cloud based network to the machine learning center.

6. The method of claim 4, further comprising detecting an event of distracted driver in response to the correlated labeled data updated by the cloud based network.

7. The method of claim 6, further comprising providing a warning signal to the operator indicating the AOB based on the event of the distracted driver;

and recording the event of distracted driver for future report.

8. The method of claim 1, further comprising pushing the containerized OB model to an onboard digital processing unit in the vehicle via a wireless communication network.

9. The method of claim 1, wherein monitoring operator body language of an operator captured by a set of interior sensor includes activating an interior camera to capture operator facial expression and activating a motion detector to detect operator body movement.

10. The method of claim 1, wherein capturing surrounding information observed by a set of exterior sensors onboard a vehicle includes activating outward-looking cameras situated on the vehicle to capture images as the vehicle is in motion.

11. The method of claim 1, wherein uploading the labeled data to the cloud based network includes separating real-time data from the labeled data and uploading the real-time data to the cloud based network in real-time via a wireless communication network.

12. The method of claim 11, wherein uploading the labeled data to the cloud based network includes separating batched data from the labeled data and uploading the batched data to the cloud based network at a later time.

13. The method of claim 1, wherein uploading the labeled data to the cloud based networking for facilitating a machine learning process within a virtuous cycle includes,

feeding real-time labeled data from the vehicle to the cloud based network for correlating and revising labeled data;

forwarding revised labeled data to the machine learning center for training OB model; and

pushing a trained OB model to the vehicle for continuing data collection.

14. A method configured to detecting operator behavior (“

OB”

) utilizing a plurality of sensors, machine learning center, and cloud based network, comprising;

monitoring operator body language of an operator captured by a set of interior sensors and capturing surrounding information observed by a set of exterior sensors onboard a vehicle as the vehicle is in motion;

selectively recording data relating to the operator body language and the surrounding information in accordance with a containerized OB model generated by a machine learning center (“

MLC”

);

detecting an abnormal OB (“

AOB”

) in accordance with vehicular status signals received by the OB model while the vehicle is in operating;

rewinding recorded operator body language and the surrounding information leading up to detection of the AOB and generating labeled data associated with the AOB; and

uploading the labeled data to the cloud based network for facilitating OB model training at the MLC via a virtuous cycle;

wherein uploading the labeled data to the cloud based networking for facilitating a machine learning process within a virtuous cycle includes,

feeding real-time labeled data from the vehicle to the cloud based network for correlating and revising labeled data;

forwarding revised labeled data to the machine learning center for training OB model; and

pushing a trained OB model to the vehicle for continuing data collection.