Method and System for Providing Behavior of Vehicle Operator Using Virtuous Cycle

• US 20170357866A1
• Filed: 06/13/2017
• Published: 12/14/2017
• Est. Priority Date: 06/13/2016
• Status: Active Grant

• Current Assignee: Xevo, Inc. (Lear Corporation)
• Original Assignee: Surround.io Corporation (Lear Corporation)
• Inventors: Welland, Robert Victor, McKelvie, Samuel James, Tong, Richard Chia-Tsing, Fradin, Noah Harrison, Sadovsky, Vladimir
• Granted Patent: US 10,460,183 B2
• Time in Patent Office: Days
• CPC Class Codes: B60R 2300/105 using multiple cameras
• CPC Class Codes: B60R 2300/30 characterised by the type of image processing
• CPC Class Codes: B60R 2300/50 characterised by the display information being shared, e.g. external display, data transfer to other traffic participants or centralised traffic controller
• CPC Class Codes: G06F 18/256 of results relating to different input data, e.g. multimodal recognition
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06V 10/772 Determining representative reference patterns, e.g. averaging or distorting patterns; Generating dictionaries
• CPC Class Codes: G06V 10/774 Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06V 10/811 the classifiers operating on different input data, e.g. multi-modal recognition
• CPC Class Codes: G06V 10/95 structured as a network, e.g. client-server architectures
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/586 of parking space
• CPC Class Codes: G06V 20/597 Recognising the driver's state or behaviour, e.g. attention or drowsiness
• CPC Class Codes: H04L 67/12 specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks

Abstract

A method or system is capable of detecting operator behavior (“OB”) utilizing a virtuous cycle containing sensors, machine learning center (“MLC”), and cloud based network (“CBN”). In one aspect, the process monitors operator body language captured by interior sensors and captures surrounding information observed by exterior sensors onboard a vehicle as the vehicle is in motion. After selectively recording the captured data in accordance with an OB model generated by MLC, an abnormal OB (“AOB”) is detected in accordance with vehicular status signals received by the OB model. Upon rewinding recorded operator body language and the surrounding information leading up to detection of AOB, labeled data associated with AOB is generated. The labeled data is subsequently uploaded to CBN for facilitating OB model training at MLC via a virtuous cycle.

Claims

1. A method configured to detecting operator behavior (“

OB”

) utilizing a plurality of sensors, machine learning center, and cloud based network, comprising;

monitoring operator body language of an operator captured by a set of interior sensors and capturing surrounding information observed by a set of exterior sensors onboard a vehicle as the vehicle is in motion;

selectively recording data relating to the operator body language and the surrounding information in accordance with a containerized OB model generated by a machine learning center (“

MLC”

);

detecting an abnormal OB (“

AOB”

) in accordance with vehicular status signals received by the OB model while the vehicle is in operating;

rewinding recorded operator body language and the surrounding information leading up to detection of the AOB and generating labeled data associated with the AOB; and

uploading the labeled data to the cloud based network for facilitating OB model training at the MLC via a virtuous cycle.

2. The method of claim 1, further comprising correlating the labeled data with location information, time stamp, and vicinity traffic condition obtained from the cloud based network to update correlated labeled data relating to the AOB.

3. The method of claim 2, further comprising correlating the labeled data with local events, additional sampling data, and weather conditions obtained from the cloud based network to update the correlated labeled data relating to the AOB.

4. The method of claim 2, further comprising correlating the labeled data with historical body language samples relating to the operator body language of OB samples obtained from the cloud based network for update the correlated labeled data relating to the AOB.

5. The method of claim 4, wherein correlating the labeled data with historical body language samples includes revising labeled data in response to one of historical samples relating to facial expression, hand movement, body temperature, and audio recording retrieved from the cloud based network.

6. The method of claim 5, further comprising training the containerized OB model in accordance with the correlated labeled data forwarded from the cloud based network to the machine learning center.

7. The method of claim 5, further comprising detecting an event of distracted driver in response to the correlated labeled data updated by the cloud based network.

8. The method of claim 7, further comprising providing a warning signal to the operator indicating the AOB based on the event of the distracted driver;

and recording the event of distracted driver for future report.

9. The method of claim 1, further comprising pushing the containerized OB model to an onboard digital processing unit in the vehicle via a wireless communication network.

10. The method of claim 1, wherein monitoring operator body language of an operator captured by a set of interior sensor includes activating an interior camera to capture operator facial expression and activating a motion detector to detect operator body movement.

11. The method of claim 1, wherein capturing surrounding information observed by a set of exterior sensors onboard a vehicle includes activating outward-looking cameras situated on the vehicle to capture images as the vehicle is in motion.

12. The method of claim 1, wherein uploading the labeled data to the cloud based network includes separating real-time data from the labeled data and uploading the real-time data to the cloud based network in real-time via a wireless communication network.

13. The method of claim 12, wherein uploading the labeled data to the cloud based network includes separating batched data from the labeled data and uploading the batched data to the cloud based network at a later time.

14. The method of claim 1, wherein uploading the labeled data to the cloud based networking for facilitating a machine learning process within a virtuous cycle includes,

feeding real-time labeled data from the vehicle to the cloud based network for correlating and revising labeled data;

forwarding revised labeled data to the machine learning center for training OB model; and

pushing a trained OB model to the vehicle for continuing data collection.

15. A network configuration able to detect operator behavior (“

OB”

) via a virtuous cycle, comprising;

a vehicle operated by a driver containing a sensing device configured to collect data relating to operator body language and surrounding information, the vehicle configured to selectively record surrounding information observed by a plurality of onboard sensors in accordance with instructions from an OB model when the vehicle is in motion;

a cloud based network wirelessly coupled to the sensing device and configured to correlate and generate labeled data associated with OB based on historical OB cloud data and the collected data; and

a machine learning center coupled to the cloud based network and configured to train and improve the OB model based on the labeled data from the cloud based network.

16. The network configuration of claim 15, wherein the vehicle includes forward-looking cameras configured to collect real-time images as the vehicle moves across a geographical area.

17. The network configuration of claim 16, wherein the sensing device of the vehicle includes a memory, controller, and transmitter, wherein the memory stores at least a portion of real-time images collected by the forward-looking cameras installed on the vehicle.

18. The network configuration of claim 15, wherein the vehicle includes inward-looking cameras configured to collect real-time images relating to driver body language during the vehicle moves across a geographical area.

19. The network configuration of claim 15, wherein the vehicle is able to detect an abnormal OB (“

AOB”

) based on vehicular status signals generated by a head unit of the vehicle.

20. A method configured to detect a sign utilizing a plurality of sensors, machine learning center, and cloud based network, comprising:

storing real-time data captured by an onboard outward-looking cameras installed on the vehicle based on instructions from a sign model when the vehicle is driving;

detecting a sign image when the vehicle captures a predefined sample image and retrieving stored real-time data from a local memory to compare the predefined sample image against the captured sign image;

generating labeled data associated with the sign in response to the stored real-time data and historical cloud data; and

uploading the labeled data to the cloud based network for facilitating model training relating to the sign model at a machine learning process via a virtuous cycle.