Method and system for monitoring user activity

• US 10,528,047 B1
• Filed: 01/31/2019
• Issued: 01/07/2020
• Est. Priority Date: 01/31/2019
• Status: Active Grant

• Current Assignee: Chongqing Jinkang Seres New Energy Vehicle Research Institute Company Limited
• Original Assignee: Chongqing Jinkang New Energy Automobile Co., Ltd., SF Motors, Inc. (SERES Group Co., Ltd.)
• Inventors: Trujillo, Edgar, Camhi, Jaime
• Primary Examiner(s): Previl, Daniel
• Application Number:
  US16/264,557
• Time in Patent Office: 341 Days
• Field of Search: 340439, 340436, 340435, 340431, 340447, 340463, 340516, 340521, 34053922, 3405391, 34063619, 3406866, 340 52
• CPC Class Codes: B60W 2040/0872 Driver physiology
• CPC Class Codes: B60W 2050/0029 Mathematical model of the driver
• CPC Class Codes: B60W 2552/10 Number of lanes
• CPC Class Codes: B60W 2552/15 Road slope, i.e. the inclination of a road segment in the longitudinal direction
• CPC Class Codes: B60W 2552/30 Road curve radius
• CPC Class Codes: B60W 2552/35 Road bumpiness, e.g. potholes
• CPC Class Codes: B60W 2554/406 Traffic density
• CPC Class Codes: B60W 2555/20 Ambient conditions, e.g. wind or rain
• CPC Class Codes: B60W 40/02 related to ambient conditions
• CPC Class Codes: B60W 40/08 related to drivers or passengers
• CPC Class Codes: B60W 40/09 Driving style or behaviour
• CPC Class Codes: B60W 50/14 Means for informing the driver, warning the driver or prompting a driver intervention
• CPC Class Codes: B60W 60/0053 from vehicle to occupant
• CPC Class Codes: G05D 1/0061 for transition from automatic pilot to manual pilot and vice versa
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/597 : Recognising the driver's state or behaviour, e.g. attention or drowsiness

Abstract

Embodiments provide a method and system for monitoring user activity in a vehicle. A first sensor in the vehicle acquiring current activity data regarding the user. A second sensor in the vehicle can acquire environment data regarding an environment surrounding the user. A first set of classifiers can generate key point data regarding the user, which indicates body hand/eye movement points of the user. A second set of classifiers can assign the current activity to the activity group. A storage can store response information regarding a user, which includes mappings between activity groups and corresponding response time duration for the user to regain control of the vehicle. The system can generate a take-over request for the user to regain control of the vehicle based on the mapping and current activity.

Claims

1. A method for monitoring user activity in a vehicle, the method being implemented by one or more processors configured to execute machine-readable instructions, the method comprising:

storing, in a storage associated with the vehicle, response information regarding a user of the vehicle, the response information includes mappings between activity groups and corresponding response time duration for the user to regain a control of the vehicle, wherein the mappings include a first mapping between a first activity group and a first response time duration;

acquiring, from a first sensor in the vehicle, current activity data regarding the user, the current activity data of the user indicating a first current activity engaged by the user in the vehicle;

generating, using a first set of one or more classifiers, key point data regarding the user, the key point data indicating a first set of body and/or hand and/or eye movement points of the user during the first current activity;

acquiring, from a second sensor in the vehicle, environment data regarding the vehicle during the first current activity, the environment data includes information regarding an environment surrounding the user during first current activity;

assigning, using a second set of one or more classifiers, based on the key point data, the first current activity to the first activity group; and

generating a take-over request for the user to regain the control of the vehicle based on the first mapping and first current activity being assigned to the first activity group and the environment data.

2. The method of claim 1, wherein the first set of one or more classifiers are included in a convolutional neural network (CNN), the CNN including a first branch of a human body part confidence map, and a second branch of human body part affinity field.

3. The method of claim 1, wherein the second set of one or more classifiers are included in a real-time neural network configured to classify a set of body and/or hand and/or eye movement points of the user into a corresponding activity group.

4. The method of claim 1, wherein the first response time duration is a mean time duration for the user to regain the control of the vehicle when engaging an activity in the first activity group.

5. The method of claim 1, wherein generating the take-over request includes generating a first alert for presentation to the user, the first alert includes information facilitating the user to regain the control of the vehicle within the first response time duration.

6. The method of claim 1, wherein the method further comprises determining an actual time duration for the user to regain the control of the vehicle after the take-over request is generated, and updating the first response time duration based on the actual time duration.

7. The method of claim 1, wherein the response information further includes mappings between the activity groups and corresponding stimulus information for the user to regain a control of the vehicle, wherein the mappings includes a second mapping between the first activity group and corresponding first stimulus information, the first stimulus information indicating a first instruction to cause the user to regain the control of the vehicle when engaging the first activity, and wherein generating the take-over request for the user to regain the control of the vehicle is further based on the second mapping and first current activity being assigned to the first activity group.

8. The method of claim 7, wherein generating the take-over request for the user to regain the control of the vehicle includes generating an alert for presentation to the user according to the first instruction.

9. The method of claim 7, wherein the first instruction includes one or more commands for execution by the processor to deactivate or activate an component of the vehicle.

10. The method of claim 1, wherein the environment data includes at least one of cabin noise, cabin temperature, time of day, weather condition, road condition, local condition, and traffic condition.

11. A system for monitoring user activity in a vehicle, the system comprising one or more processors configured to execute machine-readable instructions, wherein when the machine-readable instructions are executed, the one or more processors are caused to perform:

storing, in a storage associated with the vehicle, response information regarding a user of the vehicle, the response information includes mappings between activity groups and corresponding response time duration for the user to regain a control of the vehicle, wherein the mappings include a first mapping between a first activity group and a first response time duration;

acquiring, from a first sensor in the vehicle, current activity data regarding the user, the current activity data of the user indicating a first current activity engaged by the user in the vehicle;

generating, using a first set of one or more classifiers, key point data regarding the user, the key point data indicating a first set of body and/or hand and/or eye movement points of the user during the first current activity;

acquiring, from a second sensor in the vehicle, environment data regarding the vehicle during the first current activity, the environment data includes information regarding an environment surrounding the user during first current activity;

assigning, using a second set of one or more classifiers, based on the key point data, the first current activity to the first activity group; and

generating a take-over request for the user to regain the control of the vehicle based on the first mapping and first current activity being assigned to the first activity group and the environment data.

12. The system of claim 11, wherein the first set of one or more classifiers are included in a convolutional neural network (CNN), the CNN including a first branch of a human body part confidence map, and a second branch of human body part affinity field.

13. The system of claim 11, wherein the second set of one or more classifiers are included in a real-time neural network configured to classify a set of body and/or hand and/or eye movement points of the user into a corresponding activity group.

14. The system of claim 11, wherein the first response time duration is a mean time duration for the user to regain the control of the vehicle when engaging an activity in the first activity group.

15. The system of claim 11, wherein generating the take-over request includes generating a first alert for presentation to the user, the first alert includes information facilitating the user to regain the control of the vehicle within the first response time duration.

16. The system of claim 11, wherein the method further comprises determining an actual time duration for the user to regain the control of the vehicle after the take-over request is generated, and updating the first response time duration based on the actual time duration.

17. The system of claim 11, wherein the response information further includes mappings between the activity groups and corresponding stimulus information for the user to regain a control of the vehicle, wherein the mappings includes a second mapping between the first activity group and corresponding first stimulus information, the first stimulus information indicating a first instruction to cause the user to regain the control of the vehicle when engaging the first activity, and wherein generating the take-over request for the user to regain the control of the vehicle is further based on the second mapping and first current activity being assigned to the first activity group.

18. The system of claim 17, wherein generating the take-over request for the user to regain the control of the vehicle includes generating an alert for presentation to the user according to the first instruction.

19. The system of claim 17, wherein the first instruction includes one or more commands for execution by the processor to deactivate or activate an component of the vehicle.

20. The system of claim 11, wherein the environment data includes at least one of cabin noise, cabin temperature, time of day, weather condition, road condition, local condition, and traffic condition.