Apparatuses, systems, and methods for vehicle operator gesture recognition and transmission of related gesture data

• US 10,296,785 B1
• Filed: 07/24/2017
• Issued: 05/21/2019
• Est. Priority Date: 07/24/2017
• Status: Active Grant

• Current Assignee: State Farm Mutual Automobile Insurance Company (State Farm)
• Original Assignee: State Farm Mutual Automobile Insurance Company (State Farm)
• Inventors: Chan, Aaron Scott, Sanchez, Kenneth J.
• Primary Examiner(s): Abdi, Amara
• Application Number:
  US15/657,550
• Time in Patent Office: 666 Days
• CPC Class Codes: G06F 18/23 Clustering techniques
• CPC Class Codes: G06F 18/2415 based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
• CPC Class Codes: G06Q 40/08 Insurance
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30268 Vehicle interior
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 20/59 inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
• CPC Class Codes: G06V 20/597 Recognising the driver's state or behaviour, e.g. attention or drowsiness
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

Apparatuses, systems and methods are provided for vehicle operator gesture recognition and transmission of related gesture data. More particularly, apparatuses, systems and methods are provided for vehicle operator gesture recognition and transmission of related gesture data based on classification of image data.

Claims

1. A device for determining vehicle operator gestures and providing the vehicle operator gestures to at least one individual other than the vehicle operator, the device comprising:

a previously classified image data receiving module stored on a memory that, when executed by a processor, causes the processor to receive previously classified image data from at least one previously classified image database, wherein the previously classified image data is representative of previously classified vehicle operator gestures;

a current image data receiving module stored on a memory that, when executed by a processor, causes the processor to receive current image data from at least one vehicle interior sensor, wherein the current image data is representative of current vehicle operator gestures, and wherein the current image data is representative of rotated and scaled postures that are normalized for a range of different vehicle operators;

a current image data classification module stored on a memory that, when executed by a processor, causes the processor to classify the current image data by comparing the current image data to the previously classified image data, wherein the currently classified image data is representative of at least one current vehicle operator gesture; and

a currently classified image data transmission module stored on a memory that, when executed by a processor, causes the processor to transmit the currently classified image data to at least one individual other than the vehicle operator.

2. The device as in claim 1, wherein the at least one vehicle interior sensor is selected from:

at least one digital image sensor, at least one ultra-sonic sensor, at least one radar-sensor, at least one infrared light sensor, or at least one laser light sensor.

3. The device as in claim 1, wherein at least one vehicle operator gesture is determined using a probability function.

4. The device as in claim 1, wherein the current image data is representative of a three-dimensional representation of at least one occupant within the vehicle interior.

5. The device as in claim 1, wherein the previously classified image data is representative of a three-dimensional representation of at least one occupant within the vehicle interior.

6. The device as in claim 1, wherein the current image data includes images and/or extracted image features that are representative of a vehicle occupant interacting with an electronic device, a vehicle occupant interacting with an onboard computer interface, a vehicle occupant using a cellular telephone, a vehicle occupant looking out a vehicle side window, a vehicle occupant adjusting a vehicle radio, a vehicle occupant adjusting a vehicle heating, ventilation and air conditioning system, two vehicle occupants talking with one-another, a vehicle occupant reading a book or magazine, a vehicle occupant putting on makeup, a vehicle occupant looking at themselves in a mirror, a vehicle occupant eating, or a vehicle occupant drinking.

7. The device as in claim 1, wherein the previously classified image data includes images and/or extracted image features that have previously been classified as being representative of a vehicle occupant interacting with an electronic device, a vehicle occupant interacting with an onboard computer interface, a vehicle occupant using a mobile device, a vehicle occupant looking out a vehicle side window, a vehicle occupant adjusting a vehicle radio, a vehicle occupant adjusting a vehicle heating, ventilation and air conditioning system, two vehicle occupants talking with one-another, a vehicle occupant reading a book or magazine, a vehicle occupant putting on makeup, a vehicle occupant looking at themselves in a mirror, a vehicle occupant eating, or a vehicle occupant drinking.

8. A computer-implemented method for determining vehicle operator gestures and for transmitting the vehicle operator gestures to at least one individual other than the vehicle operator, the method comprising:

receiving, at a processor of a computing device, previously classified image data from at least one previously classified image database in response to the processor executing a previously classified image data receiving module, wherein the previously classified image data is representative of previously classified vehicle operator gestures;

receiving, at a processor of a computing device, current image data from at least one vehicle interior sensor a current image data receiving module, in response to the processor executing a current image data receiving module, wherein the current image data is representative of at least one current vehicle operator gesture, and wherein the current image data is representative of rotated and scaled postures that are normalized for a range of different vehicle operators;

classifying, using a processor of a computing device, at least one gesture associated with a vehicle operator, based on a comparison of the current image data with the previously classified image data, in response to the processor executing a current image data classification module; and

transmitting, using a processor of a computing device, the currently classified image data to at least one individual other than the vehicle operator, in response to the processor executing a currently classified image data transmission module.

9. The method as in claim 8, wherein the at least one vehicle interior sensor is selected from:

at least one digital image sensor, at least one ultra-sonic sensor, at least one radar-sensor, at least one infrared light sensor, or at least one laser light sensor.

10. The method as in claim 8, wherein the current image data is representative of either a two-dimensional or a three-dimensional representation of at least one occupant within the vehicle interior.

11. The method as in claim 8, wherein at least one vehicle operator gesture is determined using a probability function.

12. The method as in claim 8, wherein the previously classified image data is representative of a three-dimensional representation of at least one occupant within the vehicle interior.

13. The method as in claim 8, wherein the current image data includes images and/or extracted image features that are representative of a vehicle occupant interacting with an onboard computer interface, a vehicle occupant interacting with an electronic device, a vehicle occupant interacting with an onboard computer interface, vehicle occupant locations/orientations, cellular telephone locations/orientations, vehicle occupant eye locations/orientations, vehicle occupant head location/orientation, vehicle occupant hand location/orientation, a vehicle occupant torso location/orientation, a seat belt location, or a vehicle seat location/orientation.

14. The method as in claim 8, wherein the previously classified image data includes images and/or extracted image features that have previously been classified as being representative of known vehicle occupant interacting with an electronic device, vehicle occupant interacting with an onboard computer interface, vehicle occupant locations/orientations, known cellular telephone locations/orientations, known vehicle occupant eye locations/orientations, known vehicle occupant head location/orientation, known vehicle occupant hand location/orientation, a known vehicle occupant torso location/orientation, a known seat belt location, or a known vehicle seat location/orientation.

15. A non-transitory computer-readable medium storing computer-readable instructions that, when executed by a processor, cause the processor to determine degrees of vehicle operator risk, the non-transitory computer-readable medium comprising:

a previously classified image data receiving module that, when executed by a processor, causes the processor to receive previously classified image data from at least one previously classified image database, wherein the previously classified image data is representative of previously classified vehicle operator risk;

a current image data receiving module that, when executed by a processor, causes the processor to receive current image data from at least one vehicle interior sensor, wherein the current image data is representative of current vehicle operator risk, and wherein the current image data is representative of rotated and scaled postures that are normalized for a range of different vehicle operators;

a current image data classification module that, when executed by a processor, causes the processor to classify the current image data by comparing the current image data to the previously classified image data, wherein the currently classified image data is representative of at least one current vehicle operator gesture; and

a currently classified image data transmission module that, when executed by a processor, causes the processor to transmit the currently classified image data to at least one individual other than the vehicle operator.

16. The non-transitory computer-readable medium as in claim 15, wherein the vehicle operator degree of risk is determined using a probability function.

17. The non-transitory computer-readable medium as in claim 15, wherein the current image data is representative of a three-dimensional representation of at least one occupant within the vehicle interior.

18. The non-transitory computer-readable medium as in claim 15, wherein the current image data includes images and/or extracted image features that are representative of a vehicle occupant interacting with an electronic device, a vehicle occupant interacting with an onboard computer interface, a vehicle occupant using a cellular telephone, a vehicle occupant looking out a vehicle side window, a vehicle occupant adjusting a vehicle radio, a vehicle occupant adjusting a vehicle heating, ventilation and air conditioning system, two vehicle occupants talking with one-another, a vehicle occupant reading a book or magazine, a vehicle occupant putting on makeup, or a vehicle occupant looking at themselves in a mirror, vehicle occupant locations/orientations, cellular telephone locations/orientations, vehicle occupant eye locations/orientations, vehicle occupant head location/orientation, vehicle occupant hand location/orientation, a vehicle occupant torso location/orientation, a seat belt location, or a vehicle seat location/orientation.

19. The non-transitory computer-readable medium as in claim 15, wherein the previously classified image data includes images and/or extracted image features that have previously been classified as being representative of a vehicle occupant interacting with an electronic device, a vehicle occupant interacting with an onboard computer interface, a vehicle occupant using a cellular telephone, a vehicle occupant looking out a vehicle side window, a vehicle occupant adjusting a vehicle radio, a vehicle occupant adjusting a vehicle heating, ventilation and air conditioning system, two vehicle occupants talking with one-another, a vehicle occupant reading a book or magazine, a vehicle occupant putting on makeup, a vehicle occupant looking at themselves in a mirror, known vehicle occupant locations/orientations, known cellular telephone locations/orientations, known vehicle occupant eye locations/orientations, known vehicle occupant head location/orientation, known vehicle occupant hand location/orientation, a known vehicle occupant torso location/orientation, a known seat belt location, or a known vehicle seat location/orientation.

20. The non-transitory computer-readable medium as in claim 15, wherein the previously classified image data is representative of a three-dimensional representation of at least one occupant within the vehicle interior.