Gesture recognition for on-board display

• US 9,449,516 B2
• Filed: 12/01/2014
• Issued: 09/20/2016
• Est. Priority Date: 11/16/2011
• Status: Active Grant

• Current Assignee: AutoConnect Holdings LLC
• Original Assignee: AutoConnect Holdings LLC
• Inventors: Ricci, Christopher P.
• Primary Examiner(s): Dager, Jonathan M
• Application Number:
  US14/557,427
• Publication Number:
  US 20150097798A1
• Time in Patent Office: 659 Days
• US Class Current: 1/1
• CPC Class Codes: B60K 2360/11 Instrument graphical user interfaces or menu aspects
• CPC Class Codes: B60K 2360/1438 Touch screens
• CPC Class Codes: B60K 2360/146 Instrument input by gesture
• CPC Class Codes: B60K 2360/1464 3D-gesture
• CPC Class Codes: B60K 2360/1472 Multi-touch gesture
• CPC Class Codes: B60K 2360/1476 Handwriting
• CPC Class Codes: B60K 2360/164 Infotainment
• CPC Class Codes: B60K 2360/166 Navigation
• CPC Class Codes: B60K 2360/21 using cameras
• CPC Class Codes: B60K 2360/566 Mobile devices displaying vehicle information
• CPC Class Codes: B60K 2360/5894 SIM cards
• CPC Class Codes: B60K 2360/5899 Internet
• CPC Class Codes: B60K 2360/691 Housings
• CPC Class Codes: B60K 2370/11 Graphical user interfaces or menu aspects
• CPC Class Codes: B60K 2370/1438 Touch screens
• CPC Class Codes: B60K 2370/146 Input by gesture
• CPC Class Codes: B60K 2370/1464 3D-gesture
• CPC Class Codes: B60K 2370/1472 Multi-touch gesture
• CPC Class Codes: B60K 2370/1476 Handwriting
• CPC Class Codes: B60K 2370/164 Infotainment
• CPC Class Codes: B60K 2370/166 : Navigation
• CPC Class Codes: B60K 2370/21 : using cameras
• CPC Class Codes: B60K 2370/566 : Mobile devices displaying vehicle information
• CPC Class Codes: B60K 2370/5894 : SIM cards
• CPC Class Codes: B60K 2370/5899 : Internet
• CPC Class Codes: B60K 2370/691 : Housings
• CPC Class Codes: B60K 2370/81 : Fastening of instruments, e.g. to dashboard
• CPC Class Codes: B60K 35/00 : Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
• CPC Class Codes: B60K 35/10 : Input arrangements, i.e. from user to vehicle, associated with vehicle functions or specially adapted therefor
• CPC Class Codes: B60K 35/28 : characterised by the type of the output information, e.g. video entertainment or vehicle dynamics information; characterised by the purpose of the output information, e.g. for attracting the attention of the driver
• CPC Class Codes: B60K 35/50 : Instruments characterised by their means of attachment to or integration in the vehicle B60K35/231 takes precedence
• CPC Class Codes: B60K 35/60 : Instruments characterised by their location or relative disposition in or on vehicles arrangements of lighting devices on dashboards B60Q3/10
• CPC Class Codes: B60K 35/80 : Arrangements for controlling instruments
• CPC Class Codes: B60K 35/85 : Arrangements for transferring vehicle- or driver-related data
• CPC Class Codes: B60K 37/00 : Dashboards as road-vehicle superstructure sub-units B62D25/14
• CPC Class Codes: B60R 16/037 : for occupant comfort , e.g. for automatic adjustment of appliances according to personal settings, e.g. seats, mirrors, steering wheel
• CPC Class Codes: B60R 21/01512 : Passenger detection systems
• CPC Class Codes: B60R 7/04 : in driver or passenger space , e.g. using racks B60R7/005, B60R7/08 take precedence
• CPC Class Codes: B60W 2040/0872 : Driver physiology
• CPC Class Codes: B60W 2040/0881 : Seat occupation; Driver or passenger presence
• CPC Class Codes: B60W 2050/0095 : Automatic control mode change
• CPC Class Codes: B60W 2540/043 : Identity of occupants
• CPC Class Codes: B60W 2540/22 : Psychological state; Stress level or workload
• CPC Class Codes: B60W 2540/221 : Physiology, e.g. weight, heartbeat, health or special needs
• CPC Class Codes: B60W 2540/26 : Incapacity
• CPC Class Codes: B60W 2555/60 : Traffic rules, e.g. speed limits or right of way
• CPC Class Codes: B60W 30/182 : Selecting between different operative modes, e.g. comfort and performance modes
• CPC Class Codes: B60W 40/04 : Traffic conditions
• CPC Class Codes: B60W 40/08 : related to drivers or passengers
• CPC Class Codes: B60W 40/09 : Driving style or behaviour
• CPC Class Codes: B60W 50/0098 : Details of control systems ensuring comfort, safety or stability not otherwise provided for
• CPC Class Codes: B60W 50/08 : Interaction between the driver and the control system
• CPC Class Codes: G01C 21/20 : Instruments for performing navigational calculations G01C21/24, G01C21/26 take precedence
• CPC Class Codes: G01S 19/13 : Receivers
• CPC Class Codes: G02B 27/0101 : characterised by optical features G02B27/0172 takes precedence
• CPC Class Codes: G06F 11/2023 : Failover techniques
• CPC Class Codes: G06F 11/3013 : where the computing system is an embedded system, i.e. a combination of hardware and software dedicated to perform a certain function in mobile devices, printers, automotive or aircraft systems testing or monitoring of control systems or parts thereof G05B23/02
• CPC Class Codes: G06F 11/3065 : Monitoring arrangements determined by the means or processing involved in reporting the monitored data error or fault reporting or logging G06F11/0766
• CPC Class Codes: G06F 11/328 : Computer systems status display G06F11/327 takes precedence
• CPC Class Codes: G06F 13/14 : Handling requests for interconnection or transfer
• CPC Class Codes: G06F 13/364 : using independent requests or grants, e.g. using separated request and grant lines
• CPC Class Codes: G06F 16/24 : Querying
• CPC Class Codes: G06F 16/29 : Geographical information databases
• CPC Class Codes: G06F 17/00 : Digital computing or data processing equipment or methods, specially adapted for specific functions information retrieval, database structures or file system structures therefor G06F16/00
• CPC Class Codes: G06F 21/1062 : Editing
• CPC Class Codes: G06F 21/121 : Restricting unauthorised execution of programs
• CPC Class Codes: G06F 21/32 : using biometric data, e.g. fingerprints, iris scans or voiceprints
• CPC Class Codes: G06F 21/335 : for accessing specific resources, e.g. using Kerberos tickets
• CPC Class Codes: G06F 21/42 : using separate channels for security data
• CPC Class Codes: G06F 21/629 : to features or functions of an application
• CPC Class Codes: G06F 2203/04808 : Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/017 : Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0481 : based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/04817 : using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/0482 : Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/0484 : for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/04842 : Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04847 : Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06F 3/0486 : Drag-and-drop
• CPC Class Codes: G06F 3/0488 : using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/04883 : for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 8/61 : Installation
• CPC Class Codes: G06F 9/44505 : Configuring for program initiating, e.g. using registry, configuration files
• CPC Class Codes: G06F 9/54 : Interprogram communication
• CPC Class Codes: G06N 5/02 : Knowledge representation; Symbolic representation
• CPC Class Codes: G06N 5/048 : Fuzzy inferencing
• CPC Class Codes: G06Q 30/0265 : Vehicular advertisement
• CPC Class Codes: G06Q 40/08 : Insurance
• CPC Class Codes: G06Q 50/26 : Government or public services business processes related to the transportation industry G06Q50/40
• CPC Class Codes: G06T 19/006 : Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06V 20/59 : inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
• CPC Class Codes: G06V 40/10 : Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G06V 40/15 : Biometric patterns based on physiological signals, e.g. heartbeat, blood flow
• CPC Class Codes: G06V 40/28 : Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: G07C 5/006 : Indicating maintenance
• CPC Class Codes: G07C 5/008 : communicating information to a remotely located station transmission systems for measured values G08C
• CPC Class Codes: G07C 5/02 : Registering or indicating driving, working, idle, or waiting time only apparatus forming part of taximeters G07B13/00
• CPC Class Codes: G07C 5/08 : Registering or indicating performance data other than driving, working, idle, or waiting time, with or without registering driving, working, idle or waiting time
• CPC Class Codes: G07C 5/0808 : Diagnosing performance data testing of vehicles G01M17/00; testing of electrical installation on vehicles G01R31/005
• CPC Class Codes: G07C 5/0825 : using optical means
• CPC Class Codes: G07C 5/0833 : using audio means
• CPC Class Codes: G07C 5/085 : using electronic data carriers
• CPC Class Codes: G07C 5/0866 : the electronic data carrier being a digital video recorder in combination with video camera
• CPC Class Codes: G08C 19/00 : Electric signal transmission systems G08C17/00 takes precedence
• CPC Class Codes: G08G 1/0112 : from the vehicle, e.g. floating car data [FCD]
• CPC Class Codes: G08G 1/0116 : from roadside infrastructure, e.g. beacons
• CPC Class Codes: G08G 1/012 : from other sources than vehicle or roadside beacons, e.g. mobile networks
• CPC Class Codes: G08G 1/0129 : for creating historical data or processing based on historical data
• CPC Class Codes: G08G 1/0133 : for classifying traffic situation
• CPC Class Codes: G08G 1/0137 : for specific applications
• CPC Class Codes: G08G 1/0141 : for traffic information dissemination
• CPC Class Codes: G08G 1/017 : identifying vehicles G08G1/015, G08G1/054 take precedence
• CPC Class Codes: G08G 1/09 : Arrangements for giving variable traffic instructions railroad crossing signals B61L; reflectors E01F, G08B; indicating arrangements for variable information by selection or combination of individual elements G09F9/00
• CPC Class Codes: G08G 1/096716 : where the received information does not generate an automatic action on the vehicle control
• CPC Class Codes: G08G 1/096725 : where the received information generates an automatic action on the vehicle control
• CPC Class Codes: G08G 1/096775 : where the origin of the information is a central station
• CPC Class Codes: G08G 1/096783 : where the origin of the information is a roadside individual element
• CPC Class Codes: G08G 1/096791 : where the origin of the information is another vehicle
• CPC Class Codes: G08G 1/096827 : where the route is computed onboard
• CPC Class Codes: G08G 1/096844 : where the complete route is dynamically recomputed based on new data
• CPC Class Codes: G08G 1/143 : inside the vehicles
• CPC Class Codes: G08G 1/16 : Anti-collision systems road vehicle drive control systems for predicting or avoiding probable or impending collision otherwise than by control of a particular sub-unit B60W30/08
• CPC Class Codes: H04L 41/14 : Network analysis or design
• CPC Class Codes: H04L 51/56 : Unified messaging, e.g. interactions between e-mail, instant messaging or converged IP messaging [CPM]
• CPC Class Codes: H04L 63/10 : for controlling access to devices or network resources
• CPC Class Codes: H04L 67/02 : based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04L 67/12 : specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
• CPC Class Codes: H04L 67/125 : involving control of end-device applications over a network
• CPC Class Codes: H04N 21/41422 : located in transportation means, e.g. personal vehicle arrangements specially adapted for transportation systems in broadcast systems H04H20/62
• CPC Class Codes: H04N 21/4331 : Caching operations, e.g. of an advertisement for later insertion during playback
• CPC Class Codes: H04N 21/482 : End-user interface for program selection broadcast systems using EPGs H04H60/72
• CPC Class Codes: H04W 4/40 : for vehicles, e.g. vehicle-to-pedestrians [V2P]
• CPC Class Codes: H04W 4/48 : for in-vehicle communication
• CPC Class Codes: H04W 4/50 : Service provisioning or reconfiguring
• CPC Class Codes: H04W 4/80 : Services using short range communication, e.g. near-field communication [NFC], radio-frequency identification [RFID] or low energy communication
• CPC Class Codes: H04W 4/90 : Services for handling of emergency or hazardous situations, e.g. earthquake and tsunami warning systems [ETWS]
• CPC Class Codes: H04W 8/22 : Processing or transfer of terminal data, e.g. status or physical capabilities
• CPC Class Codes: H04W 84/005 : Moving wireless networks
• CPC Class Codes: H04W 84/12 : WLAN [Wireless Local Area Networks]

Abstract

Methods and systems for a complete vehicle ecosystem are provided. Specifically, systems that when taken alone, or together, provide an individual or group of individuals with an intuitive and comfortable vehicular environment. The present disclosure includes a system to recognize the drivers and/or passengers within the automobile. Based on the recognition, the vehicle may change a configuration of the automobile to match predetermined preferences for the driver and/or passenger. The configurations may also include the recognition of a unique set of gestures for the person. Further, the configuration can also include the tracking of health data related to the person.

Claims

1. A method, comprising:

automatically identifying, by one or more processors of a vehicle control system of a vehicle, a person;

determining, by the one or more processors of the vehicle control system of the vehicle, which one of two or more zones within the vehicle is occupied by the identified person;

determining, by the one or more processors of the vehicle control system, if there is a gesture to be stored for the identified person;

receiving a gesture made upon a receiving surface of an input device, wherein the received gesture configures the vehicle;

associating the received gesture with the identified person and with the determined zone occupied by the person, wherein the received gesture, if associated with a first zone of the two or more zones, has a first vehicle response and a same gesture, if associated with a second zone of the two or more zones, has a second vehicle response, and wherein the first and second vehicle responses are different; and

storing, by the one or more processors of the vehicle control system, the received gesture in a user record associated with the identified person.

2. The method of claim 1, wherein the received gesture that is made upon the receiving surface of the input device is a gesture in a gesture capture region or a gesture in a touch-sensitive display.

3. The method of claim 1, wherein automatically identifying the person comprises:

receiving sensor data from at least one sensor;

comparing the sensor data to feature data in a data structure; and

determining if the sensor data is substantially similar to the feature data.

4. The method of claim 1, wherein determining if there is a gesture to be stored for the person comprises determining if the person has created a gesture.

5. The method of claim 4, wherein determining if there is a gesture to be stored for the person further comprises receiving an indication from the person that a gesture is to be created.

6. The method of claim 1, wherein the gesture is stored in cloud storage.

7. A non-transitory computer-readable medium having stored thereon computer-executable instructions, the computer-executable instructions causing the one or more processors to execute the method of claim 1.

14. The method of claim 1, wherein a first portion of the user record includes information identifying the person, wherein a second portion of the user record includes the received gesture, and wherein a third portion of the user record includes the determined zone occupied by the identified person.

15. The method of claim 1, wherein the received gesture is made upon the receiving surface of the input device in a gesture capture region outside of a touch sensitive display.

8. A vehicle system of a vehicle, comprising:

one or more sensors;

a vehicle control system comprising;

one or more processors;

a memory;

wherein the one or more processors are configured to;

receive sensor data from the one or more sensors;

automatically identify a person;

determine which one of two or more zones within the vehicle is occupied by the identified person;

determine if the identified person has made a gesture to be stored;

receive the gesture, wherein the gesture is made upon a receiving surface of an input device and the gesture automatically configures the vehicle;

associate the received gesture with the identified person and with the determined zone occupied by the person, wherein the received gesture, if associated with a first zone of the two or more zones, has a first vehicle response and a same gesture, if associated with a second zone of the two or more zones, has a second vehicle response, and wherein the first and second vehicle responses are different; and

store the received gesture in a user record associated with the identified person.

9. The system of claim 8, wherein the received gesture that is made upon the receiving surface of the input device is a gesture in a gesture capture region or a gesture in a touch sensitive display.

10. The system of claim 8, wherein in determining if there is a gesture to be stored for the person, the one or more processors of the vehicle control system are further configured to determine if the person has created a gesture.

11. The system of claim 10, wherein in determining if there is a gesture to be stored for the person, the one or more processors of the vehicle control system are further configured to receive an indication from the person that a gesture is to be created.

12. The system of claim 8, wherein the gesture is stored in cloud storage.

16. The system of claim 8, wherein a first portion of the user record includes information identifying the person, wherein a second portion of the user record includes the received gesture, and wherein a third portion of the user record includes the determined zone occupied by the identified person.

17. The system of claim 8, wherein the received gesture is made upon the receiving surface of the input device in a gesture capture region outside of a touch sensitive display.

13. A method, comprising:

receiving sensor data from one or more sensors;

based on the sensor data, automatically identifying a person;

based on the sensor data, automatically determining which one of two or more zones within a vehicle is occupied by the identified person;

retrieving two or more gestures associated with the determined zone and with the identified person, wherein a retrieved gesture, if associated with a first zone of the two or more zones, has a first vehicle response and a same gesture, if associated with a second zone of the two or more zones, has a second vehicle response, and wherein the first and second vehicle responses are different;

receiving a gesture made upon a receiving surface of an input device, wherein the received gesture automatically configures the vehicle;

comparing the received gesture to the two or more retrieved gestures associated with the determined zone and with the identified person; and

based on the comparison, automatically configuring the vehicle according to the received gesture.

18. The method of claim 13, wherein a first portion of a user record associated with the identified person includes information identifying the person, wherein a second portion of the user record includes the two or more retrieved gestures, and wherein a third portion of the user record includes the determined zone occupied by the identified person.