Gesture recognition for on-board display

• US 9,977,593 B2
• Filed: 09/19/2016
• Issued: 05/22/2018
• Est. Priority Date: 11/16/2011
• Status: Active Grant

• Current Assignee: AutoConnect Holdings LLC
• Original Assignee: AutoConnect Holdings LLC
• Inventors: Ricci, Christopher P.
• Primary Examiner(s): Dager, Jonathan M
• Application Number:
  US15/269,617
• Publication Number:
  US 20170068438A1
• Time in Patent Office: 610 Days
• Field of Search: None
• CPC Class Codes: B60K 2360/11 Instrument graphical user interfaces or menu aspects
• CPC Class Codes: B60K 2360/1438 Touch screens
• CPC Class Codes: B60K 2360/146 Instrument input by gesture
• CPC Class Codes: B60K 2360/1464 3D-gesture
• CPC Class Codes: B60K 2360/1472 Multi-touch gesture
• CPC Class Codes: B60K 2360/1476 Handwriting
• CPC Class Codes: B60K 2360/164 Infotainment
• CPC Class Codes: B60K 2360/166 Navigation
• CPC Class Codes: B60K 2360/21 using cameras
• CPC Class Codes: B60K 2360/566 Mobile devices displaying vehicle information
• CPC Class Codes: B60K 2360/5894 SIM cards
• CPC Class Codes: B60K 2360/5899 Internet
• CPC Class Codes: B60K 2360/691 Housings
• CPC Class Codes: B60K 35/00 Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
• CPC Class Codes: B60K 35/10 Input arrangements, i.e. from user to vehicle, associated with vehicle functions or specially adapted therefor
• CPC Class Codes: B60K 35/28 characterised by the type of the output information, e.g. video entertainment or vehicle dynamics information; characterised by the purpose of the output information, e.g. for attracting the attention of the driver
• CPC Class Codes: B60K 35/50 Instruments characterised by their means of attachment to or integration in the vehicle B60K35/231 takes precedence
• CPC Class Codes: B60K 35/60 Instruments characterised by their location or relative disposition in or on vehicles arrangements of lighting devices on dashboards B60Q3/10
• CPC Class Codes: B60K 35/80 Arrangements for controlling instruments
• CPC Class Codes: B60K 35/85 Arrangements for transferring vehicle- or driver-related data
• CPC Class Codes: B60K 37/00 : Dashboards as road-vehicle superstructure sub-units B62D25/14
• CPC Class Codes: B60R 16/037 : for occupant comfort , e.g. for automatic adjustment of appliances according to personal settings, e.g. seats, mirrors, steering wheel
• CPC Class Codes: B60R 21/01512 : Passenger detection systems
• CPC Class Codes: B60R 7/04 : in driver or passenger space , e.g. using racks B60R7/005, B60R7/08 take precedence
• CPC Class Codes: B60W 2040/0872 : Driver physiology
• CPC Class Codes: B60W 2040/0881 : Seat occupation; Driver or passenger presence
• CPC Class Codes: B60W 2050/0095 : Automatic control mode change
• CPC Class Codes: B60W 2540/043 : Identity of occupants
• CPC Class Codes: B60W 2540/22 : Psychological state; Stress level or workload
• CPC Class Codes: B60W 2540/221 : Physiology, e.g. weight, heartbeat, health or special needs
• CPC Class Codes: B60W 2540/26 : Incapacity
• CPC Class Codes: B60W 2555/60 : Traffic rules, e.g. speed limits or right of way
• CPC Class Codes: B60W 30/182 : Selecting between different operative modes, e.g. comfort and performance modes
• CPC Class Codes: B60W 40/04 : Traffic conditions
• CPC Class Codes: B60W 40/08 : related to drivers or passengers
• CPC Class Codes: B60W 40/09 : Driving style or behaviour
• CPC Class Codes: B60W 50/0098 : Details of control systems ensuring comfort, safety or stability not otherwise provided for
• CPC Class Codes: B60W 50/08 : Interaction between the driver and the control system
• CPC Class Codes: G01C 21/20 : Instruments for performing navigational calculations G01C21/24, G01C21/26 take precedence
• CPC Class Codes: G01S 19/13 : Receivers
• CPC Class Codes: G02B 27/0101 : characterised by optical features G02B27/0172 takes precedence
• CPC Class Codes: G06F 11/2023 : Failover techniques
• CPC Class Codes: G06F 11/3013 : where the computing system is an embedded system, i.e. a combination of hardware and software dedicated to perform a certain function in mobile devices, printers, automotive or aircraft systems testing or monitoring of control systems or parts thereof G05B23/02
• CPC Class Codes: G06F 11/3065 : Monitoring arrangements determined by the means or processing involved in reporting the monitored data error or fault reporting or logging G06F11/0766
• CPC Class Codes: G06F 11/328 : Computer systems status display G06F11/327 takes precedence
• CPC Class Codes: G06F 13/14 : Handling requests for interconnection or transfer
• CPC Class Codes: G06F 13/364 : using independent requests or grants, e.g. using separated request and grant lines
• CPC Class Codes: G06F 16/24 : Querying
• CPC Class Codes: G06F 16/29 : Geographical information databases
• CPC Class Codes: G06F 17/00 : Digital computing or data processing equipment or methods, specially adapted for specific functions information retrieval, database structures or file system structures therefor G06F16/00
• CPC Class Codes: G06F 21/1062 : Editing
• CPC Class Codes: G06F 21/121 : Restricting unauthorised execution of programs
• CPC Class Codes: G06F 21/32 : using biometric data, e.g. fingerprints, iris scans or voiceprints
• CPC Class Codes: G06F 21/335 : for accessing specific resources, e.g. using Kerberos tickets
• CPC Class Codes: G06F 21/42 : using separate channels for security data
• CPC Class Codes: G06F 21/629 : to features or functions of an application
• CPC Class Codes: G06F 2203/04808 : Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/017 : Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0481 : based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/04817 : using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/0482 : Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/0484 : for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/04842 : Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04847 : Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06F 3/0486 : Drag-and-drop
• CPC Class Codes: G06F 3/0488 : using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/04883 : for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 8/61 : Installation
• CPC Class Codes: G06F 9/44505 : Configuring for program initiating, e.g. using registry, configuration files
• CPC Class Codes: G06F 9/54 : Interprogram communication
• CPC Class Codes: G06N 5/02 : Knowledge representation; Symbolic representation
• CPC Class Codes: G06N 5/048 : Fuzzy inferencing
• CPC Class Codes: G06Q 30/0265 : Vehicular advertisement
• CPC Class Codes: G06Q 40/08 : Insurance
• CPC Class Codes: G06Q 50/26 : Government or public services business processes related to the transportation industry G06Q50/40
• CPC Class Codes: G06T 19/006 : Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06V 20/59 : inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
• CPC Class Codes: G06V 40/10 : Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G06V 40/15 : Biometric patterns based on physiological signals, e.g. heartbeat, blood flow
• CPC Class Codes: G06V 40/28 : Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: G07C 5/006 : Indicating maintenance
• CPC Class Codes: G07C 5/008 : communicating information to a remotely located station transmission systems for measured values G08C
• CPC Class Codes: G07C 5/02 : Registering or indicating driving, working, idle, or waiting time only apparatus forming part of taximeters G07B13/00
• CPC Class Codes: G07C 5/08 : Registering or indicating performance data other than driving, working, idle, or waiting time, with or without registering driving, working, idle or waiting time
• CPC Class Codes: G07C 5/0808 : Diagnosing performance data testing of vehicles G01M17/00; testing of electrical installation on vehicles G01R31/005
• CPC Class Codes: G07C 5/0825 : using optical means
• CPC Class Codes: G07C 5/0833 : using audio means
• CPC Class Codes: G07C 5/085 : using electronic data carriers
• CPC Class Codes: G07C 5/0866 : the electronic data carrier being a digital video recorder in combination with video camera
• CPC Class Codes: G08C 19/00 : Electric signal transmission systems G08C17/00 takes precedence
• CPC Class Codes: G08G 1/0112 : from the vehicle, e.g. floating car data [FCD]
• CPC Class Codes: G08G 1/0116 : from roadside infrastructure, e.g. beacons
• CPC Class Codes: G08G 1/012 : from other sources than vehicle or roadside beacons, e.g. mobile networks
• CPC Class Codes: G08G 1/0129 : for creating historical data or processing based on historical data
• CPC Class Codes: G08G 1/0133 : for classifying traffic situation
• CPC Class Codes: G08G 1/0137 : for specific applications
• CPC Class Codes: G08G 1/0141 : for traffic information dissemination
• CPC Class Codes: G08G 1/017 : identifying vehicles G08G1/015, G08G1/054 take precedence
• CPC Class Codes: G08G 1/09 : Arrangements for giving variable traffic instructions railroad crossing signals B61L; reflectors E01F, G08B; indicating arrangements for variable information by selection or combination of individual elements G09F9/00
• CPC Class Codes: G08G 1/096716 : where the received information does not generate an automatic action on the vehicle control
• CPC Class Codes: G08G 1/096725 : where the received information generates an automatic action on the vehicle control
• CPC Class Codes: G08G 1/096775 : where the origin of the information is a central station
• CPC Class Codes: G08G 1/096783 : where the origin of the information is a roadside individual element
• CPC Class Codes: G08G 1/096791 : where the origin of the information is another vehicle
• CPC Class Codes: G08G 1/096827 : where the route is computed onboard
• CPC Class Codes: G08G 1/096844 : where the complete route is dynamically recomputed based on new data
• CPC Class Codes: G08G 1/143 : inside the vehicles
• CPC Class Codes: G08G 1/16 : Anti-collision systems road vehicle drive control systems for predicting or avoiding probable or impending collision otherwise than by control of a particular sub-unit B60W30/08
• CPC Class Codes: H04L 41/14 : Network analysis or design
• CPC Class Codes: H04L 51/56 : Unified messaging, e.g. interactions between e-mail, instant messaging or converged IP messaging [CPM]
• CPC Class Codes: H04L 63/10 : for controlling access to devices or network resources
• CPC Class Codes: H04L 67/02 : based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04L 67/12 : specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
• CPC Class Codes: H04L 67/125 : involving control of end-device applications over a network
• CPC Class Codes: H04N 21/41422 : located in transportation means, e.g. personal vehicle arrangements specially adapted for transportation systems in broadcast systems H04H20/62
• CPC Class Codes: H04N 21/4331 : Caching operations, e.g. of an advertisement for later insertion during playback
• CPC Class Codes: H04N 21/482 : End-user interface for program selection broadcast systems using EPGs H04H60/72
• CPC Class Codes: H04W 4/40 : for vehicles, e.g. vehicle-to-pedestrians [V2P]
• CPC Class Codes: H04W 4/48 : for in-vehicle communication
• CPC Class Codes: H04W 4/50 : Service provisioning or reconfiguring
• CPC Class Codes: H04W 4/80 : Services using short range communication, e.g. near-field communication [NFC], radio-frequency identification [RFID] or low energy communication
• CPC Class Codes: H04W 4/90 : Services for handling of emergency or hazardous situations, e.g. earthquake and tsunami warning systems [ETWS]
• CPC Class Codes: H04W 8/22 : Processing or transfer of terminal data, e.g. status or physical capabilities
• CPC Class Codes: H04W 84/005 : Moving wireless networks
• CPC Class Codes: H04W 84/12 : WLAN [Wireless Local Area Networks]

Abstract

Methods and systems for a complete vehicle ecosystem are provided. Specifically, systems that when taken alone, or together, provide an individual or group of individuals with an intuitive and comfortable vehicular environment. The present disclosure includes a system to recognize the drivers and/or passengers within the automobile. Based on the recognition, the vehicle may change a configuration of the automobile to match predetermined preferences for the driver and/or passenger. The configurations may also include the recognition of a unique set of gestures for the person. Further, the configuration can also include the tracking of health data related to the person.

Claims

1. A method comprising:

receiving first sensor data from one or more vehicle sensors in a vehicle;

based on the first sensor data, determining a first zone of two or more zones in which a first occupant occupies;

determining, using the first sensor data, one or more characteristics of the first occupant and identifying the first occupant as a user based on the one or more characteristics;

receiving and storing first health data from the one or more vehicle sensors, wherein the received and stored health data is associated with the user and the determined first zone;

comparing the first health data received from the one or more vehicle sensors to a first baseline based on the determined first zone;

at a time subsequent to receiving the first sensor data, receiving second sensor data from the one or more vehicle sensors;

based on the second sensor data, determining a second zone of the two or more zones in which a second occupant occupies;

identifying the second occupant as the user;

receiving and storing second health data from the one or more vehicle sensors, wherein the second health data is associated with the user and the determined second zone;

comparing the second health data received from the one or more vehicle sensors to a second baseline based on the determined second zone; and

based on at least one of the comparison of the first health data to the first baseline based on the determined first zone and the comparison of the second health data to the second baseline based on the determined second zone, modifying an operation of the vehicle, including at least one of modifying steering, acceleration, and braking of the vehicle.

2. The method of claim 1, further comprising:

storing one or more of the first health data and the second health data in a storage location outside a vehicle having the one or more vehicle sensors.

3. The method of claim 1, further comprising:

creating a new record for one or more of the first and the second occupant if the one or more characteristics do not match characteristics in the one or more records.

4. The method of claim 1, further comprising:

determining if the health data is to be stored for one or more of the first and the second occupant; and

storing the health data.

5. The method of claim 4, wherein determining if the health data is to be stored for one or more of the first and the second occupant comprises determining if a second baseline data pattern for one or more of the first and the second occupant can be provided based on an amount of the health data.

6. The method of claim 1, further comprising:

receiving additional health data from the one or more vehicle sensors; and

storing the additional health data.

7. The method of claim 6, further comprising:

determining if the additional health data is within acceptable limits; and

modifying at least one of a function and operation of the vehicle if the additional health data is not within acceptable limits.

8. The method of claim 1, wherein the health data includes at least one of:

heart rate, blood pressure, blood presence, blood composition, temperature, a presence of liquid, weight, weight shift, position, breathing rate, anxiety level, state of awareness, consciousness, and reaction from a driver.

9. The method of claim 1, further comprising:

receiving additional sensor data from at least one vehicle sensor that indicates one or more of the first and the second occupant is within the vehicle.

10. The method of claim 1, wherein the baseline is based on sensor data received from the one or more vehicle sensors that measure health data over time.

11. The method of claim 1, further comprising:

storing one or more of the health data in a storage location outside the vehicle.

12. A non-transitory computer readable storage medium comprising processor executable instruction operable to perform the method of claim 1.