GESTURE RECOGNITION FOR ON-BOARD DISPLAY

• US 20130151031A1
• Filed: 11/16/2012
• Published: 06/13/2013
• Est. Priority Date: 11/16/2011
• Status: Active Grant

• Current Assignee: AutoConnect Holdings LLC
• Original Assignee: Flextronics America LLC (Flextronics International Limited)
• Inventors: Ricci, Christopher P.
• Granted Patent: US 8,831,826 B2
• Time in Patent Office: Days
• US Class Current: 701/1
• CPC Class Codes: B60K 2360/11 Instrument graphical user interfaces or menu aspects
• CPC Class Codes: B60K 2360/1438 Touch screens
• CPC Class Codes: B60K 2360/146 Instrument input by gesture
• CPC Class Codes: B60K 2360/1464 3D-gesture
• CPC Class Codes: B60K 2360/1472 Multi-touch gesture
• CPC Class Codes: B60K 2360/1476 Handwriting
• CPC Class Codes: B60K 2360/164 Infotainment
• CPC Class Codes: B60K 2360/166 Navigation
• CPC Class Codes: B60K 2360/21 using cameras
• CPC Class Codes: B60K 2360/566 Mobile devices displaying vehicle information
• CPC Class Codes: B60K 2360/5894 SIM cards
• CPC Class Codes: B60K 2360/5899 Internet
• CPC Class Codes: B60K 2360/691 Housings
• CPC Class Codes: B60K 2370/11 Graphical user interfaces or menu aspects
• CPC Class Codes: B60K 2370/1438 Touch screens
• CPC Class Codes: B60K 2370/146 Input by gesture
• CPC Class Codes: B60K 2370/1464 3D-gesture
• CPC Class Codes: B60K 2370/1472 Multi-touch gesture
• CPC Class Codes: B60K 2370/1476 Handwriting
• CPC Class Codes: B60K 2370/164 Infotainment
• CPC Class Codes: B60K 2370/166 : Navigation
• CPC Class Codes: B60K 2370/21 : using cameras
• CPC Class Codes: B60K 2370/566 : Mobile devices displaying vehicle information
• CPC Class Codes: B60K 2370/5894 : SIM cards
• CPC Class Codes: B60K 2370/5899 : Internet
• CPC Class Codes: B60K 2370/691 : Housings
• CPC Class Codes: B60K 2370/81 : Fastening of instruments, e.g. to dashboard
• CPC Class Codes: B60K 35/00 : Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
• CPC Class Codes: B60K 35/10 : Input arrangements, i.e. from user to vehicle, associated with vehicle functions or specially adapted therefor
• CPC Class Codes: B60K 35/28 : characterised by the type of the output information, e.g. video entertainment or vehicle dynamics information; characterised by the purpose of the output information, e.g. for attracting the attention of the driver
• CPC Class Codes: B60K 35/50 : Instruments characterised by their means of attachment to or integration in the vehicle B60K35/231 takes precedence
• CPC Class Codes: B60K 35/60 : Instruments characterised by their location or relative disposition in or on vehicles arrangements of lighting devices on dashboards B60Q3/10
• CPC Class Codes: B60K 35/80 : Arrangements for controlling instruments
• CPC Class Codes: B60K 35/85 : Arrangements for transferring vehicle- or driver-related data
• CPC Class Codes: B60K 37/00 : Dashboards as road-vehicle superstructure sub-units B62D25/14
• CPC Class Codes: B60R 16/037 : for occupant comfort , e.g. for automatic adjustment of appliances according to personal settings, e.g. seats, mirrors, steering wheel
• CPC Class Codes: B60R 21/01512 : Passenger detection systems
• CPC Class Codes: B60R 7/04 : in driver or passenger space , e.g. using racks B60R7/005, B60R7/08 take precedence
• CPC Class Codes: B60W 2040/0872 : Driver physiology
• CPC Class Codes: B60W 2040/0881 : Seat occupation; Driver or passenger presence
• CPC Class Codes: B60W 2050/0095 : Automatic control mode change
• CPC Class Codes: B60W 2540/043 : Identity of occupants
• CPC Class Codes: B60W 2540/22 : Psychological state; Stress level or workload
• CPC Class Codes: B60W 2540/221 : Physiology, e.g. weight, heartbeat, health or special needs
• CPC Class Codes: B60W 2540/26 : Incapacity
• CPC Class Codes: B60W 2555/60 : Traffic rules, e.g. speed limits or right of way
• CPC Class Codes: B60W 30/182 : Selecting between different operative modes, e.g. comfort and performance modes
• CPC Class Codes: B60W 40/04 : Traffic conditions
• CPC Class Codes: B60W 40/08 : related to drivers or passengers
• CPC Class Codes: B60W 40/09 : Driving style or behaviour
• CPC Class Codes: B60W 50/0098 : Details of control systems ensuring comfort, safety or stability not otherwise provided for
• CPC Class Codes: B60W 50/08 : Interaction between the driver and the control system
• CPC Class Codes: G01C 21/20 : Instruments for performing navigational calculations G01C21/24, G01C21/26 take precedence
• CPC Class Codes: G01S 19/13 : Receivers
• CPC Class Codes: G02B 27/0101 : characterised by optical features G02B27/0172 takes precedence
• CPC Class Codes: G06F 11/2023 : Failover techniques
• CPC Class Codes: G06F 11/3013 : where the computing system is an embedded system, i.e. a combination of hardware and software dedicated to perform a certain function in mobile devices, printers, automotive or aircraft systems testing or monitoring of control systems or parts thereof G05B23/02
• CPC Class Codes: G06F 11/3065 : Monitoring arrangements determined by the means or processing involved in reporting the monitored data error or fault reporting or logging G06F11/0766
• CPC Class Codes: G06F 11/328 : Computer systems status display G06F11/327 takes precedence
• CPC Class Codes: G06F 13/14 : Handling requests for interconnection or transfer
• CPC Class Codes: G06F 13/364 : using independent requests or grants, e.g. using separated request and grant lines
• CPC Class Codes: G06F 16/24 : Querying
• CPC Class Codes: G06F 16/29 : Geographical information databases
• CPC Class Codes: G06F 17/00 : Digital computing or data processing equipment or methods, specially adapted for specific functions information retrieval, database structures or file system structures therefor G06F16/00
• CPC Class Codes: G06F 21/10 : Protecting distributed programs or content, e.g. vending or licensing of copyrighted material protection in video systems or pay television H04N7/16 ; Digital rights management [DRM]
• CPC Class Codes: G06F 21/1062 : Editing
• CPC Class Codes: G06F 21/121 : Restricting unauthorised execution of programs
• CPC Class Codes: G06F 21/32 : using biometric data, e.g. fingerprints, iris scans or voiceprints
• CPC Class Codes: G06F 21/335 : for accessing specific resources, e.g. using Kerberos tickets
• CPC Class Codes: G06F 21/42 : using separate channels for security data
• CPC Class Codes: G06F 21/629 : to features or functions of an application
• CPC Class Codes: G06F 2203/04808 : Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/017 : Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0481 : based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/04817 : using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/0482 : Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/0484 : for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/04842 : Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04847 : Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06F 3/0486 : Drag-and-drop
• CPC Class Codes: G06F 3/0488 : using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/04883 : for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 8/61 : Installation
• CPC Class Codes: G06F 9/44505 : Configuring for program initiating, e.g. using registry, configuration files
• CPC Class Codes: G06F 9/54 : Interprogram communication
• CPC Class Codes: G06N 5/02 : Knowledge representation; Symbolic representation
• CPC Class Codes: G06N 5/048 : Fuzzy inferencing
• CPC Class Codes: G06Q 30/0265 : Vehicular advertisement
• CPC Class Codes: G06Q 40/08 : Insurance
• CPC Class Codes: G06Q 50/26 : Government or public services business processes related to the transportation industry G06Q50/40
• CPC Class Codes: G06T 19/006 : Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06V 20/59 : inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
• CPC Class Codes: G06V 40/10 : Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G06V 40/15 : Biometric patterns based on physiological signals, e.g. heartbeat, blood flow
• CPC Class Codes: G06V 40/28 : Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: G07C 5/006 : Indicating maintenance
• CPC Class Codes: G07C 5/008 : communicating information to a remotely located station transmission systems for measured values G08C
• CPC Class Codes: G07C 5/02 : Registering or indicating driving, working, idle, or waiting time only apparatus forming part of taximeters G07B13/00
• CPC Class Codes: G07C 5/08 : Registering or indicating performance data other than driving, working, idle, or waiting time, with or without registering driving, working, idle or waiting time
• CPC Class Codes: G07C 5/0808 : Diagnosing performance data testing of vehicles G01M17/00; testing of electrical installation on vehicles G01R31/005
• CPC Class Codes: G07C 5/0825 : using optical means
• CPC Class Codes: G07C 5/0833 : using audio means
• CPC Class Codes: G07C 5/085 : using electronic data carriers
• CPC Class Codes: G07C 5/0866 : the electronic data carrier being a digital video recorder in combination with video camera
• CPC Class Codes: G08C 19/00 : Electric signal transmission systems G08C17/00 takes precedence
• CPC Class Codes: G08G 1/0112 : from the vehicle, e.g. floating car data [FCD]
• CPC Class Codes: G08G 1/0116 : from roadside infrastructure, e.g. beacons
• CPC Class Codes: G08G 1/012 : from other sources than vehicle or roadside beacons, e.g. mobile networks
• CPC Class Codes: G08G 1/0129 : for creating historical data or processing based on historical data
• CPC Class Codes: G08G 1/0133 : for classifying traffic situation
• CPC Class Codes: G08G 1/0137 : for specific applications
• CPC Class Codes: G08G 1/0141 : for traffic information dissemination
• CPC Class Codes: G08G 1/017 : identifying vehicles G08G1/015, G08G1/054 take precedence
• CPC Class Codes: G08G 1/09 : Arrangements for giving variable traffic instructions railroad crossing signals B61L; reflectors E01F, G08B; indicating arrangements for variable information by selection or combination of individual elements G09F9/00
• CPC Class Codes: G08G 1/096716 : where the received information does not generate an automatic action on the vehicle control
• CPC Class Codes: G08G 1/096725 : where the received information generates an automatic action on the vehicle control
• CPC Class Codes: G08G 1/096775 : where the origin of the information is a central station
• CPC Class Codes: G08G 1/096783 : where the origin of the information is a roadside individual element
• CPC Class Codes: G08G 1/096791 : where the origin of the information is another vehicle
• CPC Class Codes: G08G 1/096827 : where the route is computed onboard
• CPC Class Codes: G08G 1/096844 : where the complete route is dynamically recomputed based on new data
• CPC Class Codes: G08G 1/143 : inside the vehicles
• CPC Class Codes: G08G 1/16 : Anti-collision systems road vehicle drive control systems for predicting or avoiding probable or impending collision otherwise than by control of a particular sub-unit B60W30/08
• CPC Class Codes: H04L 41/14 : Network analysis or design
• CPC Class Codes: H04L 51/56 : Unified messaging, e.g. interactions between e-mail, instant messaging or converged IP messaging [CPM]
• CPC Class Codes: H04L 63/10 : for controlling access to devices or network resources
• CPC Class Codes: H04L 67/02 : based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04L 67/12 : specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
• CPC Class Codes: H04L 67/125 : involving control of end-device applications over a network
• CPC Class Codes: H04N 21/41422 : located in transportation means, e.g. personal vehicle arrangements specially adapted for transportation systems in broadcast systems H04H20/62
• CPC Class Codes: H04N 21/4331 : Caching operations, e.g. of an advertisement for later insertion during playback
• CPC Class Codes: H04N 21/482 : End-user interface for program selection broadcast systems using EPGs H04H60/72
• CPC Class Codes: H04W 4/40 : for vehicles, e.g. vehicle-to-pedestrians [V2P]
• CPC Class Codes: H04W 4/48 : for in-vehicle communication
• CPC Class Codes: H04W 4/50 : Service provisioning or reconfiguring
• CPC Class Codes: H04W 4/80 : Services using short range communication, e.g. near-field communication [NFC], radio-frequency identification [RFID] or low energy communication
• CPC Class Codes: H04W 4/90 : Services for handling of emergency or hazardous situations, e.g. earthquake and tsunami warning systems [ETWS]
• CPC Class Codes: H04W 8/22 : Processing or transfer of terminal data, e.g. status or physical capabilities
• CPC Class Codes: H04W 84/005 : Moving wireless networks
• CPC Class Codes: H04W 84/12 : WLAN [Wireless Local Area Networks]

Abstract

Methods and systems for a complete vehicle ecosystem are provided. Specifically, systems that when taken alone, or together, provide an individual or group of individuals with an intuitive and comfortable vehicular environment. The present disclosure includes a system to recognize the drivers and/or passengers within the automobile. Based on the recognition, the vehicle may change a configuration of the automobile to match predetermined preferences for the driver and/or passenger. The configurations may also include the recognition of a unique set of gestures for the person. Further, the configuration can also include the tracking of health data related to the person.

Claims

1. A method, comprising:

automatically determining a person is within a vehicle;

automatically identifying the person;

determining if there is a gesture to be stored for the person; and

storing the gesture.

2. The method of claim 1, further comprising:

determining if the person can be identified.

3. The method of claim 2, further comprising:

if the person cannot be identified, characterizing the person.

4. The method of claim 3, wherein automatically determining a person is within a vehicle comprises receiving sensor data from at least one sensor that indicates the person is within the vehicle.

5. The method of claim 4, wherein automatically identifying the person comprises:

comparing the sensor data to feature data in a data structure; and

determining if the sensor data is substantially similar to the feature data.

6. The method of claim 5, wherein determining if there is a gesture to be stored for the person comprises determining if the person has created a gesture.

7. The method of claim 6, wherein determining if there is a gesture to be stored for the person further comprises receiving an indication from the person that a gesture is to be created.

8. The method of claim 1, wherein the gesture is stored in cloud storage.

9. The method of claim 1, wherein the gesture is one of a gesture in a gesture capture region, a gesture in a touch sensitive display, or a movement in a three-dimensional space.

10. The method of claim 1, wherein the person occupies an area or zone within the vehicle, and wherein the gesture is associated with the person in the area or zone.

11. A vehicle system, comprising:

one or more sensors;

a vehicle control system comprising;

a processor;

a memory;

wherein the processor is operable to;

receive sensor data from the one or more sensors;

based on the sensor data, automatically determine a person is within a vehicle;

based on the sensor data, automatically identify the person;

determine if the person has made a gesture; and

store the gesture.

12. The device of claim 11, wherein the sensor data is an image of the person.

13. The device of claim 12, wherein in automatically identifying the person the processor is operable to match the image to a stored image in a data structure.

14. The device of claim 11, wherein the sensor data is voice data from the person.

15. The device of claim 14, wherein in automatically identifying the person the processor is operable to match the voice data to stored voice data in a data structure.

16. A computer readable medium having stored thereon computer-executable instructions, the computer executable instructions causing a processor to execute a method for providing a unified desktop, the computer-executable instructions comprising:

instructions to receive sensor data from the one or more sensors;

based on the sensor data, instructions to automatically determine a person is within a vehicle;

based on the sensor data, instructions to automatically identify the person;

instructions to retrieve one or more gestures associated with the identified person;

instructions to receive a gesture; and

instructions to automatically configure the vehicle according to the received gesture.

17. The computer readable medium of claim 16, wherein in automatically determining the person is within a vehicle comprises instructions to determine an area or zone in which the person occupies.

18. The computer readable medium of claim 17, wherein the retrieved gestures are associated with the determined area or zone.

19. The computer readable medium of claim 18, wherein the gestures are retrieved from a data structure.

20. The computer readable medium of claim 19, wherein the data structure is stored in cloud storage.