System and Method for Gesture-Based Point of Interest Search

• US 20140330515A1
• Filed: 07/16/2014
• Published: 11/06/2014
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: Edge 3 Technologies, LLC, Honda Motor Co., Ltd. (Honda Motor Company)
• Original Assignee: Edge Technologies, Inc, Honda Motor Co., Ltd. (Honda Motor Company)
• Inventors: El Dokor, Tarek A., Holmes, James E., Vaghefinazari, Pedram, Yamamoto, Stuart M., Cluster, Jordon
• Granted Patent: US 9,541,418 B2
• Time in Patent Office: Days
• US Class Current: 701/516
• CPC Class Codes: B60K 2360/146 Instrument input by gesture
• CPC Class Codes: B60K 2360/1464 3D-gesture
• CPC Class Codes: B60K 2360/148 Instrument input by voice
• CPC Class Codes: B60K 2360/166 Navigation
• CPC Class Codes: B60K 2360/589 Wireless data transfers
• CPC Class Codes: B60K 2360/592 Data transfer involving external databases
• CPC Class Codes: B60K 2360/595 Data transfer involving internal databases
• CPC Class Codes: B60K 35/00 Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
• CPC Class Codes: B60K 35/10 Input arrangements, i.e. from user to vehicle, associated with vehicle functions or specially adapted therefor
• CPC Class Codes: B60K 35/22 Display screens
• CPC Class Codes: B60K 35/28 characterised by the type of the output information, e.g. video entertainment or vehicle dynamics information; characterised by the purpose of the output information, e.g. for attracting the attention of the driver
• CPC Class Codes: B60K 35/85 Arrangements for transferring vehicle- or driver-related data
• CPC Class Codes: G01C 21/3664 Details of the user input interface, e.g. buttons, knobs or sliders, including those provided on a touch screen; remote controllers; input using gestures
• CPC Class Codes: G01C 21/3679 Retrieval, searching and output of POI information, e.g. hotels, restaurants, shops, filling stations, parking facilities G01C21/3611 takes precedence
• CPC Class Codes: G01C 21/3682 output of POI information on a road map G01C21/3614, G01C21/3685 take precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 40/113 Recognition of static hand signs

Abstract

A user, such as the driver of a vehicle, to retrieve information related to a point of interest (POI) near the vehicle by pointing at the POI or performing some other gesture to identify the POI. Gesture recognition is performed on the gesture to generate a target region that includes the POI that the user identified. After generating the target region, information about the POI can be retrieved by querying a server-based POI service with the target region or by searching in a micromap that is stored locally. The retrieved POI information can then be provided to the user via a display and/or speaker in the vehicle. This process beneficially allows a user to rapidly identify and retrieve information about a POI near the vehicle without having to navigate a user interface by manipulating a touchscreen or physical buttons.

Claims

1-20. -20. (canceled)

21. A vehicle-based computer-implemented method for retrieving information associated with a point of interest (POI), the method comprising:

accessing location data identifying a current location of the vehicle;

analyzing the location data to find a micromapped region;

retrieving, from a POI data server, a micromap corresponding to the micromapped region, the micromap including information associated with one or more points of interest inside the micromapped region;

receiving a data signal representing an identifying gesture performed by a user inside the vehicle, the identifying gesture oriented in a direction and identifying an object exterior to the vehicle;

performing gesture recognition on the data signal to determine a direction vector representing the direction of the identifying gesture;

analyzing the direction vector and the location data to generate a target region corresponding to the object identified by the identifying gesture;

retrieving, from the micromap, information associated with one or more points of interest located in the target region; and

providing the retrieved information to the user, the retrieved information including information associated with the object identified by the identifying gesture.

22. The computer-implemented method of claim 21, further comprising:

receiving a voice command, the voice command issued by the user inside the vehicle; and

performing voice recognition on the voice command to determine one or more words spoken by the user as part of the voice command;

wherein the analyzing step comprises analyzing the voice command words in conjunction with the direction vector and the location data to generate the target region.

23. The computer-implemented method of claim 21, wherein the identifying gesture comprises a hand with an outstretched index finger pointing in the direction.

24. The computer-implemented method of claim 21, wherein a corner of the target region corresponds to the current location of the vehicle, and wherein the target region aligns with the direction vector.

25. The computer-implemented method of claim 21, further comprising retrieving additional information associated with the one or more points of interest located in the target region, by:

sending the target region to the POI data server over a network; and

receiving, from the POI data server, the additional information associated with the one or more points of interest located in the target region.

26. The computer-implemented method of claim 21, wherein retrieving, from the micromap, information associated with one or more points of interest located in the target region comprises:

determining whether the target region overlaps with the micromapped region;

responsive to determining that the target region overlaps with the micromapped region, accessing the micromap to retrieve the information associated with the one or more points of interest located within the target region.

27. The computer-implemented method of claim 26, wherein retrieving, from the micromap, information associated with one or more points of interest located in the target region comprises:

sending the target region to the POI data server over a network;

receiving, from the POI data server, additional information associated with additional points of interest located in the target region; and

merging the information associated with the one or more points of interest located in the target region and the additional information associated with the additional points of interest located in the target region into a single set of POI information.

28. The computer-implemented method of claim 21, wherein providing the retrieved information to the user comprises sending at least part of the retrieved information to be shown on a display in the vehicle.

29. The computer-implemented method of claim 21, wherein finding a micromapped region comprises:

identifying a plurality of candidate micromapped regions; and

selecting the micromapped region from among the plurality of candidate micromapped regions.

30. The computer-implemented method of claim 21, wherein analyzing the location data to find a micromapped region comprises:

analyzing the current location of the vehicle to generate a retrieval region in front of the vehicle; and

identifying a micromapped region within the retrieval region as the micromapped region.

31. A non-transitory computer-readable storage medium for storing computer program instructions for retrieving information associated with a point of interest (POI), the program instructions when executed by a processor cause the processor to perform steps including:

accessing location data identifying a current location of the vehicle;

analyzing the location data to find a micromapped region;

retrieving, from a POI data server, a micromap corresponding to the micromapped region, the micromap including information associated with one or more points of interest inside the micromapped region;

receiving a data signal representing an identifying gesture performed by a user inside the vehicle, the identifying gesture oriented in a direction and identifying an object exterior to the vehicle;

performing gesture recognition on the data signal to determine a direction vector representing the direction of the identifying gesture;

analyzing the direction vector and the location data to generate a target region corresponding to the object identified by the identifying gesture;

retrieving information associated with one or more points of interest located in the target region; and

providing the retrieved information to the user, the retrieved information including information associated with the object identified by the identifying gesture.

32. The storage medium of claim 31, wherein the program instructions further cause the processor to perform steps including:

receiving a voice command, the voice command issued by the user inside the vehicle; and

performing voice recognition on the voice command to determine one or more words spoken by the user as part of the voice command;

wherein the analyzing step comprises analyzing the voice command words in conjunction with the direction vector and the location data to generate the target region.

33. The storage medium of claim 31, wherein a corner of the target region corresponds to the current location of the vehicle, and wherein the target region aligns with the direction vector.

34. The storage medium of claim 31, wherein the program instructions further cause the processor to retrieve additional information associated with the one or more points of interest located in the target region by:

sending the target region to the POI data server over a network; and

receiving, from the POI data server, additional information associated with the one or more points of interest located in the target region.

35. The storage medium of claim 31, wherein retrieving, from the micromap, information associated with one or more points of interest comprises:

determining whether the target region overlaps with the micromapped region; and

responsive to determining that the target region overlaps with the micromapped region, accessing the micromap to retrieve the information associated with the one or more points of interest located within the target region.

36. A vehicle-based computing system for retrieving information associated with a point of interest (POI), the system comprising:

a location module configured to access location data identifying a current location of a vehicle;

a micromap management module configured to;

analyze the location data to find a micromapped region;

retrieve, from a POI data server, a micromap corresponding to the micromapped region, the micromap including information associated with one or more points of interest inside the micromapped region;

a gesture recognition module configured to;

receive a data signal representing an identifying gesture performed by a user inside the vehicle, the identifying gesture oriented in a direction and identifying an object exterior to the vehicle; and

perform gesture recognition on the data signal to determine a direction vector representing the direction of the identifying gesture;

an input analysis module configured to analyze the direction vector and the location data to generate a target region corresponding to the object identified by the identifying gesture;

a search module configured to retrieve information associated with one or more points of interest located in the target region; and

a data output module configured to provide the retrieved information to the user, the retrieved information including information associated with the object identified by the identifying gesture.

37. The computing system of claim 36, further comprising a voice recognition module configured to:

receive a voice command, the voice command issued by the user inside the vehicle; and

perform voice recognition on the voice command to determine one or more words spoken by the user as part of the voice command; and

wherein the input analysis module is further configured to analyze the voice command words in conjunction with the direction vector and the location data to generate the target region.

38. The computing system of claim 36, wherein a corner of the target region corresponds to the current location of the vehicle, and wherein the target region aligns with the direction vector.

39. The computing system of claim 36, wherein the search module comprises a POI search module configured to:

send the target region to the POI data server over a network; and

receive, from the POI data server, additional information associated with the one or more points of interest located in the target region.

40. The computing system of claim 36, wherein the input analysis module is further configured to determine whether the target region overlaps with the micromapped region, and wherein the search module comprises a micromap search module configured to access the micromap to retrieve the information associated with the one or more points of interest located within the target region.