Information query by pointing

• US 9,477,315 B2
• Filed: 02/27/2014
• Issued: 10/25/2016
• Est. Priority Date: 03/13/2013
• Status: Active Grant

• Current Assignee: Honda Motor Co., Ltd. (Honda Motor Company)
• Original Assignee: Honda Motor Co., Ltd. (Honda Motor Company)
• Inventors: Fujimura, Kikuo, Ng-Thow-Hing, Victor, Dariush, Behzad
• Primary Examiner(s): Duong, Hien
• Application Number:
  US14/192,499
• Publication Number:
  US 20140282259A1
• Time in Patent Office: 971 Days
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/01 Head-up displays
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/0487 using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06V 20/597 Recognising the driver's state or behaviour, e.g. attention or drowsiness
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

Navigating through objects or items in a display device using a first input device to detect pointing of a user'"'"'s finger to an object or item and using a second input device to receive the user'"'"'s indication on the selection of the object or the item. An image of the hand is captured by the first input device and is processed to determine a location on the display device corresponding to the location of the fingertip of the pointing finger. The object or the item corresponding to the location of the fingertip is selected after the second input device receives predetermined user input from the second input device.

Claims

1. A method, comprising:

capturing a depth image by a depth camera placed in a vehicle, the depth image covering a hand of a user and comprising pixels representing distances from the depth camera to parts of the user;

determining a location of a fingertip of a pointing finger of the hand by processing the captured depth image;

determining a location or region of a display device corresponding to the location of the fingertip, wherein the display device is divided into a plurality of equiangular sections of a semicircle along a center portion of the display device, each equiangular section having a width corresponding to (sine (n×

θ

)−

sine (n−

1)×

θ

) where n is the sequence of the plurality of equiangular sections starting from 0 of a center equiangular section and increasing towards left or right ends of the display device, and θ

is a rotation angle of the fingertip of the pointing finger;

receiving a selection signal indicating selection of an item or an object at the determined location or the region of the display device based on a sensor signal received from an input device in the vehicle; and

performing an operation associated with the selected item or object responsive to receiving the selection signal.

2. The method of claim 1, wherein determining the location of the fingertip comprises generating a skeleton structure of the hand including a plurality of line segments, each of the plurality of line segments representing different parts of the hand.

3. The method of claim 1, further comprising tracking the location of the fingertip to recognize a character or symbol drawn by the pointing finger.

4. The method of claim 1, wherein the input device comprises a microphone for receiving a verbal command from the user, and the selection signal is generated by performing speech recognition on the verbal command.

5. The method of claim 1, wherein the input device comprises a button or switch, and the selection signal indicates that the input device was operated.

6. The method of claim 1, further comprising displaying a result of the operation on the display device.

7. The method of claim 1, wherein the plurality of equiangular sections have horizontal widths that are determined based on horizontal locations of the plurality of equiangular sections, each of a plurality of locations of the fingertip mapped to one of the plurality of the equiangular sections.

8. The method of claim 1, wherein the display device comprises a volumetric head-up display (HUD) device projecting artificial graphical elements corresponding to items or objects onto a windshield of the vehicle.

9. The method of claim 1, wherein the location or the region of the display device corresponding to the location of the fingertip is determined by identifying one of a plurality of virtual areas mapped to the fingertip by a first mapping.

10. The method of claim 9, wherein the location or the region of the display device corresponding to the location of the fingertip is further determined by:

identifying the location or the region of the display device corresponding to the identified virtual area by a second mapping.

11. The method of claim 9, wherein each of the plurality of virtual areas is a projection of one of the equiangular sections of the semicircle onto a plane.

12. A command processing system, comprising:

a depth camera placed in a vehicle and configured to capture a depth image covering a hand of a user and comprising pixels representing distances from the depth camera to parts of the user;

an image processing module configured to;

determine a location of a fingertip of a pointing finger of the hand by processing the captured depth image, and

determine a location or region of a display device corresponding to the location of the fingertip, wherein the display device is divided into a plurality of equiangular sections of a semicircle along a center portion of the display device, each equiangular section having a width corresponding to (sine (n×

θ

)−

sine (n−

1)×

θ

) where n is the sequence of the plurality of equiangular sections starting from 0 of a center equiangular section and increasing towards left or right ends of the display device, and A is a rotation angle of the fingertip of the pointing finger; and

an information query system configured to;

receive a selection signal indicating selection of an item or an object at the determined location or the region of the display device based on a sensor signal received from an input device in the vehicle; and

perform an operation associated with the selected item or object responsive to receiving the selection signal.

13. The system of claim 12, wherein the image processing module is configured to:

generate a skeleton structure of the hand including a plurality of line segments, each of the plurality of line segments representing different parts of the hand.

14. The system of claim 12, further comprising tracking the location of the fingertip to recognize a character or symbol drawn by the pointing finger.

15. The system of claim 12, further comprising a selection detection module configured to perform speech recognition on a verbal command received by the input device.

16. The system of claim 12, further comprising the display device configured to display a result of the operation.

17. The system of claim 12, wherein the display plurality of equiangular sections have horizontal widths that are determined based on the horizontal locations of the plurality of equiangular sections, each of a plurality of locations of the fingertip mapped to one of the plurality of the equiangular sections.

18. The system of claim 12, wherein the display device comprises a volumetric head-up display (HUD) device projecting artificial graphical elements corresponding to items or objects onto a windshield of the vehicle.

19. The system of claim 12, wherein the image processing module is configured to determine the location or the region of the display device corresponding to the location of the fingertip by identifying one of a plurality of virtual areas mapped to the fingertip by a first mapping.

20. The system of claim 19, wherein the image processing module is further configured to determine the location or the region of the display device corresponding to the location of the fingertip is further determined by:

identifying the location or the region of the display device corresponding to the identified virtual area by a second mapping.