User Interface Device

• US 20150035746A1
• Filed: 12/27/2011
• Published: 02/05/2015
• Est. Priority Date: 12/27/2011
• Status: Abandoned Application

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Mandalapu, Dinesh, Cockburn, Andy, Billinghurst, Mark
• Application Number:
  US14/369,479
• Publication Number:
  US 20150035746A1
• Time in Patent Office: Days
• US Class Current: 345/156
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/0487 using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: H04N 13/20 Image signal generators

Abstract

A user interface device (105) comprising a camera (110, 205) that captures an image of a user'"'"'s (115) face and fingers (125) and a processor (210, 230) that determines the spatial location of the user'"'"'s (115) farcical features (120) and fingers (125) using the captured image, in which the processor (210, 230) further determines where on a screen (130) of the user interface device (105) the user (115) is viewing and in which the processor (210, 230) monitors the user'"'"'s (115) facial features (120) and fingers (125) for indications of manipulation of on-screen content of the user interface device (105).

Claims

1. A user interface device (105) comprising:

a camera (110, 205) that captures an image of a user'"'"'s (115) face and fingers (125); and

a processor (210, 230) that determines the spatial location of the user'"'"'s (115) facial features (120) and fingers (125) using the captured image;

in which the processor (210, 230) further determines where on a screen (130) of the user interface device (105) the user (115) is viewing; and

in which the processor (210, 230) monitors the user'"'"'s (115) facial features (120) and fingers (125) for indications of manipulation of on-screen content of the user interface device (105).

2. The user interface device (105) of claim 1, in which the camera (110, 205) is a three-dimensional camera.

3. The user interface device (105) of claim 1, in which the processor (210, 230) determines the spatial location of the user'"'"'s (115) facial features (120) by determining the location of the user'"'"'s (115) eye socket, eyeball, pupil, or combinations thereof.

4. The user interface device of claim 1, in which the processor (210, 230) determines where on the screen (130) of the user interface device (105) the user (115) is viewing by calculating a vector consisting of the spatial positions of the user'"'"'s (115) facial features (120) and fingers (125).

5. The user interface device (105) of claim 1, in which the processor (210, 230) changes the on-screen content of the user interface device (105) when the processor (210, 230) has detected indications of manipulation of on-screen content.

6. The user interface device (105) of claim 1, in which indications of manipulation of on-screen content of the user interface device (105) comprises a blink of the eye (120), an increase in the distance between two of the user'"'"'s (115) fingers 125), a decrease in the distance between two of the user'"'"'s (115) fingers (125), an increase in the distance between the user'"'"'s (115) finger (125) and the screen (130), a decrease in the distance between the user'"'"'s (115) finger (125) and the screen (130), a change in spatial location of a user'"'"'s (115) fingers (125), or combinations of these.

7. The user interface device (105) of claim 1, in which the processor (210, 230) further monitors audible cues from the user (115) for indications of manipulation of on-screen content of the user interface device (105).

8. The user interface device (105) of claim 1, in which the camera (110, 205) captures a number of sequential images of a user'"'"'s (115) facial features (120) and fingers (125) at a frame rate of thirty frames per second or greater.

9. A method of manipulating on-screen content on a user interface device 105), comprising:

capturing an image of the user'"'"'s (115) eye (120) and fingers (125) with a camera (110, 205);

determining the spatial location of the user'"'"'s (115) eye (120) and fingers (125) with a processor (210, 230);

determining, with the processor (210, 230), where on a screen (130) of the user interface device (105) the user (115) is viewing using the spatial location of the user'"'"'s (115) eye (120) and fingers (125);

monitoring the user'"'"'s (115) eye (120) and fingers (125) for indications of manipulation of on-screen content; and

manipulating the on-screen content based on the user'"'"'s (115) indications of manipulation of on-screen content.

10. The method of claim 9, in which the camera (110, 205) is a three-dimensional camera.

11. The method of claim 9, in which determining where on the screen of the user interface device (105) the user (105) is viewing is accomplished by calculating a vector consisting of the spatial positions of the user'"'"'s (115) eye (120) and fingers (125).

12. The method of claim 9, further comprising changing the on-screen content of the user interface device (105) when the processor (210, 230) has detected indications of manipulation of on-screen content.

13. The method of claim 9, in which the processor (210, 230) further monitors audible cues from the user (115) for indications of manipulation of on-screen content of the user interface device (105).

14. A computer program product for manipulating on-screen content on a user interface device (105), the computer program product comprising:

a non-transitory computer usable storage medium (240, 245, 250, 255) having computer usable program code embodied therewith, the computer usable program code comprising;

computer usable program code that determines the spatial location of a user'"'"'s (115) eye (120) and fingers (125) using an image of the user'"'"'s (115) facial features (120) and fingers (125) captured by a camera (110, 205),

computer usable program code that determines where on a screen (130) of the user interface device (105) the user (115) is viewing using the spatial location of the user'"'"'s (115) facial features (120) and fingers (125);

computer usable program code that monitors the user'"'"'s (115) facial features (120) and fingers (125) for indications of manipulation of on-screen content; and

computer usable program code that manipulates the on-screen content based on the user'"'"'s (115) indications of manipulation of on-screen content.

15. The computer program product of claim 14, in which the camera (110, 205) is a three-dimensional camera.