POSITION CAPTURE INPUT APPARATUS, SYSTEM, AND METHOD THEREFOR

• US 20110225534A1
• Filed: 03/11/2011
• Published: 09/15/2011
• Est. Priority Date: 03/12/2010
• Status: Active Grant

• Current Assignee: Shafa Wala
• Original Assignee: Shafa Wala
• Inventors: Wala, Shafa
• Granted Patent: US 9,250,746 B2
• Time in Patent Office: Days
• US Class Current: 715/772
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 2203/014 Force feedback applied to GUI
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0418 for error correction or compensation, e.g. based on parallax, calibration or alignment
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence

Abstract

According to various embodiments, a position capture input system uses a camera to capture an image of a displayed graphical user interface that may be partially obstructed by an object, such as a user'"'"'s hand or other body part. The position capture input system also includes a software component that causes a computing device to compare the captured image with a displayed image to determine which portion, if any, of the graphical user interface is obstructed. The computing device can then identify any user interface elements with which the user is attempting to interact. The position capture input system may also include an accelerometer or accelerometers for detecting gestures performed by the user to, for example, select or otherwise interact with a user interface element. The position capture input system may also include a haptic feedback module to provide confirmation, for example, that a user interface element has been selected.

Claims

1. A position capture input system comprising:

a first camera configured to capture an image of a graphical user interface from a perspective of a user, the graphical user interface comprising a plurality of user interface elements; and

a computing device in communication with the first camera and configured to

receive the captured image;

compare the captured image to a displayed image generated by the computing device;

based on the comparison between the captured image and the displayed image, determine a position of an obstruction relative to the graphical user interface; and

based on the position of the obstruction relative to the graphical user interface, determine whether a user is interacting with a user interface element of the graphical user interface.

2. The position capture input system of claim 1, further comprising an additional camera separated from the first camera and in communication with the computing device and configured to capture an additional image of the graphical user interface from the perspective of the user and to provide the additional image to the computing device.

3. The position capture input system of claim 1, further comprising a plurality of lenses adapted to cause the displayed image to have a three-dimensional appearance.

4. The position capture input system of claim 1, further comprising at least one accelerometer in communication with the computing device and configured to detect motion of the obstruction and to provide at least one signal to the computing device for sensing motion of the obstruction in at least one dimension, wherein the computing device is configured to determine whether the user is interacting with a user interface element of the graphical user interface based in part on the at least one signal.

5. The position capture input system of claim 1, further comprising a haptic feedback module in communication with the computing device and configured to provide haptic feedback to the user.

6. The position capture input system of claim 1, further comprising a calibration control configured to initiate a calibration process.

7. The position capture input system of claim 1, wherein the computing device consists of a display device.

8. A method of processing user input received in a computing device, the method comprising:

receiving a captured image in the computing device from a camera, the captured image representing a graphical user interface comprising a plurality of user interface elements;

comparing the captured image to a displayed image generated by the computing device;

based on the comparison between the captured image and the displayed image, determining a position of an obstruction relative to the graphical user interface; and

based on the position of the obstruction relative to the graphical user interface, determining whether a user is interacting with a user interface element of the graphical user interface.

9. The method of claim 8, further comprising moving a cursor to a location on the displayed image associated with the determined position of the obstruction relative to the graphical user interface.

10. The method of claim 8, further comprising identifying a gesture.

11. The method of claim 8, further comprising determining respective positions of multiple obstructions relative to the graphical user interface based on the comparison between the captured image and the displayed image.

12. The method of claim 11, further comprising controlling a virtual keyboard displayed in the graphical user interface based on the respective determined positions of the multiple obstructions relative to the graphical user interface.

13. The method of claim 8, further comprising receiving at least one signal from at least one accelerometer corresponding to motion of the obstruction in at least one dimension.

14. The method of claim 8, further comprising providing haptic feedback to the user via a haptic feedback module in communication with the computing device.

15. A computer-readable storage medium, other than a signal, storing computer-executable instructions that, when executed by a computing device, cause the computing device to perform a method comprising:

receiving a captured image in the computing device from a camera, the captured image representing a graphical user interface comprising a plurality of user interface elements;

comparing the captured image to a displayed image generated by the computing device;

based on the comparison between the captured image and the displayed image, determining a position of an obstruction relative to the graphical user interface; and

based on the position of the obstruction relative to the graphical user interface, determining whether a user is interacting with a user interface element of the graphical user interface.

16. The computer-readable storage medium of claim 15, storing further computer-executable instructions for moving a cursor to a location on the displayed image corresponding to the determined position of the obstruction relative to the graphical user interface.

17. The computer-readable storage medium of claim 15, storing further computer-executable instructions for identifying a gesture.

18. The computer-readable storage medium of claim 15, storing further computer-executable instructions for determining respective positions of multiple obstructions relative to the graphical user interface based on the comparison between the captured image and the displayed image.

19. The computer-readable storage medium of claim 18, storing further computer-executable instructions for controlling a virtual keyboard displayed in the graphical user interface based on the respective determined positions of the multiple obstructions relative to the graphical user interface.

20. The computer-readable storage medium of claim 15, storing further computer-executable instructions for receiving at least one signal from at least one accelerometer corresponding to motion of the obstruction in at least one dimension.