Position capture input apparatus, system, and method therefor

• US 9,250,746 B2
• Filed: 03/11/2011
• Issued: 02/02/2016
• Est. Priority Date: 03/12/2010
• Status: Active Grant

• Current Assignee: Shafa Wala
• Original Assignee: Shafa Wala
• Inventors: Wala, Shafa
• Primary Examiner(s): ULRICH, NICHOLAS S
• Application Number:
  US13/046,452
• Publication Number:
  US 20110225534A1
• Time in Patent Office: 1,789 Days
• Field of Search: 715/773, 345/156, 351/41
• US Class Current: 1/1
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 2203/014 Force feedback applied to GUI
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0418 for error correction or compensation, e.g. based on parallax, calibration or alignment
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence

Abstract

According to various embodiments, a position capture input system uses a camera to capture an image of a displayed graphical user interface that may be partially obstructed by an object, such as a user'"'"'s hand or other body part. The position capture input system also includes a software component that causes a computing device to compare the captured image with a displayed image to determine which portion, if any, of the graphical user interface is obstructed. The computing device can then identify any user interface elements with which the user is attempting to interact. The position capture input system may also include an accelerometer or accelerometers for detecting gestures performed by the user to, for example, select or otherwise interact with a user interface element. The position capture input system may also include a haptic feedback module to provide confirmation, for example, that a user interface element has been selected.

Claims

1. A position capture input system comprising:

a headworn device supportable on a head of a human user;

a first camera located in the headworn device pointing forward and configured to capture an image of a field of view from a perspective of the user, wherein the first camera is viewing at least part of what the user is viewing; and

a computing device in communication with the first camera and configured to;

perform a calibration process comprising displaying a calibration screen and receiving, from the first camera, a captured image of the calibration screen in the field of view from the perspective of the user, wherein the calibration process is performed on an ongoing basis to accommodate for the user changing perspective or moving;

display a graphical user interface, wherein the graphical user interface comprises a plurality of user interface elements;

receive, from the first camera, a captured image of the graphical user interface in the field of view from the perspective of the user;

compare the captured image of the graphical user interface to an image of the graphical user interface to detect an obstruction in the captured image of the graphical user interface;

map a position of the detected obstruction to a location in the graphical user interface using results obtained from the calibration process; and

determine whether a user is interacting with one of the plurality of user interface elements of the graphical user interface based on the mapped position of the obstruction to the location in the graphical user interface.

2. The position capture input system of claim 1, further comprising an additional camera located in the headworn device separated from the first camera and in communication with the computing device and configured to capture an additional image of the graphical user interface from the perspective of the user and to provide the additional image to the computing device, wherein the user interacting with a user interface element of the graphical user interface can manipulate an object displayed in the graphical user interface to provide a perception of three-dimensional movement of the object.

3. The position capture input system of claim 1, further comprising a plurality of lenses supported on the headworn device to cover eyes of the user, the lenses being adapted to cause the displayed image to have a three-dimensional appearance.

4. The position capture input system of claim 1, further comprising at least one accelerometer in communication with the computing device, the accelerometer being wearable by the user on a limb of the user and configured to detect motion of the user'"'"'s limb and to provide at least one signal to the computing device for sensing motion of the user'"'"'s limb in at least one dimension, wherein the computing device is configured to determine whether the user is interacting with the one of the plurality of user interface elements of the graphical user interface based in part on the at least one signal without contacting the graphical user interface.

5. The position capture input system of claim 1, further comprising a haptic feedback module in communication with the computing device and configured to provide haptic feed back to the user when moving without contacting the graphical user interface.

6. The position capture input system of claim 1, wherein the calibration process determines scale and location of the graphic user interface relative to the forward looking direction of the first camera on the headworn device.

7. The position capture input system of claim 1, further comprising a monitor displaying the graphical user interface.

8. A method of processing user input received in a computing device, the method comprising:

in the computing device;

receiving captured images from a camera located in a headworn device supportable on a head of a human user, wherein the camera points forward and is configured to capture an image of a field of view from a perspective of the user and wherein the camera is viewing at least part of what the user is viewing;

performing a calibration process comprising displaying a calibration screen and receiving, from the camera, a captured image of the calibration screen in the field of view from the perspective of the user, wherein the calibration process is performed on an ongoing basis to accommodate for the user changing perspective or moving;

displaying a graphical user interface, wherein the graphical user interface comprises a plurality of user interface elements;

receiving, from the camera, a captured image of the graphical user interface in the field of view from the perspective of the user;

comparing the captured image of the graphical user interface to an image of the graphical user interface to detect an obstruction in the captured image of the graphical user interface;

mapping a position of the detected obstruction to a location in the graphical user interface using results obtained from the calibration process; and

determining whether the user is interacting with one of the plurality of user interface elements of the graphical user interface based on the mapped position of the obstruction to the location in the graphical user interface.

9. The method of claim 8, further comprising moving a cursor to the location in the graphical user interface.

10. The method of claim 8, wherein the mapping a position of the detected obstruction to a location in the graphical user interface using results obtained from the calibration process is performed on an ongoing basis and comprises identifying a gesture.

11. The method of claim 8, further comprising mapping positions of multiple obstructions to the graphical user interface.

12. The method of claim 11, further comprising controlling a virtual keyboard displayed in the graphical user interface based on the mapped positions of the multiple obstructions to the graphical user interface.

13. The method of claim 8, further comprising receiving at least one signal from at least one accelerometer worn by the user on a limb of the user, the accelerometer being configured to detect motion of the user'"'"'s limb in at least one dimension without having the user contact the graphical user interface.

14. The method of claim 8, further comprising providing haptic feedback to the user, without having the user contact the graphical user interface, via a haptic feedback module in communication with the computing device.

15. The method of claim 8, wherein the calibration process further comprises identifying scale and/or location of the graphical user interface on an ongoing basis.

16. The method of claim 8, wherein the calibration process further comprises producing a homography on an ongoing basis that allows for a direct comparison between pixels of the captured image of the graphical user interface and the displayed graphical user interface.

17. A non-transitory computer-readable storage medium storing computer-executable instructions that, when executed by a computing device, causes the computing device to perform a method comprising:

receiving captured images from a camera located in a headworn device supportable on a head of a human user, wherein the camera points forward and is configured to capture an image of a field of view from a perspective of the user and wherein the camera is viewing at least part of what the user is viewing;

performing a calibration process comprising displaying a calibration screen and receiving, from the camera, a captured image of the calibration screen in the field of view from the perspective of the user, wherein the calibration process is performed on an ongoing basis to accommodate for the user changing perspective or moving;

displaying a graphical user interface, wherein the graphical user interface comprises a plurality of user interface elements;

receiving, from the camera, a captured image of the graphical user interface in the field of view from the perspective of the user;

comparing the captured image of the graphical user interface to an image of the graphical user interface to detect an obstruction in the captured image of the graphical user interface;

mapping a position of the detected obstruction to a location in the graphical user interface using results obtained from the calibration process; and

determining whether the user is interacting with one of the plurality of user interface elements of the graphical user interface based on the mapped position of the obstruction to the location in the graphical user interface.

18. The computer-readable storage medium of claim 17, storing further computer-executable instructions for moving a cursor to the location in the graphical user interface.

19. The computer-readable storage medium of claim 17, storing further computer-executable instructions for mapping the position of the obstruction on an ongoing basis and for identifying a gesture.

20. The computer-readable storage medium of claim 17, storing further computer-executable instructions for mapping positions of multiple obstructions to the graphical user interface, and for controlling a virtual keyboard displayed in the graphical user interface based on the mapped positions of the multiple obstructions to the graphical user interface.

21. The computer-readable storage medium of claim 17, storing further computer-executable instructions for receiving at least one signal from at least one accelerometer corresponding to motion of the accelerometer in at least one dimension, the signal being processed to compute motion of the obstruction relative to the graphical user interface without having the obstruction contact the graphical user interface.