Methods and Systems for a Virtual Input Device

• US 20130016070A1
• Filed: 06/26/2012
• Published: 01/17/2013
• Est. Priority Date: 07/12/2011
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Starner, Thad Eugene, Chi, Liang-Yu (Tom), Gomez, Luis Ricardo Prada
• Granted Patent: US 9,069,164 B2
• Time in Patent Office: Days
• US Class Current: 345/175
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G03B 17/54 with projector
• CPC Class Codes: G03B 21/2033 LED or laser light sources
• CPC Class Codes: G03B 29/00 Combinations of cameras, projectors or photographic printing apparatus with non-photographic non-optical apparatus, e.g. clocks or weapons; Cameras having the shape of other objects combinations with flash apparatus G03B15/03
• CPC Class Codes: G06F 3/01 Input arrangements or combined input and output arrangements for interaction between user and computer G06F3/16 takes precedence
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0426 tracking fingers with respect to a virtual keyboard projected or printed on the surface virtual keyboards on touch screens G06F3/04886
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
• CPC Class Codes: H04N 9/3173 wherein the projection device is specially adapted for enhanced portability
• CPC Class Codes: H04N 9/3185 Geometric adjustment, e.g. keystone or convergence optical or mechanical adjustment of convergence H04N9/317; using scanning means H04N3/22; optical or mechanical adjustments of projectors not peculiar to the presence of an electronic spatial light modulator G03B21/14
• CPC Class Codes: H04N 9/3194 including sensor feedback

Abstract

The present application discloses systems and methods for a virtual input device. In one example, the virtual input device includes a projector and a camera. The projector projects a pattern onto a surface. The camera captures images that can be interpreted by a processor to determine actions. The projector may be mounted on an arm of a pair of eyeglasses and the camera may be mounted on an opposite arm of the eyeglasses. A pattern for a virtual input device can be projected onto a “display hand” of a user, and the camera may be able to detect when the user uses an opposite hand to select items of the virtual input device. In another example, the camera may detect when the display hand is moving and interpret display hand movements as inputs to the virtual input device, and/or realign the projection onto the moving display hand.

Claims

1. A method comprising:

providing an input device onto a surface, the input device comprising a pattern of objects;

receiving an image of the input device;

determining an approximate location of the surface based on a brightness level of the input device within the image of the input device; and

modifying a projection of the input device based on the approximate location of the surface.

2. The method of claim 1, further comprising determining an area of the surface, wherein modifying the projection of the input device is further based on the area of the surface.

3. The method of claim 1, further comprising determining a selection of an object of the pattern of objects based at least in part on movement of the surface.

4. The method of claim 1, further comprising:

receiving a reference background image including the input device;

receiving a given image of the input device depicting a selection of an object of the pattern of objects; and

determining the selection of the object based on a comparison of the given image to the reference background image.

5. The method of claim 1, further comprising determining that the input device is in use based on a comparison of the brightness level of the input device within the image of the input device to a predetermined brightness threshold.

6. The method of claim 1, further comprising adjusting, based on the approximate location of the surface, a direction of the projection of the input device so as to substantially provide the input device onto the surface while the surface is in motion.

7. A non-transitory computer readable medium having stored thereon instructions executable by a computing device to cause the computing device to perform functions comprising:

providing a projected input device onto a surface, the projected input device comprising a pattern of objects;

receiving an image of the projected input device;

determining an approximate location of the surface based on a brightness level of the projected input device within the image of the projected input device; and

modifying the brightness level of the projected input device based on the approximate location of the surface.

8. The non-transitory computer readable medium of claim 7, wherein the pattern of objects includes a selector, and wherein the instructions are further executable by the computing device to cause the computing device to perform functions comprising determining a rotation of the surface as a horizontal movement of the selector.

9. The non-transitory computer readable medium of claim 7, wherein the pattern of objects includes a selector, and wherein the instructions are further executable by the computing device to cause the computing device to perform functions comprising determining a tilt of the surface as a vertical movement of the selector.

10. A system comprising:

a projector; and

a computing device in communication with the projector, the computing device is configured to;

cause the projector to provide a projected input device onto a surface, the projected input device comprising a pattern of objects;

receive a reference background image including the projected input device;

receive an image of the projected input device depicting a selection of an object of the pattern of objects;

determine the selection of the object based on a comparison of the image to the reference background image; and

modify a brightness level of the projected input device based on one or more of the image and the reference background image.

11. The system of claim 10, wherein the computing device is further configured to cause the projector to change a direction of projection of the projected input device so as to substantially provide the projected input device onto the surface while the surface is in motion.

12. The system of claim 10, wherein the projector is configured to provide the projected input device onto a hand of a user.

13. The system of claim 12, wherein the pattern of objects includes a selector, and wherein the computing device is further configured to determine a hand gesture as a user selection of an object indicated by the selector.

14. The system of claim 13, wherein the hand gesture is making a fist.

15. The system of claim 10, further comprising:

a camera coupled to the computing device and configured to capture one or more images of the projected input device; and

a sensor coupled to the computing device, wherein the computing device is further configured to receive one or more outputs from the sensor and to determine movement of the system based on the one or more outputs, and wherein the camera is configured to begin to capture the one or more images of the projected input device when the one or more outputs indicate that the system is substantially stationary.

16. The system of claim 15, further comprising a head mounted device, wherein the projector and the camera are coupled to the head mounted device.

17. The system of claim 10, wherein the computing device is further configured to:

generate a display of an input device on a display device coupled to the computing device; and

receive a request to project the displayed input device onto the surface, wherein the computing device is configured to cause the projector to provide the projected input device onto the surface based on the request.

18. The system of claim 10, wherein the computing device is further configured to determine an area of the surface, wherein the computing device is configured to cause the projector to provide the projected input device onto the surface based on the area of the surface.

19. The system of claim 10, wherein the computing device is configured to determine an approximate location of the surface based on the reference background image, wherein the computing device is configured to modify the brightness level of the projected input device based on the approximate location of the surface.

20. The system of claim 19, wherein the computing device is configured to cause the projector to provide the projected input device such that the projected input device has a given coverage area on the surface, wherein the given coverage area is based on the approximate location of the surface.