Methods and systems for a virtual input device

• US 9,069,164 B2
• Filed: 06/26/2012
• Issued: 06/30/2015
• Est. Priority Date: 07/12/2011
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Starner, Thad Eugene, Chi, Liang-Yu (Tom), Gomez, Luis Ricardo Prada
• Primary Examiner(s): Boddie, William
• Assistant Examiner(s): Edwards, Carolyn R
• Application Number:
  US13/533,120
• Publication Number:
  US 20130016070A1
• Time in Patent Office: 1,099 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G03B 17/54 with projector
• CPC Class Codes: G03B 21/2033 LED or laser light sources
• CPC Class Codes: G03B 29/00 Combinations of cameras, projectors or photographic printing apparatus with non-photographic non-optical apparatus, e.g. clocks or weapons; Cameras having the shape of other objects combinations with flash apparatus G03B15/03
• CPC Class Codes: G06F 3/01 Input arrangements or combined input and output arrangements for interaction between user and computer G06F3/16 takes precedence
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0426 tracking fingers with respect to a virtual keyboard projected or printed on the surface virtual keyboards on touch screens G06F3/04886
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
• CPC Class Codes: H04N 9/3173 wherein the projection device is specially adapted for enhanced portability
• CPC Class Codes: H04N 9/3185 Geometric adjustment, e.g. keystone or convergence optical or mechanical adjustment of convergence H04N9/317; using scanning means H04N3/22; optical or mechanical adjustments of projectors not peculiar to the presence of an electronic spatial light modulator G03B21/14
• CPC Class Codes: H04N 9/3194 including sensor feedback

Abstract

The present application discloses systems and methods for a virtual input device. In one example, the virtual input device includes a projector and a camera. The projector projects a pattern onto a surface. The camera captures images that can be interpreted by a processor to determine actions. The projector may be mounted on an arm of a pair of eyeglasses and the camera may be mounted on an opposite arm of the eyeglasses. A pattern for a virtual input device can be projected onto a “display hand” of a user, and the camera may be able to detect when the user uses an opposite hand to select items of the virtual input device. In another example, the camera may detect when the display hand is moving and interpret display hand movements as inputs to the virtual input device, and/or realign the projection onto the moving display hand.

Claims

1. A method comprising:

receiving information indicative of an amount of movement of a head-mountable computing device; and

determining that the indicated amount of movement is within a range of movement associated with the head-mountable computing device being substantially stationary and responsively;

providing, by a projector, an input device onto a surface, wherein the input device comprises a pattern of objects, and wherein the pattern of objects includes a plurality of regions;

receiving, from a camera, an image of the input device;

determining, by the head-mountable computing device, an approximate location of the surface based on a brightness level of the input device within the image of the input device;

modifying a power of the projector to adjust brightness of a projection of the input device based on the brightness level indicating that a distance between the surface and the camera exceeds a predetermined threshold distance;

highlighting a first region of the plurality of regions;

detecting that a size of a second region increased relative to the first region; and

highlighting the second region instead of the first region.

2. The method of claim 1, further comprising determining an area of the surface, wherein modifying the power of the projector to adjust brightness of the projection of the input device is further based on the area of the surface.

3. The method of claim 1, further comprising determining a selection of an object of the pattern of objects based at least in part on movement of the surface.

4. The method of claim 1, wherein providing the input device on the surface comprises projecting laser lines on the surface, the method further comprising:

receiving a given image of the input device depicting a discontinuity of at least one laser line of the laser lines projected on the surface, wherein the discontinuity of the at least one laser line is indicated by curvature of the at least one laser line; and

detecting selection of an object of the pattern of objects based on a location of the at least one laser line having the discontinuity.

5. The method of claim 1, further comprising determining that the input device is in use based on a comparison of the brightness level of the input device within the image of the input device to a predetermined brightness threshold.

6. The method of claim 1, further comprising adjusting, based on the approximate location of the surface, a direction of the projection of the input device so as to substantially provide the input device onto the surface while the surface is in motion.

7. A non-transitory computer readable medium having stored thereon instructions executable by a head-mountable computing device to cause the head-mountable computing device to perform functions comprising:

receiving information indicative of an amount of movement of the head-mountable computing device;

determining that the indicated amount of movement is within a range of movement associated with the head-mountable computing device being substantially stationary and responsively;

providing, by a projector coupled to the head-mountable computing device, a projected input device onto a surface, wherein the projected input device comprises a pattern of objects, and wherein the pattern of objects includes a plurality of regions;

receiving, from a camera coupled to the head-mountable computing device, an image of the projected input device;

determining an approximate location of the surface based on a brightness level of the projected input device within the image of the projected input device;

modifying a power of the projector to adjust the brightness level of the projected input device based on the brightness level indicating that a distance between the surface and the camera exceeds a predetermined threshold distance;

highlighting a first region of the plurality of regions;

detecting that a size of a second region increased relative to the first region; and

highlighting the second region instead of the first region.

8. The non-transitory computer readable medium of claim 7, wherein the instructions are further executable by the head-mountable computing device to cause the head-mountable computing device to perform functions comprising:

determining, based on a location of the second region relative to the first region, whether the surface rotated or tilted.

9. A system comprising:

an image-capture device;

a projector; and

a computing device in communication with the projector, wherein the computing device is configured to detect that the system is substantially stable and responsively;

cause the projector to provide a projected input device onto a surface, wherein the projected input device comprises a pattern of objects, and wherein the pattern of objects includes a plurality of regions;

receive from the image-capture device a reference background image including the projected input device;

determine an approximate location of the surface based on a brightness level of the projected input device within the reference background image;

modify a power of the projector to adjust the brightness level of the projected input device based on the brightness level indicating that a distance between the surface and the image-capture device exceeds a predetermined threshold distance;

highlighting a first region of the plurality of regions;

detecting that a size of a second region increased relative to the first region; and

highlighting the second region instead of the first region.

10. The system of claim 9, wherein the computing device is further configured to cause the projector to change a direction of projection of the projected input device so as to substantially provide the projected input device onto the surface while the surface is in motion.

11. The system of claim 9, wherein the projector is configured to provide the projected input device onto a hand of a user.

12. The system of claim 11, wherein the pattern of objects includes a selector, and wherein the computing device is further configured to determine a hand gesture as a user selection of an object indicated by the selector.

13. The system of claim 12, wherein the hand gesture is making a fist.

14. The system of claim 9, further comprising:

a movement sensor coupled to the computing device, wherein the computing device is further configured to;

receive one or more outputs from the movement sensor indicating an amount of movement of the system, wherein the computing device is configured to detect that the system is substantially stable based on the one or more outputs indicating that the amount of movement is within a predetermined range of movement.

15. The system of claim 14, further comprising a head mounted device, wherein the projector and the image-capture device are coupled to the head mounted device.

16. The system of claim 9, wherein the computing device is further configured to:

generate a display of an input device on a display device coupled to the computing device; and

receive a request to project the displayed input device onto the surface, wherein the computing device is configured to cause the projector to provide the projected input device onto the surface based on the request.

17. The system of claim 9, wherein the computing device is further configured to determine an area of the surface, wherein the computing device is configured to cause the projector to provide the projected input device onto the surface based on the area of the surface.

18. The system of claim 9, wherein the computing device is configured to determine an approximate location of the surface based on the reference background image, wherein the computing device is configured to modify the power of the projector to adjust the brightness level of the projected input device further based on the approximate location of the surface.

19. The system of claim 18, wherein the computing device is configured to cause the projector to provide the projected input device such that the projected input device has a given coverage area on the surface, wherein the given coverage area is based on the approximate location of the surface.