Methods and Systems for a Virtual Input Device

• US 20150268799A1
• Filed: 06/10/2015
• Published: 09/24/2015
• Est. Priority Date: 07/12/2011
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Starner, Thad Eugene, Chi, Liang-Yu (Tom), Gomez, Luis Ricardo Prada
• Granted Patent: US 9,727,174 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G03B 17/54 with projector
• CPC Class Codes: G03B 21/2033 LED or laser light sources
• CPC Class Codes: G03B 29/00 Combinations of cameras, projectors or photographic printing apparatus with non-photographic non-optical apparatus, e.g. clocks or weapons; Cameras having the shape of other objects combinations with flash apparatus G03B15/03
• CPC Class Codes: G06F 3/01 Input arrangements or combined input and output arrangements for interaction between user and computer G06F3/16 takes precedence
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0426 tracking fingers with respect to a virtual keyboard projected or printed on the surface virtual keyboards on touch screens G06F3/04886
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
• CPC Class Codes: H04N 9/3173 wherein the projection device is specially adapted for enhanced portability
• CPC Class Codes: H04N 9/3185 Geometric adjustment, e.g. keystone or convergence optical or mechanical adjustment of convergence H04N9/317; using scanning means H04N3/22; optical or mechanical adjustments of projectors not peculiar to the presence of an electronic spatial light modulator G03B21/14
• CPC Class Codes: H04N 9/3194 including sensor feedback

Abstract

The present application discloses systems and methods for a virtual input device. In one example, the virtual input device includes a projector and a camera. The projector projects a pattern onto a surface. The camera captures images that can be interpreted by a processor to determine actions. The projector may be mounted on an arm of a pair of eyeglasses and the camera may be mounted on an opposite arm of the eyeglasses. A pattern for a virtual input device can be projected onto a “display hand” of a user, and the camera may be able to detect when the user uses an opposite hand to select items of the virtual input device. In another example, the camera may detect when the display hand is moving and interpret display hand movements as inputs to the virtual input device, and/or realign the projection onto the moving display hand.

Claims

1. A method comprising:

projecting, by a projector coupled to a head-mountable device, a virtual input device onto a movable projection surface, wherein the virtual input device comprises a plurality of objects;

visually highlighting, by the projector, a center object located at a center of the plurality of objects;

detecting a change in size, on the moveable projection surface, of one or more objects of the plurality of objects, relative to a size of the highlighted center object on the moveable projection surface;

determining, based on detecting the change, that the movable projection surface has moved relative to the projected virtual input device; and

in response to (i) detecting the change in size, and (ii) determining that the movable projection surface has moved relative to the projected virtual input device, visually highlighting another object from the one or more objects instead of the center object, wherein the other object has an increased size relative to the center object.

2. The method of claim 1, wherein visually highlighting the center object comprises:

projecting a selector on the movable projection surface at the center object.

3. The method of claim 2, wherein determining that the movable projection surface has moved relative to the projected virtual input device comprises:

determining that the selector has moved relative to the movable projection surface.

4. The method of claim 3, further comprising:

determining that the movable projection surface has rotated based on determining that that selector has moved horizontally relative to the movable projection surface.

5. The method of claim 3, further comprising:

determining that the movable projection surface has tilted based on determining that that selector has moved vertically relative to the movable projection surface.

6. The method of claim 1, wherein determining that the movable projection surface has moved relative to the projected virtual input device comprises:

capturing a sequence of images depicting the movable projection surface and the virtual input device projected thereon; and

determining a change in location of an object of the plurality of objects relative to the movable projection surface from one image to a consecutive image in the sequence of images.

7. The method of claim 1, further comprising determining a selection of an object of the plurality of objects based at least in part on movement of the movable projection surface.

8. The method of claim 1, wherein the movable projection surface is a hand of a user.

9. A non-transitory computer readable medium having stored thereon instructions that, when executed by a computing device, causes the computing device to perform operations comprising:

projecting, by a projector coupled to a head-mountable device, a virtual input device onto a movable projection surface, wherein the virtual input device comprises a plurality of objects;

visually highlighting, by the projector, a center object located at a center of the plurality of objects;

detecting a change in size, on the moveable projection surface, of one or more objects of the plurality of objects, relative to a size of the highlighted center object on the moveable projection surface;

determining, based on detecting the change, that the movable projection surface has moved relative to the projected virtual input device; and

in response to (i) detecting the change in size, and (ii) determining that the movable projection surface has moved relative to the projected virtual input device, visually highlighting another object from the one or more objects instead of the center object, wherein the other object has an increased size relative to the center object.

10. The non-transitory computer readable medium of claim 9, wherein the operation of visually highlighting the center object comprises:

projecting a selector on the movable projection surface at the center object.

11. The non-transitory computer readable medium of claim 10, wherein the operation of determining that the movable projection surface has moved relative to the projected virtual input device comprises:

determining that the selector has moved relative to the movable projection surface.

12. The non-transitory computer readable medium of claim 11, wherein the operations further comprise:

determining that the movable projection surface has rotated based on determining that that selector has moved horizontally relative to the movable projection surface.

13. The non-transitory computer readable medium of claim 11, wherein the operations further comprise:

determining that the movable projection surface has tilted based on determining that that selector has moved vertically relative to the movable projection surface.

14. The non-transitory computer readable medium of claim 9, wherein the operation of determining that the movable projection surface has moved relative to the projected virtual input device comprises:

capturing, by way of a camera coupled to the head-mountable device, a sequence of images depicting the movable projection surface and the virtual input device projected thereon over time; and

detecting a change in location of an object of the plurality of objects relative to the movable projection surface from one image to a consecutive image in the sequence of images.

15. A system comprising:

a head-mountable device;

a projector coupled to the head-mountable device;

an image-capture device coupled to the head-mountable device; and

a computing device in communication with the projector and the image-capture device, the computing device is configured to perform operations comprising;

projecting, by way of the projector, a virtual input device onto a movable projection surface, wherein the virtual input device comprises a plurality of objects,

visually highlighting, by the projector, a center object located at a center of the plurality of objects,

detecting a change in size, on the moveable projection surface, of one or more objects of the plurality of objects, relative to a size of the highlighted center object on the moveable projection surface,

determining, based on detecting the change, that the movable projection surface has moved relative to the projected virtual input device, and

in response to (i) detecting the change in size, and (ii) determining that the movable projection surface has moved relative to the projected virtual input device, visually highlighting another object from the one or more objects instead of the center object, wherein the other object has an increased size relative to the center object.

16. The system of claim 15, wherein the operation of determining that the movable projection surface has moved relative to the projected virtual input device comprises:

determining that the highlighted center object has moved relative to the movable projection surface

17. The system of claim 16, wherein the movable projection surface is a hand of a user.

18. The system of claim 16, wherein the operations further comprise.

determining that the movable projection surface has rotated based on determining that that highlighted center object has moved horizontally relative to the movable projection surface.

19. The system of claim 16, wherein the operations further comprise:

determining that the movable projection surface has tilted based on determining that that highlighted center object has moved vertically relative to the movable projection surface.

20. The system of claim 15, wherein the operation of determining that the movable projection surface has moved relative to the projected virtual input device comprises:

capturing, by way of the camera, a sequence of images depicting the movable projection surface and the virtual input device projected thereon over time; and

detecting a change in location of an object of the plurality of objects relative to the movable projection surface from one image to a consecutive image in the sequence of images.