Wearable gestural interface

• US 9,569,001 B2
• Filed: 02/03/2010
• Issued: 02/14/2017
• Est. Priority Date: 02/03/2009
• Status: Active Grant

• Current Assignee: Massachusetts Institute of Technology
• Original Assignee: Massachusetts Institute of Technology
• Inventors: Mistry, Pranav, Maes, Patricia
• Primary Examiner(s): Gaffin, Jeffrey A
• Assistant Examiner(s): Heffington, John M
• Application Number:
  US12/699,803
• Publication Number:
  US 20100199232A1
• Time in Patent Office: 2,568 Days
• Field of Search: 715/863, 715/718
• US Class Current: 1/1
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 1/1639 the display being based on projection
• CPC Class Codes: G06F 1/1673 Arrangements for projecting a virtual keyboard
• CPC Class Codes: G06F 1/1696 the I/O peripheral being a printing or scanning device
• CPC Class Codes: G06F 16/9554 by using bar codes
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06F 3/0426 tracking fingers with respect to a virtual keyboard projected or printed on the surface virtual keyboards on touch screens G06F3/04886
• CPC Class Codes: H04L 67/60 Scheduling or organising the servicing of application requests, e.g. requests for application data transmissions using the analysis and optimisation of the required network resources admission control or resource allocation H04L47/70
• CPC Class Codes: H04M 1/0264 for a camera module assembly
• CPC Class Codes: H04M 1/0272 for a projector or beamer module assembly constructional features of projectors per se G03B21/00; constructional details of projectors using electronic spatial light modulators per se H04N9/31
• CPC Class Codes: H04N 9/3173 wherein the projection device is specially adapted for enhanced portability
• CPC Class Codes: H04N 9/3194 including sensor feedback

Abstract

This invention may be implemented as a wearable apparatus comprised of a camera, a projector, a mirror, a microphone and a digital computer. The camera captures visual data. This data is analyzed by the digital computer to recognize objects and hand gestures, using color tracking and edge detection techniques. The projector is used, along with a mirror to adjust the direction of the projected light, to project images on objects in the user'"'"'s environment. For example, the images may be projected on surfaces such as a wall, table, or piece of paper. The projected images may contain information relevant to the object being augmented. Indeed, the information may include current data obtained from the Internet. Also, the projected images may comprise graphical interfaces, with which a user may interact by making hand gestures.

Claims

1. Apparatus comprising:

(a) a camera;

(b) a projector for projecting images on a surface, which surface is separate from the apparatus and is not dedicated to electronic visual display;

(c) a transceiver for connecting to a global network; and

(d) a digital computer for

(i) performing analysis of visual data captured by the camera (A) to identify a solid object or a feature of the solid object, neither the object nor the feature being an image projected by the projector, and (B) to recognize a gesture made by a human in the air while the human is neither in physical contact with, nor holding an object that is in physical contact with, a solid surface, and

(ii) based at least in part on the analysis;

outputting instructions to obtain, from the global network, information about the object or feature;

receiving and processing data indicative of the information, and

outputting instructions to make changes to the images, in a way that the changes depend at least in part on the information and the gesture;

wherein

(A) the global network comprises the Internet, the World-Wide Web, or any other global system of interconnected computer networks, and

(B) the apparatus is wearable in a manner in which weight of the apparatus is not supported by a human user'"'"'s hands.

2. Apparatus as set forth in claim 1, wherein the projector is adapted for projecting images which convey the information.

3. Apparatus as set forth in claim 1, wherein the computer is adapted for outputting signals indicative of audio output of a transducer, in such a way that the transducer outputs sound that conveys the information audibly to a user.

4. Apparatus as set forth in claim 1, wherein:

(a) the feature is the surface of a product or of the product'"'"'s packaging;

(b) the information is about the product; and

(c) the projector is adapted for projecting, on the surface, images that convey the information.

5. Apparatus as set forth in claim 1, wherein:

(a) the feature comprises text or a graphical pattern, which text or graphical patttern is not an image projected by the projector, and which text or graphical pattern has a subject;

(b) the information is about the subject; and

(c) the projector is adapted for projecting, on the surface, images that convey the information.

6. Apparatus as set forth in claim 1, wherein the computer is adapted for analyzing the visual data to identify a specific person, and further adapted for outputting instructions for altering audio or visual output of a transducer in such a way as to convey the information, and the information is about the specific person.

7. Apparatus as set forth in claim 1, wherein the computer is adapted for outputting instructions to project an image of a graphical user interface, and further adapted for analyzing the visual data captured by the camera to detect movements comprising a user'"'"'s interaction with the projected image of the graphical user interface, wherein the graphical user interface displays a plurality of graphical icons or other visual symbols, each of which represent a different input that may be selected by a gesture made by the user.

8. Apparatus as set forth in claim 1, wherein the computer is adapted for outputting instructions to project images that display an email message obtained through the global network.

9. Apparatus comprising:

(a) a projector for projecting images on at least one surface, which surface is not part of the apparatus or any other electronic visual display device; and

(b) a camera for

(i) capturing visual data, and

(ii) outputting signals indicative of the visual data, for transmission to one or more digital processors adapted for (A) analyzing the visual data captured by the camera to recognize a gesture made by a human in the air while the human is neither touching, nor holding an object that touches, a solid surface, and (B) outputting instructions to change the projected images, in such a way as to alter the projected images in response to the gesture;

wherein the apparatus is wearable by a person in a manner in which weight of the apparatus is not supported by the person'"'"'s hands.

10. Apparatus as set forth in claim 9, wherein at least one said digital processor is adapted for

outputting, in response to the gesture, instructions to project unto the surface a visual pattern captured by the camera, which visual pattern is on a different surface when captured by the camera, which different surface is not part of an electronic visual display device.

11. Apparatus as set forth in claim 9, wherein at least one said-digital processor is adapted for outputting, in response to the gesture, instructions for the camera to take a photograph.

12. Apparatus as set forth in claim 9, wherein at least one said digital processor is adapted for outputting, in response to the gesture, instructions to project, on the user, an image of a timepiece.

13. Apparatus as set forth in claim 9, wherein at least one said digital processor is adapted for analyzing said visual data to detect changes in position of a body part of a user or of an object held by or affixed to the user, and further adapted for outputting instructions to alter the projected images in response to the changes in position.

14. Apparatus as set forth in claim 9, wherein at least one said digital processor is adapted for analyzing the visual data to recognize a tangible object and for outputting instructions to project on the object information about the object.