AR glasses with event and user action control of external applications

• US 10,180,572 B2
• Filed: 12/30/2011
• Issued: 01/15/2019
• Est. Priority Date: 02/28/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Osterhout, Ralph F., Haddick, John D., Lohse, Robert Michael, Cella, Charles, Nortrup, Robert J., Nortrup, Edward H.
• Primary Examiner(s): Weng, Peiyong
• Application Number:
  US13/341,818
• Publication Number:
  US 20120194419A1
• Time in Patent Office: 2,573 Days
• Field of Search: 715773, 715863, 715864
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising

Abstract

This disclosure concerns an interactive head-mounted eyepiece with an integrated processor for handling content for display and an integrated image source for introducing the content to an optical assembly through which the user views a surrounding environment and the displayed content, wherein the eyepiece includes event and user action control of external applications.

Claims

1. A system comprising:

an interactive head-mounted device including an optical assembly configured to display virtual content and to enable viewing of at least a portion of a surrounding environment, an integrated processor for processing the virtual content for display, an integrated image source for introducing the virtual content to the optical assembly, a communications facility configured to connect the interactive head-mounted device to an external device, and a camera, wherein the interactive head-mounted display is configured to detect a user action as input via image data from the camera,

wherein the interactive head-mounted device is configured to detect, via a sensor of the interactive head-mounted device, a target icon that indicates that instructions are available to a user, and in response enable a command and control scheme for command and control of an external application resident on the external device, and

wherein the command and control scheme of the external application is configured to use user actions captured by the camera as input to the external application, to send a request for the instructions to the external application in response to the user actions captured by the camera of the interactive head-mounted device, and to present the instructions to the user of the interactive head- mounted device responsive to the external application sending the instructions.

2. The system of claim 1, wherein the external application stores the instructions.

3. The system of claim 1, wherein the user actions comprise one or more of user head movements, user eye movements, user voice commands, and user finger movements.

4. The system of claim 1, wherein the external device comprises one or more of a computer, a smart phone, a storage-enabled device, and a communications system.

5. The system of claim 1, wherein the camera is a first camera, and further comprising a second camera arranged in a stereo camera arrangement with the first camera.

10. The system of claim 1, wherein the camera comprises an eye-tracking camera.

6. On a head-mounted display device including an optical assembly through which a surrounding environment and displayed content are viewable, an integrated processor for handling content for display to a user, an integrated image source for introducing the content to the optical assembly, a communications facility configured to connect an external device to the interactive head-mounted display device, a sensor configured to detect an event or condition, and a camera to provide image data for tracking a user action as input, a method comprising:

detecting, via data from the sensor, a target icon in the surrounding environment that indicates that instructions are available to the user;

connecting to the external device via the communications facility;

in response to detecting the target icon, activating a command and control scheme for command and control of an external application resident on the external device;

detecting a user action via the camera;

translating the user action into input to the external application;

sending a request to the external application for the instructions;

receiving instructions from the external application in response to the input; and

presenting the instructions on the optical assembly.

7. The method of claim 6, wherein detecting, via data from the sensor, the target icon comprises detecting, via data from the camera, the target icon.

8. The method of claim 6, wherein detecting the user action via the camera on the head-mounted display device comprises

tracking a user eye gaze direction via the camera.

9. The method of claim 6, wherein detecting the user action via the camera on the head-mounted display device comprises detecting one or more of a user head movement and a user hand gesture.