AR glasses with predictive control of external device based on event input

• US 10,860,100 B2
• Filed: 02/15/2017
• Issued: 12/08/2020
• Est. Priority Date: 02/28/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Osterhout, Ralph F., Haddick, John D., Lohse, Robert Michael
• Primary Examiner(s): Nunez, Jordany
• Application Number:
  US15/433,757
• Publication Number:
  US 20170168566A1
• Time in Patent Office: 1,392 Days
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 13/00 Interconnection of, or transfer of information or other signals between, memories, input/output devices or central processing units interface circuits for specific input/output devices G06F3/00 ; multiprogram control therefor G06F9/46; multiprocessor systems G06F15/16
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/014 Hand-worn input/output arrangements, e.g. data gloves
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0227 Cooperation and interconnection of the input arrangement with other functional units of a computer G06F3/023 - G06F3/037 take precedence
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/03547 Touch pads, in which fingers can move on a surface
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

This disclosure concerns an interactive head-mounted eyepiece with an integrated processor for handling content for display and an integrated image source for introducing the content to an optical assembly through which the user views a surrounding environment and the displayed content, wherein the eyepiece includes predictive control of external device based on an event input.

Claims

1. A system, comprising:

a head-mounted display device including an optical assembly through which at least a part of a surrounding environment and also displayed content are viewable, a camera configured to image the surrounding environment, and an image source configured to introduce content for display via the optical assembly; and

a peripheral hand motion sensing device configured to provide input to the head-mounted display device, the peripheral hand motion sensing device comprising one or more motion sensors configured to sense and send motion data to the head-mounted display device, the head-mounted display device being configured to use motion data from the peripheral hand motion sensing device for one or more control functions of the head-mounted display device when a hand is not in a field of view of the camera, wherein the head-mounted display device is configured to detect a motion of the hand coming into the field of view of the camera from out of the field of view of the camera, in response disable a control function of the peripheral hand motion sensing device, and after disabling, detect a control input to the head-mounted display device based on motion of the hand in the field of view of the camera.

2. The system of claim 1, wherein the head-mounted display device is configured to enable the control function of the peripheral hand motion sensing device when the hand is no longer detected in a field of view of the camera of the head-mounted display device.

3. The system of claim 1, wherein the peripheral hand motion sensing device is configured to sense motion of a finger and send motion data regarding the motion of the finger to the head-mounted display device.

4. The system of claim 1, wherein the peripheral hand motion sensing device is configured to receive input to control a cursor displayed by the head-mounted display device.

5. The system of claim 4, wherein the head-mounted display device is configured to enable control of the cursor when the motion of the hand is detected within the field of view of the camera of the head-mounted display device.

6. The system of claim 4, wherein the head-mounted display device is configured to disable the hand motion sensing device from controlling the cursor when the motion of the hand is detected within the field of view of the camera of the head-mounted display device.

7. The system of claim 1, wherein the peripheral hand motion sensing device comprises one or more of a gyroscope, an accelerometer, a velocity sensor, a force sensor, a pressure sensor, an optical sensor, and a proximity sensor.

8. The system of claim 1, wherein the head-mounted display device comprises an optical transmitter configured to emit signals detectable by the peripheral hand motion sensing device, and wherein the peripheral hand motion sensing device is configured to track hand motions based upon the signals emitted by the optical transmitter.

9. The system of claim 1, wherein the peripheral hand motion sensing device is configured to track hand gestures within the field of view of the camera of the head-mounted display device and provide control instructions to the head-mounted display device based upon the hand gestures.

10. The system of claim 1, wherein the peripheral hand motion sensing device comprises a user-actuatable input device configured to receive input to control a user interface of the head-mounted display device.

11. A system, comprising:

a head-mounted display device including an optical assembly through which at least a part of a surrounding environment and also displayed content are viewable, a camera configured to image the surrounding environment, and an image source configured to introduce content for display via the optical assembly;

a peripheral hand motion sensing device comprising one or more motion sensors configured to sense hand motion data as input to control a user interface of the head-mounted display device, the head-mounted display device being configured to use hand motion data from the peripheral hand motion sensing device for one or more control functions of the head-mounted display device when a hand is not in a field of view of the camera;

a processor; and

a storage device comprising instructions executable by the processor to

detect a motion of the hand coming into the field of view of the camera of the head-mounted display device from out of the field of view of the camera,

in response, disable a control function of the peripheral hand motion sensing device,

after disabling, detect a control input to the head-mounted display device based on motion of the hand in the field of view of the camera, and

enable the control function of the peripheral hand motion sensing device when the hand is no longer detected in the field of view of the camera.

12. The system of claim 11, wherein the instructions are executable to disable control of a cursor by the peripheral hand motion sensing device when the motion of a hand is detected in the field of view of the camera of the head-mounted display device.

13. The system of claim 11, wherein the motion of the hand is a first motion, and wherein the instructions are further executable to detect a second motion of the hand in the field of view of the camera of the head-mounted display device, and in response, enable the control function of the peripheral hand motion sensing device.

14. The system of claim 11, wherein the peripheral hand motion sensing device is configured to sense motion data of a finger and send the motion data of the finger to the head-mounted display device.

15. The system of claim 11, wherein the peripheral hand motion sensing device comprises one or more of a gyroscope, an accelerometer, a velocity sensor, a force sensor, a pressure sensor, an optical sensor, and a proximity sensor.

16. The system of claim 11, wherein the head-mounted display device comprises an optical transmitter configured to emit signals detectable by the peripheral hand motion sensing device, and wherein the peripheral hand motion sensing device is configured to track hand motions based upon the signals emitted by the optical transmitter.

17. The system of claim 11, wherein the peripheral hand motion sensing device is configured to track hand gestures within the field of view of the camera of the head-mounted display device and provide control instructions to the head-mounted display device based upon the hand gestures.

18. On a system comprising a head-mounted display device including an optical assembly through which at least a part of a surrounding environment and also displayed content are viewable, a camera configured to image the surrounding environment, and an image source configured to introduce content for display via the optical assembly, and the system also comprising a peripheral hand motion sensing device comprising one or more motion sensors, a method comprising:

using motion data from the peripheral hand motion sensing device for one or more control functions of the head-mounted display device when a hand is not in a field of view of the camera,

detecting a motion of the hand coming into the field of view of the camera of the head-mounted display device from out of the field of view of the camera,

in response, disabling a control function of the peripheral hand motion sensing device,

after disabling, detecting a control input to the head-mounted display device based on motion of the hand in the field of view of the camera, and

enabling the control function of the peripheral hand motion sensing device when the hand is no longer detected in the field of view of the camera.

19. The method of claim 18, wherein disabling the control function of the peripheral hand motion sensing device comprises disabling control of a cursor by the peripheral hand motion sensing device when the motion of a hand is detected within the field of view of the camera of the head-mounted display device.