Portals: registered objects as virtualized, personalized displays

• US 9,342,610 B2
• Filed: 08/25/2011
• Issued: 05/17/2016
• Est. Priority Date: 08/25/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Liu, James Chia-Ming, Andrews, Anton Oguzhan Alford, Maitlen, Craig R., Novak, Christopher M., Bennett, Darren A., Small, Sheridan Martin
• Primary Examiner(s): Perromat, Carlos
• Application Number:
  US13/218,298
• Publication Number:
  US 20130050258A1
• Time in Patent Office: 1,727 Days
• Field of Search: 345/633, 701/533, 701/400, 701/25, 701/300, 701/420, 701/423, 701/428, 701/538, 701/26, 701/418, 701/436, 701/467, 701/487, 701/516
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/0127 comprising devices increasing the depth of field
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 16/9537 Spatial or temporal dependent retrieval, e.g. spatiotemporal queries
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 13/366 using viewer tracking
• CPC Class Codes: H04N 13/383 for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes

Abstract

A see-through head-mounted display (HMD) device provides an augmented reality image which is associated with a real-world object, such as a picture frame, wall or billboard. Initially, the object is identified by a user, e.g., based on the user gazing at the object for a period of time, making a gesture such as pointing at the object and/or providing a verbal command. The location and visual characteristics of the object are determined by a front-facing camera of the HMD device, and stored in a record. The user selects from among candidate data streams, such as a web page, game feed, video or stocker ticker. Subsequently, when the user is in the location of the object and looks at the object, the HMD device matches the visual characteristics to the record to identify the data stream, and displays corresponding augmented reality images registered to the object.

Claims

1. A head-mounted display device, comprising:

a microdisplay;

a forward-facing camera having a field of view of an environment of a user;

a location detection device, the location detection device is configured to determine a location of a user; and

a control circuit associated with the microdisplay, the forward-facing camera and the location detection device, the control circuit is configured to;

process images from the forward-facing camera to make an identification of a real-world object at the location,

in response to the identification, control the microdisplay to concurrently project to an eye of the user at a first time;

an augmented reality image which highlights and overlaps the real-world object and which is registered to the real-world object,

a first data stream of augmented reality images which do not overlap the real-world object, and

a second data stream of augmented reality images which do not overlap the real-world object,

receive a selection of the first data stream and the second data stream from the user, and

in response to the selection, control the microdisplay to concurrently project to the eye of the user, at a second time after the first time, in place of the augmented reality image which highlights the real-world object, and registered to the real-world object;

a formatted version of the first data stream of augmented reality images, formatted to a size of the real-world object and comprising fewer lines of text than the first augmented reality image which is displayed at the first time, and

the second data stream of augmented reality images formatted to the size of the real-world object.

2. The head-mounted display device of claim 1, further comprising:

an eye tracking camera which tracks a gaze direction of the eye of the user, wherein the control circuit, before the first time, is configured to access the gaze direction and make a determination as to whether the gaze direction is in a direction of the real-world object, and if the determination indicates the gaze direction is in the direction of the real-world object, cause the microdisplay to project the augmented reality image which highlights the real-world object.

3. The head-mounted display device of claim 1, further comprising:

an inertial sensor, the control circuit is configured to access the inertial sensor to determine an orientation of the user'"'"'s head and control the microdisplay to project the augmented reality image which highlights the real-world object based on a determination that the orientation of the user'"'"'s head indicates that the user is looking in a direction of the real-world object.

4. The head-mounted display device of claim 1, wherein:

the second data stream identifies a session of another user in an application; and

the control circuit is configured to receive a command from the user, and, in response to the command, transmit a signal from the head-mounted display device to another computing device of the another user to allow the another user to join the session of the application.

5. The head-mounted display device of claim 1, wherein:

the formatted version of the first data stream of augmented reality images and the second data stream of augmented reality images overlap the real-world object.

6. The head-mounted display device of claim 1, wherein:

the first augmented reality image and the second augmented reality image are not registered to the real-world object before the selection.

7. The head-mounted display device of claim 1, wherein:

the formatted version of the first augmented reality image and the second augmented reality image are displayed concurrently in respective regions within a shape of the real-world object.

8. A method for controlling a head-mounted display device, comprising:

determining a location of user of the head-mounted display device;

processing image data from a forward-facing camera of the head-mounted display device to make an identification of a real-world object at the location;

in response to the identification, concurrently projecting to an eye of the user at a first time;

(a) an augmented reality image which highlights and overlaps the real-world object and which is registered to the real-world object, (b) a first data stream of augmented reality images which do not overlap the real-world object, and (c) a second data stream of augmented reality images which do not overlap the real-world object;

receiving a selection of the first data stream and the second data stream from the user; and

in response to the selection, concurrently projecting to the eye of the user, at a second time after the first time, in place of the augmented reality image which highlights the real-world object, and registered to the real-world object;

(d) a formatted version of the first data stream of augmented reality images, formatted to a size of the real-world object and comprising fewer lines of text than the first augmented reality image which is displayed at the first time, and (e) the second data stream of augmented reality images formatted to the size of the real-world object.

9. The method of claim 8, further comprising:

tracking a gaze direction of the eye of the user; and

accessing the gaze direction and making a determination as to whether the gaze direction is in a direction of the real-world object, and if the determination indicates the gaze direction is in the direction of the real-world object, projecting the augmented reality image which highlights the real-world object.

10. The method of claim 8, further comprising:

determining an orientation of the user'"'"'s head; and

projecting the augmented reality image which highlights the real-world object based on a determination that the orientation of the user'"'"'s head indicates that the user is looking in a direction of the real-world object.

11. The method of claim 8, wherein the second data stream identifies a session of another user in an application, the method further comprising:

transmitting a signal from the head-mounted display device to another computing device of the another user to allow the another user to join the session of the application.

12. The method of claim 8, wherein:

the formatted version of the first data stream of augmented reality images and the second data stream of augmented reality images overlap the real-world object.

13. The method of claim 8, wherein:

the first augmented reality image and the second augmented reality image are not registered to the real-world object before the selection.

14. The method of claim 8, wherein:

the formatted version of the first augmented reality image and the second augmented reality image are displayed concurrently in respective regions within a shape of the real-world object.

15. The method of claim 8, further comprising:

tracking a gaze direction of the eye of the user; and

making a determination as to whether the gaze direction is in a direction of the real-world object, and if the determination indicates the gaze direction is in the direction of the real-world object, projecting the augmented reality image which highlights the real-world object.

16. The method of claim 8, further comprising:

determining an orientation of the user'"'"'s head; and

projecting the augmented reality image which highlights the real-world object based on a determination that the orientation of the user'"'"'s head indicates that the user is looking in a direction of the real-world object.

17. A computer readable memory having computer readable software embodied thereon for programming a processor to perform a method for controlling a head-mounted display device, the method comprising:

determining a location of user of the head-mounted display device;

processing image data from a forward-facing camera of the head-mounted display device to make an identification of a real-world object at the location;

in response to the identification, concurrently projecting to an eye of the user at a first time;

(a) an augmented reality image which highlights and overlaps the real-world object and which is registered to the real-world object, (b) a first data stream of augmented reality images which do not overlap the real-world object, and (c) a second data stream of augmented reality images which do not overlap the real-world object;

receiving a selection of the first data stream and the second data stream from the user; and

in response to the selection, concurrently projecting to the eye of the user, at a second time after the first time, in place of the augmented reality image which highlights the real-world object, and registered to the real-world object;

(d) a formatted version of the first data stream of augmented reality images, formatted to a size of the real-world object and comprising fewer lines of text than the first augmented reality image which is displayed at the first time, and (e) the second data stream of augmented reality images formatted to the size of the real-world object.

18. The computer readable memory of claim 17, wherein:

the formatted version of the first data stream of augmented reality images and the second data stream of augmented reality images overlap the real-world object.

19. The computer readable memory of claim 17, wherein:

the first augmented reality image and the second augmented reality image are not registered to the real-world object before the selection.

20. The computer readable memory of claim 17, wherein:

the formatted version of the first augmented reality image and the second augmented reality image are displayed concurrently in respective regions within a shape of the real-world object.