Portals: Registered Objects As Virtualized, Personalized Displays

• US 20130050258A1
• Filed: 08/25/2011
• Published: 02/28/2013
• Est. Priority Date: 08/25/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: James Chia-Ming Liu
• Inventors: Liu, James Chia-Ming, Andrews, Anton Oguzhan Alford, Maitlen, Craig R., Novak, Christopher M., Bennett, Darren A., Small, Sheridan Martin
• Granted Patent: US 9,342,610 B2
• Time in Patent Office: Days
• US Class Current: 345/633
• CPC Class Codes: G02B 2027/0127 comprising devices increasing the depth of field
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 16/9537 Spatial or temporal dependent retrieval, e.g. spatiotemporal queries
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 13/366 using viewer tracking
• CPC Class Codes: H04N 13/383 for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes

Abstract

A see-through head-mounted display (HMD) device provides an augmented reality image which is associated with a real-world object, such as a picture frame, wall or billboard. Initially, the object is identified by a user, e.g., based on the user gazing at the object for a period of time, making a gesture such as pointing at the object and/or providing a verbal command. The location and visual characteristics of the object are determined by a front-facing camera of the HMD device, and stored in a record. The user selects from among candidate data streams, such as a web page, game feed, video or stocker ticker. Subsequently, when the user is in the location of the object and looks at the object, the HMD device matches the visual characteristics to the record to identify the data stream, and displays corresponding augmented reality images registered to the object.

Claims

1. A head-mounted display device for wear on a user'"'"'s head, comprising:

at least one see-through lens;

at least one microdisplay;

at least one forward-facing camera having a field of view;

a location-detection device which determines a location of the user; and

at least one control circuit, the at least one control circuit determines if at least one criterion is met based at least on the location, and if the at least one criterion is met, processes image data from the forward-facing camera to identify a real-world object in the field of view, identify a data stream which is associated with the real-world object, and control the at least one microdisplay to project augmented reality images from the data stream through the at least one see-through lens to at least one eye of the user, so that the augmented reality images are registered to the real-world object.

2. The head-mounted display device of claim 1, further comprising:

the data stream comprises a first data stream; and

the at least one forward-facing camera comprises at least one forward-facing depth sensor, the at least one control circuit processes depth data from the at least one forward-facing depth sensor to detect a gesture of the user during the projecting of the augmented reality images from the data stream, and, in response to detecting the gesture, identify a second data stream, and control the at least one microdisplay to project augmented reality images from the second data stream instead of the first data stream through the at least one see-through lens to the at least one eye of the user.

3. The head-mounted display device of claim 1, wherein:

a plurality of available data streams are associated with the real-world object; and

the identified data stream is identified from among the plurality of available data streams based on an identifier of the user.

4. The head-mounted display device of claim 1, wherein:

the at least one criterion comprises the location of the user matching a unique geographic location.

5. The head-mounted display device of claim 1, wherein:

the at least one criterion comprises the location of the user matching a class of geographic locations.

6. The head-mounted display device of claim 1, further comprising:

at least one eye tracking camera which tracks a gaze direction of the at least one eye of the user, the at least one control circuit accesses the gaze direction and determines that the at least one criterion is met based on whether the gaze direction is in a direction of the real-world object.

7. The head-mounted display device of claim 1, further comprising:

at least one inertial sensor, the at least one control circuit accesses the least one inertial sensor to determine an orientation of the user'"'"'s head and determines that the at least one criterion is met based on whether the orientation of the user'"'"'s head indicates that the user is looking in a direction of the real-world object.

8. The head-mounted display device of claim 1, wherein:

the data stream identifies a session of one or more other users in an application; and

the at least one control circuit receives a command from the user, and, in response to the command, transmits a signal from the head-mounted display device to another computing device of the one or more other users to allow the user to join the session of the application.

9. A processor-implemented method for controlling a head-mounted display device for wear on a user'"'"'s head, comprising the processor-implemented steps of:

processing image data from at least one forward-facing camera of the head-mounted display device to recognize a real-world object in a field of view of the at least one forward-facing camera;

controlling at least one microdisplay of the head-mounted display device to visually identify the real-world object to the user using augmented reality images, via a see-through lens of the head-mounted display device;

via a user interface of the head-mounted display device, receiving a command from the user which selects the real-world object as a target for receiving augmented reality images;

determining a location of the user at a time of the command; and

in response to the command, storing at least one record for the real-world object, the at least one record associates;

(a) at least one visual characteristic of the real-world object based on the image data and (b) the location.

10. The processor-implemented method of claim 9, further comprising:

determining a gaze direction of the user at the time of the command; and

configuring the at least one record to associate the gaze direction with the location.

11. The processor-implemented method of claim 9, wherein:

the at least one visual characteristic of the real-world object comprises boundaries of the real-world object.

12. The processor-implemented method of claim 9, wherein:

the at least one visual characteristic of the real-world object comprises a pattern of the real-world object.

13. The processor-implemented method of claim 9, further comprising:

controlling the at least one microdisplay to visually display augmented reality images of one or more data streams;

via the user interface of the head-mounted display device, receiving at least one command from the user which selects at least one of the one or more data streams; and

in response to the at least one command which selects at least one of the one or more data streams, configuring the at least one record to associate the one or more data streams with the real-world object.

14. The processor-implemented method of claim 13, further comprising:

the at least one command sets a privacy setting of the user for the data stream, the privacy setting indicates whether the at least one record which associates the data stream with the real-world object can be shared with one or more other users.

15. The processor-implemented method of claim 13, further comprising:

the at least one command provides a setting which indicates whether the data stream is to be associated with the real-world object as a unique object, or the data stream is to be associated with a class of objects of which the real-world object is an instance of the class.

16. The processor-implemented method of claim 15, wherein:

the class of objects comprises vertical planar objects having an aspect ratio within a specified range of aspect ratios.

17. The processor-implemented method of claim 13, further comprising:

the at least one command sets a priority setting, the priority setting indicating a priority of a personal data stream which is associated with the real-world object relative to a public data stream which is associated with the real-world object.

18. A computer readable storage device having computer readable software embodied thereon for programming at least one processor to perform a method for controlling a head-mounted display device for wear on a user'"'"'s head, the method comprising:

processing image data from at least one forward-facing camera of the head-mounted display device to identify a subject real-world object;

accessing respective records for respective real-world objects, each respective record indicates at least one visual characteristic of the respective real-world object, and is associated with a respective data stream;

identifying one of the respective records for which the at least one visual characteristic is a match to the image data; and

controlling at least one microdisplay of the head-mounted display device to project augmented reality images from the respective data stream of the one of the respective records, through at least one see-through lens of the head-mounted display device to at least one eye of the user, so that the augmented reality images are registered to the real-world object.

19. The computer readable storage device of claim 18, wherein the respective records include location data, and the method performed further comprises:

determining a location of the user, the identifying filters out one or more of the respective records whose location data is inconsistent with the location of the user.

20. The computer readable storage device of claim 18, wherein the method performed further comprises:

in response to the identifying, accessing the respective data stream of the one of the respective records via a wireless link of a network, the controlling is responsive to the accessing.