Head-mounted display device which provides surround video

• US 8,576,276 B2
• Filed: 11/18/2010
• Issued: 11/05/2013
• Est. Priority Date: 11/18/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Bar-Zeev, Avi, Kipman, Alex Aben-Athar
• Primary Examiner(s): Patel, Jay
• Assistant Examiner(s): Aghevli, Reza
• Application Number:
  US12/949,147
• Publication Number:
  US 20120127284A1
• Time in Patent Office: 1,083 Days
• Field of Search: None
• US Class Current: 348/53
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G09G 2340/12 Overlay of images, i.e. displayed pixel being the result of switching between the corresponding input pixels
• CPC Class Codes: G09G 3/003 to produce spatial visual effects

Abstract

A see-through head-mounted display (HMD) device, e.g., in the form of augmented reality glasses, allows a user to view a video display device and an associated augmented reality image. In one approach, the augmented reality image is aligned with edges of the video display device to provide a larger, augmented viewing region. The HMD can include a camera which identifies the edges. The augmented reality image can be synchronized in time with content of the video display device. In another approach, the augmented reality image video provides a virtual audience which accompanies a user in watching the video display device. In another approach, the augmented reality image includes a 3-D which appears to emerge from the video display device, and which is rendered from a perspective of a user'"'"'s location. In another approach, the augmented reality image can be rendered on a vertical or horizontal surface in a static location.

Claims

1. A user display apparatus, comprising:

a head-mounted display unit including a see-through lens;

an augmented reality emitter, associated with the head-mounted display unit, which emits light to a user'"'"'s eye, the light represents augmented reality video images;

a camera which provides an image of a video display screen the user is looking at;

at least one sensor which tracks an orientation and location of a head of the user; and

at least one control circuit which identifies edges of the video display screen based on the image provided by the camera, and controls the augmented reality emitter responsive to the at least one sensor, to display the augmented reality video images spatially synchronized with the edges of the video display screen and temporally synchronized with video content displayed by the video display screen.

2. The user display apparatus of claim 1, wherein:

the augmented reality video images augment the video content displayed by the video display screen and appear to the user to be in a static location relative to the video display screen.

3. The user display apparatus of claim 2, wherein:

the augmented reality video images are displayed in a region of space which appears to the user to be adjacent to, and extend outward from, at least one of the edges of the video display screen, do not overlap the video display screen, and appear to the user as a continuation of the content displayed by the video display screen.

4. The user display apparatus of claim 3, wherein:

the augmented reality video images appear as the continuation of the content displayed by the video display screen by including at least one of a color or a lighting element of the video content displayed by the video display screen.

5. The user display apparatus of claim 1, further comprising:

a wireless communication interface which receives timing data, where the augmented reality video images are displayed in temporal synchronism with the video content displayed by the video display screen, in response to the received timing data.

6. The user display apparatus of claim 1, further comprising:

an opacity filter associated with the head-mounted display unit which blocks the user'"'"'s eye from seeing the video display screen, where the augmented reality video images replace the video content displayed by the video display screen with a larger sized image.

18. The user display apparatus of claim 1, wherein:

the at least one control circuit recognizes the video display screen in the image provided by the camera based on an expected shape of the video display screen.

19. The user display apparatus of claim 1, wherein:

the at least one control circuit recognizes the video display screen in the image provided by the camera based on a greater light intensity which is seen at a location of the video display screen.

20. The user display apparatus of claim 3, wherein:

the augmented reality video images are displayed on a common vertical plane as the video display screen.

7. A user display apparatus, comprising:

a head-mounted display unit including a see-through lens;

an augmented reality emitter, associated with the head-mounted display unit, which emits light to a user'"'"'s eye, the light represents augmented reality video images;

at least one sensor which tracks an orientation and location of a head of the user; and

at least one control circuit which operates in temporal and spatial synchronism with video content displayed by a video display screen, the at least one control circuit controls the augmented reality emitter to provide a 3-D display, responsive to the at least one sensor, which is rendered from a perspective of the user, based on the orientation and location of the user'"'"'s head.

8. The user display apparatus of claim 7, wherein:

the 3-D display comprises at least one audience member which appears to the user to be co-located with the user and to be viewing the video display screen with the user, and the at least one control circuit provides spatialized audio based on a virtual location of the at least one audience member.

9. The user display apparatus of claim 7, further comprising:

a wireless communication interface which receives data, the data comprising timing data, where the 3-D display is provided in synchronism with the video content displayed by the video display screen, in response to the received data.

10. The user display apparatus of claim 7, wherein:

the 3-D display comprises a 3-D object moving through space, away from the video display screen, where the object is rendered from the perspective of the user, based on the orientation and location of the user'"'"'s head, and is synchronized with the video content displayed by the video display screen.

11. The user display apparatus of claim 10, wherein:

the at least one control circuit controls the augmented reality emitter to display the 3-D object moving through space in response to a detected location of the user.

12. The user display apparatus of claim 10, wherein:

the 3-D object is displayed using a spherical or cylindrical virtual surface, where the display of the object is responsive to rotation of the user'"'"'s head.

13. A user display apparatus, comprising:

a head-mounted display unit including a see-through lens;

an augmented reality emitter, associated with the head-mounted display unit, which emits light to a user'"'"'s eye, the light represents 3-D augmented reality images;

at least one sensor which tracks an orientation and location of a head of the user; and

at least one control circuit which controls the augmented reality emitter, responsive to the at least one sensor, to display the augmented reality images, where the augmented reality images appear to the user to be in a static location in a substantially vertical plane when the user looks at a video display screen, the static location in the substantially vertical plane is a specified location relative to the video display screen, and the augmented reality images are temporally synchronized with video content of the video display screen.

14. The user display apparatus of claim 13, wherein:

the augmented reality images appear to the user to be in a static location in a substantially horizontal plane when the user looks down.

15. The user display apparatus of claim 13, wherein:

the augmented reality images are responsive to a streaming layer of web-based information which is associated with video content displayed by the video display screen.

16. The user display apparatus of claim 13, wherein:

the augmented reality images block the video display screen and provide a same video content as the video display screen in a different size than the video display screen.

17. The user display apparatus of claim 13, wherein:

the augmented reality images comprises instructional text which is temporally synchronized with video content displayed by the video display screen.