AUGMENTED BOOKS IN A MIXED REALITY ENVIRONMENT

• US 20130321390A1
• Filed: 05/31/2012
• Published: 12/05/2013
• Est. Priority Date: 05/31/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Latta, Stephen G., Hastings, Ryan L., Brown, Cameron G., Krauss, Aaron, McCulloch, Daniel J., Sugden, Ben J.
• Granted Patent: US 9,165,381 B2
• Time in Patent Office: Days
• US Class Current: 345/419
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 16/338 Presentation of query results
• CPC Class Codes: G06F 16/9038 Presentation of query results
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/147 using display panels
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G09G 2380/14 Electronic books and readers
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence

Abstract

A system and method are disclosed for augmenting a reading experience in a mixed reality environment. In response to predefined verbal or physical gestures, the mixed reality system is able to answer a user'"'"'s questions or provide additional information relating to what the user is reading. Responses may be displayed to the user on virtual display slates in a border or around the reading material without obscuring text or interfering with the user'"'"'s reading experience.

Claims

1. A system for presenting a mixed reality experience to one or more users, the system comprising:

a display device for a user of the one or more users, the display device including a display unit for displaying a virtual image to the user of the display device; and

a computing system operatively coupled to the display device, the computing system generating the virtual image for display on the display device, the virtual image added in relation to reading material the user is reading or an image the user is viewing.

2. The system of claim 1, the computing system comprises at least one of a hub computing system and one or more processing units.

3. The system of claim 1, the virtual image added in relation to reading material including a response to a query from the user relating to the reading material.

4. The system of claim 3, wherein the response is one of text, an image and a video.

5. The system of claim 1, the virtual image added in relation to reading material including an annotation with user-defined content.

6. The system of claim 5, wherein the annotation is one of text, an image, a video, a data file, and audio file and an executable software application file.

7. The system of claim 1, wherein the reading material or image is one of a tangible reading material or image, an electronic reading material or image, or a virtual reading material or image.

8. A method of presenting a mixed reality experience to a user viewing a reading material including at least one of text and an image, the method comprising:

(a) receiving a request for information regarding a portion of the reading material;

(b) searching for a response to the request for information received in said step (a);

(c) formatting a virtual object for displaying content with the response if a response is found in said step (b); and

(d) displaying the virtual object to a user at a position in the user'"'"'s field of view that does not overlap with text and or images from the reading material.

9. The method recited in claim 8, wherein said step (a) of receiving a request for information comprises the step of receiving one of a plurality of predefined gestures associated with specific requests for information.

10. The method recited in claim 9, wherein the plurality of predefined gestures are at least one of physical gestures and verbal gestures.

11. The method recited in claim 9, wherein the plurality of predefined gestures are gestures for requesting at least one of:

i) a definition of selected text, ii) accumulated knowledge on a person, character, place or thing selected in the reading material, iii) a map of a place mentioned in the reading material, iv) a map of relations between people or characters from the reading material, and v) a graphical representation of a person, character, place or thing recited in the text.

12. The method recited in claim 8, wherein said step (a) of receiving a request for information regarding a portion of the reading material comprises the step of receiving one of a plurality of predefined gestures for identifying the portion of reading material on which information is sought.

13. The method recited in claim 12, wherein the plurality of predefined gestures for identifying the portion of reading material including at least one of:

i) pointing at the portion of reading material, ii) gazing at the portion of reading material, iii) speaking the portion of reading material, and iv) touching a display screen to highlight the portion of reading material.

14. The method recited in claim 8, wherein said step (b) of searching for a response to the request for information includes the step of searching for a response in a memory of a computing device assisting in the presentation of the mixed reality experience.

15. The method recited in claim 14, further comprising the step of executing a software application on the computing device, the software application including queriable information relating to the reading material.

16. The method recited in claim 8, wherein said step (b) of searching for a response to the request for information includes the step of searching for a response at a designated website accessible via a network connection.

17. The method recited in claim 8, wherein said step (c) of formatting a virtual object for displaying content with the response comprises the step of formatting a virtual display slate for display with a size and at a location within a user'"'"'s field of view that does not obscure text or images from the reading material, or overlap with another virtual object displaying content with a response to an earlier request for information relating to the reading material.

18. A method of presenting a mixed reality experience to a user viewing a reading material including at least one of text and an image, the method comprising:

(a) identifying a position of the reading material in three-dimensional space;

(b) tracking a position of a user'"'"'s eyes with respect to text or images in the reading material;

(c) receiving a request for information regarding a portion of the reading material;

(d) searching for a response to the request for information received in said step (a);

(e) displaying the response to a user, if found in said step (d), in a virtual display slate at a position in the user'"'"'s field of view that does not overlap with text and or images from the reading material; and

(f) pinning the position of the virtual display slate in a fixed position with respect to the reading material in the user'"'"'s field of view.

19. The method recited in claim 18, wherein response displayed in said step (e) includes one of:

i) a definition of selected text, ii) accumulated knowledge on a person, character, place or thing selected in the reading material, iii) a map of a place mentioned in the reading material, iv) a map of relations between people or characters from the reading material, v) a graphical representation of a person, character, place or thing recited in the text, vi) a synopsis of a story set forth in the reading material, the synopsis not including subject matter from the story set forth in the story after a point of the selected text.

20. The method recited in claim 18, the virtual display slate recited in step (d) comprising a first virtual display slate, the method further comprising displaying a second virtual display slate to the user, the second virtual display slate included by an author of a software application relating to the reading material, the software application executing on a computing system associated with the mixed reality environment, the second virtual display slate displaying additional information about a portion of the reading material when the user reaches the portion of the reading material.