HOLOGRAPHIC STORY TELLING

• US 20130307855A1
• Filed: 05/16/2012
• Published: 11/21/2013
• Est. Priority Date: 05/16/2012
• Status: Abandoned Application

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Robert L. Crocco Jr
• Inventors: Lamb, Mathew J., Sugden, Ben J., Crocco, Robert L. Jr., Keane, Brian E., Miles, Christopher E., Perez, Kathryn Stone, Kipman, Alex Aben-Athar, Massey, Laura K.
• Application Number:
  US13/473,241
• Publication Number:
  US 20130307855A1
• Time in Patent Office: Days
• US Class Current: 345/473
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0141 characterised by the informative content of the display
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G03H 1/0005 Adaptation of holography to specific applications holographic optical element G02B5/32; holographic scanner G02B26/106; recognition using holographic mask G06V10/88; holographic memories G11B7/0065, G11C13/042
• CPC Class Codes: G03H 1/2294 Addressing the hologram to an active spatial light modulator
• CPC Class Codes: G03H 2001/2284 Superimposing the holobject with other visual information
• CPC Class Codes: G03H 2227/02 Handheld portable device, e.g. holographic camera, mobile holographic display
• CPC Class Codes: G03H 2270/55 being an optical element, e.g. spectacles
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: H04N 13/221 using the relative movement between cameras and objects
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 13/383 for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes
• CPC Class Codes: H04N 2213/001 Constructional or mechanical details

Abstract

A system for generating and displaying holographic visual aids associated with a story to an end user of a head-mounted display device while the end user is reading the story or perceiving the story being read aloud is described. The story may be embodied within a reading object (e.g., a book) in which words of the story may be displayed to the end user. The holographic visual aids may include a predefined character animation that is synchronized to a portion of the story corresponding with the character being animated. A reading pace of a portion of the story may be used to control the playback speed of the predefined character animation in real-time such that the character is perceived to be lip-syncing the story being read aloud. In some cases, an existing book without predetermined AR tags may be augmented with holographic visual aids.

Claims

1. A method for generating and displaying holographic visual aids associated with a work of literature, comprising:

identifying a reading object within a reading distance of a mobile device;

acquiring a first virtual object associated with the reading object, the first virtual object is associated with a first triggering event;

detecting a portion of the reading object being read by an end user of the mobile device;

determining whether the first triggering event has been satisfied based on the detecting a portion of the reading object being read;

determining a reading pace associated with the portion of the reading object read by the end user in response to the first triggering event being satisfied;

generating a holographic animation associated with the first virtual object based on the reading pace; and

displaying at the mobile device the holographic animation.

2. The method of claim 1, wherein:

the reading object comprises a book; and

the portion of the reading object comprises a sentence from the book.

3. The method of claim 1, wherein:

the detecting a portion of the reading object being read includes acquiring one or more images associated with the reading object, identifying a page of the reading object based on the one or more images, and detecting the end user of the mobile device gazing at a particular sequence of words on the page.

4. The method of claim 1, wherein:

the detecting a portion of the reading object being read includes detecting a first utterance corresponding with the portion of the reading object.

5. The method of claim 1, wherein:

the determining whether the first triggering event has been satisfied includes determining whether a first utterance corresponding with the portion of the reading object has been spoken by the end user and whether a first gesture has been performed by a second person different from the first person.

6. The method of claim 1, wherein:

the determining a reading pace includes determining the amount of time the end user took to speak a plurality of words corresponding with the portion of the reading object.

7. The method of claim 1, wherein:

the determining a reading pace includes determining the amount of time the end user took to focus on a plurality of words corresponding with the portion of the reading object.

8. The method of claim 1, wherein:

the holographic animation includes a sequence of mouth shape images.

9. The method of claim 1, wherein:

the mobile device comprises a see-through HMD.

10. One or more storage devices containing processor readable code for programming one or more processors to perform a method for generating and displaying holographic visual aids associated with a work of literature comprising the steps of:

acquiring a user profile associated with an end user of a mobile device, the user profile includes a virtual object viewing history; and

identifying a reading object within a viewing distance of the mobile device;

acquiring one or more virtual objects associated with the reading object, the one or more virtual objects include a first set of virtual objects associated with a first triggering event;

detecting a first utterance spoken by a particular person reading from the reading object;

determining whether the first triggering event has been satisfied based on the detecting a first utterance;

determining a reading pace associated with the first utterance in response to the first triggering event being satisfied;

determining a first virtual object of the first set of virtual objects based on the virtual object viewing history;

generating a holographic animation associated with the first virtual object based on the reading pace; and

displaying at the mobile device the holographic animation.

11. The one or more storage devices of claim 10, wherein:

the reading object comprises a book;

the first utterance comprises a sentence from the book; and

the first virtual object comprises a virtual object that has not been previously viewed by the end user.

12. The one or more storage devices of claim 10, wherein:

the particular person is the end user.

13. The one or more storage devices of claim 10, wherein:

the detecting a first utterance includes detecting a sequence of one or more words spoken by the end user.

14. The one or more storage devices of claim 10, wherein:

the determining a reading pace includes determining the amount of time the particular person took to speak a plurality of words corresponding with the first utterance.

15. The one or more storage devices of claim 10, wherein:

the holographic animation includes a sequence of mouth shape images.

16. The one or more storage devices of claim 10, wherein:

the mobile device comprises a see-through HMD.

17. An electronic device for generating and displaying holographic visual aids associated with a work of literature, comprising:

a memory, the memory stores a first virtual object associated with a reading object, the first virtual object is associated with a first triggering event;

one or more processors, the one or more processors identify the reading object within a reading distance of the electronic device, the one or more processors detect a portion of the reading object being read by an end user of the electronic device, the one or more processors determine whether the first triggering event has been satisfied in response to detecting the portion of the reading object being read by the end user, the one or more processors determine a reading pace associated with the portion of the reading object read by the end user, the one or more processors generate a holographic animation associated with the first virtual object based on the reading pace; and

a see-through display, the see-through display displays the holographic animation.

18. The electronic device of claim 17, wherein:

the reading object comprises a book; and

the portion of the reading object comprises a sentence from the book.

19. The electronic device of claim 18, wherein:

the electronic device comprises a see-through HMD; and

the holographic animation includes a sequence of mouth shape images.

20. The electronic device of claim 19, wherein:

the one or more processors determine the reading pace by determining the amount of time the end user took to speak a plurality of words corresponding with the portion of the reading object.