Synchronizing virtual actor's performances to a speaker's voice

• US 9,035,955 B2
• Filed: 05/16/2012
• Issued: 05/19/2015
• Est. Priority Date: 05/16/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Keane, Brian E., Sugden, Ben J., Crocco, Robert L. Jr., Miles, Christopher E., Perez, Kathryn Stone, Massey, Laura K., Lamb, Mathew J., Kipman, Alex Aben-Athar
• Primary Examiner(s): BROOME, SAID A
• Application Number:
  US13/473,268
• Publication Number:
  US 20130307856A1
• Time in Patent Office: 1,098 Days
• Field of Search: None
• US Class Current: 345/473
• CPC Class Codes: G03H 1/2294 Addressing the hologram to an active spatial light modulator
• CPC Class Codes: G03H 2001/2284 Superimposing the holobject with other visual information
• CPC Class Codes: G03H 2227/02 Handheld portable device, e.g. holographic camera, mobile holographic display
• CPC Class Codes: G03H 2270/55 being an optical element, e.g. spectacles
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0483 Interaction with page-structured environments, e.g. book metaphor
• CPC Class Codes: G06F 3/1415 with means for detecting differences between the image stored in the host and the images displayed on the displays
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G09B 5/062 Combinations of audio and printed presentations, e.g. magnetically striped cards, talking books, magnetic tapes with printed texts thereon
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 2021/105 Synthesis of the lips movements from speech, e.g. for talking heads
• CPC Class Codes: G10L 21/10 Transforming into visible information
• CPC Class Codes: G10L 25/03 characterised by the type of extracted parameters

Abstract

A system for generating and displaying holographic visual aids associated with a story to an end user of a head-mounted display device while the end user is reading the story or perceiving the story being read aloud is described. The story may be embodied within a reading object (e.g., a book) in which words of the story may be displayed to the end user. The holographic visual aids may include a predefined character animation that is synchronized to a portion of the story corresponding with the character being animated. A reading pace of a portion of the story may be used to control the playback speed of the predefined character animation in real-time such that the character is perceived to be lip-syncing the story being read aloud. In some cases, an existing book without predetermined AR tags may be augmented with holographic visual aids.

Claims

1. A method for generating and displaying holographic visual aids, comprising:

capturing one or more images of a reading object using a mobile device;

identifying the reading object using the one or more images;

detecting at the mobile device that a first phrase of the reading object has been spoken by a particular person;

determining a reading pace corresponding with the first phrase spoken by the particular person;

determining a location of a second phrase different from the first phrase as it appears in the reading object; and

displaying a sequence of character images at a rate corresponding with the reading pace of the first phrase, the sequence of character images includes a sequence of mouth shape images displayed such that a character associated with the location appears to speak the second phrase at the reading pace.

2. The method of claim 1, wherein:

the reading object comprises a book;

the first phrase comprises a sentence from the book; and

the displaying a sequence of character images includes detecting a page of the book within a field of view of the mobile device and identifying the character based on the page of the book.

3. The method of claim 1, further comprising:

detecting a failure of the second phrase being completely spoken; and

displaying an idling holographic animation in response to detecting the failure of the second phrase being completely spoken.

4. The method of claim 1, wherein:

the determining a reading pace includes detecting a first utterance corresponding with a portion of the first phrase.

5. The method of claim 1, wherein:

the displaying a sequence of character images includes detecting a sequence of keywords corresponding with a portion of the second phrase and displaying the sequence of character images in response to detecting the sequence of keywords.

6. The method of claim 1, wherein:

the determining a reading pace includes determining the amount of time the particular person took to speak a plurality of words corresponding with the first phrase.

7. The method of claim 2, wherein:

the identifying the character includes detecting an augmented reality tag on the page of the book and identifying the character based on the augmented reality tag.

8. The method of claim 1, wherein:

the mobile device comprises a see-through HMD worn by a first person different from the particular person, the displaying a sequence of character images includes displaying the sequence of character images using the see-through HMD.

9. One or more storage devices containing processor readable code for programming one or more processors to perform a method for generating and displaying holographic visual aids comprising the steps of:

acquiring a user profile associated with an end user of a mobile device, the user profile includes a voice model corresponding with a particular person different from the end user;

capturing one or more images of a reading object using the mobile device;

identifying the reading object using the one or more images;

detecting a first phrase of the reading object spoken by the particular person based on the voice model;

determining a reading pace corresponding with the first phrase spoken by the particular person;

determining a location of a second phrase different from the first phrase as it appears in the reading object; and

displaying a sequence of character images at a rate corresponding with the reading pace of the first phrase, the sequence of character images includes a sequence of mouth shape images displayed such that a character associated with the location appears to speak the second phrase at the reading pace.

10. The one or more storage devices of claim 9, further comprising:

detecting a failure of the second phrase being completely spoken; and

displaying an idling holographic animation in response to detecting the failure of the second phrase being completely spoken.

11. The one or more storage devices of claim 9, wherein:

the reading object comprises a book;

the first phrase comprises a sentence from the book; and

the displaying a sequence of character images includes detecting a page of the book within a field of view of the mobile device and identifying the character based on the page of the book.

12. The one or more storage devices of claim 11, wherein:

the identifying the character includes detecting an augmented reality tag on the page of the book and identifying the character based on the augmented reality tag.

13. The one or more storage devices of claim 9, wherein:

the determining a reading pace includes detecting a first utterance corresponding with a portion of the first phrase.

14. The one or more storage devices of claim 9, wherein:

the displaying a sequence of character images includes detecting a sequence of keywords corresponding with a portion of the second phrase and displaying the sequence of character images in response to detecting the sequence of keywords.

15. The one or more storage devices of claim 9, wherein:

the determining a reading pace includes determining the amount of time the particular person took to speak a plurality of words corresponding with the first phrase.

16. The one or more storage devices of claim 9, wherein:

the sequence of mouth shape images are displayed prior to the second phrase being completely spoken by the particular person.

17. The one or more storage devices of claim 9, wherein:

the mobile device comprises a see-through HMD worn by the end user different from the particular person, the displaying a sequence of character images includes displaying the sequence of character images using the see-through HMD.

18. An electronic device for generating and displaying holographic visual aids, comprising:

a memory, the memory stores a voice model associated with a particular person;

one or more processors, the one or more processors identify a reading object, the one or more processors detect that a first phrase of the reading object has been spoken by the particular person using the voice model, the one or more processors determine a reading pace corresponding with the first phrase spoken by the particular person, the one or more processors determine a location of a second phrase different from the first phrase as it appears in the reading object, the one or more processors generate a sequence of character images; and

a see-through display, the see-through display displays the sequence of character images at a rate corresponding with the reading pace of the first phrase, the sequence of character images includes a sequence of mouth shape images displayed such that a character associated with the location appears to speak the second phrase at the reading pace.

19. The electronic device of claim 18, wherein:

the reading object comprises a book;

the first phrase comprises a sentence from the book;

the second phrase comprises a second sentence from the book different from the sentence; and

the one or more processors detect a page of the book within a field of view of the electronic device and identify the character based on the page of the book.

20. The electronic device of claim 18, wherein:

the one or more processors detect a failure of the second phrase being completely spoken and cause an idling holographic animation to be displayed using the see-through display in response to detecting the failure of the second phrase being completely spoken.