Synchronizing virtual actor's performances to a speaker's voice

• US 9,524,081 B2
• Filed: 04/15/2015
• Issued: 12/20/2016
• Est. Priority Date: 05/16/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Miles, Christopher E., Keane, Brian E., Sugden, Ben J., Crocco, Robert L. Jr., Perez, Kathryn Stone, Massey, Laura K., Lamb, Mathew J., Kipman, Alex Aben-Athar
• Primary Examiner(s): BROOME, SAID A
• Application Number:
  US14/687,877
• Publication Number:
  US 20150220231A1
• Time in Patent Office: 615 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G03H 1/2294 Addressing the hologram to an active spatial light modulator
• CPC Class Codes: G03H 2001/2284 Superimposing the holobject with other visual information
• CPC Class Codes: G03H 2227/02 Handheld portable device, e.g. holographic camera, mobile holographic display
• CPC Class Codes: G03H 2270/55 being an optical element, e.g. spectacles
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0483 Interaction with page-structured environments, e.g. book metaphor
• CPC Class Codes: G06F 3/1415 with means for detecting differences between the image stored in the host and the images displayed on the displays
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G09B 5/062 Combinations of audio and printed presentations, e.g. magnetically striped cards, talking books, magnetic tapes with printed texts thereon
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 2021/105 Synthesis of the lips movements from speech, e.g. for talking heads
• CPC Class Codes: G10L 21/10 Transforming into visible information
• CPC Class Codes: G10L 25/03 characterised by the type of extracted parameters

Abstract

A system for generating and displaying holographic visual aids associated with a story to an end user of a head-mounted display device while the end user is reading the story or perceiving the story being read aloud is described. The story may be embodied within a reading object (e.g., a book) in which words of the story may be displayed to the end user. The holographic visual aids may include a predefined character animation that is synchronized to a portion of the story corresponding with the character being animated. A reading pace of a portion of the story may be used to control the playback speed of the predefined character animation in real-time such that the character is perceived to be lip-syncing the story being read aloud. In some cases, an existing book without predetermined AR tags may be augmented with holographic visual aids.

Claims

1. A method for synchronizing performance of a virtual object to performance of a person, comprising:

capturing a first audio signal associated with sounds made by the person during a first time period;

detecting that a first utterance has been spoken by the person using the captured first audio signal;

determining a speed at which the person spoke the first utterance;

capturing a second audio signal associated with sounds made by the person during a second time period subsequent to the first time period;

detecting that a portion of a second utterance has been spoken by the person using the captured second audio signal; and

displaying, using a display, a sequence of images of the virtual object at a rate corresponding with the determined speed of the first utterance in response to detecting that the portion of the second utterance has been spoken by the person, the sequence of images including a sequence of mouth shape images displayed such that the virtual object appears to speak the second utterance at the determined speed of the first utterance.

2. The method of claim 1, wherein:

the capturing a first audio signal includes capturing the first audio signal using a microphone; and

the detecting that a first utterance has been spoken by the person is performed by a mobile device, the detecting that a first utterance has been spoken by the person includes applying speech recognition techniques to the captured first audio signal; and

the displaying a sequence of images of the virtual object includes displaying the sequence of images using the mobile device.

3. The method of claim 2, wherein:

the mobile device comprises a head-mounted display device.

4. The method of claim 2, further comprising:

identifying a reading object within a field of view of the mobile device, the first utterance corresponds with a first sentence from the reading object.

5. The method of claim 4, wherein:

the reading object comprises a book.

6. The method of claim 4, wherein:

the second utterance comprises a second sentence from the reading object that is spoken by the person.

7. The method of claim 4, further comprising:

detecting a page of the reading object within the field of view of the mobile device;

identifying the virtual object based on the page of the reading object; and

the displaying the sequence of images includes displaying the sequence of images such that the virtual object appears to be attached to the reading object.

8. The method of claim 1, wherein:

the displaying a sequence of images of the virtual object includes displaying the sequence of mouth shape images prior to the second utterance being completely spoken by the person.

9. The method of claim 1, further comprising:

detecting a failure of the second utterance being completely spoken by the person; and

displaying an idling holographic animation in response to detecting the failure of the second phrase being completely spoken by the person.

10. The method of claim 1, wherein:

the detecting that a portion of a second utterance has been spoken by the person includes detecting a sequence of keywords corresponding with the portion of the second utterance.

11. The method of claim 1, wherein:

the determining a speed at which the person spoke the first utterance includes determining an amount of time that the person took to speak a plurality of words corresponding with the first utterance.

12. The method of claim 2, wherein:

the mobile device comprises a see-through head-mounted display device worn by a first person different from the person.

13. An electronic device for synchronizing performance of a virtual object to performance of a person, comprising:

one or more processors, the one or more processors detect that a first utterance has been spoken by a person and determine a speed at which the person spoke the first utterance, the one or more processors detect that a portion of a second utterance has been spoken by the person and generate a sequence of images of the virtual object in response to detecting that the portion of the second utterance has been spoken by the person; and

a see-through display in communication with the one or more processors, the see-through display displays the sequence of images of the virtual object at a rate corresponding with the speed of the first utterance, the sequence of images including a sequence of mouth shape images that are displayed using the see-through display such that the virtual object appears to speak the second utterance at the speed of the first utterance.

14. The electronic device of claim 13, wherein:

the electronic device comprises a head-mounted display device.

15. The electronic device of claim 13, wherein:

the one or more processors identify a reading object within a field of view of the electronic device, the second utterance comprises a second sentence from the reading object that is spoken by the person.

16. The electronic device of claim 15, wherein:

the reading object comprises a book.

17. The electronic device of claim 13, wherein:

the see-through display displays the sequence of mouth shape images prior to the second utterance being completely spoken by the person.

18. One or more hardware storage devices containing processor readable code for programming one or more processors to perform a method for synchronizing performance of a virtual object to performance of a person, the processor readable code comprising:

processor readable code configured to detect that a first utterance has been spoken by the person;

processor readable code configured to determine a speed at which the person spoke the first utterance;

processor readable code configured to detect that a portion of a second utterance has been spoken by the person; and

processor readable code configured to cause a sequence of images of the virtual object to be displayed at a rate corresponding with the speed of the first utterance subsequent to detecting that the portion of the second utterance has been spoken by the person, the sequence of images including a sequence of mouth shape images that are displayed such that the virtual object appears to speak the second utterance at the speed of the first utterance.

19. The one or more hardware storage devices of claim 18, wherein the processor readable code further comprises:

processor readable code configured to detect that the first utterance has been spoken using a head-mounted display device; and

processor readable code configured to cause the sequence of images of the virtual object to be displayed using a display of the head-mounted display device.

20. The one or more hardware storage devices of claim 19, wherein the processor readable code further comprises:

processor readable code configured to cause the sequence of images of the virtual object to be displayed using the display of the head-mounted display device such that the sequence of mouth shape images are displayed prior to the second utterance being completely spoken by the person.