Natural user input for driving interactive stories

• US 8,381,108 B2
• Filed: 06/21/2010
• Issued: 02/19/2013
• Est. Priority Date: 06/21/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Fuller, Andrew, Poat, Rudy, Kipman, Alex Aben-Athar, Perez, Kathryn Stone
• Primary Examiner(s): RIES, LAURIE ANNE
• Application Number:
  US12/819,983
• Publication Number:
  US 20110314381A1
• Time in Patent Office: 974 Days
• Field of Search: 715/727, 715/728, 715/716, 715/720, 463/31, 463/36, 463/37, 463/38
• US Class Current: 715/727
• CPC Class Codes: A63F 13/10 Control of the course of the game, e.g. start, progess, end
• CPC Class Codes: A63F 13/12 involving interaction between a plurality of game devices, e.g. transmisison or distribution systems protocols for networked virtual reality, networked games H04L29/06034
• CPC Class Codes: A63F 13/20 Input arrangements for video game devices
• CPC Class Codes: A63F 13/21 characterised by their sensors, purposes or types
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/215 comprising means for detecting acoustic signals, e.g. using a microphone
• CPC Class Codes: A63F 13/30 Interconnection arrangements between game servers and game devices; Interconnection arrangements between game devices; Interconnection arrangements between game servers
• CPC Class Codes: A63F 13/424 involving acoustic input signals, e.g. by using the results of pitch or rhythm extraction or voice recognition
• CPC Class Codes: A63F 13/45 Controlling the progress of the video game
• CPC Class Codes: A63F 13/47 involving branching, e.g. choosing one of several possible scenarios at a given point in time
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: A63F 2300/6045 for mapping control signals received from the input arrangement into game commands
• CPC Class Codes: A63F 2300/632 by branching, e.g. choosing one of several possible story developments at a given point in time
• CPC Class Codes: G06F 16/4393 Multimedia presentations, e.g. slide shows, multimedia albums
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence

Abstract

A system and method are disclosed for combining interactive gaming aspects into a linear story. A user may interact with the linear story via a NUI system to alter the story and the images that are presented to the user. In an example, a user may alter the story by performing a predefined exploration gesture. This gesture brings the user into the 3-D world of the displayed image. In particular, the image displayed on the screen changes to create the impression that a user is stepping into the 3-D virtual world to allow a user to examine virtual objects from different perspectives or to peer around virtual objects.

Claims

1. In a system comprising a computing environment coupled to a capture device for capturing user motion and an audiovisual device for displaying images and/or providing audio, a method of combining interactive gaming aspects into a linear story, comprising:

a) presenting the linear story via the audiovisual device using at least one of images and an audio narrative, the linear story having a default story and an altered story, the default story including a default set of images and/or narrative that is presented to a user if a user does not interact with the story;

b) detecting at least one of a movement and a voice command by a user via the capture device;

c) identifying whether the movement and/or the voice command detected in said step b) is an interaction with the linear story that alters the linear story to include a story branch having at least one of additional images and additional narrative not included in the default set of images and/or narrative; and

d) altering the linear story to the altered story by presenting at least one of the additional images and additional narrative to a user via the audiovisual device.

2. The method of claim 1, wherein the linear story includes images having story scenes defined by three-dimensional descriptions of the scene in virtual machine space, said step d) of altering the linear story comprising the step e) of presenting additional images showing a scene from the default set of images from a different three dimensional viewing perspective within the virtual machine space.

3. The method of claim 2, the additional images presented in said step e) creating the impression of a user moving into the scene.

4. The method of claim 2, the additional images presented in said step e) creating the impression of a user examining an object from the scene more closely.

5. The method of claim 2, the additional images presented in said step e) creating the impression of a user examining an object from the scene from a different perspective.

6. The method of claim 2, the additional images presented in said step e) creating the impression of a user looking behind an object within the scene.

7. The method of claim 1, wherein said step d) of altering the linear story comprises the step f) of a user interacting with a character displayed within a scene.

8. The method of claim 1, wherein said step d) of altering the linear story occurs where it is determined that a movement and/or voice command is not an interaction that alters the linear story, but a predefined event occurs which alters the linear story, the predefined event relating to receiving the linear story a predetermined number of times without altering the story.

9. The method of claim 1, wherein the linear story includes an audio narrative and images, said step a) of presenting the linear story comprising the steps of:

a1) a user voicing the narrative,

a2) matching the user-voiced narrative to corresponding images of the linear story, and

a3) presenting the images at a pace determined by a pace of the user-voiced narrative.

10. The method of claim 1, wherein the linear story includes an audio narrative and images presented to a user, said step a) of presenting the linear story comprising the steps of:

a1) a third party voicing the narrative, the third party not present with the user and the third party'"'"'s voice provided as the narrative via a speaker in a vicinity of the user,

a2) matching the third party-voiced narrative to corresponding images of the linear story, and

a3) presenting the images at a pace determined by a pace of the third party-voiced narrative.

11. A processor-readable storage medium for programming a processor to perform a method of combining interactive gaming aspects into a linear story, comprising:

a) presenting the linear story via an audiovisual device using at least one of images and an audio narrative, the linear story presented as a complete story, beginning to end and including a default set of images, in the event no interaction by a user is perceived by a capture device monitoring user movements;

b) detecting a movement by a user via a capture device associated with the audiovisual device;

c) identifying whether the movement detected in said step b) is an interaction with the linear story requiring a modification from the linear story to include a story branch having additional images not included in the default set of images;

d) determining whether the additional images may be rendered based on stored or processed data;

e) altering the linear story by presenting the additional images to a user via the audiovisual device if it is determined the additional images may be rendered in said step d); and

f) scoring a user'"'"'s interaction where the interaction corresponds to awarding or taking away a predetermined number of points based on how the user interacts with the story.

12. The processor-readable storage medium of claim 11, wherein the linear story includes images having story scenes defined by three-dimensional descriptions of the scene in virtual machine space, said step e) of altering the linear story comprising the step g) of presenting additional images showing a scene from the default set of images from a different three dimensional viewing perspective within the virtual machine space.

13. The processor-readable storage medium of claim 11, wherein said step b) comprises the step h) of a user taking over at least partial control of a character displayed as part of the linear story.

14. The processor-readable storage medium of claim 13, wherein said step h) comprises the step of a user controlling movement of a character displayed as part of the linear story in a monkey-see-monkey-do fashion, and/or a user controlling talking of the character.

15. The processor-readable storage medium of claim 11, wherein said steps a) and e) comprise the step of displaying the linear story and/or story branch in at least one of still-image panels, dynamic computer graphics animation and linear video.

16. A system for combining interactive gaming aspects into a linear story, comprising:

an audiovisual device for presenting at least one of images and an audio narration;

an image capture device for capturing at least one of image and audio data from a user; and

a computing environment coupled to the audiovisual device and image capture device, the computing environment;

a) presenting the linear story via the audiovisual device using at least one of images and an audio narrative, the linear story presented as a complete story, beginning to end and including a default set of images, in the event no interaction by a user is perceived by the capture device;

b) detecting a movement by a user via the capture device;

c) identifying whether the detected movement is an exploration gesture;

d) branching from the linear story to a story branch upon identifying the movement as an exploration gesture, the branching including;

d1) sensing a point on the audiovisual device indicated by the user,

d2) calculating a 3-D real world vector from a portion of the user to the point on the audiovisual device indicated by the user,

d3) translating the 3-D real world vector to a 3-D machine space vector having a start point at the current view perspective and an end point at the virtual object in 3-D machine space, and

d4) displaying the virtual object from a viewing perspective at the end point of the 3-D machine space vector.

17. The system of claim 16, the computing environment sensing the point on the audiovisual device indicated by the user by the capture device sensing a position of the user'"'"'s head.

18. The system of claim 16, the computing environment sensing the point on the audiovisual device indicated by the user by the capture device sensing a point indicated by the user'"'"'s hand.

19. The system of claim 16, the computing environment branching back to the linear story when the user gestures that they are finished examining the virtual object.

20. The system of claim 16, wherein the user is able to augment a score associated with the user'"'"'s experience in interacting with the linear and branched story.

21. In a system comprising a computing environment coupled to a capture device for capturing user motion and an audiovisual device for displaying images and/or providing audio, a method of combining interactive gaming aspects into a linear story, comprising:

a) presenting the linear story via the audiovisual device using at least an audio narrative, the audio narrative voice by a third party not present with the user and the third party'"'"'s voice provided as the narrative via a speaker in a vicinity of the user;

b) detecting at least one of a movement and a voice command by a user via the capture device;

c) identifying whether the movement and/or the voice command detected in said step b) is an interaction with the linear story that alters the linear story to include a story branch having at least one of additional images and additional narrative not included in the default set of images and/or narrative; and

d) altering the linear story by presenting at least one of the additional images and additional narrative to a user via the audiovisual device.

22. A method of combining interactive gaming aspects into a linear story, comprising:

a) presenting the linear story via an audiovisual device using at least one of images and an audio narrative, the linear story presented as a complete story, beginning to end and including a default set of images, in the event no interaction by a user is perceived by a capture device monitoring user movements;

b) detecting a movement by a user via a capture device associated with the audiovisual device, wherein said step b) comprises the step of a user taking over at least partial control of a character displayed as part of the linear story,

c) identifying whether the movement detected in said step b) is an interaction with the linear story requiring a modification from the linear story to include a story branch having additional images not included in the default set of images;

d) determining whether the additional images may be rendered based on stored or processed data; and

e) altering the linear story by presenting the additional images to a user via the audiovisual device if it is determined the additional images may be rendered in said step d).