System and method for generating a virtual reality scene based on individual asynchronous motion capture recordings

• US 10,664,045 B2
• Filed: 03/07/2019
• Issued: 05/26/2020
• Est. Priority Date: 05/23/2017
• Status: Active Grant

• Current Assignee: Mindshow, Inc.
• Original Assignee: Visionary VR, Inc.
• Inventors: Ross, Jonathan Michael, Baron, Gil
• Primary Examiner(s): Hailu, Tadesse
• Assistant Examiner(s): Hope, Darrin
• Application Number:
  US16/295,376
• Publication Number:
  US 20190204906A1
• Time in Patent Office: 446 Days
• Field of Search: 715728
• CPC Class Codes: A63F 13/00 Video games, i.e. games using an electronically generated display having two or more dimensions gaming systems which provide a financial reward G07F17/32
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/212 using sensors worn by the player, e.g. for measuring heart beat or leg activity
• CPC Class Codes: A63F 13/25 Output arrangements for video game devices
• CPC Class Codes: A63F 13/5255 according to dedicated instructions from a player, e.g. using a secondary joystick to rotate the camera around a player's character
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/16 Sound input; Sound output speech processing G10L
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2219/024 Multi-user, collaborative environment

Abstract

A system configured to generate and/or modify three-dimensional scenes comprising animated character(s) based on individual asynchronous motion capture recordings. The system may comprise sensor(s), display(s), and/or processor(s). The system may receive selection of a first character to virtually embody within the virtual space, receive a first request to capture the motion and/or the sound for the first character, and/or record first motion capture information characterizing the motion and/or the sound made by the first user as the first user virtually embodies the first character. The system may receive selection of a second character to virtually embody, receive a second request to capture the motion and/or the sound for the second character, and/or record second motion capture information. The system may generate a compiled virtual reality scene wherein the first character and the second character appear animated within the compiled virtual reality scene contemporaneously.

Claims

1. A system configured to generate and/or modify three-dimensional scenes comprising animated characters based on individual asynchronous motion capture recordings, the system comprising:

one or more sensors configured to generate output signals conveying information related to motion and/or sound made by one or more users in physical space, the sensors being configured to capture the motion and/or the sound made by the one or more users;

one or more displays that present virtual reality content to one or more users, wherein presentation of the virtual reality content via a display simulates presence of a user within a virtual space that is fixed relative to physical space, wherein the one or more displays are configured to present options for recording the motion and/or the sound for one or more of the characters within the virtual space;

one or more processors configured by machine-readable instructions to;

receive selection of a first character to virtually embody within the virtual space such that virtually embodying the first character enables a first user to record the motion and/or the sound to be made by the first character within the compiled virtual reality scene based on an existing animated character;

receive a first request to capture the motion and/or the sound for the first character;

present, via the one or more displays, existing motion and/or sound of the existing animated character prior to and/or during recording of the motion and/or the sound for the first character;

record first motion capture information characterizing the motion and/or the sound made by the first user as the first user virtually embodies the first character by following one or more motion and/or sound of the existing animated character, wherein the first motion capture information is captured in a manner such that actions of the first user are manifested by the first character within the compiled virtual reality scene;

receive selection of a second character to virtually embody, wherein the second character is separate and distinct from the first character, and wherein virtually embodying the second character enables the first user or another user to record one or more of the motion and/or the sound to be made by the second character within the compiled virtual reality scene;

receive a second request to capture the motion and/or the sound for the second character;

record second motion capture information that characterizes the motion and/or the sound made by the first user or other user as the first user or the other user virtually embodies the second character, wherein the second motion capture information is captured in a manner such that actions of the first user or the other user are manifested by the second character contemporaneously with the actions of the first user manifested by the first character within the compiled virtual reality scene; and

generate the compiled virtual reality scene including animation of the first character and the second character such that the first character and the second character appear animated within the compiled virtual reality scene contemporaneously.

2. The system of claim 1, wherein selection of the second character is received from a second display for a second computing device associated with a second user and the second motion capture information characterizes the motion or sound made by the second user, wherein the second user is a different user than the first user.

3. The system of claim 1, wherein the one or more processors are further configured by machine-readable instructions to:

receive selection of a third character to virtually embody, wherein the third character is separate and distinct from the first character and the second character, and wherein virtually embodying the third character enables the first user or a different user to record one or more of the motion and/or the sound to be made by the third character within the compiled virtual reality scene;

receive a third request to capture the motion and/or the sound for the third character;

record third motion capture information that characterizes the motion and/or the sound made by the first user or the different user as the first user or the different user virtually embodies the third character, wherein the third motion capture information is captured in a manner such that the actions of the first user and/or the different user are manifested by the third character contemporaneously with the actions of the first user manifested by the first character and the actions of the first user or the other user manifested by the second character within the compiled virtual reality scene,

such that the compiled virtual reality scene includes animation of the first character, the second character, and the third character such that the first character, the second character, and the third character appear animated within the compiled virtual reality scene contemporaneously.

4. The system of claim 3, wherein the first motion capture information characterizes the motion or the sound made by the first user, the second motion capture information characterizes the motion and/or the sound made by a second user, and the third motion capture information characterizes the motion and/or the sound made by a third user, wherein the first user, the second user, and the third user are different.

5. The system of claim 4, wherein the first user, the second user, and the third user are associated with different computing devices located at different physical locations.

6. The system of claim 3, wherein the first motion capture information characterizes the motion or the sound made by the first user, the second motion capture information characterizes the motion and/or the sound made by the second user, and the third motion capture information characterizes the motion and/or the sound made by the first user.

7. The system of claim 1, wherein the virtual reality content includes an editing scene, wherein the editing scene is an editing version of the compiled scene that is presented to the one or more users while recording motion capture information for one or more characters, such that responsive to receiving the second request to capture the motion and/or sound for the second character, the editing scene includes a manifestation of the first user'"'"'s actions by the first character presented contemporaneously to the first user or the other user while the second motion capture information is being recorded, and wherein the displays are configured to present the editing scene including one or more manifestations of one or more users'"'"' actions by one or more characters based on previously recorded motion capture information while recording subsequent motion capture information characterizing the motion and/or the sound made by the users as the users virtually embody one or more characters.

8. The system of claim 1, wherein the one or more processors are further configured by machine-readable instructions to receive selection of a start time within a timeline of the compiled scene that indicates when the first character should start manifesting the actions of the first user within the compiled virtual reality scene during playback of the compiled virtual reality scene.

9. The system of claim 1, wherein the one or more processors are further configured by machine-readable instructions to receive selection of one or more virtual objects, virtual scenery themes, and/or virtual scenery items for placement within an editing scene to be reflected in the compiled virtual reality scene.

10. The system of claim 1, wherein the one or more processors are further configured by machine-readable instructions to receive selection of a first facial expression for the first character, such that the first motion capture information includes the first facial expression for the first character and one or more of the actions of the first user are manifested in the compiled virtual reality scene by the first character with the first facial expression.

11. A method for generating and/or modifying three-dimensional scenes comprising animated characters based on individual asynchronous motion capture recordings, the method being implemented by one or more sensors, displays, and/or processors configured to perform the method, the method comprising:

generating, by the one or more sensors, output signals conveying information related to motion and/or sound made by one or more users in physical space, the sensors being configured to capture the motion and/or the sound made by the one or more users;

presenting, via the one or more displays, virtual reality content to one or more users, wherein presentation of the virtual reality content via a display simulates presence of a user within a virtual space that is fixed relative to physical space, wherein the one or more displays are configured to present options for recording the motion and/or the sound for one or more of the characters within the virtual space;

receiving, at the one or more processors, selection of a first character to virtually embody within the virtual space such that virtually embodying the first character enables a first user to record the motion and/or the sound to be made by the first character within the compiled virtual reality scene based on an existing animated character;

receiving, at the one or more processors, a first request to capture the motion and/or the sound for the first character;

presenting, via the one or more displays, existing motion and/or sound of the existing animated character prior to and/or during recording of the motion and/or the sound for the first character;

recording, by the one or more processors, first motion capture information characterizing the motion and/or the sound made by the first user as the first user virtually embodies the first character by following one or more motion and/or sound of the existing animated character, wherein the first motion capture information is captured in a manner such that actions of the first user are manifested by the first character within the compiled virtual reality scene;

receiving, at the one or more processors, selection of a second character to virtually embody, wherein the second character is separate and distinct from the first character, and wherein virtually embodying the second character enables the first user or another user to record one or more of the motion and/or the sound to be made by the second character within the compiled virtual reality scene;

receiving, at the one or more processors, a second request to capture the motion and/or the sound for the second character;

recording, by the one or more processors, second motion capture information that characterizes the motion and/or the sound made by the first user or other user as the first user or the other user virtually embodies the second character, wherein the second motion capture information is captured in a manner such that actions of the first user or the other user are manifested by the second character contemporaneously with the actions of the first user manifested by the first character within the compiled virtual reality scene; and

generating, by the one or more processors, the compiled virtual reality scene including animation of the first character and the second character such that the first character and the second character appear animated within the compiled virtual reality scene contemporaneously.

12. The method of claim 11, wherein selection of the second character is received from a second display for a second computing device associated with a second user and the second motion capture information characterizes the motion or sound made by the second user, wherein the second user is a different user than the first user.

13. The method of claim 11, further comprising:

receiving, at the one or more processors, selection of a third character to virtually embody, wherein the third character is separate and distinct from the first character and the second character, and wherein virtually embodying the third character enables the first user or a different user to record one or more of the motion and/or the sound to be made by the third character within the compiled virtual reality scene;

receiving, at the one or more processors, a third request to capture the motion and/or the sound for the third character;

recording, by the one or more processors, third motion capture information that characterizes the motion and/or the sound made by the first user or the different user as the first user or the different user virtually embodies the third character, wherein the third motion capture information is captured in a manner such that the actions of the first user and/or the different user are manifested by the third character contemporaneously with the actions of the first user manifested by the first character and the actions of the first user or the other user manifested by the second character within the compiled virtual reality scene,

such that the compiled virtual reality scene includes animation of the first character, the second character, and the third character such that the first character, the second character, and the third character appear animated within the compiled virtual reality scene contemporaneously.

14. The method of claim 13, wherein the first motion capture information characterizes the motion or the sound made by the first user, the second motion capture information characterizes the motion and/or the sound made by a second user, and the third motion capture information characterizes the motion and/or the sound made by a third user, wherein the first user, the second user, and the third user are different.

15. The method of claim 14, wherein the first user, the second user, and the third user are associated with different computing devices located at different physical locations.

16. The method of claim 13, wherein the first motion capture information characterizes the motion or the sound made by the first user, the second motion capture information characterizes the motion and/or the sound made by the second user, and the third motion capture information characterizes the motion and/or the sound made by the first user.

17. The method of claim 11, wherein the virtual reality content includes an editing scene, wherein the editing scene is an editing version of the compiled scene that is presented to the one or more users while recording motion capture information for one or more characters, such that responsive to receiving the second request to capture the motion and/or sound for the second character, the editing scene includes a manifestation of the first user'"'"'s actions by the first character presented contemporaneously to the first user or the other user while the second motion capture information is being recorded, and wherein the displays are configured to present the editing scene including one or more manifestations of one or more users'"'"' actions by one or more characters based on previously recorded motion capture information while recording subsequent motion capture information characterizing the motion and/or the sound made by the users as the users virtually embody one or more characters.

18. The method of claim 11, further comprising receiving, at the one or more processors, selection of a start time within a timeline of the compiled scene that indicates when the first character should start manifesting the actions of the first user within the compiled virtual reality scene during playback of the compiled virtual reality scene.

19. The method of claim 11, further comprising receiving, at the one or more processors, selection of one or more virtual objects, virtual scenery themes, and/or virtual scenery items for placement within an editing scene to be reflected in the compiled virtual reality scene.

20. The method of claim 11, further comprising receiving, at the one or more processors, selection of a first facial expression for the first character, such that the first motion capture information includes the first facial expression for the first character and one or more of the actions of the first user are manifested in the compiled virtual reality scene by the first character with the first facial expression.