Control system for virtual characters

• US 10,489,957 B2
• Filed: 11/07/2016
• Issued: 11/26/2019
• Est. Priority Date: 11/06/2015
• Status: Active Grant

• Current Assignee: Mursion, Inc.
• Original Assignee: Mursion, Inc.
• Inventors: Zelenin, Alex, Kelly, Brian D., Nagendran, Arjun
• Primary Examiner(s): Nguyen, Kimbinh T
• Application Number:
  US15/344,969
• Publication Number:
  US 20170132828A1
• Time in Patent Office: 1,114 Days
• Field of Search: 345473
• CPC Class Codes: A63F 13/45 Controlling the progress of the video game
• CPC Class Codes: A63F 13/63 by the player, e.g. authoring using a level editor
• CPC Class Codes: A63F 2300/6607 for animating game characters, e.g. skeleton kinematics
• CPC Class Codes: A63F 2300/8082 Virtual reality
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 7/00 Image analysis

Abstract

A control system provides an interface for virtual characters, or avatars, during live avatar-human interactions. A human interactor can select facial expressions, poses, and behaviors of the virtual character using an input device mapped to menus on a display device.

Claims

1. A system for controlling a virtual character, comprising:

one or more processors and memory located at an interactor node, the memory including a stored library comprising a plurality of facial expressions, a plurality of poses, and a plurality of behaviors associated with a virtual character; and

an input device in communication with the one or more processors and comprising input elements mapped to the plurality of facial expressions, the plurality of poses, and the plurality of behaviors associated with the virtual character, the input elements selectable by a user to select a facial expression, a pose, and a behavior for the virtual character;

wherein the one or more processors are operable, via machine-readable instructions stored in the memory, to;

display a menu on a display device located at the interactor node, the menu listing selections from one of the plurality of facial expressions, the plurality of poses, and the plurality of behaviors associated with a virtual character,

receive from the input device at the interactor node one or more of a user-selected facial expression, a user-selected pose, and a user-selected behavior for the virtual character selected from the menu via the mapped input elements,

provide in real time an animation of movement of the virtual character from a prior position to a position exhibiting the one or more user-selected facial expression, user-selected pose, and user-selected behavior, and

display in real time, on the display device located at the interactor node and on a further display device located at an end user node located remotely from the interactor node, the movement of the virtual character from the prior position to the position exhibiting the one or more user-selected facial expression, user-selected pose, and user-selected behavior.

2. The system of claim 1, wherein at least one input element of the input device comprises a control stick movable in radial directions, each radial direction mapped to a selection from one of the plurality of facial expressions, the plurality of poses, or the plurality of behaviors associated with the virtual character.

3. The system of claim 2, wherein radial directions of the control stick are discretized into spaces, each space mapped to a selection from one of the plurality of facial expressions, the plurality of poses, or the plurality of behaviors associated with the virtual character.

4. The system of claim 1, wherein the menu comprises a radial menu, and each selection on the radial menu is bound to a direction.

5. The system of claim 4, wherein the input device comprises at least one control stick movable in radial directions, wherein the radial directions of the control stick are mapped to a corresponding direction on the radial menu, wherein movement of the control stick in one of the radial directions moves a pointer on the radial menu to the selection bound to the corresponding direction on the radial menu.

6. The system of claim 5, wherein the mapping includes converting rectangular positional coordinates of the control stick into polar coordinates of the radial menu.

7. The system of claim 6, further comprising moving the pointer on the radial menu to a new direction when a square magnitude of the rectangular positional coordinates of the control stick exceeds a determined threshold.

8. The system of claim 1, wherein the input elements of the input device comprise at least one input element configured to select one of a plurality of menus for display on the display device, the plurality of menus includes a menu listing the plurality of facial expressions, a menu listing the plurality of poses, and a menu listing the plurality of behaviors associated with a virtual character.

9. The system of claim 8, wherein the input device includes two input elements configured so that simultaneous activation of the two elements selects a different menu for display on the display device.

10. The system of claim 1, further comprising, selecting one or more of a further facial expression, a further pose, and a further behavior, and displaying on the display device in real time movement of the virtual character to the selected one or more further facial expression, further pose, and further behavior.

11. The system of claim 1, wherein the memory further includes a stored library comprising a plurality of virtual characters, each virtual character associated with a cognitive and emotional profile, and the operations include selecting one of the virtual characters for display on the display device.

12. The system of claim 1, wherein the operations further comprise applying a velocity clamp to prevent movement of the virtual character from changing too quickly.

13. The system of claim 1, wherein movement of the virtual character to a new position is provided by an animation, stored in the memory, of the virtual character moving from a prior position to a selected position, the animation providing a change in pose or a change in behavior of the virtual character.

14. The system of claim 13, further comprising moving a joint of a virtual character from a prior animation position to a new animation position.

15. The system of claim 13, wherein a weight of an exiting animation is decreased while a while a weight of an entering animation is increased.

16. The system of claim 1, wherein movement of a virtual character from a selected facial position to a new facial expression comprises a transition of a plurality of blend shapes.

17. The system of claim 16, wherein in the transition from the selected facial expression to the new facial expression, deactivated blend shapes are weighted to decrease at a constant velocity and active blend shapes are weighted to increase at a constant velocity.

18. The system of claim 16, wherein different blend shapes are weighted to increase or decrease at different velocities.

19. The system of claim 16, wherein in the transition from the selected facial expression to a new facial expression with a different mouth shape, a last selected mouth shape is retained until its weight reaches about 0.

20. The system of claim 1, wherein the input device comprises a further input element configured to control mouth motion of the virtual character.

21. The system of claim 1, wherein the input device comprises a further input element configured to control timing of the movement of the virtual character.

22. The system of claim 1, wherein the input device comprises a hand held controller or joystick.

23. A method for controlling a virtual character, comprising:

displaying a virtual character on a display device located at an interactor node and on an end user display device located remotely from the interactor node;

displaying a menu on the interactor node display device, the menu listing selections from one of a plurality of facial expressions, a plurality of poses, and a plurality of behaviors associated with the virtual character;

at the interactor node including one or more processors and memory, receiving, from a user input device, one or more of a user-selected facial expression, a user-selected pose, and a user-selected behavior for the virtual character selected from the menu;

providing an animation of movement of the virtual character from a prior position to a position exhibiting the one or more user-selected facial expression, user-selected pose, and user-selected behavior; and

displaying, on the interactor node display device and the end user display device, the movement of the virtual character in real time from a prior position to a position exhibiting the one or more user-selected facial expression, user-selected pose, and user-selected behavior.

24. The method of claim 23, further comprising activating an input element on an input device to select a selection on the menu displayed on the interactor node display device.

25. The method of claim 24, wherein the menu comprises a radial menu, and each selection on the radial menu is bound to a direction, and wherein the input device includes at least one control stick movable in radial directions each mapped to a corresponding selection on the radial menu.

26. The method of claim 23, further comprising selecting the virtual character from among a plurality of virtual characters stored in memory.

27. A method of creating a virtual character for a system for virtual character interaction, comprising:

creating, at a computer comprising one or more processors and a non-transitory computer-readable memory, a virtual character with an emotional and cognitive profile, further comprising;

creating a set of facial expressions for the virtual character comprising capturing an image of each facial expression and storing each facial expression in a library of facial expressions in the non-transitory computer-readable memory;

creating a set of poses for the virtual character comprising creating an animation for each pose from a sequence of video images and storing each animation in a library of poses in the non-transitory computer-readable memory;

creating a set of behaviors for the virtual character comprising capturing each behavior with a motion capture system, separating each behavior into one or more motion clips, and storing each clip in a library of motions in the non-transitory computer-readable memory; and

categorizing the facial expressions, poses, and behaviors into response states based on the emotional and cognitive profile of the virtual character and storing the response states in the non-transitory computer-readable memory,

wherein the libraries and the responses states are stored in a configuration system of the computer in communication with an input system, and each facial expression, pose, and behavior is bound to a selection of the input system for selection by a user.

28. The method of claim 27, wherein the step of creating the set of facial expressions further comprises:

providing a plurality of reference photographs of a human model'"'"'s face posing in a variety of predetermined expressions;

create a model of a face of the virtual character with a polygonal mesh based on a neutral facial expression; and

blending the neutral facial expression into a target facial expression by deforming vertices of the polygonal mesh from the neutral facial expression to the target facial expression.

29. The method of claim 28, wherein the step of blending the neutral facial expression into the target facial expression uses a blend shapes algorithm.

30. The method of claim 27, wherein the step of creating the set of poses further comprises:

recording a live action video of a human actor performance;

determining start and end poses from still images of the video recording; and

generating the animation of the virtual character by interpolating between the start and end poses.

31. The method of claim 27, wherein the step of creating the set of behaviors further comprises:

providing a video recording of a human performance of a behavior; and

generating a skeletal frame animation from the video recording.