Control System for Virtual Characters

• US 20170132828A1
• Filed: 11/07/2016
• Published: 05/11/2017
• Est. Priority Date: 11/06/2015
• Status: Active Grant

• Current Assignee: Mursion, Inc.
• Original Assignee: Mursion, Inc.
• Inventors: Zelenin, Alex, Kelly, Brian D., Nagendran, Arjun
• Granted Patent: US 10,489,957 B2
• Time in Patent Office: Days
• CPC Class Codes: A63F 13/45 Controlling the progress of the video game
• CPC Class Codes: A63F 13/63 by the player, e.g. authoring using a level editor
• CPC Class Codes: A63F 2300/6607 for animating game characters, e.g. skeleton kinematics
• CPC Class Codes: A63F 2300/8082 Virtual reality
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 7/00 Image analysis

Abstract

A control system provides an interface for virtual characters, or avatars, during live avatar-human interactions. A human interactor can select facial expressions, poses, and behaviors of the virtual character using an input device mapped to menus on a display device.

Claims

1. A system for controlling a virtual character, comprising:

one or more processors and memory, the memory including a stored library comprising a plurality of facial expressions, a plurality of poses, and a plurality of behaviors associated with a virtual character;

machine-readable instructions stored in the memory that, upon execution by the one or more processors cause the system to carry out operations comprising;

selecting, for the virtual character, one or more of a facial expression, a pose, and a behavior, and

displaying, on a display device in real time, movement of the virtual character from a prior position to a position exhibiting the selected one or more facial expression, pose, and behavior; and

an input device comprising one or more input elements mapped to selections from one or more of the plurality of facial expressions, the plurality of poses, and the plurality of behaviors associated with the virtual character.

2. The system of claim 1, wherein at least one input element of the input device comprises a control stick movable in radial directions, each radial direction mapped to a selection from one of the plurality of facial expressions, the plurality of poses, or the plurality of behaviors associated with the virtual character.

3. The system of claim 2, wherein radial directions of the control stick are discretized into spaces, each space mapped to a selection from one of the plurality of facial expressions, the plurality of poses, or the plurality of behaviors associated with the virtual character.

4. The system of claim 1, wherein the operations further comprise displaying a menu on the display device, the menu listing selections from one of the plurality of facial expressions, the plurality of poses, and the plurality of behaviors associated with a virtual character.

5. The system of claim 4, wherein the menu comprises a radial menu, and each selection on the radial menu is bound to a direction.

6. The system of claim 5, wherein the input device comprises at least one control stick movable in radial directions, wherein the radial directions of the control stick are mapped to a corresponding direction on the radial menu, wherein movement of the control stick in one of the radial directions moves a pointer on the radial menu to the selection bound to the corresponding direction on the radial menu.

7. The system of claim 6, wherein the mapping includes converting rectangular positional coordinates of the control stick into polar coordinates of the radial menu.

8. The system of claim 7, further comprising moving the pointer on the radial menu to a new direction when a square magnitude of the rectangular positional coordinates of the control stick exceeds a determined threshold.

9. The system of claim 1, wherein the input device comprises one or more input elements, at least one input element configured to select one of a plurality of menus for display on the display device, the plurality of menus includes a menu listing the plurality of facial expressions, a menu listing the plurality of poses, and a menu listing the plurality of behaviors associated with a virtual character.

10. The system of claim 9, wherein the input device includes two input elements configured so that simultaneous activation of the two elements selects a different menu for display on the display device.

11. The system of claim 1, further comprising, selecting one or more of a further facial expression, a further pose, and a further behavior, and displaying on the display device in real time movement of the virtual character to the selected one or more further facial expression, further pose, and further behavior.

12. The system of claim 1, wherein the memory further includes a stored library comprising a plurality of virtual characters, each virtual character associated with a cognitive and emotional profile, and the operations include selecting one of the virtual characters for display on the display device.

13. The system of claim 1, wherein the operations further comprise applying a velocity clamp to prevent movement of the virtual character from changing too quickly.

14. The system of claim 1, wherein movement of the virtual character to a new position is provided by an animation, stored in the memory, of the virtual character moving from a prior position to a selected position, the animation providing a change in pose or a change in behavior of the virtual character.

15. The system of claim 14, further comprising moving a joint of a virtual character from a prior animation position to a new animation position.

16. The system of claim 14, wherein a weight of an exiting animation is decreased while a while a weight of an entering animation is increased.

17. The system of claim 1, wherein movement of a virtual character from a selected facial position to a new facial expression comprises a transition of a plurality of blend shapes.

18. The system of claim 17, wherein in the transition from the selected facial expression to the new facial expression, deactivated blend shapes are weighted to decrease at a constant velocity and active blend shapes are weighted to increase at a constant velocity.

19. The system of claim 17, wherein different blend shapes are weighted to increase or decrease at different velocities.

20. The system of claim 17, wherein in the transition from the selected facial expression to a new facial expression with a different mouth shape, a last selected mouth shape is retained until its weigh reaches about 0.

21. The system of claim 1, wherein the input device comprises a further input element configured to control mouth motion of the virtual character.

22. The system of claim 1, wherein the input device comprises a further input element configured to control timing of the movement of the virtual character.

23. The system of claim 1, wherein the input device comprises a hand held controller or joystick.

24. The system of claim 1, wherein the display device and the input device are located at an interactor node, and a further display device is provided at an end user node located remotely from the interactor node and in communication with the one or more processors to receive a display of the movement of the virtual character.

25. The system of claim 1, wherein the virtual character is displayed on the end user node display device in real time.

26. A method for controlling a virtual character, comprising:

displaying a virtual character on an interactor node display device and on an end user display device;

at an interactor node including one or more processors and memory, selecting one or more of a facial expression, a pose, and a behavior for the virtual character; and

displaying, on the interactor node display device and the end user display device, movement of the virtual character from a prior position to a position exhibiting the selected one or more of the facial expression, the pose, and the behavior.

27. The method of claim 26, further comprising displaying a menu on the interactor node display device, the menu listing selections from one of the plurality of facial expressions, the plurality of poses, and the plurality of behaviors associated with a virtual character.

28. The method of claim 27, further comprising activating an input element on an input device to select a selection on the menu displayed on the interactor node display device.

29. The method of claim 28, wherein the menu comprises a radial menu, and each selection on the radial menu is bound to a direction, and wherein the input device includes at least one control stick movable in radial directions each mapped to a corresponding selection on the radial menu.

30. The method of claim 26, further comprising selecting the virtual character from among a plurality of virtual characters stored in memory.

31. A method of creating a virtual character for a system for virtual character interaction, comprising:

creating a virtual character with an emotional and cognitive profile;

creating a set of facial expressions for the virtual character;

creating a set of poses for the virtual character;

creating a set of behaviors for the virtual character;

categorizing the facial expressions, poses, and behaviors into response states based on the emotional and cognitive profile of the virtual character; and

storing the sets of facial expressions, poses, and behaviors in a non-transitory computer memory.

32. The method of claim 31, wherein the step of creating the set of facial expressions comprises:

providing a plurality of reference photographs of a human model'"'"'s face posing in a variety of predetermined expressions;

create a model of a face of the virtual character with a polygonal mesh based on a neutral facial expression; and

blending the neutral facial expression into a target facial expression by deforming vertices of the polygonal mesh from the neutral facial expression to the target facial expression.

33. The method of claim 32, wherein the step of blending the neutral facial expression into the target facial expression uses a blend shapes algorithm.

34. The method of claim 31, wherein the step of creating the set of poses comprises:

recording a live action video of a human actor performance;

determining start and end poses from still images of the video recording; and

generating an animation of the virtual character by interpolating between the start and end poses.

35. The method of claim 31, wherein the step of creating the set of behaviors comprises:

providing a video recording of a human performance of a behavior; and

generating a skeletal frame animation from the video recording.