Method and system for controlling an avatar using computer vision

• US 20020135581A1
• Filed: 03/26/2001
• Published: 09/26/2002
• Est. Priority Date: 03/26/2001
• Status: Active Grant

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Russell, Ryan S., Zipperer, John B.
• Granted Patent: US 6,697,072 B2
• Time in Patent Office: Days
• US Class Current: 345/474
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation

Abstract

A method and system for controlling an avatar using computer vision is presented. A video stream representing a background and a foreground is received. A user in the foreground is segmented from the background and classified to produce effector information. An avatar may be controlled based on the effector information.

Claims

1. A method for controlling an avatar using computer vision, said method comprising:

receiving a video stream representing a background and a foreground;

segmenting a user in said foreground from said background and classifying said user to produce effector information; and

controlling an avatar based on said effector information.

2. The method according to claim 1, wherein said foreground includes a moving user.

3. The method according to claim 2, wherein said effector information includes three-dimensional coordinates of the head position of said moving user.

4. The method according to claim 3, wherein said segmenting said user includes segmenting the head of said moving user.

5. The method according to claim 2, wherein said effector information includes three-dimensional coordinates of the hand positions of said moving user.

6. The method according to claim 5, wherein said segmenting said user includes segmenting the hands of said moving user.

7. The method according to claim 1, further comprising:

calibrating a video camera with respect to said background to produce a background reference image; and

capturing said video stream.

8. The method according to claim 7, wherein said segmenting said user comprises:

separating, in said video stream, said user from said background; and

tracking the head and the hands of said user to produce two-dimensional image coordinates of said head and said hands.

9. The method according to claim 8, wherein said separating said user from said background includes subtracting said background reference image from each frame of said video stream.

10. The method according to claim 8, wherein said controlling further comprises:

receiving said effector information;

performing inverse kinematics based on said effector information to produce a movement of said avatar; and

rendering said movement of said avatar on a display screeen.

11. The method according to claim 10, further comprising converting said two-dimensional image coordinates to three-dimensional coordinates before said performing.

12. The method according to claim 11, wherein said converting includes converting said two-dimensional image coordinates to said three-dimensional coordinates using a constant depth coordinate value.

13. The method according to claim 10, wherein said effector information includes at least one of coordinates of the left hand of said user, coordinates of the right hand of said user, and coordinates of the head of said user.

14. The method according to claim 13, wherein said movement of said avatar includes at least one of head movement, left arm movement, and right arm movement.

15. The method according to claim 14, wherein said controlling further comprises:

controlling said right arm movement of said avatar using said coordinates of the left hand of said user; and

controlling said left arm movement of said avatar using said coordinates of the right hand of said user.

16. The method according to claim 10, wherein bones of said avatar are effected, and wherein said performing includes performing kinematics for each of the effected bones of said avatar.

17. The method according to claim 10, wherein joints of said avatar are effected, and wherein said performing includes performing kinematics for each of the effected joints of said avatar.

19. The system according to claim 18, further comprising a video camera calibrated with respect to said background to produce a background reference image, wherein said video stream is captured using said video camera.

20. The system according to claim 19, wherein said segmentor separates, in said video stream, said user from said background, and tracks the head and hands of said user to produce two-dimensional image coordinates of said head and said hands.

21. The system according to claim 20, wherein said segmentor subtracts said background reference image from each frame of said video stream.

22. The system according to claim 20, wherein said controller is further configured to receive said effector information, perform inverse kinematics based on said effector information to produce a movement of said avatar, and render said movement of said avatar on a display screeen.

23. The system according to claim 22, wherein said controller is further configured to convert said two-dimensional image coordinates to three-dimensional coordinates.

24. The system according to claim 23, wherein said effector information includes at least one of three-dimensional coordinates of the left hand of said user, three-dimensional coordinates of the right hand of said user, and three-dimensional coordinates of the head of said user.

26. The computer-readable medium of claim 25, further comprising processor-executable instructions for:

calibrating a video camera with respect to said background to produce a background reference image; and

capturing said video stream.

27. The computer-readable medium of claim 26, wherein said segmenting said user comprises:

separating, in said video stream, said user from said background; and

tracking the head and the hands of said user to produce two-dimensional image coordinates of said head and said hands.

28. The computer-readable medium according to claim 27, wherein said separating said user from said background includes subtracting said background reference image from each frame of said video stream.

18. A system for controlling an avatar using computer vision, said system comprising:

a video segmentor configured to receive a video stream representing a background and a foreground, and to segment a user in said foreground from said background and classify said user to produce effector information; and

a controller configured to control an avatar based on said effector information.

25. A computer-readable medium encoded with a plurality of processor-executable instructions for:

receiving a video stream representing a background and a foreground;

segmenting a user in said foreground from said background and classifying said user to produce effector information; and

controlling an avatar based on said effector information.