Method and system for controlling an avatar using computer vision

• US 6,697,072 B2
• Filed: 03/26/2001
• Issued: 02/24/2004
• Est. Priority Date: 03/26/2001
• Status: Active Grant

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Russell, Ryan S., Zipperer, John B.
• Primary Examiner(s): Nguyen, Phu K.
• Application Number:
  US09/816,158
• Publication Number:
  US 20020135581A1
• Time in Patent Office: 1,065 Days
• Field of Search: 345/473, 345/474, 345/475, 382/103, 382/115, 348/169
• US Class Current: 345/474
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation

Abstract

A method and system for controlling an avatar using computer vision is presented. A video stream representing a background and a foreground is received. A user in the foreground is segmented from the background and classified to produce effector information. An avatar may be controlled based on the effector information.

Claims

1. A method for controlling an avatar using computer vision, said method comprising:

receiving a video stream representing a background and a foreground;

segmenting a user in said foreground from said background;

tracking a head position of the user to produce effector information;

performing coordinate conversion on the effector information;

performing inverse kinematics on the effector inforamtion; and

controlling said avatar based on said effector information.

2. The method according to claim 1, wherein said foreground includes a moving user.

3. The method according to claim 2, wherein said effector information includes three-dimensional coordinates of the head position of said moving user.

4. The method according to claim 3, wherein said segmenting said user includes segmenting the head of said moving user.

5. The method according to claim 2, wherein said effector information includes three-dimensional coordinates of at least one hand position of said moving user.

6. The method according to claim 5, wherein said segmenting said user includes segmenting at least one hand of said moving user.

7. A method for controlling an avatar using computer vision, said method comprising:

calibrating a video camera with respect to a background to produce a background reference image;

receiving a video stream representing said background and a foreground;

segmenting a user in said foreground from said background and classifying said user to produce effector information; and

controlling said avatar based on said effector information.

8. The method according to claim 7, wherein said segmenting said user comprises:

separating, in said video stream, said user from said background; and

tracking the head and at least one hand of said user to produce two-dimensional dimensional image coordinates of said head and said at least one hand.

9. The method according to claim 8, wherein said separating said user from said background includes subtracting said background reference image from each frame of said video stream.

10. The method according to claim 8, wherein said controlling further comprises:

receiving said effector information;

performing inverse kinematics based on said effector information to produce a movement of said avatar; and

rendering said movement of said avatar on a display screen.

11. The method according to claim 10, further comprising converting said two-dimensional image coordinates to three-dimensional coordinates before said performing inverse kinematics.

12. The method according to claim 11, wherein said converting includes converting said two-dimensional image coordinates to said three-dimensional coordinates using a constant depth coordinate value.

13. The method according to claim 10, wherein said effector information includes at least one of coordinates of the left hand of said user, coordinates of the right hand of said user, and coordinates of the head of said user.

14. The method according to claim 13, wherein said movement of said avatar includes at least one of head movement, left arm movement, and right arm movement.

15. The method according to claim 14, wherein said controlling further comprises:

controlling said right arm movement of said avatar using said coordinates of the left hand of said user; and

controlling said left arm movement of said avatar using said coordinates of the right hand of said user.

16. The method according to claim 10, wherein bones of said avatar are effected, and wherein said performing includes performing kinematics for each of the effected bones of said avatar.

17. The method according to claim 10, wherein joints of said avatar are effected, and wherein said performing includes performing kinematics for each of the effected joints of said avatar.

18. A system for controlling an avatar using computer vision, said system comprising:

a video segmentor configured to receive a video stream representing a background and a foreground and produce effector information, said video segmentor having a background/foreground classifier and a hand and head tracker

a controller configured to control said avatar based on said effector information, said controller having a coordinate converter, a inverse kinematics engine, and a graphics rendering engine; and

a display.

19. A system for controlling an avatar using computer vision, said system comprising:

a video camera calibrated with respect to a background to produce a background reference image, wherein a video stream is captured using said video camera;

a video segmentor configured to receive the video stream representing the background and a foreground, and to segment a user in said foreground from said background and classify said user to produce effector information; and

a controller configured to control said avatar based on said effector information.

20. The system according to claim 19, wherein said segmentor separates, in said video stream, said user from said background, and tracks the head and at least one hand of said user to produce two-dimensional image coordinates of said head and said at least one hand.

21. The system according to claim 20, wherein said segmentor subtracts said background reference image from each frame of said video stream.

22. The system according to claim 20, wherein said controller is further configured to receive said effector information, perform inverse kinematics based on said effector information to produce a movement of said avatar, and render said movement of said avatar on a display screen.

23. The system according to claim 22, wherein said controller is further configured to convert said two-dimensional image coordinates to three-dimensional coordinates.

24. The system according to claim 23, wherein said effector information includes at least one of three-dimensional coordinates of the left hand of said user, three-dimensional coordinates of the right hand of said user, and three-dimensional coordinates of the head of said user.

25. A computer-readable medium having encoded thereon a computer-readable program code which when executed causes a computer to:

receive a video stream representing a background and a foreground;

segment a user in said foreground from said background; and

track a head position of the user to produce effector information;

perform coordinate conversion on effector information;

perform inverse kinematics on effector inforamtion; and

control an avatar based on said effector information.

26. A computer-readable medium having encoded thereon a computer-readable program code which when executed causes a computer to:

calibrate a video camera with respect to a background to produce a background reference image;

capture a video stream with said video camera;

receive the video stream representing said background and a foreground;

segment a user in said foreground from said background;

track a head position of the user to produce effector information;

perform coordinate conversion on effector information;

perform inverse kinematics on effector inforamtion; and

control said avatar based on said effector information.

27. The computer-readable medium of claim 26, wherein said segmenting said user comprises:

separating, in said video stream, said user from said background; and

tracking the head and at least one hands of said user to produce two-dimensional image coordinates of said head and at least one hands.

28. The computer-readable medium according to claim 27, wherein said separating said user from said background includes subtracting said background reference image from each frame of said video stream.