Image processing device and method, and distribution medium

• US 7,200,247 B2
• Filed: 11/26/2003
• Issued: 04/03/2007
• Est. Priority Date: 05/19/1998
• Status: Expired due to Term

• Current Assignee: Sony Computer Entertainment Incorporated (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment Incorporated (Sony Group Corp.)
• Inventors: Ohba, Akio
• Primary Examiner(s): CHAWAN, SHEELA C
• Application Number:
  US10/723,874
• Publication Number:
  US 20040240740A1
• Time in Patent Office: 1,224 Days
• Field of Search: 382/103, 382/144, 382/117, 382/118, 382/162, 382/165, 382/173, 382/181, 382/190, 382/199, 382/240, 382/284, 382/260, 382/294, 382/283, 382/274, 382/214, 348/14.16, 348/169, 348/588, 348/36, 348/87, 345/428, 345/31, 345/592, 345/630, 345/204, 345/629
• US Class Current: 382/103
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06V 30/2504 Coarse or fine approaches, e.g. resolution of ambiguities or multiscale approaches
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: H04N 19/17 the unit being an image region, e.g. an object
• CPC Class Codes: H04N 19/20 using video object coding
• CPC Class Codes: H04N 7/142 Constructional details of the terminal equipment, e.g. arrangements of the camera and the display
• CPC Class Codes: H04N 7/15 Conference systems
• CPC Class Codes: H04N 7/157 defining a virtual conference space and using avatars or agents computer conference optimisation or adaptation H04L12/1827

Abstract

To make it possible to accurately track a prescribed part. a low-resolution overall image is taken in from an input image, and a medium-resolution image, whose center is the head part, is extracted from it. When it is possible to extract a medium-resolution image in the middle of which is the head part, an even higher-resolution image is extracted, in the middle of which is the two-eyes part.

Claims

1. An image processing device comprising:

a means for tracking an image of a prescribed part from an image of a user and extracting it as image and converting said image of the user to a left-right reversed mirror image;

a means for superimposing and drawing a CG image of a mask on said image of the user; and

a means for changing the corresponding part of said CG image of the mask in response to the action of the user;

whereby an image of the user can be obtained who puts on a mask whose expression changes in real time in correspondence to a change of the prescribed part of the user.

2. The image processing device recited in claim 1 which further comprises:

a display means for displaying said mirror image; and

a position adjustment means;

whereby the user can appropriately adjust his own position or a position of a video camera which extracts said image of the user.

3. The image processing device recited in claim 1 which further comprises:

a display means for displaying said mirror image; and

a means) for superimposing a display which indicates a region of said prescribed part onto said mirror image

whereby by means of the display indicating the region of said prescribed part, the user can recognize that said region is extracted and the tracking operation is carried out.

4. The image processing device recited in claim 1 which further comprises:

a display means for displaying an image of a mask whose expression changes in correspondence to a change of said prescribed part of the user;

whereby the user can recognize identity between own action change and an action change of the mask displayed on said display means.

5. The image processing device recited in claim 1 which further comprises:

a means for outputting correction instruction such that position extraction of said prescribed part takes place correctly when an action change of the user and an action change of the mask displayed on said display means do not correspond.

6. The image processing device recited in claim 1 which further comprises:

a means for outputting correction instruction such that expression extraction of said prescribed part takes place correctly when an action change of the user and an action change of the mask displayed on said display means do not correspond.

7. The image processing device recited in claim 1 wherein:

said image processing device is connected with a plurality of other image processing devices via a network line and said image of the mask is displayed on a display device each of said plurality of other image processing devices.

8. The image processing device recited in claim 1 further characterized by said means for tracking an image of a prescribed part from an image of a user and extracting it as image, wherein said means for tracking comprises:

a first extraction means that extracts an image data of a prescribed part from an image inputted from a video camera;

a second extraction means that extracts an image data of a portion of said prescribed part from said image of the prescribed part; and

a tracking means for automatically tracking said image data of the prescribed part and extracting it as image;

whereby even if the user moves relative to said video camera, said prescribed part can be automatically tracked and extracted as image.

9. The image processing device recited in claim 1 further characterized by said means for tracking an image of a prescribed part from an image of a user and extracting it as image

wherein said means tracks images of a plurality of said prescribed parts and extracts them as images.

10. The image processing device recited in claim 9 wherein said plurality of prescribed parts are regions of eyes, mouth and eyebrows.

11. The image processing device recited in claim 1 further characterized by said means for superimposing and drawing a CG image of a mask on said image of the user

wherein said means comprises:

a means for extracting a region of said prescribed part by extracting as image change a change of operation by the user in response to action instruction and

a drawing means for generating said CG image of the mask and superimposing and drawing said CG image of the mask at the display position of the image of the user

whereby an image is obtained in which the image of the user is replaced by said mask.

12. The image processing device recited in claim 11 wherein said action instruction is transmitted to the user via voice from said image processing device.

13. The image processing device recited in claim 11 wherein said action instruction is transmitted to the user by displaying a prescribed message to a display device of said image processing device.

14. The image processing device recited in claim 1 further characterized by said means for changing the corresponding part of said CG image of the mask in response to the action of the user;

wherein said means detects a region of change of operation by the user in response to action instruction and changes the corresponding part of the mask in correspondence to the detection result.

15. The image processing device recited in claim 14 wherein said action instruction is transmitted to the user via voice from said image processing device.

16. The image processing device recited in claim 14 wherein said action instruction is transmitted to the user by displaying a prescribed message to a display device of said image processing device.

17. An image processing method by which an image of a user is obtained who puts on a mask whose expression changes in real time in correspondence to a change of a prescribed part of the user, said method comprising the steps of:

tracking an image of the prescribed part from the image of the user and extracting it as image;

converting said image of the user to a left-right reversed mirror image;

superimposing and drawing a CG image of a mask on said image of the user; and

changing the corresponding part of said CG image of the mask in response to the action of the user.

18. The image processing method recited in claim 17 which further comprises the steps of:

displaying said mirror image; and

taking a position adjustment means whereby the user appropriately adjusts his own position or a position of a video camera which extracts said image of the user.

19. The image processing method recited in claim 17 which further comprises the steps of:

displaying said mirror image; and

superimposing a display which indicates a region of said prescribed part onto said mirror image;

whereby by means of the display indicating the region of said prescribed part, the user can recognize that said region is extracted and the tracking operation is carried out.

20. The image processing method recited in claim 17 which further comprises the step of:

displaying an image of a mask whose expression changes correspondence to a change of said prescribed part of the user;

whereby the user can recognize identity between own action change and an action change of the mask displayed on said display means.

21. The image processing method recited in claim 17 which further comprises the step of:

outputting correction instruction such that position extraction of said prescribed part takes place correctly when an action change of the user and an action change of the mask displayed on said display means do not correspond.

22. The image processing method recited in claim 17 which further comprises the step of:

outputting correction instruction such that expression extraction of said prescribed part takes place correctly when an action change of the user and an action change of the mask displayed on said display means do not correspond.

23. The image processing method recited in claim 17 wherein:

said image processing device is connected with a plurality of other image processing devices via a network line and said image of the mask is displayed on a display device each of said plurality of other image processing devices.

24. The image processing method recited in claim 17 further characterized by said step of tracking an image of a prescribed part from an image of a user and extracting it as image, wherein said step includes:

a first extraction step that extracts an image data of a prescribed part from an image inputted from a video camera;

a second extraction step that extracts an image data of a portion of said prescribed part from said image of the prescribed part; and

a tracking step of automatically tracking said image data of the prescribed part and extracting it as image;

whereby even if the user moves relative to said video camera, said prescribed part can be automatically tracked and extracted as image.

25. The image processing method recited in claim 17 further characterized by said step of tracking an image of a prescribed part from an image of a user and extracting it as image;

wherein said step tracks images of a plurality of said prescribed parts and extracts them as images.

26. The image processing method recited in claim 25 wherein said plurality of prescribed parts are regions of eyes, mouth and eyebrows.

27. The image processing method recited in claim 17 further characterized by said step of superimposing and drawing a CG image of a mask on said image of the user, wherein said step includes the steps of:

extracting a region of said prescribed part by extracting as image change a change of operation by the user in response to action instruction; and

generating said CG image of the mask and superimposing and drawing said CG image of the mask at the display position of the image of the user;

whereby an image is obtained in which the image of the user is replaced by said mask.

28. The image processing method recited in claim 27 wherein said action instruction is transmitted to the user via voice from said image processing device.

29. The image processing method recited in claim 27 wherein said action instruction is transmitted to the user by displaying a prescribed message to a display device of said image processing device.

30. The image processing method recited in claim 17 further characterized by said step of changing the corresponding part of said CG image of the mask in response to the action of the user;

wherein said step detects a region of change of operation by the user in response to action instruction and changes the corresponding part of the mask in correspondence to the detection result.

31. The image processing method recited in claim 30 wherein said action instruction is transmitted to the user via voice from said image processing device.

32. The image processing method recited in claim 30 wherein said action instruction is transmitted to the user by displaying a prescribed message to a display device of said image processing device.

33. A computer having a processor, stored program control and memory, said computer being operated by a computer readable and executable program, whereby said computer is operated to perform the image processing method described in any one of claims 17–

32.

34. A recording medium on which is recorded a computer readable and executable program wherein said program is operable to execute the image processing method described in any one of claims 17–

32.