IMAGE PROCESSING DEVICE, IMAGE PROCESSING METHOD AND PROGRAM

• US 20110096093A1
• Filed: 09/10/2010
• Published: 04/28/2011
• Est. Priority Date: 10/27/2009
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Guo, Jingjing, OI, Kenichiro, Suto, Yasuhiro
• Granted Patent: US 8,933,966 B2
• Time in Patent Office: Days
• US Class Current: 345/633
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/20164 Salient point detection; Corner detection
• CPC Class Codes: G06T 7/74 involving reference images or patches

Abstract

There is provided an image processing device, including: a data storage unit storing feature data indicating a feature of appearance of an object; an environment map generating unit for generating an environment map representing a position of one or more objects existing in a real space based on an input image obtained by imaging the real space using an imaging device and the feature data stored in the data storage unit; and an output image generating unit for generating an output image obtained by erasing an erasing target object from the input image based on a position of the erasing target object specified out of objects present in the input image represented in the environment map and a position of the imaging device.

Claims

1. An image processing device, comprising:

a data storage unit storing feature data indicating a feature of appearance of an object;

an environment map generating unit for generating an environment map representing a position of one or more objects existing in a real space based on an input image obtained by imaging the real space using an imaging device and the feature data stored in the data storage unit; and

an output image generating unit for generating an output image obtained by erasing an erasing target object from the input image based on a position of the erasing target object specified out of objects present in the input image represented in the environment map and a position of the imaging device.

2. The image processing device according to claim 1, further comprising:

a detecting unit for dynamically detecting the position of the imaging device based on the input image and the feature data.

3. The image processing device according to claim 2, wherein the output image generating unit includes:

an erased area determining unit for determining an erased area being an area in the input image corresponding to the erasing target object based on the position of the erasing target object in the environment map and the position of the imaging device;

a CG image generating unit for generating a CG image to be superimposed on the input image; and

an image superimposing unit for generating the output image by superimposing the CG image generated by the CG image generating unit on the input image for at least a part of the erased area.

4. The image processing device according to claim 3, wherein the CG image generating unit generates the CG image based on a position of an object other than the erasing target object represented by the environment map and the position of the imaging device.

5. The image processing device according to claim 4, wherein

the output image generating unit further includes a position estimating unit for estimating a position of a floor surface or a wall surface in the real space based on a position of a point on a surface of the object represented by the environment map, and

the CG image generating unit generates the CG image further based on the position of the floor surface or the wall surface in the real space estimated by the position estimating unit.

6. The image processing device according to claim 5, wherein

the feature data includes data indicating, for one or more points on the surface of each object, whether each point is likely to come in contact with the floor surface or the wall surface in the real space,

and the position estimating unit estimates the position of the floor surface or the wall surface in the real space further based on the feature data.

7. The image processing device according to claim 5, wherein the CG image generating unit generates the CG image of a portion included in the erased area and the portion in which the floor surface or the wall surface in the real space should be shown according to an image of a portion in which the floor surface or the wall surface in the real space is estimated to be shown of the input image.

8. The image processing device according to claim 3, further comprising:

a virtual object selecting unit for selecting an object to be virtually displayed in the output image,

wherein the CG image generating unit additionally arranges an object selected by the virtual object selecting unit in the CG image.

9. The image processing device according to claim 8, wherein the object to be virtually displayed, corresponds to a specified object after being moved out of the objects present in the input image and the erasing target object corresponds to the specified object before being moved.

10. An image processing method performed using an image processing device having a data storage unit storing feature data indicating a feature of appearance of an object, the method comprising steps of:

generating an environment map representing a position of one or more objects existing in a real space based on an input image obtained by imaging the real space using an imaging device and the feature data stored in the data storage unit; and

generating an output image obtained by erasing an erasing target object from the input image based on a position in the environment map of the erasing target object specified out of objects present in the input image and a position of the imaging device.

11. A program for causing a computer for controlling an image processing device to function as:

a data storage unit storing feature data indicating a feature of appearance of an object;

an environment map generating unit for generating an environment map representing a position of one or more objects existing in a real space based on an input image obtained by imaging the real space using an imaging device and the feature data stored in the data storage unit; and

an output image generating unit for generating an output image obtained by erasing an erasing target object from the input image based on a position of the erasing target object specified out of objects present in the input image represented in the environment map and a position of the imaging device.