IMAGE PROCESSING DEVICE AND IMAGE PROCESSING METHOD

• US 20170140574A1
• Filed: 01/31/2017
• Published: 05/18/2017
• Est. Priority Date: 03/20/2015
• Status: Active Grant

• Current Assignee: Kabushiki Kaisha Toshiba (Toshiba Corporation)
• Original Assignee: Kabushiki Kaisha Toshiba (Toshiba Corporation)
• Inventors: SEKINE, Masahiro, SUGITA, Kaoru, NISHIYAMA, Masashi, OHIRA, Hidetaka
• Granted Patent: US 10,366,533 B2
• Time in Patent Office: Days
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: G06T 2210/16 Cloth
• CPC Class Codes: G06T 2219/2021 Shape modification

Abstract

According to one embodiment, an image processing device includes at least one processor. The at least one processor is configured to acquire a first three-dimensional model regarding a subject, set a plurality of first control points on the first three-dimensional model, acquire mesh data of a meshed image of a region of clothing extracted from a captured image, acquire a second three-dimensional model, modify the mesh data based on an amount of movement from each of the plurality of first control points, to each respective one of a plurality of second control points, and generate an image of the clothing using the captured image and the modified mesh data.

Claims

1. An image processing device comprising at least one processor configured to:

acquire a first three-dimensional model regarding a subject;

set a plurality of first control points on the first three-dimensional model;

acquire mesh data of a meshed image of a region of clothing extracted from a captured image containing the subject wearing the clothing;

acquire a second three-dimensional model by modifying the first three-dimensional model;

modify the mesh data based on an amount of movement from each of the plurality of first control points set on the first three-dimensional model, to each respective one of a plurality of second control points on the second three-dimensional model, respectively corresponding to the plurality of first control points; and

generate an image of the clothing using the captured image and the modified mesh data.

2. The image processing device of claim 1, wherein

the acquired mesh data comprises a plurality of vertexes, and

the at least one processor is configured to:

calculate an influence degree indicating a degree of an influence on each of the plurality of vertexes of the acquired mesh data due to a moving of each of the plurality of first control points set on the first three-dimensional model when the first three-dimensional model is modified, for each combination of the first control point and the vertex, and

modify the mesh data using the calculated influence degree.

3. The image processing device of claim 2, wherein

the at least one processor is configured to calculate, as the influence degree for each combination of the first control point and the vertex, a distance between the first control point and the vertex in a total region of the region of the first three-dimensional model and the region of the clothing represented by the mesh data in a case where the generated mesh data is superimposed on the first three-dimensional model to represent a state where the subject wears the clothing.

4. The image processing device of claim 3, wherein

the plurality of first control points set on the first three-dimensional model comprises a physical feature point in the first three-dimensional model.

5. The image processing device of claim 3, wherein

the plurality of first control points are positioned on an outer periphery of the first three-dimensional model in a case where the first three-dimensional model is projected.

6. The image processing device of claim 3, wherein

the at least one processor is configured to:

acquire boundary information indicating a boundary of the region of the clothing,

calculate a distance between the first control point and the vertex based on the boundary indicated by the boundary information.

7. The image processing device of claim 1, wherein

the at least one processor is configured to acquire the first three-dimensional model regarding the subject captured by an image capture unit,

each of the plurality of first control points set on the first three-dimensional model contains depth information indicating a position of the first control point in a depth direction with respect to the image capture unit, and

the at least one processor is configured to:

calculate the depth information corresponding to each of the plurality of vertexes of the acquired mesh data based on the depth information contained in each of the plurality of first control points set on the first three-dimensional model, and

perform a hidden surface elimination process based on the depth information corresponding to each of the plurality of vertexes of the modified mesh data when the image of the clothing is generated.

8. An image processing device comprising at least one processor configured to:

acquire a first three-dimensional model regarding a subject in a first state;

set a plurality of first control points on the first three-dimensional model;

acquire mesh data of a meshed image of a region of clothing extracted from a first captured image containing the subject in the first state, wearing the clothing;

acquire a second three-dimensional model of the object in a second state by modifying the first three-dimensional model;

modify the mesh data based on an amount of movement from each of the plurality of first control points set on the first three-dimensional model, to each respective one of a plurality of second control points on the second three-dimensional model, respectively corresponding to the plurality of first control points; and

generate an image of the clothing using a second captured image including the subject in the second state, wearing the clothing and the modified mesh data.

9. The image processing device of claim 8, wherein

the at least one processor is configured to:

generate a first image using the first captured image and the modified mesh data;

generate a second image using the second captured image and the modified mesh data; and

generate an image of the clothing based on the first image and the second image.

10. An image processing method to be executed by an image processing device, comprising:

acquiring a first three-dimensional model regarding a subject;

setting a plurality of first control points on the acquired first three-dimensional model;

acquiring mesh data of a meshed image of a region of clothing extracted from a captured image containing the subject wearing the clothing;

acquiring a second three-dimensional model by modifying the first three-dimensional model;

modifying the mesh data based on an amount of movement from each of the plurality of first control points set on the first three-dimensional model, to each respective one of a plurality of second control points on the second three-dimensional model, respectively corresponding to the plurality of first control points; and

generating an image of the clothing using the captured image and the modified mesh data.

11. The image processing method of claim 10, further comprising:

calculating an influence degree,

wherein

the acquired mesh data comprises a plurality of vertexes, and

the calculating the influence degree comprises calculating an influence degree indicating a degree of an influence on each of the plurality of vertexes of the acquired mesh data due to a moving of each of the plurality of first control points set on the first three-dimensional model when the first three-dimensional model is modified, for each combination of the first control point and the vertex, and

the modifying comprises modifying the mesh data using the calculated influence degree.

12. The image processing method of claim 11, wherein

the calculating comprises calculating, as the influence degree for each combination of the first control point and the vertex, a distance between the first control point and the vertex in a total region of the region of the first three-dimensional model and the region of the clothing represented by the mesh data in a case where the generated mesh data is superimposed on the first three-dimensional model to represent a state where the subject wears the clothing.

13. The image processing method of claim 12, wherein

the plurality of first control points set on the first three-dimensional model comprise a physical feature point in the first three-dimensional model.

14. The image processing method of claim 12, wherein

the plurality of first control points are positioned on an outer periphery of the first three-dimensional model in a case where the first three-dimensional model is projected.

15. The image processing device of claim 12, further comprising:

acquiring boundary information indicating a boundary of the region of the clothing,

wherein

the calculating comprises calculating a distance between the first control point and the vertex based on the boundary indicated by the boundary information.

16. The image processing method of claim 10, further comprising:

calculating a depth information,

wherein

the acquiring the first three-dimensional model comprises acquiring the first three-dimensional model regarding the subject captured by an image capture unit,

each of the plurality of first control points set on the first three-dimensional model comprises depth information indicating a position of the first control point in a depth direction with respect to the image capture unit, and

the calculating the depth information comprises calculating the depth information corresponding to each of the plurality of vertexes of the acquired mesh data based on the depth information contained in each of the plurality of first control points set on the first three-dimensional model, and

the generating comprises performing a hidden surface elimination process based on the depth information corresponding to each of the plurality of vertexes of the modified mesh data when the image of the clothing is generated.