Image processing device and image processing method

• US 10,366,533 B2
• Filed: 01/31/2017
• Issued: 07/30/2019
• Est. Priority Date: 03/20/2015
• Status: Active Grant

• Current Assignee: Kabushiki Kaisha Toshiba (Toshiba Corporation)
• Original Assignee: Kabushiki Kaisha Toshiba (Toshiba Corporation)
• Inventors: Sugita, Kaoru, Nishiyama, Masashi, Sekine, Masahiro, Ohira, Hidetaka
• Primary Examiner(s): Yang, Andrew G
• Application Number:
  US15/420,545
• Publication Number:
  US 20170140574A1
• Time in Patent Office: 910 Days
• Field of Search: None
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: G06T 2210/16 Cloth
• CPC Class Codes: G06T 2219/2021 Shape modification

Abstract

According to one embodiment, an image processing device includes at least one processor. The at least one processor is configured to acquire a first three-dimensional model regarding a subject, set a plurality of first control points on the first three-dimensional model, acquire mesh data of a meshed image of a region of clothing extracted from a captured image, acquire a second three-dimensional model, modify the mesh data based on an amount of movement from each of the plurality of first control points, to each respective one of a plurality of second control points, and generate an image of the clothing using the captured image and the modified mesh data.

Claims

1. An image processing device comprising at least one processor configured to:

acquire a first three-dimensional model regarding a subject;

set a plurality of first control points on the first three-dimensional model;

acquire mesh data of a mesh image of a region of clothing extracted from a captured image including the subject wearing the clothing, the acquired mesh data comprising a plurality of vertexes;

acquire a plurality of second three-dimensional models by modifying a direction, a posture, and/or a body type of the first three-dimensional model;

generate modified mesh data for each respective one of the plurality of second three-dimensional models based on an amount of movement from each of the plurality of first control points set on the first three-dimensional model, to each respective one of a plurality of second control points, respectively corresponding to the plurality of first control points, on the respective one of the plurality of second three-dimensional models; and

generate a plurality of images of the clothing using the captured image and the modified mesh data, wherein

the at least one processor is configured to;

calculate, as an influence degree for each of a plurality of respective first control point-vertex combinations, a distance between the first control point and the vertex of the respective combination in a total region of a region of the first three-dimensional model and a region of the clothing represented by the mesh data in a case in which the modified mesh data is superimposed on the first three-dimensional model to represent a state in which the subject wears the clothing; and

acquire boundary information indicating a boundary of the region of the clothing, the boundary being designated on the region of the clothing, and each distance is based on the shortest path which does not cross the boundary indicated by the boundary information.

2. The image processing device of claim 1, wherein

the mesh data is modified using the calculated influence degrees.

3. The image processing device of claim 1, wherein

the plurality of first control points set on the first three-dimensional model comprises a physical feature point in the first three-dimensional model.

4. The image processing device of claim 1, wherein

the plurality of first control points is positioned on an outer periphery of the first three-dimensional model in a case in which the first three-dimensional model is projected.

5. The image processing device of claim 1, wherein

the at least one processor is configured to acquire the first three-dimensional model regarding the subject captured by an image capture unit including a camera,

each of the plurality of first control points set on the first three-dimensional model contains depth information indicating a position of the first control point in a depth direction with respect to the image capture unit, and

the at least one processor is configured to:

calculate depth information corresponding to each of the plurality of vertexes of the acquired mesh data based on the depth information contained in each of the plurality of first control points set on the first three-dimensional model; and

perform a hidden surface elimination process based on depth information corresponding to each of the plurality of vertexes of the modified mesh data when the plurality of images of the clothing is generated.

6. An image processing method to be executed by an image processing device, comprising:

acquiring a first three-dimensional model regarding a subject;

setting a plurality of first control points on the acquired first three-dimensional model;

acquiring mesh data of a mesh image of a region of clothing extracted from a captured image including the subject wearing the clothing, the acquired mesh data comprising a plurality of vertexes;

acquiring a plurality of second three-dimensional models by modifying a direction, a posture, and/or a body type of the first three-dimensional model;

generating modified mesh data for each respective one of the plurality of second three-dimensional models based on an amount of movement from each of the plurality of first control points set on the first three-dimensional model, to each respective one of a plurality of second control points, respectively corresponding to the plurality of first control points, on the respective one of the plurality of second three-dimensional models; and

generating a plurality of images of the clothing using the captured image and the modified mesh data, wherein

the method further comprises;

calculating, as an influence degree for each of a plurality of respective first control point-vertex combinations, a distance between the first control point and the vertex of the respective combination in a total region of a region of the first three-dimensional model and a region of the clothing represented by the mesh data in a case in which the modified mesh data is superimposed on the first three-dimensional model to represent a state in which the subject wears the clothing; and

acquiring boundary information indicating a boundary of the region of the clothing, the boundary being designated on the region of the clothing, and

each distance is based on the shortest path which does not cross the boundary indicated by the boundary information.

7. The image processing method of claim 6, wherein

the mesh data is modified using the calculated influence degrees.

8. The image processing method of claim 6, wherein

the plurality of first control points set on the first three-dimensional model comprises a physical feature point in the first three-dimensional model.

9. The image processing method of claim 6, wherein

the plurality of first control points are positioned on an outer periphery of the first three-dimensional model in a case in which the first three-dimensional model is projected.

10. The image processing method of claim 6, further comprising:

calculating depth information,

wherein

the acquiring of the first three-dimensional model comprises acquiring the first three-dimensional model regarding the subject captured by an image capture unit including a camera,

each of the plurality of first control points set on the first three-dimensional model comprises depth information indicating a position of the first control point in a depth direction with respect to the image capture unit, and

the calculating of the depth information comprises calculating depth information corresponding to each of the plurality of vertexes of the acquired mesh data based on the depth information contained in each of the plurality of first control points set on the first three-dimensional model, and

the generating comprises performing a hidden surface elimination process based on depth information corresponding to each of the plurality of vertexes of the modified mesh data when the plurality of images of the clothing is generated.