Virtual clothing modeling apparatus and method

• US 20060202986A1
• Filed: 03/10/2006
• Published: 09/14/2006
• Est. Priority Date: 03/11/2005
• Status: Active Grant

• Current Assignee: Kabushiki Kaisha Toshiba (Toshiba Corporation), Toshiba Digital Solutions Corporation (Toshiba Corporation)
• Original Assignee: Kabushiki Kaisha Toshiba (Toshiba Corporation)
• Inventors: Okada, Ryuzo, Kondo, Nobuhiro
• Granted Patent: US 7,308,332 B2
• Time in Patent Office: Days
• US Class Current: 345/419
• CPC Class Codes: A47F 2007/195 Virtual display of clothes on the wearer by means of a mirror, screen or the like
• CPC Class Codes: G06F 2113/12 Cloth
• CPC Class Codes: G06F 30/20 Design optimisation, verification or simulation optimisation, verification or simulation of circuit designs G06F30/30
• CPC Class Codes: G06T 13/20 3D [Three Dimensional] animation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2210/16 Cloth

Abstract

Trying on the clothes is simulated by processing an image sequence by using a posture dictionary, and a three-dimensional figure, obtained in advance, of a person who wishes to try on clothing. The process includes estimating the posture of the person, and simulating the motions of the clothes, in agreement with estimated time series posture data. A virtual modeling apparatus comprises an imaging device configured to take an image sequence, a posture-estimator which estimates the posture of a person in the image sequence from the three-dimensional figure data of the person and from data for various postures, a posture corrector which generates smooth and natural posture time series data from the time series of estimated postures, a clothes simulator which calculates the motions of the clothing being modeled, and a display device which displays images of the person as if they were actually wearing the clothing.

Claims

1. A virtual clothing modeling apparatus comprising:

a camera to obtain an image sequence including a plurality of images of a person who tries on clothing;

a storage unit to store clothes data including three-dimensional figures of clothes and patterns thereof, three-dimensional figure data of the person, and posture data of the person;

a posture-estimating unit to estimate the posture of the person in the images within the image sequence by using the image sequence and the stored posture data;

a time series posture data generating unit to generate time series posture data by using the estimated posture data of the person;

a clothes-simulating unit to estimate the motions of the clothes tried on by the person by using the three-dimensional figure data of the person, the clothes data, and the time series posture data;

a clothes image-generating unit to generate images of the clothes based on the estimated motions of the clothes; and

a synthesizing unit to synthesize images of the clothes on the person in the image sequence by using the estimated motions of the clothes and the clothes data.

2. The virtual clothing modeling apparatus according to claim 1, wherein the posture-estimating unit includes:

a candidate mode selecting unit to select posture data for a plurality of posture candidates which are similar to the previously estimated posture data of the person;

a candidate model generating unit to generate a plurality of candidate models by modifying the three-dimensional figure data using a plurality of posture data candidates; and

a present posture-estimating unit to estimate the present posture of the person based on evaluation values which indicate similarities between the person in the images and the candidate models;

wherein the previously estimated posture data is selected from posture data representing a plurality of postures; and

wherein the posture data candidates selected by the candidate mode selecting unit are candidates whose similarities exceed a threshold.

3. The virtual clothing modeling apparatus according to claim 2, wherein the present posture-estimating unit in the posture-estimating unit includes:

a silhouette extracting unit to extract a silhouette of the person, the silhouette representing a region occupied by the person in the images;

a silhouette calculation unit to calculate candidate model silhouettes which represent the regions occupied by the candidate models on an image plane, the candidate model silhouettes being obtained by projecting the candidate models onto the image plane;

an evaluation value calculation unit to calculate silhouette evaluation values relating to the silhouettes of the candidate models; and

a model selection unit to select a model from the candidate models, the selected model having the silhouette evaluation value satisfying a predetermined condition;

wherein the evaluation values are based upon areas where inner regions of the candidate model silhouettes and inner regions of the silhouette of the person overlap and where outer regions of the candidate model silhouettes and the inner region of the silhouette of the person overlap; and

wherein the present posture-estimating unit estimates the present posture data of the person by using the posture data candidate of the selected model.

4. The virtual clothing modeling apparatus according to claim 3, wherein:

the storage unit further stores preset time series posture data including time series posture data for the person, and

the time series posture data generating unit includes;

a speed-setting unit for setting a range of speed for motions that can be expressed based on the preset time series posture data;

a data set selecting unit to select a set of posture data for the present frame in the image sequence from the preset time series posture data, based upon the posture data of the person in the image one frame before the present frame of the image sequence, and based upon the speed range; and

a correction data selecting unit to select, from the set of posture data, correction data which is posture data having similarities to the posture data of the person;

wherein the correction data includes those postures determined by the posture-estimating unit to have similarities to the posture data of the person that are higher than a threshold; and

wherein the time series posture data-generating unit generates the time series posture data by correcting the posture data of the person with the correction data.

5. The virtual clothing modeling apparatus according to claim 4, further comprising:

a texture-obtaining unit to obtain texture data of the person from the image of the present frame in the image sequence;

a data input unit to input modification data for modifying the three-dimensional figure data of the person;

a figure-modifying unit to modify the three-dimensional figure data of the person based on the modification data; and

a modified figure image generating unit to generate a modified figure image, which is an image of the person after the figure is modified by using the modified three-dimensional figure data of the person and the texture data of the person;

wherein the storage unit further stores a background image obtained in advance;

wherein the clothes-simulating unit estimates the motions of the clothes by using the modified three-dimensional figure data of the person; and

wherein the synthesizing unit synthesizes the modified figure image and the images of the clothes on the background image.

6. The virtual clothing modeling apparatus according to claim 3, wherein the present posture-estimating unit in the posture-estimating unit further includes:

a contour extracting unit to extract one or more contours of the person in the images; and

a contour calculation unit to calculate the contours of the candidate models by projecting the candidate models onto the image plane;

wherein the evaluation value calculation unit in the present posture-estimating unit further calculates contour evaluation values, based on the distances from points on the contours of the candidate models to the closest points on the contour of the person; and

wherein the model selection unit selects, from the candidate models, a model with a silhouette evaluation value and a contour evaluation value satisfying predetermined conditions.

7. The virtual clothing modeling apparatus according to claim 6, wherein the storage unit further stores preset time series posture data, including time series posture data for the person trying on clothing, and the time series posture data generating unit includes:

a speed-setting unit to set a range of speed for motions that can be expressed based on the preset time series posture data;

a data set selecting unit to select a set of posture data for the present frame in the image sequence from the preset time series posture data, based upon the posture data of the person in the image one frame before the present frame of the image sequence, and based upon the speed range; and

a correction data selecting unit to select, from the set of posture data, correction data which is posture data having similarities to the posture data of the person;

wherein the correction data consists of those postures determined by the posture-estimating unit to have similarities to the posture data for the person that are higher than a threshold; and

wherein the time series posture data-generating unit generates the time series posture data by correcting the posture data of the person with the correction data.

8. The virtual clothing modeling apparatus according to claim 6, further comprising:

a texture-obtaining unit to obtain texture data of the person from the image of the present frame in the image sequence;

a data input unit to input modification data for modifying the three-dimensional figure data of the person;

a figure-modifying unit to modify the three-dimensional figure data of the person based on the modification data; and

a modified figure image-generating unit to generate a modified figure image, which is an image of the person after the figure is modified by using the modified three-dimensional figure data of the person and the texture data of the person;

wherein the storage unit further stores a background image obtained in advance;

wherein the clothes-simulating unit estimates the motions of the clothes by using the modified three-dimensional figure data of the person; and

wherein the synthesizing unit synthesizes the modified figure image and the images of the clothes on the background image.

9. The virtual clothing modeling apparatus according to claim 2, wherein the present posture-estimating unit in the posture-estimating unit includes:

a contour extracting unit to extract one or more contours of the person in the images in the image sequence;

a contour calculation unit to calculate contours of the candidate models by projecting the candidate models onto the image plane;

an evaluation value calculation unit to calculate contour evaluation values for the candidate models, based on the distances from points on the contours of the candidate models to the closest points on the contour of the person; and

a model selection unit to select, from the candidate models, a model with a contour evaluation value satisfying predetermined conditions;

wherein the present posture-estimating unit estimates the present posture data of the person by using the posture data candidate of the selected model.

10. The virtual clothing modeling apparatus according to claim 1, wherein the storage unit further stores preset time series posture data including time series posture data for the person, and the time series posture data generating unit includes:

a speed-setting unit to set a range of speed for motions that can be expressed based on the preset time series posture data;

a data set selecting unit to select a set of posture data for the present frame in the image sequence from the preset time series posture data, based upon the posture data of the person in the image one frame before the present frame of the image sequence, and based upon the speed range; and

a correction data selecting unit to select, from the set of posture data, correction data which is posture data having similarities to the posture data of the person;

wherein the correction data includes those postures determined by the posture-estimating unit to have similarities the posture data of the person that are higher than a threshold; and

wherein the time series posture data-generating unit generates the time series posture data by correcting the posture data of the person with the correction data.

11. The virtual clothing modeling apparatus according to claim 1, further comprising:

a texture-obtaining unit to obtain texture data of the person from the image of the present frame in the image sequence;

a data input unit to input modification data for modifying the three-dimensional figure data of the person;

a figure-modifying unit to modify the three-dimensional figure data of the person based on the modification data; and

a modified figure image-generating unit to generate a modified figure image, which is an image of the person after the figure is modified by using the modified three-dimensional figure data of the person and the texture data of the person;

wherein the storage unit further stores a background image obtained in advance;

wherein the clothes-simulating unit estimates the motions of the clothes by using the modified three-dimensional figure data of the person; and

wherein the synthesizing unit synthesizes the modified figure image and the images of the clothes on the background image.

12. A computer readable medium storing a virtual clothing modeling computer program for causing a computer to execute instructions to perform steps of:

inputting the image sequence including a plurality of images of a person who tries on clothing;

making a reference to clothes data including three-dimensional figures of clothes and patterns thereof, three-dimensional figure data of the person, and posture data of the person stored in a storage unit;

estimating the posture data of the person in the images within the image sequence by using the image sequence and the posture data of the person;

generating time series posture data by using the estimated posture data of the person;

estimating the motions of the clothes tried on by the person by using the three-dimensional figure data of the person, the clothes data, and the generated time series posture data;

generating images of the clothes based on the estimated motions of the clothes; and

synthesizing images of the clothes on the person in the image sequence by using the estimated motions of the clothes and the clothes data stored in the storage unit.

13. The computer readable medium according to claim 12, wherein the step of estimating the posture includes the steps of:

selecting a plurality of posture data candidates which are similar to the previously estimated posture data of the person;

generating a plurality of candidate models by modifying the three-dimensional figure data using the data of the posture data candidates; and

estimating the present posture of the person based on evaluation values which indicate similarities between the person in the images in the image sequence and the candidate models generated by the candidate model-generating unit;

wherein the previously estimated posture data is selected from the posture data representing a plurality of postures; and

wherein the posture data candidates selected by the candidate mode selecting unit are candidates whose similarities exceed a threshold.

14. The computer readable medium according to claim 13, wherein the step of estimating the present posture in the step of estimating the posture includes the steps of:

extracting a silhouette of the person, where the silhouette represents a region occupied by the person in the images;

calculating candidate model silhouettes which represent the regions occupied by the candidate models on an image plane, the candidate model silhouettes being obtained by projecting the candidate models onto the image plane of the image sequence;

calculating silhouette evaluation values relating to the silhouettes of the candidate models; and

selecting, from the candidate models, a model having the silhouette evaluation value satisfying a predetermined condition;

wherein the evaluation values are based upon areas where inner regions of the candidate model silhouettes and inner regions of the silhouette of the person overlap and where outer regions of the candidate model silhouettes and the inner region of the silhouette of the person overlap; and

wherein the step of estimating the present posture estimates the present posture data of the person by using the posture data candidate of the selected model.

15. The computer readable medium according to claim 14, wherein:

the step of making a reference involves referring to the preset time series posture data, including time series posture data for the person that has been stored in advance in the storage unit, and

the step of generating the time series posture data includes the steps of;

setting a range of speed for motions that can be expressed based on the preset time series posture data, including time series posture data for the person trying on clothing, that is stored in the storage unit;

selecting a set of posture data for the present frame in the image sequence from the preset time series posture data, based upon the posture data of the person in the image one frame before the present frame of the image sequence, and based upon the speed range; and

selecting, from the set of posture data, correction data which is posture data having similarities to the posture data of the person;

wherein the correction data includes those postures determined to have similarities to the posture data of the person that are higher than a threshold;

and wherein the step of generating the time series posture data forms the time series posture data by correcting the posture data of the person with the correction data.

16. The computer readable medium according to claim 15, for causing a computer to execute instructions to perform the further steps of:

obtaining texture data of the person from the image of the present frame in the image sequence;

inputting modification data for modifying the three-dimensional figure data of the person;

modifying the three-dimensional figure data of the person based on the modification data; and

generating a modified figure image, which is an image of the person after the figure is modified, by using the modified three-dimensional figure data of the person and the texture data of the person;

wherein the step of making a reference further involves referring to a background image obtained in advance and stored in the storage unit;

the step of estimating the motion estimates the motions of the clothes by using the modified three-dimensional figure data of the person; and

the step of synthesis synthesizes the modified figure image and the images of the clothes on the background image.

17. The computer readable medium according to claim 14, wherein the step of estimating the present posture in the step of estimating the posture further includes the steps of:

extracting one or more contours of the person in the images; and

calculating the contours of the candidate models by projecting the candidate models onto the image plane of the image sequence;

wherein the step of calculating silhouette evaluation values within the step of estimating the posture further includes calculating contour evaluation values, based on the distances from points on the contours of the candidate models to the closest points on the contour of the person; and

the step of selecting a model involves selecting, from the candidate models, the model with a silhouette evaluation value and a contour evaluation value satisfying predetermined conditions.

18. The computer readable medium according to claim 17, wherein:

the step of making a reference further involves referring to the preset time series posture data stored in the storage unit, including time series posture data, for the person trying on clothing, and

the step of generating the time series posture data includes the steps of;

setting a range of speed for motions that can be expressed based on the preset time series posture data;

selecting a set of posture data for the present frame in the image sequence from the preset time series posture data, based upon the posture data of the person in the image one frame before the present frame of the image sequence, and based upon the speed range; and

selecting, from the set of posture data, correction data which is posture data having similarities to the posture data of the person;

wherein the correction data includes those postures determined by the posture-estimating unit to have similarities to the posture data for the person that are higher than a threshold; and

wherein the step of generating the time series posture data involves generating the time series posture data by correcting the posture data of the person with the correction data.

19. The computer readable medium according to claim 17, for causing a computer to execute instructions to perform the further steps of:

obtaining texture data of the person from the image of the present frame in the image sequence;

inputting modification data for modifying the three-dimensional figure data of the person;

modifying the three-dimensional figure data of the person based on the modification data; and

generating a modified figure image, which is an image of the person after the figure is modified by using the modified three-dimensional figure data of the person and the texture data of the person;

wherein the step of making a reference further involves making a reference to a background image obtained in advance and stored in the storage unit;

wherein the step of estimating the motion of the clothes does so by using the modified three-dimensional figure data of the person ; and

the step of synthesis includes synthesizing the modified figure image and the images of the clothes on the background image.

20. The computer readable medium according to claim 13, wherein the step of estimating the present posture in the step of estimating the posture includes the steps of:

extracting one or more contours of the person in the images in the image sequence;

calculating the contours of the candidate models by projecting the candidate models onto the image plane of the image sequence;

calculating contour evaluation values for the candidate models based on the distances from points on the contours of the candidate models to the closest points on the contour of the person; and

selecting, from the candidate models, a model with a contour evaluation value satisfying predetermined conditions;

wherein the step of estimating the present posture includes estimating the present posture data of the person by using the posture data candidate of the selected model.