DEPTH CAMERA-BASED HUMAN-BODY MODEL ACQUISITION METHOD AND NETWORK VIRTUAL FITTING SYSTEM

• US 20170140578A1
• Filed: 05/21/2015
• Published: 05/18/2017
• Est. Priority Date: 06/12/2014
• Status: Active Grant

• Current Assignee: Shenzhen Orbbec Co., Ltd.
• Original Assignee: Shenzhen Orbbec Co., Ltd.
• Inventors: XIAO, ZHENZHONG, XU, HONGHUAI, LIU, LONG, HUANG, YUANHAO
• Granted Patent: US 10,217,293 B2
• Time in Patent Office: Days
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: G06T 2200/32 involving image mosaicing
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2210/16 Cloth
• CPC Class Codes: G06T 2219/2012 Colour editing, changing, or manipulating; Use of colour codes
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: H04N 13/257 Colour aspects
• CPC Class Codes: H04N 13/282 for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems

Abstract

The present invention relates to a human body model acquisition method and a virtual network fitting system based on depth cameras. The method includes the following steps: step S1: generating mark points covering a model body surface and used for determining features of the model body surface on the model body surface, meanwhile collecting depth images of the model body from a plurality of angles by using the depth cameras to acquire a depth image sequence that covers the model body surface and contains the mark points; step S2: carrying target depth information point cloud grid reconstruction on frames of depth images in the depth image sequence; and step S3: mosaicing the reconstructed frames of depth images into a three-dimensional model of the model body according to the mark points in the reconstructed frames of depth images. Compared with the prior art, by adopting the human body model acquisition method and the virtual network fitting system based on the depth cameras provided by the present invention, the accurate three-dimensional human body model and the accurate three-dimensional garment model can be acquired, and virtual fitting is carried out accordingly to acquire a real fitting effect.

Claims

1. A human body model acquisition method based on depth cameras, comprising the following steps:

step S1;

generating mark points covering a model body surface and for determining features of the model body surface on the model body surface, and collecting depth images of the model body from a plurality of angles by using the depth cameras to acquire a depth image sequence that covers the model body surface and contains the mark points;

step S2;

carrying target depth information point cloud grid reconstruction on frames of depth images in the depth image sequence; and

step S3;

mosaicing the reconstructed frames of depth images into a three-dimensional model of the model body according to the mark points in the reconstructed frames of depth images;

wherein prior to step S2, the method further comprises the following steps;

preprocessing the frames of depth images in the depth image sequence acquired in step S1 according to the following steps;

estimating the position of the model body in the depth image through segmenting a foreground by a median;

placing a model body outline generating seed at the center of the position;

diffusing the model body outline by depth search based on smoothness constraint, generating an accurate model body outline, extracting the model body outline, and meanwhile removing the rest parts in the depth image as backgrounds;

wherein when the model body outline of the next frame of depth image is extracted, extracting the model body outline in combination with the outline extraction result of the previous frame;

wherein when the depth image is a rigid-body depth image;

the step S1 is specifically as follows;

irradiating the model body from 360 degrees by using structured light generated by the depth cameras, so as to generate the mark points for determining the feature information of irradiation sites on the irradiation sites; and

collecting the depth images of the irradiation sites by using the depth cameras during the irradiation to acquire the depth image sequence that covers the model body surface and contains the mark points;

the step S3 is specifically as follows;

mosaicing the mark points with correlation higher than a set value according to the correlation of the mark points in every two adjacent frames of depth images in the reconstructed frames of depth images, and deleting the mark points with correlation lower than the set value, so as to mosaic the frames of depth images into the three-dimensional model of the model body;

when the depth image is a non-rigid-body depth image;

the step S1 is specifically as follows;

irradiating the model body from 360 degrees by using light spots of random lattices generated by the depth cameras, so as to generate the mark points for determining the feature information of irradiation sites on the irradiation sites; and

collecting the depth images of the irradiation sites by using the depth cameras during the irradiation to acquire the depth image sequence that covers the model body from 360 degrees and contains the mark points; and

the step S3 is specifically as follows;

mosaicing the mark points with correlation higher than the set value according to the correlation of the mark points in every two adjacent frames of depth images in the reconstructed frames of depth images, and deleting the mark points with correlation lower than the set value, so as to mosaic the frames of depth images into the three-dimensional model of the model body.

2. The human body model acquisition method based on the depth cameras of claim 1, wherein during the mosaicing, the three-dimensional model is generated according to weights of the rigid-body depth images and the non-rigid-body depth images in the depth image sequence.

3. The human body model acquisition method based on the depth cameras of claim 1, wherein when step S1 is implemented:

the model body is photographed by a plurality of depth cameras arranged at different angle positions according to a set photographing frequency, and the model body rotates automatically according to a set angular speed during photography until the plurality of depth cameras arranged at different angle positions acquire the depth image sequence that covers the model body from 360 degrees together.

4. The human body model acquisition method based on the depth cameras of claim 3, wherein three depth cameras are arranged, and the arrangement positions of the three depth cameras are as follows:

two depth cameras are arranged to scan an upper part and a lower part of the model body respectively, and the upper part and the lower part are not overlapped;

the other depth camera is arranged to scan a middle part of the model body, and the middle part is partially overlapped with the upper part and the lower part; and

the three depth cameras are arranged in such a manner that any one of the depth cameras cannot photograph the other two depth cameras when photographing the model body.

5. The human body model acquisition method based on the depth cameras of claim 1, further comprising the following steps:

acquiring a color image sequence that covers the model body from 360 degrees by color cameras; and

matching the color image sequence with the depth image sequence, and fusing the color image sequence into the depth image sequence.

6. A human body model acquisition method based on depth cameras, comprising the following steps:

step S1;

generating mark points covering a model body surface and used for determining features of the model body surface on the model body surface, meanwhile collecting depth images of the model body from a plurality of angles by using the depth cameras to acquire a depth image sequence that covers the model body surface and contains the mark points;

step S2;

carrying target depth information point cloud grid reconstruction on frames of depth images in the depth image sequence; and

step S3;

mosaicing the reconstructed frames of depth images into a three-dimensional model of the model body according to the mark points in the reconstructed frames of depth images.

7. The human body model acquisition method based on the depth cameras of claim 6, wherein:

when the depth image is a rigid-body depth image;

the step S1 is specifically as follows;

irradiating the model body from 360 degrees by using structured light generated by the depth cameras, so as to generate the mark points for determining the feature information of irradiation sites on the irradiation sites; and

collecting the depth images of the irradiation sites by using the depth cameras during the irradiation to acquire the depth image sequence that covers the model body surface and contains the mark points;

the step S3 is specifically as follows;

mosaicing the mark points with correlation higher than a set value according to the correlation of the mark points in every two adjacent frames of depth images in the reconstructed frames of depth images, and deleting the mark points with correlation lower than the set value, so as to mosaic the frames of depth images into the three-dimensional model of the model body;

when the depth image is a non-rigid-body depth image;

the step S1 is specifically as follows;

irradiating the model body from 360 degrees by using light spots of random lattices generated by the depth cameras, so as to generate the mark points for determining the feature information of irradiation sites on the irradiation sites; and

collecting the depth images of the irradiation sites by using the depth cameras during the irradiation to acquire the depth image sequence that covers the model body from 360 degrees and contains the mark points; and

the step S3 is specifically as follows;

mosaicing the mark points with correlation higher than the set value according to the correlation of the mark points in every two adjacent frames of depth images in the reconstructed frames of depth images, and deleting the mark points with correlation lower than the set value, so as to mosaic the frames of depth images into the three-dimensional model of the model body.

8. The human body model acquisition method based on the depth cameras of claim 7, wherein during the mosaicing, the three-dimensional model is generated according to weights of the rigid-body depth images and the non-rigid-body depth images in the depth image sequence.

9. The human body model acquisition method based on the depth cameras of claim 6, wherein prior to step S2, the method further comprises the following steps:

preprocessing the frames of depth images in the depth image sequence acquired in step S1 according to the following steps;

estimating the position of the model body in the depth image through segmenting a foreground by a median;

placing a model body outline generating seed at the center of the position;

diffusing the model body outline by depth search based on smoothness constraint, generating an accurate model body outline, extracting the model body outline, and meanwhile removing the rest parts in the depth image as backgrounds; and

when the model body outline of the next frame of depth image is extracted, extracting the model body outline in combination with the outline extraction result of the previous frame.

10. The human body model acquisition method based on the depth cameras of claim 6, wherein when step S1 is implemented:

the model body is photographed by a plurality of depth cameras arranged at different angle positions according to a set photographing frequency, and the model body autorotates according to a set angular speed during photography until the plurality of depth cameras arranged at different angle positions acquire the depth image sequence that covers the model body from 360 degrees together.

11. The human body model acquisition method based on the depth cameras of claim 10, wherein three depth cameras are arranged, and the arrangement positions of the three depth cameras are as follows:

two depth cameras are arranged to scan an upper part and a lower part of the model body respectively, and the upper part and the lower part are not overlapped;

the other depth camera is arranged to scan a middle part of the model body, and the middle part is partially overlapped with the upper part and the lower part; and

the three depth cameras are arranged in such a manner that any one of the depth cameras cannot photograph the other two depth cameras when photographing the model body.

12. The human body model acquisition method based on the depth cameras of claim 6, further comprising the following steps:

acquiring a color image sequence that covers the model body from 360 degrees by color cameras; and

matching the color image sequence with the depth image sequence, and fusing the color image sequence into the depth image sequence.

13. The human body model acquisition method based on the depth cameras of claim 6, further comprising the following steps:

acquiring a plurality of three-dimensional models of the model body, and establishing a database of the three-dimensional models of the model body;

acquiring three-dimensional models of model fitting effects when a model tries garments with different sizes and at different postures, and establishing virtual fitting training samples accordingly; and

carrying out fitting training according to the virtual fitting training samples, and establishing a database of the three-dimensional models of the virtual model fitting effects.

14. The human body model acquisition method based on the depth cameras of claim 6, further comprising the following steps:

acquiring a three-dimensional garment model, and matching the three-dimensional garment model with the three-dimensional model of the model body to generate a three-dimensional model of virtual fitting effects.

15. A virtual network fitting system based on depth cameras, comprising

a depth image collection module, used for collecting a depth image sequence that covers a model body from 360 degrees;

a point cloud grid reconstruction module, used for carrying target depth information point cloud grid reconstruction on each frame of depth image in the depth image sequence;

a three-dimensional human body model establishment module, used for mosaicing the reconstructed frames of depth images into a three-dimensional model of the model body;

a three-dimensional garment model collection module, used for acquiring a three-dimensional garment model;

a fitting module, used for matching the three-dimensional garment model with the three-dimensional model of the model body to generate a three-dimensional model of fitting effects; and

a network interface module, used for outputting the three-dimensional model of fitting effects through a network.

16. The virtual network fitting system based on the depth cameras of claim 15, wherein the fitting module is used for generating the three-dimensional garment model according to the specification and the size of the garment.

17. The virtual network fitting system based on the depth cameras of claim 15, wherein the fitting module is used for generating the three-dimensional garment model according to a garment depth image.

18. The virtual network fitting system based on the depth cameras of claim 15, wherein the three-dimensional model of the model body comprises three-dimensional models of the model body of a plurality of postures, and the system further comprises:

a database of the three-dimensional models of the model body, used for storing the three-dimensional models of the model body of different postures.

19. The virtual network fitting system based on the depth cameras of claim 15, further comprising a database of three-dimensional models of virtual fitting effects, wherein the database of three-dimensional models of virtual fitting effects is generated according to a plurality of fitting training samples;

and the fitting training samples are generated according to a plurality of three-dimensional models of fitting effects.

20. The virtual network fitting system based on the depth cameras of claim 15, further comprising:

a color image collection module, used for collecting color images of the model body and the garment and respectively matching and fusing the color images into the three-dimensional model of the model body and the three-dimensional garment model.