BODY CONTOUR KEY POINT DETECTION METHODS, APPARATUSES, AND DEVICES

• US 20190318194A1
• Filed: 06/26/2019
• Published: 10/17/2019
• Est. Priority Date: 01/19/2018
• Status: Active Grant

• Current Assignee: Beijing SenseTime Technology Development Co., Ltd. (SenseTime group Limited)
• Original Assignee: Beijing SenseTime Technology Development Co., Ltd. (SenseTime group Limited)
• Inventors: LIU, Wentao, QIAN, Chen, XU, Qinqin
• Granted Patent: US 11,113,560 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/20116 Active contour; Active surface; Snakes
• CPC Class Codes: G06T 7/12 Edge-based segmentation
• CPC Class Codes: G06T 7/143 involving probabilistic approaches, e.g. Markov random field [MRF] modelling
• CPC Class Codes: G06V 10/44 Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
• CPC Class Codes: G06V 10/46 Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features colour feature extraction G06V10/56
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition

Abstract

Body contour key point detection methods, image processing methods, neural network training methods, apparatuses, electronic devices, computer-readable storage media, and computer programs include: obtaining an image feature of an image block including a body; obtaining a body contour key point prediction result of the body by means of a first neural network according to the image feature; and obtaining a body contour key point in the image block according to the body contour key point prediction result; where the body contour key point is used for representing an outer contour of the body.

Claims

1. A body contour key point detection method, performed by an electronic device, comprising:

obtaining an image feature of an image block comprising a body;

obtaining, by means of a first neural network, a body contour key point prediction result of the body according to the image feature; and

obtaining, according to the body contour key point prediction result, a body contour key point in the image block, wherein the body contour key point is used for representing an outer contour of the body.

2. The method according to claim 1, wherein the obtaining the image feature of the image block comprising the body comprises:

performing body detection on an image to be processed; and

obtaining the image feature of the image block comprising the body in the image to be processed according to a body detection result;

wherein the image block is the image to be processed, the image block is a partial image comprising a body in the image to be processed, or the image block is an image block obtained by processing the partial image comprising the body in the image to be processed.

3. The method according to claim 2, wherein the performing the body detection on the image to be processed comprises:

performing the body detection on the image to be processed using a body detector, to obtain a central position of a body bounding box and a body scale factor;

wherein the central position of the body bounding box and the body scale factor are used for determining a position of the body bounding box in the image to be processed.

4. The method according to claim 1, wherein the obtaining, by means of the first neural network, the body contour key point prediction result of the body according to the image feature comprises:

forming, by means of the first neural network, at least one body contour key point response diagram respectively corresponding to at least one body contour key point according to the image feature.

5. The method according to claim 4, wherein each of the at least one body contour key point response diagram comprises a body contour key point confidence diagram, wherein the obtaining the body contour key point in the image block according to the body contour key point prediction result comprises:

selecting a point meeting a predetermined confidence requirement from any one body contour key point confidence diagram comprised in the at least one body contour key point response diagram; and

using a mapping position point of the point in the image block as a body contour key point corresponding to the any one body contour key point confidence diagram.

6. The method according to claim 1, wherein the body contour key point comprises at least one of the following:

a head contour key point, an arm contour key point, a hand contour key point, a shoulder contour key point, a leg contour key point, a waist contour key point, or a foot contour key point;

wherein the head contour key point comprises at least one of;

a head top key point, a nose tip key point, or a chin key point;

or

the arm contour key point comprises at least one of;

a wrist contour key point, an elbow contour key point, an arm root contour key point, a lower arm contour midpoint key point located at a midpoint position between the wrist contour key point and the elbow contour key point, and an upper arm midpoint key point located at a midpoint position between the elbow contour key point or the arm root contour key point;

or

the hand contour key point comprises at least one of;

a hand tip key point and a palm midpoint key point;

or

the shoulder contour key point comprises at least one of;

a shoulder and head intersection key point located at an intersection position of the shoulder and the head, or a shoulder contour midpoint key point located at a midpoint position between the arm root contour key point and the shoulder and head intersection key point;

or

the leg contour key point comprises at least one of;

a crotch key point, a knee contour key point, an ankle contour key point, a thigh root outside contour key point, a shank contour midpoint key point located at a midpoint position between the knee contour key point and the ankle contour key point, a thigh inner contour midpoint key point located at a midpoint position between the knee inner contour key point and the crotch key point, or a thigh outer contour midpoint key point located at a midpoint position between the knee outer contour key point and the thigh root outside contour key point;

or

the waist contour key point comprises at least one of;

N−

1 equal division points generated by dividing the thigh root outside contour key point or the arm root contour key point into N equal parts, wherein N is greater than 1;

or

the foot contour key point comprises at least one of;

a tiptoe key point or a heel key point.

7. The method according to claim 1, wherein the first neural network is trained in advance by means of a training image set comprising body contour key point marking information, wherein the training of the first neural network comprises:

obtaining an image feature of a sample image block comprising a body;

obtaining, by means of a first neural network to be trained, a body contour key point prediction result of the body according to the image feature of the sample image block; and

using a difference between the body contour key point prediction result and the body contour key point marking information as guidance information to perform supervised learning on the first neural network to be trained.

8. The method according to claim 7, wherein the obtaining the image feature of the sample image block comprising the body comprises:

obtaining an image sample from a training data set; and

obtaining an image feature of the sample image block comprising the body in the image sample.

9. The method according to claim 8, wherein the obtaining, by means of the first neural network to be trained, the body contour key point prediction result of the body according to the image feature comprises:

forming, by means of the first neural network to be trained, at least one body contour key point response diagram respectively corresponding to at least one body contour key point according to the image feature.

10. The method according to claim 9, wherein each of the body contour key point response diagram comprises a body contour key point confidence diagram, wherein the using the difference between the body contour key point prediction result and the body contour key point marking information as guidance information to perform supervised learning on the first neural network to be trained comprises:

respectively generating a generated body contour key point confidence diagram for at least one piece of body contour key point marking information; and

using a difference between the body contour key point confidence diagram and the generated body contour key point confidence diagram as guidance information to perform supervised learning on the first neural network to be trained.

11. The method according to claim 10, wherein the respectively generating the body contour key point confidence diagram for at least one piece of body contour key point marking information comprises:

respectively forming a Gaussian response in a predetermined peripheral region of a position in the sample image block corresponding to the at least one piece of body contour key point marking information, and forming a body contour key point confidence diagram separately corresponding to the at least one piece of body contour key point marking information according to the Gaussian response.

12. The method according to claim 9, wherein each of the body contour key point response diagram comprises a formed body contour key point confidence diagram, wherein the using the difference between the body contour key point prediction result and the body contour key point marking information as guidance information to perform supervised learning on the first neural network to be trained comprises:

selecting a point meeting a predetermined confidence requirement from any one formed body contour key point confidence diagram comprised in the at least one body contour key point response diagram;

using a mapping position point of the point in the image sample as a body contour key point corresponding to the any one formed body contour key point confidence diagram; and

using a difference between the body contour key point corresponding to the any one formed body contour key point confidence diagram and the body contour key point marking information of the image sample as guidance information to perform supervised learning on the first neural network to be trained.

13. The method according to claim 8, wherein the obtaining the image feature of the sample image block comprising the body comprises:

performing body detection on the image sample;

providing a body detection result and the image sample to an input neural network to be trained, to obtain, by means of the input neural network to be trained, the sample image block comprising the body, wherein the sample image block has a predetermined size; and

obtaining the image feature of the sample image block using a second neural network to be trained.

14. The method according to claim 13, wherein the training of the first neural network further comprises:

using a difference between the body contour key point prediction result and the body contour key point marking information as guidance information to perform supervised learning on the input neural network to be trained and the second neural network to be trained.

15. The method according to claim 8, wherein a configuring process for the body contour key point marking information carried in the image sample comprises:

obtaining a body skeleton key point of the image sample;

configuring an auxiliary line according to at least one of the configured body contour key point marking information in a first set or the body skeleton key point; and

forming, according to a point selected from the auxiliary line, body contour key point marking information in a second set.

16. The method according to claim 15, wherein the configuring process for the body contour key point marking information carried in the image sample further comprises:

forming body contour key point marking information in a third set according to N1 division points on a connection line between two body contour key points in at least one of the first set or the second set;

wherein N1 is an integer greater than 1.

17. The method according to claim 15, wherein the configuring the auxiliary line according to at least one of the configured body contour key point marking information in the first set or the body skeleton key point comprises:

determining, according to the body skeleton key point, a first auxiliary line passing through an arm root inner contour key point and vertical to an upper arm direction; and

the forming, according to the point selected from the auxiliary line, the body contour key point marking information in the second set comprises;

forming, according to an intersection point of the first auxiliary line and an upper arm outer contour, marking information of the arm root outer contour key point in the second set.

18. The method according to claim 15, wherein the configuring the auxiliary line according to the configured body contour key point marking information in the first set and/or the body skeleton key point comprises:

configuring a second auxiliary line passing through a crotch key point and vertical to a thigh direction formed by the body skeleton key point; and

the forming, according to the point selected from the auxiliary line, the body contour key point marking information in the second set comprises;

forming, according to an intersection point of the second auxiliary line and a thigh outer contour, marking information of the thigh root outside contour key point in the second set.

19. An electronic device, comprising:

a processor; and

a memory for storing instructions executable by the processor;

wherein execution of the instructions by the processor causes the processor to perform;

obtaining an image feature of an image block comprising a body;

obtaining, by means of a first neural network, a body contour key point prediction result of the body according to the image feature; and

obtaining, according to the body contour key point prediction result, a body contour key point in the image block, wherein the body contour key point is used for representing an outer contour of the body.

20. A non-transitory computer-readable storage medium, configured to store computer-readable instructions, wherein execution of the instructions by the processor causes the processor to perform:

obtaining an image feature of an image block comprising a body;

obtaining, by means of a first neural network, a body contour key point prediction result of the body according to the image feature; and

obtaining, according to the body contour key point prediction result, a body contour key point in the image block, wherein the body contour key point is used for representing an outer contour of the body.