Method and device for generating safe clothing patterns for rider of bike

• US 10,657,584 B1
• Filed: 01/10/2020
• Issued: 05/19/2020
• Est. Priority Date: 01/31/2019
• Status: Active Grant

• Current Assignee: Stradvision, Inc.
• Original Assignee: Stradvision, Inc.
• Inventors: Kim, Kye-Hyeon, Kim, Yongjoong, Kim, Hak-Kyoung, Nam, Woonhyun, Boo, SukHoon, Sung, Myungchul, Shin, Dongsoo, Yeo, Donghun, Ryu, Wooju, Lee, Myeong-Chun, Lee, Hyungsoo, Jang, Taewoong, Jeong, Kyungjoong, Je, Hongmo, Cho, Hojin
• Primary Examiner(s): Strege, John B
• Application Number:
  US16/740,085
• Time in Patent Office: 130 Days
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06Q 30/0643 Graphical representation of items or shoppers
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/30124 Fabrics; Textile; Paper
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/593 Recognising seat occupancy
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition

Abstract

A method for generating safe clothing patterns for a human-like figure is provided. The method includes steps of: a safe clothing-pattern generating device, (a) after acquiring an image of the human-like figure, generating a specific clothing pattern having an initial value, inputting the specific clothing pattern and the image of the human-like figure into a clothing composition network, combining the specific clothing pattern with a clothing of the human-like figure to generate a composite image; (b) inputting the composite image into an image translation network, translating surrounding environment on the composite image to generate a translated image, and inputting the translated image into an object detector to output detection information on the human-like figure; and (c) instructing a 1-st loss layer to calculate losses by referring to the detection information and a GT corresponding to the image of the human-like figure, and updating the initial value by using the losses.

Claims

1. A method for generating one or more safe clothing patterns to be used for a human-like figure, comprising steps of:

(a) if at least one image of the human-like figure is acquired, a safe clothing-pattern generating device performing a process of generating at least one specific clothing pattern having an initial value, and a process of inputting the specific clothing pattern and the image of the human-like figure into a clothing composition network, to thereby allow the clothing composition network to combine the specific clothing pattern with a clothing of the human-like figure on the image of the human-like figure, and thus to generate at least one composite image corresponding to the image of the human-like figure;

(b) the safe clothing-pattern generating device performing a process of inputting the composite image into an image translation network, to thereby allow the image translation network to generate at least one translated image by translating a surrounding environment on the composite image, and a process of inputting the translated image into an object detector, to thereby allow the object detector to output detection information on the human-like figure representing the human-like figure detected in the translated image; and

(c) the safe clothing-pattern generating device performing a process of instructing a 1-st loss layer to calculate one or more losses by referring to the detection information on the human-like figure and at least one GT corresponding to the image of the human-like figure, and a process of updating the initial value of the specific clothing pattern by using the losses such that the losses are minimized.

2. The method of claim 1, wherein, at the step of (a), the safe clothing-pattern generating device performs a process of inputting the image of the human-like figure and the specific clothing pattern into the clothing composition network, to thereby allow the clothing composition network to perform (i) a process of segmenting the clothing on the image of the human-like figure via a segmentation network and a process of acquiring one or more joint parts of the human-like figure on the image of the human-like figure via a posture estimation network, (ii) a process of acquiring at least one masked image created by masking the clothing with the specific clothing pattern by referring to the joint parts via a masking layer, (iii) a process of generating at least one concatenated image by concatenating the masked image and the image of the human-like figure, and (iv) a process of applying at least one convolution operation to the concatenated image via an encoder and a process of applying at least one deconvolution operation to an output of the encoder via a decoder, to thereby generate the composite image.

3. The method of claim 2, wherein the clothing composition network has been learned by a learning device to generate the composite image using the specific clothing pattern and the image of the human-like figure, and

wherein the learning device has learned the clothing composition network by repeating (i) a process of inputting at least one acquired training image of the human-like figure into the segmentation network, to thereby allow the segmentation network to segment at least one clothing for training on the training image of the human-like figure and a process of inputting the training image of the human-like figure into the posture estimation network, to thereby allow the posture estimation network to acquire one or more joint parts for training of at least one human-like figure for training on the training image of the human-like figure, (ii) a process of inputting the joint parts for training, the clothing for training, and at least one acquired clothing pattern for training into the masking layer, to thereby allow the masking layer to generate at least one masked image for training created by masking the clothing for training with the clothing pattern for training by referring to the joint parts for training, (iii) a process of generating at least one concatenated image for training by concatenating the masked image for training and the training image of the human-like figure, (iv) a process of inputting the concatenated image for training into the encoder, to thereby allow the encoder to apply said at least one convolution operation to the concatenated image for training and a process of inputting the output of the encoder into the decoder, to thereby allow the decoder to apply at least one deconvolution operation to the output of the encoder and thus to generate the composite image for training and (v) a process of instructing a 2-nd loss layer to calculate one or more perceptual losses by referring to (v-1) the composite image for training and (v-2) at least one GT image created as corresponding to the training image of the human-like figure and the clothing pattern for training and a process of adjusting at least one parameter of the decoder, the encoder, and the masking layer via backpropagation by referring to the perceptual losses such that the perceptual losses are minimized.

4. The method of claim 3, wherein the GT image is generated by masking the clothing for training on the training image of the human-like figure with the clothing pattern for training.

5. The method of claim 1, wherein the GT is one of (i) information corresponding to the human-like figure created by labeling the image of the human-like figure, and (ii) information corresponding to the human-like figure which is detected from the image of the human-like figure inputted into the object detector or another object detector.

6. The method of claim 1, wherein, at the step of (b), the safe clothing-pattern generating device performs a process of instructing the image translation network to translate the surrounding environment on the composite image into another surrounding environment such that a performance of a conventional object detector for detecting the human-like figure becomes lower than a preset threshold.

7. The method of claim 6, wherein the image translation network includes a CycleGAN (Generative Adversarial Network) capable of image-to-image translation of original images and translated images, which are unpaired with each other, respectively belonging to different domains.

8. A safe clothing-pattern generating device for generating one or more safe clothing patterns to be used for a human-like figure, comprising:

at least one memory that stores instructions; and

at least one processor configured to execute the instructions to perform or support another device to perform;

(I) if at least one image of the human-like figure is acquired, a process of generating at least one specific clothing pattern having an initial value, and a process of inputting the specific clothing pattern and the image of the human-like figure into a clothing composition network, to thereby allow the clothing composition network to combine the specific clothing pattern with a clothing of the human-like figure on the image of the human-like figure, and thus to generate at least one composite image corresponding to the image of the human-like figure, (II) a process of inputting the composite image into an image translation network, to thereby allow the image translation network to generate at least one translated image by translating a surrounding environment on the composite image, and a process of inputting the translated image into an object detector, to thereby allow the object detector to output detection information on the human-like figure representing the human-like figure detected in the translated image, and (III) a process of instructing a 1-st loss layer to calculate one or more losses by referring to the detection information on the human-like figure and at least one GT corresponding to the image of the human-like figure, and a process of updating the initial value of the specific clothing pattern by using the losses such that the losses are minimized.

9. The safe clothing-pattern generating device of claim 8, wherein, at the process of (I), the processor performs a process of inputting the image of the human-like figure and the specific clothing pattern into the clothing composition network, to thereby allow the clothing composition network to perform (i) a process of segmenting the clothing on the image of the human-like figure via a segmentation network and a process of acquiring one or more joint parts of the human-like figure on the image of the human-like figure via a posture estimation network, (ii) a process of acquiring at least one masked image created by masking the clothing with the specific clothing pattern by referring to the joint parts via a masking layer, (iii) a process of generating at least one concatenated image by concatenating the masked image and the image of the human-like figure, and (iv) a process of applying at least one convolution operation to the concatenated image via an encoder and a process of applying at least one deconvolution operation to an output of the encoder via a decoder, to thereby generate the composite image.

10. The safe clothing-pattern generating device of claim 9, wherein the clothing composition network has been learned by a learning device to generate the composite image using the specific clothing pattern and the image of the human-like figure, and

wherein the learning device has learned the clothing composition network by repeating (i) a process of inputting at least one acquired training image of the human-like figure into the segmentation network, to thereby allow the segmentation network to segment at least one clothing for training on the training image of the human-like figure and a process of inputting the training image of the human-like figure into the posture estimation network, to thereby allow the posture estimation network to acquire one or more joint parts for training of at least one human-like figure for training on the training image of the human-like figure, (ii) a process of inputting the joint parts for training, the clothing for training, and at least one acquired clothing pattern for training into the masking layer, to thereby allow the masking layer to generate at least one masked image for training created by masking the clothing for training with the clothing pattern for training by referring to the joint parts for training, (iii) a process of generating at least one concatenated image for training by concatenating the masked image for training and the training image of the human-like figure, (iv) a process of inputting the concatenated image for training into the encoder, to thereby allow the encoder to apply said at least one convolution operation to the concatenated image for training and a process of inputting the output of the encoder into the decoder, to thereby allow the decoder to apply at least one deconvolution operation to the output of the encoder and thus to generate the composite image for training and (v) a process of instructing a 2-nd loss layer to calculate one or more perceptual losses by referring to (v-1) the composite image for training and (v-2) at least one GT image created as corresponding to the training image of the human-like figure and the clothing pattern for training and a process of adjusting at least one parameter of the decoder, the encoder, and the masking layer via backpropagation by referring to the perceptual losses such that the perceptual losses are minimized.

11. The safe clothing-pattern generating device of claim 10, wherein the GT image is generated by masking the clothing for training on the training image of the human-like figure with the clothing pattern for training.

12. The safe clothing-pattern generating device of claim 8, wherein the GT is one of (i) information corresponding to the human-like figure created by labeling the image of the human-like figure, and (ii) information corresponding to the human-like figure which is detected from the image of the human-like figure inputted into the object detector or another object detector.

13. The safe clothing-pattern generating device of claim 8, wherein, at the process of (II), the processor performs a process of instructing the image translation network to translate the surrounding environment on the composite image into another surrounding environment such that a performance of a conventional object detector for detecting the human-like figure becomes lower than a preset threshold.

14. The safe clothing-pattern generating device of claim 13, wherein the image translation network includes a CycleGAN (Generative Adversarial Network) capable of image-to-image translation of original images and translated images, which are unpaired with each other, respectively belonging to different domains.