Artificial intelligence based generation and analysis of 3D models

• US 10,621,779 B1
• Filed: 05/25/2018
• Issued: 04/14/2020
• Est. Priority Date: 05/25/2017
• Status: Active Grant

• Current Assignee: FastVDO LLC
• Original Assignee: FastVDO LLC
• Inventors: Topiwala, Pankaj N., Krishnan, Madhu Peringassery, Dai, Wei
• Primary Examiner(s): Ge, Jin
• Application Number:
  US15/990,212
• Time in Patent Office: 690 Days
• Field of Search: 345419
• CPC Class Codes: G06F 18/213 Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 18/253 of extracted features
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 5/00 Computing arrangements using knowledge-based models
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: G06T 7/62 of area, perimeter, diameter or volume
• CPC Class Codes: G06V 10/761 Proximity, similarity or dissimilarity measures
• CPC Class Codes: G06V 10/7715 Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/653 by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces

Abstract

Artificial intelligence based techniques are used for analysis of 3D objects in conjunction with each other. A 3D model of two or more 3D objects is generated. Features of 3D objects are matched to develop a correspondence between the 3D objects. Two 3D objects are geometrically mapped and an object is overlayed on another 3D object to obtain a superimposed object. Match analysis of 3D objects is performed based on machine learning based models to determine how well the objects are spatially matched. The analysis of the objects is used in augmented reality applications.

Claims

1. A method of processing imagery, representing a plurality of 3-dimensional (3D) objects, comprising the following steps:

generating, using 3D modeling, a plurality of 3D models, each 3D model corresponding to a 3D object from the plurality of 3D objects, wherein a 3D model is represented using one of point clouds or tilings, wherein the plurality of 3D models comprises a base 3D model corresponding to a human body and a set of remaining 3D models each corresponding to a respective wearable article;

extracting features describing each 3D model;

matching the extracted features of the 3D models to develop a correspondence between the based 3D model and each of the remaining 3D models, wherein the correspondence aligns the 3D objects spatially, so that they are geometrically matched as close as possible;

geometrically mapping each of the remaining 3D models to the base 3D model by overlaying each of the remaining 3D models on the base 3D model to obtain a unified superimposed model;

using a first neural network model, extracting numerical tensor features from the 3D models that sample the geometry of the 3D models, the first neural network model having a first set of layers configured to construct and update a graph indicating geometric structures of a 3D model of the plurality of 3D models, and wherein an initial stage of first set of layers of the first neural network model generates generic features from the 3D models and subsequent stages of the first set of layers of the first neural network model generate specific features representing regions of interest in the 3D models by analyzing the generic features;

using the second neural network model, matching the extracted numerical tensor features of the of the remaining 3D models to the base 3D model to perform match analysis to determine, for each remaining 3D model, a metric indicating a degree of spatial match between the remaining 3D model and the base model, wherein the second neural network model has a second set of layers configure to determine the metric based upon differences between corresponding numerical tensor features of the remaining 3D model and the base 3D model;

generating a 2-dimensional image corresponding to a view of the superimposed model; and

sending the generated 2-dimensional image and the determined metrics of the superimposed model for presentation.

2. The method of claim 1, wherein generating using 3D modeling comprises:

obtaining imagery representing at least one of the 3D objects by use of color images obtained from a color camera and depth images obtained from a depth sensor, and

converting the color images and depth images of the 3D object into a 3D model.

3. The method of claim 2, wherein the imagery representing the 3D objects is obtained from a color camera and involves one or more 2D images of each of the 3D objects, wherein the 3D modeling phase involves shape learning based on databases storing clusters of similar shapes, to convert the imagery to 3-dimensional models.

4. The method of claim 1, further comprising classifying the base 3D model by:

storing a plurality of 3D models, each 3D model representing a type of human body or portions thereof in a population of users;

extracting numerical feature tensors from the stored 3D models, by taking horizontal cross-sections of the 3D models, and extracting features such as circumference, area, and volume between cross-sections;

clustering the numerical feature tensors to segment the stored 3D models into a plurality of distinct clusters; and

classifying the base 3D model to a particular cluster by a proximity analysis to the elements of the cluster cells.

5. The method of claim 1, wherein the extracted generic features include sets of tensors representing the corners, edges and points of inflection in the 3D model.

6. The method of claim 1, wherein the generated specific features include sets of vectors representing the geometry of unique regions in the input.

7. The method of claim 1, wherein training the first neural network model comprises:

training the first neural network model with training data comprising ground truth information regarding a set of 3D/2D inputs;

periodically updating the training data using new user 3D/2D inputs; and

periodically retraining the first neural network model with the updated training data either online or offline.

8. The method of claim 1, wherein the second neural network model matches numerical tensor features of a remaining 3D model of the set of remaining 3D models by finding corresponding sets of tensor features of the remaining 3D model and base 3D model that have the highest correlation.

9. The method of claim 1, further comprising training the second neural network model by:

training the second neural network model with training data comprising ground truth information regarding a training set of extracted features and the correlation between them;

periodically updating the training data using new extracted features and their correlation; and

periodically retraining the second neural network model with the updated training data either online or offline.

10. The method of claim 1, wherein geometrically mapping a pair of 3D models comprises, using trained deep neural networks to orient and overlay the 3D models.

11. The method of claim 10, wherein orienting and overlaying the 3D models comprises finding one or more transformations that map the matched features and applying the one or more transformation to one or more other 3D models.

12. The method of claim 10, wherein training the deep neural network comprises:

training a deep neural network with training data comprising ground truth information regarding the 3D/2D inputs, extracted features, the transformation between matched features and the mapped inputs;

periodically updating the training data using the new user 3D/2D input, extracted features and transformation between matched features; and

periodically retraining the deep neural network with the updated training data either online or offline.

13. A computer system comprising:

one or more computer processors; and

a non-transitory computer readable storage medium storing instructions for;

generating, using 3D modeling, a plurality of 3D models, each 3D model corresponding to a 3D object from the plurality of 3D objects, wherein a 3D model is represented using one of point clouds or tilings, wherein the plurality of 3D models comprises a base 3D model corresponding to a human body and a set of remaining 3D models each corresponding to a respective wearable article;

extracting features describing each 3D model;

matching the extracted features of the 3D models to develop a correspondence between the based 3D model and each of the remaining 3D models, wherein the correspondence aligns the 3D objects spatially, so that they are geometrically matched as close as possible;

geometrically mapping each of the remaining 3D models to the base 3D model by overlaying each of the remaining 3D models on the base 3D model to obtain a unified superimposed model;

using a first neural network model, extracting numerical tensor features from the 3D models that sample the geometry of the 3D models, the first neural network model having a first set of layers configured to construct and update a graph indicating geometric structures of a 3D model of the plurality of 3D models, and wherein an initial stage of first set of layers of the first neural network model generates generic features from the 3D models and subsequent stages of the first set of layers of the first neural network model generate specific features representing regions of interest in the 3D models by analyzing the generic features;

using the second neural network model, matching the extracted numerical tensor features of the of the remaining 3D models to the base 3D model to perform match analysis to determine, for each remaining 3D model, a metric indicating a degree of spatial match between the remaining 3D model and the base model, wherein the second neural network model has a second set of layers configure to determine the metric based upon <

>

corresponding numerical tensor features of the remaining 3D model and the base 3D model;

generating a 2-dimensional image corresponding to a view of the superimposed model; and

sending the generated 2-dimensional image and the determined metrics of the superimposed model for presentation.

14. The computer system of claim 13, wherein instructions for generating using 3D modeling comprise instructions for:

obtaining imagery representing at least one of the 3D objects by use of RGB (color) imaging and depth imaging, and

converting the RGB and depth images into a 3D model.

15. The computer system of claim 14, wherein the imagery representing the 3D objects is obtained from an ordinary RGB (color) camera and involves one or more images of each of the 3D objects, wherein the 3D modeling phase involves shape learning through databases, to convert the 2D imagery to 3-dimensional models.

16. The computer system of claim 13, wherein the instructions are further for classifying the base 3D model by:

storing a plurality of 3D models, each 3D model representing a type of human body or portions thereof in a population of users;

extracting numerical feature tensors from the stored 3D models, by taking horizontal cross-sections of the 3D models, and extracting features such as circumference, area, and volume between cross-sections;

clustering the numerical feature tensors to segment the stored 3D models into a plurality of distinct clusters; and

classifying the base 3D model to a particular cluster by a proximity analysis to the elements of the cluster cells.

17. The computer system of claim 13, wherein training the first neural network model comprises:

training the first neural network model with training data comprising ground truth information regarding a set of 3D/2D inputs;

periodically updating the training data using new user 3D/2D inputs; and

periodically retraining the first neural network model with the updated training data either online or offline.