MACHINE LEARNING BASED MODEL LOCALIZATION SYSTEM

• US 20180189974A1
• Filed: 02/23/2018
• Published: 07/05/2018
• Est. Priority Date: 05/19/2017
• Status: Active Grant

• Current Assignee: Manor Financial, Inc.
• Original Assignee: Manor Financial, Inc.
• Granted Patent: US 10,977,818 B2
• Time in Patent Office: Days
• CPC Class Codes: G06N 3/04 Architecture, e.g. interconnection topology
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: G06T 7/536 from perspective effects, e.g. by using vanishing points
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06T 7/77 using statistical methods

Abstract

A method for deriving an image sensor'"'"'s 3D pose estimate from a 2D scene image input includes at least one Machine Learning algorithm trained a priori to generate a 3D depth map estimate from the 2D image input, which is used in conjunction with physical attributes of the source imaging device to make an accurate estimate of the imaging device 3D location and orientation relative to the 3D content of the imaged scene. The system may optionally employ additional Machine Learning algorithms to recognize objects within the scene to further infer contextual information about the scene, such as the image sensor pose estimate relative to the floor plane or the gravity vector. The resultant refined imaging device localization data can be applied to static (picture) or dynamic (video), 2D or 3D images, and is useful in many applications, most specifically for the purposes of improving the realism and accuracy of primarily static, but also dynamic Augmented Reality (AR) applications.

Claims

1. A method of estimating the pose estimate of a source imaging device based on at least one two-dimensional (2D) image input, comprising:

an input data set further comprising an 2D image data set to be analyzed a source camera parameter information such that the angular field of view of the imaging device can be determined,

a first step comprising Machine Learning algorithm module capable of receiving said 2D image from the input data set and generating estimated depth values at least a portion of the image pixels relative to the source imaging device as output,

a second step, executed in parallel with the first step and comprising an Imaging Device angular field of view determination process, capable of receiving the input data set and generating the source imaging sensor angular field of view as output,

a third step receiving the outputs from steps one and two and generating the source imaging sensor three-dimensional (3D) pose estimate relative to all points in the image scene points as output in conjunction with the image depth values of the first step,

2. The method of claim 1, wherein the Machine Learning algorithm is an Artificial Neural Network (ANN).

3. The method of claim 1, wherein the 3D imaging device pose estimate generated in the third step is a vector with spherical coordinates.

4. The method of claim 1, further comprising a secondary Machine Learning algorithm, executed directly before, directly after, or in parallel with the first and second method steps, wherein the algorithm has been trained to recognize a floor plane in a scene image, and upon receiving an input scene image data set, is capable of segmenting and classifying a floor plane object within the image and subsequently issuing the matching pixel set as output;

5. The method of claim 4, wherein the third method step further receives the floor plane pixel values, determines the associated depth estimates for said pixels, and uses trigonometry or another suitable mathematical method to calculate the location and orientation of the floor plane relative to the imaging device;

6. The method of claim 5, wherein the third method step further encodes the floor plane relative 3D location and orientation measurements into the output imaging sensor pose estimate vector in the form of additional vector components,

7. The method of claim 1, further comprising an arbitrary number of additional Machine Learning algorithms, executed directly before, directly after, or in parallel with the first and second steps, or with each other, wherein an arbitrary Machine Learning algorithm has been trained to recognize an arbitrary object in a scene image, and upon receiving an input scene image data set, is capable of segmenting and classifying said object within the image and subsequently issuing the matching pixel set as output;

8. The method of claim 7, wherein the third method step further encodes the relative 3D location and orientation measurements of each detectable object into the output imaging sensor pose estimate vector in the form of additional vector components.

9. The method of claim 2, wherein the ANN is a Convolutional Neural Network (CNN).

10. The method of claim 1, 4, and 7, wherein a Machine Learning algorithm is trained using supervised training.

11. The method of claim 1, wherein the training data set for the Machine Learning algorithm of the first step comprises a plurality of image data set pairs further comprising a 2D image of a scene and a 3D image of the same scene;

12. The method of claim 7, wherein additional semantic context definitions of objects are store and transmitted in conjunction with the pose estimate vector outputs of the respective object detection Machine Learning algorithm;

13. The method of claim 4, wherein the third step includes an additional calculation to create a file containing all pixel not coincident with the detected floor plane;

14. The method of claim 7, wherein a Machine Learning algorithm is trained to detect wall plane objects;

15. The method of claim 7, wherein a Machine Learning algorithm is trained to detect pendulous objects;