CAMERA/OBJECT POSE FROM PREDICTED COORDINATES

• US 20140241617A1
• Filed: 02/22/2013
• Published: 08/28/2014
• Est. Priority Date: 02/22/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Shotton, Jamie Daniel Joseph, Zach, Christopher, Izadi, Shahram, Criminisi, Antonio, Fitzgibbon, Andrew William, Glocker, Benjamin Michael
• Granted Patent: US 9,940,553 B2
• Time in Patent Office: Days
• US Class Current: 382/159
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/231 Hierarchical techniques, i.e. dividing or merging pattern sets so as to obtain a dendrogram
• CPC Class Codes: G06F 18/24323 Tree-organised classifiers
• CPC Class Codes: G06V 10/7625 Hierarchical techniques, i.e. dividing or merging patterns to obtain a tree-like representation; Dendograms
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/774 Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06V 20/20 in augmented reality scenes

Abstract

Camera or object pose calculation is described, for example, to relocalize a mobile camera (such as on a smart phone) in a known environment or to compute the pose of an object moving relative to a fixed camera. The pose information is useful for robotics, augmented reality, navigation and other applications. In various embodiments where camera pose is calculated, a trained machine learning system associates image elements from an image of a scene, with points in the scene'"'"'s 3D world coordinate frame. In examples where the camera is fixed and the pose of an object is to be calculated, the trained machine learning system associates image elements from an image of the object with points in an object coordinate frame. In examples, the image elements may be noisy and incomplete and a pose inference engine calculates an accurate estimate of the pose.

Claims

1. A method of calculating pose of an entity comprising:

receiving, at a processor, at least one image where the image is either;

of the entity and captured by a fixed camera, or of a scene captured by a mobile camera in the scene;

applying image elements of the at least one image to a trained machine learning system to obtain a plurality of associations between image elements and points in either entity coordinates or scene coordinates; and

calculating the pose of the entity from the associations.

2. A method as claimed in claim 1 where the entity is a mobile camera and the pose of the camera is calculated.

3. A method as claimed in claim 1 where the entity is an object and the pose of the object is calculated using the at least one image captured by a fixed camera.

4. A method as claimed in claim 1 comprising calculating the pose of the entity as parameters having six degrees of freedom, three indicating rotation of the entity and three indicating position of the entity.

5. A method as claimed in claim 1, the machine learning system having been trained using images with image elements labeled either with scene coordinates or object coordinates.

6. A method as claimed in claim 1 where the machine learning system is a random decision forest.

7. A method as claimed in claim 1 where the machine learning system comprises a plurality of trained random forests and the method comprises applying the image elements of the at least one image to the plurality of trained random forests, each random forest having been trained using images from a different one of a plurality of scenes, and calculating which of the scenes the mobile camera was in when the at least one image was captured.

8. A method as claimed in claim 1 the machine learning system having been trained using images of a plurality of scenes with image elements labeled with scene identifiers and labeled with scene coordinates of points in the scene the image elements depict.

9. A method as claimed in claim 1 comprising applying only a subsample of the image elements of the at least one image to the trained machine learning system.

10. A method as claimed in claim 1 comprising calculating the pose by searching amongst a set of possible pose candidates and using samples of associations between image elements and points to assess the pose candidates.

11. A method as claimed in claim 1 comprising receiving at the processor, a stream of images, and calculating the pose by searching amongst a set of possible pose candidates which includes a pose calculated from another image in the stream.

12. A method as claimed in claim 1 at least partially carried out using hardware logic selected from any one or more of:

a field-programmable gate array, a program-specific integrated circuit, a program-specific standard product, a system-on-a-chip, a complex programmable logic device, a graphics processing unit.

13. A method as claimed in claim 1 where the entity is a mobile camera and the pose of the camera is calculated, the method comprising accessing a 3D model of the scene and refining the camera pose using the accessed 3D model.

14. A method comprising:

receiving, at a processor, a plurality of images, each image having a plurality of image elements labeled with coordinates of points either in a scene the image elements depict or of an object the image elements depict;

training a machine learning system using the received plurality of images such that when an image element from another image is applied to the machine learning system, an estimate of a coordinate of a point the image element depicts is produced.

15. A method as claimed in claim 14 comprising receiving a plurality of images of sub-scenes, each having a plurality of image elements labeled with scene coordinates in a space in which the sub-scenes are embedded;

and training the machine learning system using the received plurality of images of sub-scenes.

16. A pose tracker comprising:

a processor arranged to receive at least one image either of an object captured by a fixed camera, or of a scene captured by a mobile camera;

the processor arranged to apply image elements of the at least one image to a trained machine learning system to obtain a plurality of associations between image elements and points in either object coordinates or scene coordinates; and

a pose inference engine arranged to calculate a position and orientation of either the object or the mobile camera from the associations.

17. A pose tracker as claimed in claim 16 the processor arranged to apply only a subsample of the image elements of the at least one image to the trained machine learning system.

18. A pose tracker as claimed in claim 16 the pose inference engine arranged to calculate the pose by searching amongst a set of possible pose candidates and using samples of associations between image elements and points in either object coordinates or scene coordinates to assess the pose candidates.

19. A pose tracker as claimed in claim 16 the processor arranged to receive a stream of images, and comprising a pose inference engine arranged to calculate the pose by searching amongst a set of possible pose candidates which includes a pose calculated from another image in the stream.

20. A pose tracker as claimed in claim 16 at least partially implemented using hardware logic selected from any one or more of:

a field-programmable gate array, a program-specific integrated circuit, a program-specific standard product, a system-on-a-chip, a complex programmable logic device, a graphics processing unit.