Object-based localization

• US 10,726,264 B2
• Filed: 06/25/2018
• Issued: 07/28/2020
• Est. Priority Date: 06/25/2018
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Sawhney, Harpreet Singh, Pollefeys, Marc Andre Leon
• Primary Examiner(s): Bernardi, Brenda C
• Application Number:
  US16/017,893
• Publication Number:
  US 20190392212A1
• Time in Patent Office: 764 Days
• CPC Class Codes: G06F 18/2431 Multiple classes
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06T 2210/12 Bounding box
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 10/757 Matching configurations of points or features
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/64 Three-dimensional objects

Abstract

Techniques for localizing a device based on images captured by the device. The techniques include receiving, from a device via a data communication network, image frame data for frames captured by an imaging camera included in the device, the frames including a first frame, automatically detecting real-world objects captured in the image frame data, automatically classifying the detected real-world objects as being associated with respective object classes, automatically identifying object class and instance dependent keypoints for the real-world objects based on the object classes associated with the real-world objects, and estimating a pose of the device for the first frame based on at least the identified object class and instance dependent keypoints.

Claims

1. An object-based localization system comprising:

a frame data receiving module configured to receive, from a remote client device via a data communication network, image frame data for a plurality of frames captured by an imaging camera included in the remote client device, the plurality of frames including a first frame;

an object detector configured to receive the image frame data, automatically detect a plurality of real-world objects captured in the image frame data, and automatically classify the plurality of real-world objects as being associated with respective object classes;

an object class and instance dependent keypoint selector configured to automatically identify a plurality of object class and instance dependent keypoints for the plurality of real-world objects based on the object classes associated with the plurality of real-world objects;

a location selector module configured to select a current location record from a database based on at least a determination that the object classes associated with the plurality of real-world objects correlate with object classes associated with object instances identified by the selected current location record; and

a localization module configured to estimate a first pose of the remote client device for the first frame based on at least the identified plurality of object class and instance dependent keypoints and location data indicated by the selected current location record, and to transmit the estimated first pose to the remote client device.

2. The system of claim 1, wherein:

the system is configured to, using the object detector, automatically detect a first real-world object captured in the image frame data and automatically classify the first real-world object as being associated with a first object class;

the first real-world object is not included in the plurality of real-world objects;

none of the plurality of real-world objects are classified as being associated with the first object class; and

in response to a determination that the first object class associated with the first real-world object is for non-stationary physical objects, the estimation of the first pose is not based on object class and instance dependent keypoints identified for the first real-world object.

3. The system of claim 1, wherein:

the plurality of real-world objects includes a first real-world object;

the object detector is further configured to automatically determine a three-dimensional bounding box for the first real-world object; and

the automatic identification of the plurality of object class and instance dependent keypoints includes selecting one or more vertices of the bounding box as object class and instance dependent keypoints for the first real-world object.

4. The system of claim 1, further comprising a location selector module configured to:

determine a first physical arrangement of the plurality of real-world objects; and

select a current location record from a database based on at least a determination that the first physical arrangement of the plurality of real-world objects correlates with a second physical arrangement of object instances identified by the selected current location record;

wherein the estimation of the first pose is further based on location data indicated by the selected current location record.

5. The system of claim 1, further comprising:

a first machine-trained model trained to detect and classify images of real-world objects; and

a second machine-trained model trained to detect object class and instance dependent keypoints for one or more object classes;

wherein;

the object detector is further configured to use the first machine-trained model to detect and classify the plurality of real-world objects, and

the object class and instance dependent keypoint selector is further configured to use the second machine-trained model to identify the plurality of object class and instance dependent keypoints.

6. The system of claim 1, further comprising:

a location record selector configured to select, based on a correspondence with the plurality of real-world objects and/or the plurality of object class and instance dependent keypoints, a current location record from a location database; and

a local object detector configured to;

determine that an object database does not include an object instance record corresponding to a first real-world object included in the plurality of real-world objects,

create, in response to the determination that the object database does not include an object instance record corresponding to the first real-world object, a new first object instance record based on information determined for the first real-world object, and

update, in response to the determination that the object database does not include an object instance record corresponding to the first real-world object, the current location record to identify the first object instance record as being associated with the current location record.

7. A method of localizing a device, the method comprising:

receiving, from a remote client device via a data communication network, image frame data for a plurality of frames captured by an imaging camera included in the remote client device, the plurality of frames including a first frame;

automatically detecting a plurality of real-world objects captured in the image frame data;

automatically classifying the plurality of real-world objects as being associated with respective object classes;

automatically identifying a plurality of object class and instance dependent keypoints for the plurality of real-world objects based on the object classes associated with the plurality of real-world objects;

selecting a current location record from a database based on at least a determination that the object classes associated with the plurality of real-world objects correlate with object classes associated with object instances identified by the selected current location record;

estimating a first pose of the remote client device for the first frame based on at least the identified plurality of object class and instance dependent keypoints and location data indicated by the selected current location record; and

transmitting the estimated first pose to the remote client device.

8. The method of claim 7, further comprising:

automatically detecting a first real-world object captured in the image frame data, wherein the first real-world object is not included in the plurality of real-world objects; and

automatically classifying the first real-world object as being associated with a first object class, wherein;

none of the plurality of real-world objects are classified as associated with the first object class, and

in response to a determination that the first object class associated with the first real-world object is for non-stationary real-world objects, the estimation of the first pose is not based on object class and instance dependent keypoints identified for the first real-world object.

9. The method of claim 7, wherein:

the plurality of real-world objects includes a first real-world object;

the method further includes automatically determining a three-dimensional bounding box for the first real-world object; and

the automatic identification of the plurality of object class and instance dependent keypoints includes selecting one or more vertices of the bounding box as object class and instance dependent keypoints for the first real-world object.

10. The method of claim 7, further comprising:

determining a first physical arrangement of the plurality of real-world objects;

selecting a current location record from a database based on at least a determination that the first physical arrangement of the plurality of real-world objects correlates with a second physical arrangement of object instances identified by the selected current location record;

wherein the estimation of the first pose is further based on location data indicated by the selected current location record.

11. The method of claim 7, wherein:

the detection of the plurality of real-world objects includes use of a machine-trained model trained to detect classify images of real-world objects; and

the identification of the plurality of object class and instance dependent keypoints includes use of a machine-trained model trained to detect object class and instance dependent keypoints for one or more object classes.

12. The method of claim 7, further comprising:

determining that an object database does not include an object instance record corresponding to a first real-world object included in the plurality of real-world objects;

creating, in response to the determination that the object database does not include an object instance record corresponding to the first real-world object, a new first object instance record based on information determined for the first real-world object;

selecting, based on a correspondence with the plurality of real-world objects and/or the plurality of object class and instance dependent keypoints, a current location record from a location database; and

updating, in response to the determination that the object database does not include an object instance record corresponding to the first real-world object, the current location record to identify the first object instance record as being associated with the current location record.

13. A non-transitory machine-readable medium including instructions recorded thereon which, when executed by one or more processors, cause the processors to perform the method of claim 7.

14. A method of localizing a device, the method comprising:

capturing, with an imaging camera included in the device, image frame data for a plurality of frames at a rate based at least in part on whether the imaging camera is capturing image frame data associated with a region of interest, the plurality of frames including a first frame captured at a first time;

transmitting, via a data communication network, the image frame data from the device to a remote object-based localization system;

receiving, at the device at a second time, a first pose estimate generated by the remote object-based localization system for the device for the first time; and

determining, at the device, a second pose estimate of the device for a third time after the second time based on at least the received first pose estimate, wherein the second pose estimate is different than the first pose estimate.

15. The method of claim 14, further comprising:

capturing motion data with a motion sensor included in the device for motion of the device occurring after the first time and before the second time;

wherein the determination of the second pose estimate is further based on the captured motion data.

16. The method of claim 14, further comprising, for each frame in the plurality of frames:

determining a frame pose estimate for the device corresponding to a time at which the frame was captured by the imaging device; and

transmitting the frame pose estimate to the remote object-based localization system.

17. The method of claim 14, further comprising receiving, at the device, an identification of an object instance detected in the image frame data by the remote object-based localization system, wherein the object instance corresponds to a real-world object.

18. A non-transitory machine-readable medium including instructions recorded thereon which, when executed by one or more processors, cause the processors to perform the method of claim 14.