System and method for fault detection and recovery for concurrent odometry and mapping

• US 11,017,610 B2
• Filed: 05/17/2017
• Issued: 05/25/2021
• Est. Priority Date: 05/18/2016
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Nerurkar, Esha, Zhao, Sheng
• Primary Examiner(s): Salvucci, Matthew
• Application Number:
  US15/597,366
• Publication Number:
  US 20170337749A1
• Time in Patent Office: 1,469 Days
• Field of Search: 345419
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06T 17/05 Geographic models
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts

Abstract

An electronic device detects and recovers from fault conditions while tracking its motion and building a map of its environment. A front-end motion tracking module detects fault conditions while tracking motion over time and provides mapping data to a back-end mapping module. The front-end motion tracking module indicates to the back-end mapping module when a fault condition has been detected and when the fault condition is no longer detected. The back-end mapping module generates maps from the mapping data and merges the maps to build a three-dimensional representation of the environment. The back-end mapping module buffers or discards any mapping data received from the front-end motion tracking module during the existence of a fault condition. The back-end mapping module merges the maps generated based on mapping data received before and after the fault condition and adds the merged maps to the three-dimensional representation of the environment.

Claims

1. A method, comprising:

modifying, at an electronic device, a three-dimensional representation of an environment of the electronic device based on known feature descriptors of a stored plurality of maps of the environment that were previously generated by the electronic device during prior mapping sessions and a first map file comprising accumulated first mapping data received from a motion tracking module, wherein the first mapping data is based on images captured from one or more visual sensors and non-visual data from one or more non-image sensors;

in response to receiving a second map file comprising accumulated second mapping data from the motion tracking module after receiving an indication from the motion tracking module that a fault condition exists, wherein the fault condition comprises the electronic device changing direction at a rate that exceeds a rate at which the motion tracking module can generate accurate mapping data, buffering or discarding the second map file and omitting the second map file from the three-dimensional representation;

discontinuing buffering or discarding the second map file in response to receiving an indication from the motion tracking module that the fault condition has resolved; and

in response to receiving a third map file comprising accumulated third mapping data from the motion tracking module after receiving an indication from the motion tracking module that the fault condition has resolved, modifying the three-dimensional representation based on the third map file.

2. The method of claim 1, wherein the first, second, and third mapping data comprise feature descriptors of one or more spatial features representing an object in the environment of the electronic device, and wherein modifying the three-dimensional representation based on the third map file comprises matching feature descriptors of the third mapping data to feature descriptors of the first mapping data.

3. The method of claim 2, further comprising building a bundle adjustment problem comprising the feature descriptors of the three-dimensional representation and solving the bundle adjustment problem to generate a localized pose of the electronic device.

4. The method of claim 2, wherein modifying the three-dimensional representation based on the third map file comprises matching feature descriptors of the third mapping data to feature descriptors of the three-dimensional representation.

5. The method of claim 1, further comprising detecting, at the motion tracking module, existence of the fault condition.

6. The method of claim 5, wherein detecting occurs at a frequency that is substantially the same as an operating frequency of the one or more visual sensors.

7. The method of claim 1, wherein the fault condition further comprises the images captured from the one or more visual sensors comprising less than a threshold amount of information.

8. A method, comprising:

generating, at an electronic device, a three-dimensional representation of an environment of the electronic device based on known feature descriptors of a stored plurality of maps of the environment that were previously generated by the electronic device during prior mapping sessions and a first map file comprising accumulated first mapping data received from a motion tracking module of the electronic device, the first mapping data comprising feature descriptors of one or more spatial features representing an object in the environment of the electronic device based on images captured from one or more visual sensors and non-visual data from one or more non-image sensors;

in response to receiving from the motion tracking module an indication of a fault condition, wherein the fault condition comprises the electronic device changing direction at a rate that exceeds a rate at which the motion tracking module can generate accurate mapping data, interrupting generating the three-dimensional representation of the environment; and

in response to receiving an indication from the motion tracking module that the fault condition has resolved, resuming generating the three-dimensional representation based on a second map file comprising accumulated second mapping data received from the motion tracking module after the indication that the fault condition has resolved.

9. The method of claim 8, wherein resuming generating the three-dimensional representation comprises matching feature descriptors of the second mapping data to feature descriptors of the three-dimensional representation.

10. The method of claim 8, further comprising building a bundle adjustment problem comprising the feature descriptors of the three-dimensional representation and solving the bundle adjustment problem to generate a localized pose of the electronic device.

11. The method of claim 8, further comprising detecting, at the motion tracking module, existence of the fault condition.

12. The method of claim 11, wherein detecting occurs at a frequency that is substantially the same as an operating frequency of the one or more visual sensors.

13. The method of claim 8, wherein the fault condition further comprises the images captured from the one or more visual sensors comprising less than a threshold amount of information.

14. An electronic device, comprising:

a motion tracking module configured to track motion of the electronic device relative to an environment of the electronic device based on mapping data comprising feature descriptors of one or more spatial features representing one or more objects in the environment and to indicate a fault condition, wherein the feature descriptors are generated based on images captured from one or more visual sensors and non-visual data from one or more non-image sensors; and

a mapping module configured to generate a three-dimensional representation of the environment of the electronic device based on known feature descriptors of a stored plurality of maps of the environment that were previously generated by the electronic device during prior mapping sessions and first mapping data received from the motion tracking module, wherein the mapping module is further configured to interrupt generating the three-dimensional representation of the environment in response to receiving from the motion tracking module an indication of a fault condition, wherein the fault condition comprises the electronic device changing direction at a rate that exceeds a rate at which the motion tracking module can generate accurate mapping data, and, in response to receiving an indication from the motion tracking module that the fault condition has resolved, resume generating the three-dimensional representation based on the known feature descriptors of the stored plurality of maps and second mapping data received from the motion tracking module after the indication that the fault condition has resolved.

15. The electronic device of claim 14, wherein the mapping module is configured to resume generating the three-dimensional representation by matching feature descriptors of the second mapping data to feature descriptors of the three-dimensional representation.

16. The electronic device of claim 14, further comprising a loop-closure module configured to build and solve a bundle adjustment problem comprising the feature descriptors of the three-dimensional representation to generate a localized pose of the electronic device.

17. The electronic device of claim 14, wherein the motion tracking module is further configured to detect a fault condition at a frequency that is substantially the same as an operating frequency of the one or more visual sensors.

18. The electronic device of claim 14, wherein the fault condition further comprises the images captured from the one or more visual sensors comprising less than a threshold amount of information.