Electronic device pose identification based on imagery and non-image sensor data

• US 10,073,531 B2
• Filed: 10/07/2015
• Issued: 09/11/2018
• Est. Priority Date: 10/07/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Hesch, Joel, Nerurkar, Esha
• Primary Examiner(s): Flohre, Jason A
• Application Number:
  US14/877,038
• Publication Number:
  US 20170102772A1
• Time in Patent Office: 1,070 Days
• CPC Class Codes: G01C 3/08 Use of electric radiation detectors
• CPC Class Codes: G01P 15/093 by photoelectric pick-up
• CPC Class Codes: G01S 5/163 Determination of attitude using inertial means G01C9/00; control of attitude G05D1/49
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 23/951 by using two or more images to influence resolution, frame rate or aspect ratio
• CPC Class Codes: H04N 23/957 Light-field or plenoptic cameras or camera modules

Abstract

An electronic device includes one or more imaging sensors (e.g, imaging cameras) and includes one or more non-image sensors, such as an inertial measurement unit (IMU), that can provide information indicative of the pose of the electronic device. The electronic device estimates its pose based on two independent sources of pose information: pose information generated at a relatively high rate based on non-visual information generated by the non-image sensors and pose information generated at a relatively low rate based on imagery captured by the one or more imaging sensors. To achieve both a high pose-estimation rate and high degree of pose estimation accuracy, the electronic device adjusts a pose estimate based on the non-visual pose information at a high rate, and at a lower rate spatially smoothes the pose estimate based on the visual pose information.

Claims

1. A method comprising:

updating, at an electronic device, a first data structure storing a first estimated pose of the electronic device at a first rate based on received non-visual pose information obtained from non-visual sensors;

updating, at the electronic device, a second data structure storing a second estimated pose of the electronic device at a second rate based on imagery captured at the electronic device, the second rate lower than the first rate;

adjusting the first data structure to match the second data structure, comprising;

identifying a difference between the first estimated pose and the second estimated pose along a first axis of a frame of reference of the electronic device;

adjusting the first estimated pose along the first axis within a first threshold amount based on the difference between the first estimated pose and the second estimated pose, and,

in response to identifying that the difference exceeds a second threshold, setting the difference to the second threshold; and

employing the adjusted first data structure as an estimated pose of the electronic device to support location-based functionality at the electronic device.

2. The method of claim 1, wherein adjusting the first data structure to match the second data structure comprises:

adjusting the first estimated pose along each of a plurality of axes of the frame of reference based on the difference between the first estimated pose and the second estimated pose.

3. The method of claim 1 wherein adjusting the first estimated pose along the first axis comprises:

scaling the first estimated pose along the first axis based on the difference.

4. The method of claim 1, wherein adjusting the first estimated pose along the first axis comprises:

setting a value of the first estimated pose along the first axis to a corresponding value of the second estimated pose.

5. The method of claim 1, wherein adjusting the first estimated pose along the first axis comprises:

selecting the first axis based on a degree of motion along the first axis indicated by the difference between the first estimated pose and the second estimated pose.

6. The method of claim 1, wherein adjusting the first estimated pose along the first axis comprises:

selecting the first axis based upon a predicted likelihood that pose errors along the first axis will be perceived by a user of the electronic device.

7. The method of claim 1, wherein updating the first data structure based on received non-visual pose information comprises:

identifying a third estimated pose of the electronic device based on the received non-visual pose information; and

in response to determining that a difference between the first estimated pose and the third estimated pose is below a threshold, refraining from modifying the first estimated pose.

8. The method of claim 1, wherein updating the second data structure based on imagery comprises:

identifying a third estimated pose of the electronic device based on the imagery captured at the electronic device and based on non-visual pose information;

identifying a difference between the third estimated pose and the second estimated pose along a first axis of a frame of reference of the electronic device; and

adjusting the second estimated pose along the first axis based on the difference between the third estimated pose and the second estimated pose.

9. A method, comprising,

updating at an electronic device a first set of estimated poses at a first rate based on non-visual pose information obtained from non-image sensors;

updating a second set of estimated poses at a second rate lower than the first rate based on imagery captured at the electronic device; and

spatially smoothing, at the electronic device, the first set of estimated poses based on the second set of estimated poses, wherein spatially smoothing is within a first threshold amount and comprises;

scaling the first set of estimated poses based on a difference between the first set of estimated poses and the second set of estimated poses, and,

in response to identifying that the difference exceeds a second threshold, setting the difference to the second threshold.

10. The method of claim 9, wherein the first rate is at least five times the second rate.

11. The method of claim 9, further comprising:

updating the second set of estimated poses based on the imagery and based on geolocation data received at the electronic device.

12. An electronic device comprising:

an imaging camera to capture imagery;

an inertial measurement unit (IMU) to generate non-visual pose information; and

a processor to

update a first data structure storing a first estimated pose of the electronic device at a first rate based on the imagery;

update a second data structure storing a second estimated pose of the electronic device at a second rate based on received non-visual pose information obtained from non-image sensors, the second rate higher than the first rate; and

adjust, within a first threshold amount, the second data structure to match the first data structure, wherein the processor is to adjust the second data structure to match the first data structure by;

identifying a difference between the first estimated pose and the second estimated pose along a first axis of a frame of reference of the electronic device;

adjusting the first estimated pose along the first axis based on the difference between the first estimated pose and the second estimated pose; and

in response to identifying that the difference exceeds a second threshold, setting the difference to the second threshold.

13. The electronic device of claim 12, wherein the processor is to adjust the second data structure to match the first data structure by:

adjusting the second data structure along each of a plurality of axes of the frame of reference based on the difference between the first data structure and the second data structure.

14. The electronic device of claim 12, wherein the processor is to adjust the second data structure to match the first data structure by:

scaling the second data structure along the first axis based on the difference.

15. The electronic device of claim 12, wherein the processor is to adjust the second data structure along the first axis by:

setting a value of the second data structure along the first axis to a corresponding value of the first data structure.

16. The electronic device of claim 12, wherein the processor is to adjust the second data structure along the first axis by:

selecting the first axis based on a degree of motion along the first axis indicated by the difference between the first data structure and the second data structure.