Adaptive switching between vision aided INS and vision only pose

• US 9,123,135 B2
• Filed: 06/14/2012
• Issued: 09/01/2015
• Est. Priority Date: 06/14/2012
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Ramanandan, Arvind, Brunner, Christopher, Ramachandran, Mahesh, Tyagi, Abhishek, Knoblauch, Daniel, Chari, Murali Ramaswamy
• Primary Examiner(s): Patel, Jay
• Assistant Examiner(s): Aghevli, Reza
• Application Number:
  US13/523,634
• Publication Number:
  US 20130335562A1
• Time in Patent Office: 1,174 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G01S 3/786 the desired condition being maintained automatically
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 7/254 involving subtraction of images

Abstract

A mobile device tracks a relative pose between a camera and a target using Vision aided Inertial Navigation System (VINS), that includes a contribution from inertial sensor measurements and a contribution from vision based measurements. When the mobile device detects movement of the target, the contribution from the inertial sensor measurements to track the relative pose between the camera and the target is reduced or eliminated. Movement of the target may be detected by comparing vision only measurements from captured images and inertia based measurements to determine if a discrepancy exists indicating that the target has moved. Additionally or alternatively, movement of the target may be detected using projections of feature vectors extracted from captured images.

Claims

1. A method comprising:

tracking a relative pose between a camera and a target using a contribution from inertial sensor measurements and a contribution from vision based measurements,

wherein the contribution from the inertial sensor measurements is a weighted contribution and the contribution from the vision based measurements is a weighted contribution;

detecting movement of the target; and

reducing the weighted contribution from the inertial sensor measurements to track the relative pose between the camera and the target when movement of the target is detected.

2. The method of claim 1, wherein reducing the weighted contribution from the inertial sensor measurements comprises eliminating the weighted contribution from the inertial sensor measurements, wherein the relative pose between the camera and the target is tracked using only vision based measurements when the movement of the target is detected.

3. The method of claim 1, wherein detecting movement of the target comprises:

determining at least a portion of a vision based pose of the camera relative to the target;

determining a change in at least a portion of an inertia based pose of the camera based on inertia sensor measurements; and

using the at least the portion of the vision based pose and the change in the at least the portion of the inertia based pose to the detect movement of the target.

4. The method of claim 3, wherein:

the at least the portion of the vision based pose comprises a vision based attitude;

the change in the at least the portion of the inertia based pose comprises a change in an inertia based attitude; and

using the at least the portion of the vision based pose and the change in the at least the portion of the inertia based pose comprises using the vision based attitude and the change in the inertia based attitude to detect the movement of the target.

5. The method of claim 3, further comprising:

determining a change in the at least the portion of the vision based pose;

wherein using the at least the portion of the vision based pose and the change in the at least the portion of the inertia based pose comprises;

determining a difference between the change in the at least the portion of the inertia based pose and the change in the at least the portion of the vision based pose;

comparing the difference to a threshold to determine the target has moved.

6. The method of claim 3, further comprising:

determining at least a portion of an inertia based pose using the change in the at least the portion of the inertia based pose;

wherein using the at least the portion of the vision based pose and the change in the at least the portion of the inertia based pose comprises;

determining a difference between the at least the portion of the vision based pose and the at least the portion of the inertia based pose;

comparing the difference to a threshold to determine the target has moved.

7. The method of claim 1, further comprising determining that the camera is stationary before detecting the movement of the target.

8. The method of claim 1, wherein detecting movement of the target comprises using projections of feature vectors based on the inertial sensor measurements.

9. The method of claim 8, further comprising:

extracting features from captured images

predicting the projections of feature vectors based on the inertial sensor measurements;

comparing the projections of feature vectors to extracted feature vectors extracted from a captured image to determine a difference; and

comparing the difference to a threshold to determine the target has moved.

10. A mobile device comprising:

a camera capable of capturing images of a target;

inertial sensors; and

a processor coupled to receive captured images of the target and coupled to receive signals from the inertial sensors, the processor configured produce vision based measurements using the captured images of the target and inertial sensor measurements using the signals from the inertial sensors, the processor being configured to track a relative pose between the camera and the target using a contribution from the inertial sensor measurements and a contribution from the vision based measurements,

wherein the contribution from the inertial sensor measurements is a weighted contribution and the contribution from the vision based measurements is a weighted contribution,

the processor being further configured to detect movement of the target and to reduce the weighted contribution from the inertial sensor measurements to track the relative pose between the camera and the target when movement of the target is detected.

11. The mobile device of claim 10, wherein the processor is configured to reduce the weighted contribution from the inertial sensor measurements by being configured to eliminate the weighted contribution from the inertial sensor measurements, wherein the processor is configured to track the relative pose between the camera and the target with only the vision based measurements when the movement of the target is detected.

12. The mobile device of claim 10, wherein the processor is configured to detect movement of the target by being configured to:

determine at least a portion of a vision based pose of the camera relative to the target;

determine a change in at least a portion of an inertia based pose of the camera based on inertia sensor measurements; and

use the at least the portion of the vision based pose and the change in the at least the portion of the inertia based pose to the detect movement of the target.

13. The mobile device of claim 12, wherein:

the at least the portion of the vision based pose comprises a vision based attitude;

the change in the at least the portion of the inertia based pose comprises a change in an inertia based attitude; and

the processor is configured to use the at least the portion of the vision based pose and the change in the at least the portion of the inertia based pose by being configured to use the vision based attitude and the change in the inertia based attitude to detect the movement of the target.

14. The mobile device of claim 12, the processor being further configured to determine a change in the at least the portion of the vision based pose, wherein the processor is configured to use the at least the portion of the vision based pose and the change in the at least the portion of the inertia based pose by being configured to:

determine a difference between the change in the at least the portion of the inertia based pose and the change in the at least the portion of the vision based pose;

compare the difference to a threshold to determine the target has moved.

15. The mobile device of claim 12, the processor being further configured to determine at least a portion of an inertia based pose using the change in the at least the portion of the inertia based pose;

wherein the processor is configured to use the at least the portion of the vision based pose and the change in the at least the portion of the inertia based pose by being configured to;

determine a difference between the at least the portion of the vision based pose and the at least the portion of the inertia based pose;

compare the difference to a threshold to determine the target has moved.

16. The mobile device of claim 10, the processor being further configured to determine that the camera is stationary before detecting the movement of the target.

17. The mobile device of claim 10, wherein the processor is configured to detect movement of the target by being configured to use projections of feature vectors based on the inertial sensor measurements.

18. The mobile device of claim 17, wherein the processor is further configured to:

extract features from the captured images

predict the projections of feature vectors based on the inertial sensor measurements;

compare the projections of feature vectors to extracted feature vectors extracted from a captured image to determine a difference; and

compare the difference to a threshold to determine the target has moved.

19. A mobile device comprising:

means for tracking a relative pose between a camera and a target using a contribution from inertial sensor measurements and a contribution from vision based measurements,

wherein the contribution from the inertial sensor measurements is a weighted contribution and the contribution from the vision based measurements is a weighted contribution;

means for detecting movement of the target; and

means for reducing the weighted contribution from the inertial sensor measurements to track the relative pose between the camera and the target when movement of the target is detected.

20. The mobile device of claim 19, wherein the means for reducing the weighted contribution from the inertial sensor measurements eliminates the weighted contribution from the inertial sensor measurements.

21. The mobile device of claim 19, wherein the means for detecting movement of the target comprises:

means for determining at least a portion of a vision based pose of the camera relative to the target;

means for determining a change in at least a portion of an inertia based pose of the camera based on inertia sensor measurements; and

means for using the at least the portion of the vision based pose and the change in the at least the portion of the inertia based pose to the detect movement of the target.

22. The mobile device of claim 19, further comprising means for determining that the camera is stationary before detecting the movement of the target.

23. The mobile device of claim 19, wherein the means for detecting movement of the target comprises means for using projections of feature vectors based on the inertial sensor measurements.

24. A non-transitory computer-readable medium with instructions stored thereon that when executed on an electronic device causes the electronic device to perform:

tracking a relative pose between a camera and a target using a contribution from inertial sensor measurements and a contribution from vision based measurements,

wherein the contribution from the inertial sensor measurements is a weighted contribution and the contribution from the vision based measurements is a weighted contribution;

detecting movement of the target; and

reducing the weighted contribution from the inertial sensor measurements to track the relative pose between the camera and the target when movement of the target is detected.

25. The non-transitory computer-readable medium of claim 24 with instructions stored thereon that when executed on the electronic device causes the electronic device to reduce the weighted contribution from the inertial sensor measurements eliminates the weighted contribution from the inertial sensor measurements.

26. The non-transitory computer-readable medium of claim 24 with the instructions stored thereon that when executed on the electronic device that causes the electronic device to perform detecting movement of the target comprises instructions that when executed on the electronic device causes the electronic device to perform:

determining at least a portion of a vision based pose of the camera relative to the target;

determining a change in at least a portion of an inertia based pose of the camera based on inertia sensor measurements; and

using the at least the portion of the vision based pose and the change in the at least the portion of the inertia based pose to the detect movement of the target.

27. The non-transitory computer-readable medium of claim 24, with instructions stored thereon that when executed on the electronic device causes the electronic device to further determine that the camera is stationary before detecting the movement of the target.

28. The non-transitory computer-readable medium of claim 24, with instructions stored thereon that when executed on the electronic device causes the electronic device to detect movement of the target wherein the detection comprises using projections of feature vectors based on the inertial sensor measurements.