POWER EFFICIENT USE OF A DEPTH SENSOR ON A MOBILE DEVICE

• US 20150146926A1
• Filed: 11/24/2014
• Published: 05/28/2015
• Est. Priority Date: 11/25/2013
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: RAMACHANDRAN, Mahesh, SADASIVAM, Shankar, SWAMINATHAN, Ashwin
• Granted Patent: US 9,336,440 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: G06F 1/1694 the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer
• CPC Class Codes: G06F 1/325 Power saving in peripheral device
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 7/277 involving stochastic approaches, e.g. using Kalman filters
• CPC Class Codes: H04N 13/25 using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations

Abstract

Systems, apparatus and methods in a mobile device to enable and disable a depth sensor for tracking pose of the mobile device are presented. A mobile device relaying on a camera without a depth sensor may provide inadequate pose estimates, for example, in low light situations. A mobile device with a depth sensor uses substantial power when the depth sensor is enabled. Embodiments described herein enable a depth sensor only when images are expected to be inadequate, for example, accelerating or moving too fast, when inertial sensor measurements are too noisy, light levels are too low or high, an image is too blurry, or a rate of images is too slow. By only using a depth sensor when images are expected to be inadequate, battery power in the mobile device may be conserved and pose estimations may still be maintained.

Claims

1. A method in a mobile device for tracking pose of the mobile device, the method comprising:

receiving, from an image capture device, a plurality of images;

computing a pose based at least in part on one of the plurality of images;

sending the pose to a pose tracking filter module;

comparing a first measurement to a first threshold to determine that the first measurement exceeds the first threshold;

enabling a depth sensor based, at least in part, on comparing the first measurement;

receiving a first depth map;

computing a depth sensor pose based in part on the first depth map; and

providing the depth sensor pose to the pose tracking filter module.

2. The method of claim 1, further comprising disabling the depth sensor.

3. The method of claim 2, wherein disabling the depth sensor comprises:

comparing a second measurement to the first threshold to determine that the second measurement no longer exceeds the first threshold; and

disabling the depth sensor based on comparing the second measurement.

4. The method of claim 1, further comprising:

capturing a second depth map;

wherein computing the depth sensor pose based in part on the first depth map comprises computing the depth sensor pose from the first depth map and the second depth map.

5. The method of claim 1, wherein the first measurement comprises a rate of angular velocity or a rate of angular acceleration obtained from inertial sensor measurements.

6. The method of claim 1, wherein the first measurement comprises a noise level of inertial sensor measurements.

7. The method of claim 1, wherein the first measurement comprises a light level.

8. The method of claim 1, wherein the first measurement comprises an indication of blurriness.

9. The method of claim 1, wherein the first measurement comprises a frame rate of images.

10. The method of claim 1, wherein the pose tracking filter module comprises an extended Kalman filter (EKF).

11. The method of claim 1, wherein the pose tracking filter module comprises a particle filter.

12. The method of claim 1, further comprising computing an estimated pose from the pose tracking filter module.

13. A mobile device for tracking pose, the mobile device comprising:

an image capture device configured to capture a plurality of images;

a pose module coupled to the image capture device and configured to compute a pose based at least in part on one of the plurality of images;

a depth sensor configured to send a first depth map;

a comparator module configured to compare a first measurement to a first threshold to determine that the first measurement exceed the first threshold;

a depth sensor trigger module coupled to the depth sensor and the comparator module, and configured to enable the depth sensor based, at least in part, on comparing the first measurement;

a depth sensor pose module coupled to the depth sensor and configured to compute a depth sensor pose based in part on the first depth map; and

a pose tracking filter module;

coupled to the pose module and the depth sensor pose module;

configured to receive the pose and the depth sensor pose; and

provide an estimated pose.

14. The mobile device of claim 13, further comprising an inertial sensor, wherein the first measurement comprises a measurement from the inertial sensor.

15. The mobile device of claim 13, further comprising a light sensor, wherein the first measurement comprises a measurement from the light sensor.

16. The mobile device of claim 13, wherein the first measurement comprises an indication of blur from the image capture device.

17. The mobile device of claim 13, wherein the first measurement comprises a frame rate of images from the image capture device.

18. The mobile device of claim 13, further comprising a processor and memory couple to the image capture device and the depth sensor, and comprising code for:

the comparator module;

the depth sensor trigger module;

the depth sensor pose module; and

the pose tracking filter module.

19. A mobile device for tracking pose of the mobile device, the mobile device comprising:

means for receiving, from an image capture device, a plurality of images;

means for computing a pose based at least in part on one of the plurality of images;

means for sending the pose to a pose tracking filter module;

means for comparing a first measurement to a first threshold to determine that the first measurement exceeds the first threshold;

means for enabling a depth sensor based, at least in part, on comparing the first measurement;

means for receiving a first depth map;

means for computing a depth sensor pose based in part on the first depth map; and

means for providing the depth sensor pose to the pose tracking filter module.

20. The mobile device of claim 19, further comprising means for disabling the depth sensor comprises:

means for comparing a second measurement to the first threshold to determine that the second measurement exceeds the first threshold; and

means for disabling the depth sensor based on comparing the second measurement.

21. The mobile device of claim 19, wherein the first measurement comprises at least one of:

a rate of linear acceleration from inertial sensor measurements;

a rate of angular acceleration from inertial sensor measurements;

a noise level of inertial sensor measurements;

a light level;

an indication of blurriness;

or

a frame rate of images.

22. The mobile device of claim 19, further comprising means for computing an estimated pose from the pose tracking filter module.

23. A non-transient computer-readable storage medium including program code stored thereon, comprising program code for a mobile device for tracking pose of the mobile device, the program code comprising code to:

receive, from an image capture device, a plurality of images;

compute a pose based at least in part on one of the plurality of images;

send the pose to a pose tracking filter module;

compare a first measurement to a first threshold to determine that the first measurement exceed the first threshold;

enable a depth sensor based, at least in part, on comparing the first measurement;

receive a first depth map;

compute a depth sensor pose based in part on the first depth map; and

provide the depth sensor pose to the pose tracking filter module.

24. The non-transient computer-readable storage medium of claim 23, wherein the first measurement comprises at least one of:

a rate of linear acceleration from inertial sensor measurements;

a rate of angular acceleration from inertial sensor measurements;

a noise level of inertial sensor measurements;

a light level;

an indication of blurriness;

or

a frame rate of images.

25. The non-transient computer-readable storage medium of claim 23, wherein the program code further comprises code to compute an estimated pose from the pose tracking filter module.

26. The non-transient computer-readable storage medium of claim 23, wherein the program code further comprises code to:

compare a second measurement to the first threshold to set a second comparison; and

disable the depth sensor based on the second comparison.