ADAPTIVE ESTIMATION OF FRAME TIME STAMP LATENCY

• US 20130335554A1
• Filed: 06/14/2012
• Published: 12/19/2013
• Est. Priority Date: 06/14/2012
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Brunner, Christopher, Ramachandran, Mahesh, Ramanandan, Arvind, Chari, Murali Ramaswamy
• Granted Patent: US 9,116,001 B2
• Time in Patent Office: Days
• US Class Current: 348/135
• CPC Class Codes: H04N 17/002 for television cameras
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors

Abstract

A mobile device compensates for a lack of a time stamp when an image frame is captured by estimating the frame time stamp latency. The mobile device captures images frames and time stamps each frame after the frame time stamp latency. A vision based rotation is determined from a pair of frames. A plurality of inertia based rotations is measured using time stamped signals from an inertial sensor in the mobile device based on different possible delays between time stamping each frame and time stamps on the signals from the inertial sensors. The determined rotations may be about the camera'"'"'s optical axis. The vision based rotation is compared to the plurality of inertia based rotations to determine an estimated frame time stamp latency, which is used to correct the frame time stamp latency when time stamping subsequently captured frames. A median latency determined using different frame pairs may be used.

Claims

1. A method comprising:

capturing frames with a camera in a mobile device;

time stamping each frame after a frame time stamp latency from when the frame is captured by the camera;

determining a vision based rotation of the camera using a pair of frames;

measuring a plurality of inertia based rotations using signals from an inertial sensor in the mobile device, wherein the signals are time stamped when produced by the inertial sensor and the plurality of inertia based rotations are based on different possible delays between time stamping each frame and time stamps on the signals from the inertial sensor;

comparing the vision based rotation to the plurality of inertia based rotations to determine an estimated frame time stamp latency; and

using the estimated frame time stamp latency to correct the frame time stamp latency in subsequently captured frames.

2. The method of claim 1, wherein the vision based rotation is determined and the plurality of inertia based rotations is measured about a single axis normal to a lens of the camera.

3. The method of claim 1, wherein:

measuring the plurality of inertia based rotations comprises integrating signals from the inertial sensor between times time stamped on each frame, the times being shifted with the different possible delays; and

comparing the vision based rotation to the plurality of inertia based rotations to determine the estimated frame time stamp latency comprises;

comparing each of the plurality of inertia based rotations to the vision based rotation to generate a plurality of residuals;

using the plurality of residuals to determine the estimated frame time stamp latency.

4. The method of claim 3, wherein using the plurality of residuals to determine the estimated frame time stamp latency comprises interpolating the plurality of residuals to determine the estimated frame time stamp latency.

5. The method of claim 1, further comprising:

determining a plurality of estimated frame time stamp latencies, wherein each estimated frame time stamp latency in the plurality of estimated frame time stamp latencies is determined for a different pair of frames; and

calculating a median latency for the plurality of estimated frame time stamp latencies;

wherein using the estimated frame time stamp latency to correct the frame time stamp latency comprises using the median latency to correct the frame time stamp latency in subsequently captured frames.

6. The method of claim 5, further comprising determining an accuracy of poses generated by each frame in the different pair of frames, wherein determining the plurality of estimated frame time stamp latencies uses only pairs for frames with the accuracy greater than a threshold.

7. The method of claim 5, further comprising:

determining if a frame rate for the camera changes; and

determining a new plurality of estimated frame time stamp latencies for pairs of frames captured after a frame rate change;

calculating a new median latency for the new plurality of estimated frame time stamp latencies; and

using the new median latency to correct the frame time stamp latency in subsequently captured frames.

8. The method of claim 7, wherein determining if the frame rate for the camera changes comprises:

generating a first time difference by comparing frame time stamps from a first pair of frames;

generating a second time difference by comparing frame time stamps from a second pair of frames;

comparing the first time difference and the second time difference to determine if the frame rate has changed.

9. The method of claim 8, wherein the first time difference and the second time difference are rounded prior to being compared.

10. The method of claim 1, wherein the inertial sensor comprises a gyroscope.

11. A mobile device comprising:

a camera;

an inertial sensor; and

a processor coupled to receive frames captured by the camera and coupled to sample signals from the inertial sensor, wherein the frames are time stamped after a frame time stamp latency from when the frame is captured by the camera and the signals from the inertial sensor are time stamped when produced by the inertial sensor, the processor configured to determine a vision based rotation of the camera using a pair of frames, measure a plurality of inertia based rotations using the signals sampled from the inertial sensor, the plurality of inertia based rotations being based on different possible delays between each frame being time stamped and time stamps on the signals from the inertial sensor, compare the vision based rotation to the plurality of inertia based rotations to determine an estimated frame time stamp latency, and use the estimated frame time stamp latency to correct the frame time stamp latency in subsequently captured frames.

12. The mobile device of claim 11, wherein the vision based rotation is determined and the plurality of inertia based rotations is measured about a single axis normal to a lens of the camera.

13. The mobile device of claim 11, wherein the processor is configured to measure the plurality of inertia based rotations using the signals sampled from the inertial sensor by being configured to integrate the signals between times time stamped on each frame, the times shifted with the different possible delays, and the processor is configured to compare the vision based rotation to the plurality of inertia based rotations to determine the estimated frame time stamp latency by being configured to compare each of the plurality of inertia based rotations to the vision based rotation to generate a plurality of residuals, and to use the plurality of residuals to determine the estimated frame time stamp latency.

14. The mobile device of claim 13, wherein the processor is configured to use the plurality of residuals to determine the estimated frame time stamp latency by being configured to interpolate the plurality of residuals to determine the estimated frame time stamp latency.

15. The mobile device of claim 11, wherein the processor is further configured to determine a plurality of estimated frame time stamp latencies, wherein each estimated frame time stamp latency in the plurality of estimated frame time stamp latencies is determined for a different pair of frames, and calculate a median latency for the plurality of estimated frame time stamp latencies, wherein the processor is configured to use the median latency to correct the frame time stamp latency in subsequently captured frames.

16. The mobile device of claim 15, wherein the processor is further configured to determine an accuracy of poses generated by each frame in the different pair of frames, wherein the processor determines the plurality of estimated frame time stamp latencies with only pairs for frames with the accuracy greater than a threshold.

17. The mobile device of claim 15, wherein the processor is further configured to determine if a frame rate for the camera changes, determine a new plurality of estimated frame time stamp latencies for pairs of frames captured after a frame rate change, calculate a new median latency for the new plurality of estimated frame time stamp latencies;

and use the new median latency to correct the frame time stamp latency in subsequently captured frames.

18. The mobile device of claim 17, wherein the processor is configured to determine if the frame rate for the camera changes by being configured to generate a first time difference by comparing frame time stamps from a first pair of frames, generate a second time difference by comparing frame time stamps from a second pair of frames, and compare the first time difference and the second time difference to determine if the frame rate has changed.

19. The mobile device of claim 18, wherein the processor is configured to round the first time difference and the second time difference prior to comparing the first time difference and the second time difference.

20. The mobile device of claim 11, wherein the inertial sensor comprises a gyroscope.

21. A mobile device comprising:

means for capturing frames of images or video;

means for time stamping each frame after a frame time stamp latency from when the frame is captured by the camera;

means for determining a vision based rotation of the camera using a pair of frames;

means for measuring a plurality of inertia based rotations using signals from an inertial sensor in the mobile device, wherein the signals are time stamped when produced by the inertial sensor and the plurality of inertia based rotations are based on different possible delays between time stamping each frame and time stamps on the signals from the inertial sensor;

means for comparing the vision based rotation to the plurality of inertia based rotations to determine an estimated frame time stamp latency; and

means for using the estimated frame time stamp latency to correct the frame time stamp latency in subsequently captured frames.

22. The mobile device of claim 21, wherein the vision based rotation is determined and the plurality of inertia based rotations is measured about a single axis normal to a lens of the camera.

23. The mobile device of claim 21, wherein

the means for measuring the plurality of inertia based rotations integrates signals between times time stamped on each frame, the times shifted with the different possible delays;

and

the means for comparing the vision based rotation to the plurality of inertia based rotations to determine the estimated frame time stamp latency compares each of the plurality of inertia based rotations to the vision based rotation to generate a plurality of residuals, and uses the plurality of residuals to determine the estimated frame time stamp latency.

24. The mobile device of claim 21, further comprising:

means for determining a plurality of estimated frame time stamp latencies, wherein each estimated frame time stamp latency in the plurality of estimated frame time stamp latencies is determined for a different pair of frames; and

means for calculating a median latency of the plurality of estimated frame time stamp latencies;

wherein the means for using the estimated frame time stamp latency to correct the frame time stamp latency uses the median latency.

25. The mobile device of claim 24, further comprising:

means for determining if a frame rate for the camera changes; and

means for determining a new plurality of estimated frame time stamp latencies for pairs of frames captured after a frame rate change;

means for calculating a new median of the new plurality of estimated frame time stamp latencies; and

wherein the means for using the estimated frame time stamp latency to correct the frame time stamp latency uses the new median.

26. A non-transitory computer-readable medium including program code stored thereon, comprising:

program code to determine a vision based rotation of a camera using a pair of frames captured by the camera, wherein each frame is time stamped after a frame time stamp latency from when the frame is captured by the camera;

program code to measure a plurality of inertia based rotations using signals sample from an inertial sensor, wherein the signals are time stamped when produced by the inertial sensor and the plurality of inertia based rotations are based on different possible delays between time stamping each frame and time stamps on the signals from the inertial sensor;

program code to compare the vision based rotation to the plurality of inertia based rotations to determine an estimated frame time stamp latency; and

program code to use the estimated frame time stamp latency to correct the frame time stamp latency in subsequently captured frames.

27. The non-transitory computer-readable medium of claim 26, wherein the vision based rotation is determined and the plurality of inertia based rotations is measured about a single axis normal to a lens of the camera.

28. The non-transitory computer-readable medium of claim 26, wherein:

program code to measure the plurality of inertia based rotations comprises program code to integrate the signals sampled from the inertial sensor between times time stamped on each frame, the times being shifted by the different possible delays; and

program code to compare the vision based rotation to the plurality of inertia based rotations to determine the estimated frame time stamp latency comprises;

program code to compare each of the plurality of inertia based rotations to the vision based rotation to generate a plurality of residuals;

program code to use the plurality of residuals to determine the estimated frame time stamp latency.

29. The non-transitory computer-readable medium of claim 26, further comprising:

program code to determine a plurality of estimated frame time stamp latencies, wherein each estimated frame time stamp latency in the plurality of estimated frame time stamp latencies is determined for a different pair of frames; and

program code to calculate a median latency of the plurality of estimated frame time stamp latencies;

wherein the median latency is used to correct the frame time stamp latency in subsequently captured frames.

30. The non-transitory computer-readable medium of claim 29,further comprising:

program code to determine if a frame rate for the camera changes; and

program code to determine a new plurality of estimated frame time stamp latencies for pairs of frames captured after a frame rate change;

program code to calculate a new median of the new plurality of estimated frame time stamp latencies; and

wherein the new median is used to correct the frame time stamp latency in subsequently captured frames.