Adaptive estimation of frame time stamp latency

• US 9,116,001 B2
• Filed: 06/14/2012
• Issued: 08/25/2015
• Est. Priority Date: 06/14/2012
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Brunner, Christopher, Ramachandran, Mahesh, Ramanandan, Arvind, Chari, Murali Ramaswamy
• Primary Examiner(s): Czekaj, Dave
• Assistant Examiner(s): PHAM, NAM D
• Application Number:
  US13/523,638
• Publication Number:
  US 20130335554A1
• Time in Patent Office: 1,167 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: H04N 17/002 for television cameras
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors

Abstract

A mobile device compensates for a lack of a time stamp when an image frame is captured by estimating the frame time stamp latency. The mobile device captures images frames and time stamps each frame after the frame time stamp latency. A vision based rotation is determined from a pair of frames. A plurality of inertia based rotations is measured using time stamped signals from an inertial sensor in the mobile device based on different possible delays between time stamping each frame and time stamps on the signals from the inertial sensors. The determined rotations may be about the camera'"'"'s optical axis. The vision based rotation is compared to the plurality of inertia based rotations to determine an estimated frame time stamp latency, which is used to correct the frame time stamp latency when time stamping subsequently captured frames. A median latency determined using different frame pairs may be used.

Claims

1. A method comprising:

capturing frames with a camera in a mobile device;

time stamping each frame after a latency period from when the frame is captured by the camera;

determining a vision based rotation of the camera using a pair of frames;

measuring a plurality of inertia based rotations using signals from an inertial sensor in the mobile device, wherein the signals are time stamped when produced by the inertial sensor and wherein measuring the plurality of inertia based rotations comprises integrating signals from the inertial sensor between start and stop times that correspond to times time stamped on each frame in the pair of frames and that are shifted with different delays;

determining an estimate of the latency period based on differences between the vision based rotation and each of the plurality of inertia based rotations; and

using the estimate of the latency period to correct for the latency period in time stamping of subsequently captured frames.

2. The method of claim 1, wherein the vision based rotation is determined and the plurality of inertia based rotations is measured about a single axis normal to a lens of the camera.

3. The method of claim 1, wherein:

determining the estimate of the latency period based on the differences between the vision based rotation and each of the plurality of inertia based rotations comprises;

comparing each of the plurality of inertia based rotations to the vision based rotation to generate a plurality of residuals;

using the plurality of residuals to determine the estimate of the latency period.

4. The method of claim 3, wherein using the plurality of residuals to determine the estimate of the latency period comprises interpolating the plurality of residuals to determine the estimate of the latency period.

5. The method of claim 1, further comprising:

determining a plurality of estimates of the latency period, wherein each estimate of the latency period in the plurality of estimates of the latency period is determined for a different pair of frames; and

calculating a median latency for the plurality of estimates of the latency period;

wherein using the estimate of the latency period to correct for the latency period in time stamping of subsequently captured frames comprises using the median latency to correct for the latency period in time stamping of subsequently captured frames.

6. The method of claim 5, further comprising determining an accuracy of poses generated by each frame in the different pair of frames, wherein determining the plurality of estimates of the latency period uses only pairs for frames with the accuracy greater than a threshold.

7. The method of claim 5, further comprising:

determining if a frame rate for the camera changes; and

determining a new plurality of estimates of the latency period for pairs of frames captured after a frame rate change;

calculating a new median latency for the new plurality of estimates of the latency period; and

using the new median latency to correct for the latency period in time stamping of subsequently captured frames.

8. The method of claim 7, wherein determining if the frame rate for the camera changes comprises:

generating a first time difference by comparing frame time stamps from a first pair of frames;

generating a second time difference by comparing frame time stamps from a second pair of frames;

comparing the first time difference and the second time difference to determine if the frame rate has changed.

9. The method of claim 8, wherein the first time difference and the second time difference are rounded prior to being compared.

10. The method of claim 1, wherein the inertial sensor comprises a gyroscope.

11. A mobile device comprising:

a camera;

an inertial sensor; and

a processor coupled to receive frames captured by the camera and coupled to sample signals from the inertial sensor, wherein the frames are time stamped after a latency period from when the frame is captured by the camera and the signals from the inertial sensor are time stamped when produced by the inertial sensor, the processor configured to determine a vision based rotation of the camera using a pair of frames, measure a plurality of inertia based rotations using the signals sampled from the inertial sensor, wherein the plurality of inertia based rotations being measured by integrating signals from the inertial sensor between start and stop times that correspond to times time stamped on each frame in the pair of frames and that are shifted with different delays, determine an estimate of the latency period based on differences between the vision based rotation and each of the plurality of inertia based rotations, and use the estimate of the latency period to correct for the latency period in time stamping of subsequently captured frames.

12. The mobile device of claim 11, wherein the vision based rotation is determined and the plurality of inertia based rotations is measured about a single axis normal to a lens of the camera.

13. The mobile device of claim 11, wherein the processor is configured to determine the estimate of the latency period based on the differences between the vision based rotation and each of the plurality of inertia based rotations by being configured to compare each of the plurality of inertia based rotations to the vision based rotation to generate a plurality of residuals, and to use the plurality of residuals to determine the estimate of the latency period.

14. The mobile device of claim 13, wherein the processor is configured to use the plurality of residuals to determine the estimate of the latency period by being configured to interpolate the plurality of residuals to determine the estimate of the latency period.

15. The mobile device of claim 11, wherein the processor is further configured to determine a plurality of estimates of the latency period, wherein each estimate of the latency period in the plurality of estimates of the latency period is determined for a different pair of frames, and calculate a median latency for the plurality of estimates of the latency period, wherein the processor is configured to use the median latency to correct the latency period in subsequently captured frames.

16. The mobile device of claim 15, wherein the processor is further configured to determine an accuracy of poses generated by each frame in the different pair of frames, wherein the processor determines the plurality of estimates of the latency period with only pairs for frames with the accuracy greater than a threshold.

17. The mobile device of claim 15, wherein the processor is further configured to determine if a frame rate for the camera changes, determine a new plurality of estimates of the latency period for pairs of frames captured after a frame rate change, calculate a new median latency for the new plurality of estimates of the latency period;

and use the new median latency to correct the latency period in subsequently captured frames.

18. The mobile device of claim 17, wherein the processor is configured to determine if the frame rate for the camera changes by being configured to generate a first time difference by comparing frame time stamps from a first pair of frames, generate a second time difference by comparing frame time stamps from a second pair of frames, and compare the first time difference and the second time difference to determine if the frame rate has changed.

19. The mobile device of claim 18, wherein the processor is configured to round the first time difference and the second time difference prior to comparing the first time difference and the second time difference.

20. The mobile device of claim 11, wherein the inertial sensor comprises a gyroscope.

21. A mobile device comprising:

means for capturing frames of images or video;

means for time stamping each frame after a latency period from when the frame is captured;

means for determining a vision based rotation using a pair of frames;

means for measuring a plurality of inertia based rotations using signals from an inertial sensor in the mobile device, wherein the signals are time stamped when produced by the inertial sensor and wherein measuring the plurality of inertia based rotations comprises integrating signals from the inertial sensor between start and stop times that correspond to times time stamped on each frame in the pair of frames and that are shifted with different delays;

means for determining an estimate of the latency period based on differences between the vision based rotation and each of the plurality of inertia based rotations; and

means for using the estimate of the latency period to correct for the latency period in subsequently captured frames.

22. The mobile device of claim 21, wherein the vision based rotation is determined and the plurality of inertia based rotations is measured about a single axis normal to a lens of the means for capturing frames.

23. The mobile device of claim 21, wherein

the means for determining the estimate of the latency period based on the differences between the vision based rotation and each of the plurality of inertia based rotations compares each of the plurality of inertia based rotations to the vision based rotation to generate a plurality of residuals, and uses the plurality of residuals to determine the estimate of the latency period.

24. The mobile device of claim 21, further comprising:

means for determining a plurality of estimates of the latency period, wherein each estimate of the latency period in the plurality of estimates of the latency period is determined for a different pair of frames; and

means for calculating a median latency of the plurality of estimates of the latency period;

wherein the means for using the estimate of the latency period to correct the latency period uses the median latency.

25. The mobile device of claim 24, further comprising:

means for determining if a frame rate changes; and

means for determining a new plurality of estimates of the latency period for pairs of frames captured after a frame rate change;

means for calculating a new median of the new plurality of estimates of the latency period; and

wherein the means for using the estimate of the latency period to correct the latency period uses the new median.

26. A non-transitory computer-readable medium including program code stored thereon, comprising:

program code to determine a vision based rotation of a camera using a pair of frames captured by the camera, wherein each frame is time stamped after a latency period from when the frame is captured by the camera;

program code to measure a plurality of inertia based rotations using signals sample from an inertial sensor, wherein the signals are time stamped when produced by the inertial sensor and the plurality of inertia based rotations are measured by integrating signals from the inertial sensor between start and stop times that correspond to times time stamped on each frame in the pair of frames and that are shifted with different delays;

program code to determine an estimate of the latency period based on differences between the vision based rotation and each of the plurality of inertia based rotations; and

program code to use the estimate of the latency period to correct the latency period in subsequently captured frames.

27. The non-transitory computer-readable medium of claim 26, wherein the vision based rotation is determined and the plurality of inertia based rotations is measured about a single axis normal to a lens of the camera.

28. The non-transitory computer-readable medium of claim 26, wherein:

the program code to determine the estimate of the latency period based on the differences between the vision based rotation and each of the plurality of inertia based rotations comprises;

program code to compare each of the plurality of inertia based rotations to the vision based rotation to generate a plurality of residuals;

program code to use the plurality of residuals to determine the estimate of the latency period.

29. The non-transitory computer-readable medium of claim 26, further comprising:

program code to determine a plurality of estimates of the latency period, wherein each estimate of the latency period in the plurality of estimates of the latency period is determined for a different pair of frames; and

program code to calculate a median latency of the plurality of estimates of the latency period;

wherein the median latency is used to correct the latency period in subsequently captured frames.

30. The non-transitory computer-readable medium of claim 29, further comprising:

program code to determine if a frame rate for the camera changes; and

program code to determine a new plurality of estimates of the latency period for pairs of frames captured after a frame rate change;

program code to calculate a new median of the new plurality of estimates of the latency period; and

wherein the new median is used to correct the latency period in subsequently captured frames.