Synchronizing data streams

• US 10,250,868 B1
• Filed: 03/29/2017
• Issued: 04/02/2019
• Est. Priority Date: 03/29/2017
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Arnold, Brad Nathaniel, Saran, Sajesh Kumar, Burzawa, Scot, Zakian, Christopher Samuel
• Primary Examiner(s): Senfi, Behrooz M
• Application Number:
  US15/473,146
• Time in Patent Office: 734 Days
• CPC Class Codes: G11B 27/19 by using information detectable on the record carrier
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/257 Colour aspects
• CPC Class Codes: H04N 13/296 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 23/10 for generating image signals from different wavelengths
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules

Abstract

This disclosure describes techniques for synchronizing data streams. In some instances, a computing device couples to multiple sensors, such as cameras, and applies accurate timestamp information to the individual frames of sensor data from the independent sensors. After aligning these data streams by applying these accurate timestamps, the computing device may, in some instances, encode and transmit these timestamped data streams to one or more entities for further processing. In one example, a first camera (e.g., a depth camera configured to generate a depth map) may capture images of an environment, as may a second camera (e.g., an Red-Green-Blue (RGB) camera configured to generate color images). The resulting images may be temporally aligned with one another via the timestamping, and the resulting aligned images from both the depth sensor and the RGB camera may be used to create a three-dimensional (3D) model of the environment.

Claims

1. A system comprising:

a pulse-width-modulation (PWM) unit configured to generate pulses;

a camera, coupled to the PWM unit and configured to acquire respective images of an environment in response to detecting respective pulses generated by the PWM unit;

one or more processors; and

one or more computer-readable media storing computer-executable instructions that, when executed on the one or more processors, cause the one or more processors to perform acts comprising;

determining a nominal frequency at which to operate the PWM unit;

calculating a value of a first expected timestamp based at least in part on the nominal frequency and a current time;

calculating a value of a second expected timestamp based at least in part on the nominal frequency and at least one of the first expected timestamp or the current time;

configuring the PWM unit to generate pulses at the nominal frequency;

receiving a value of a first recorded timestamp, the value of the first recorded timestamp corresponding to a first time at which a first pulse was generated by the PWM unit;

comparing the value of the first expected timestamp to the value of the first recorded timestamp to generate a first error;

calculating a first amount to adjust the nominal frequency based at least in part on the first error;

configuring the PWM unit to generate pulses at a first adjusted frequency, the first adjusted frequency comprising the nominal frequency adjusted by the first amount;

receiving a value of a second recorded timestamp, the value of the second recorded timestamp corresponding to a second time at which a second pulse was generated by PWM unit;

comparing the value of the second expected timestamp to the value of the second recorded timestamp to generate a second error;

calculating a second amount to adjust nominal frequency based at least in part on the second error; and

configuring the PWM unit to generate pulses at a second adjusted frequency, the second adjusted frequency comprising the nominal frequency adjusted by the second amount.

2. The system as recited in claim 1, the acts further comprising:

storing at least one of the value of the first recorded timestamp or the value of the first expected timestamp in a timestamp queue;

storing at least one of the value of the second recorded timestamp or the value of the second expected timestamp in the timestamp queue;

receiving a first image of the environment acquired by the camera;

receiving a second image of the environment acquired by the camera;

reading the at least one of the value of the first recorded timestamp or the value of the first expected timestamp from the timestamp queue;

reading the at least one of the value of the second recorded timestamp or the value of the second expected timestamp from the timestamp queue;

adding, to the first image, metadata based on the at least one of the value of the first recorded timestamp or the value of the first expected timestamp; and

adding, to the second image, metadata based on the at least one of the value of the second recorded timestamp or the value of the second expected timestamp.

3. The system as recited in claim 1, wherein:

the calculating the second amount to adjust the nominal frequency comprises calculating the second amount by summing;

a first gain applied to a difference between the value of the second expected timestamp and the value of the second recorded timestamp; and

a second gain applied to a sum of;

(i) the difference between the value of the second expected timestamp and the value of the second recorded timestamp, and (ii) a difference between the value of the first expected timestamp and the value of the first recorded timestamp.

4. The system as recited in claim 1, the acts further comprising:

calculating a value of a third expected timestamp;

receiving a value of a third recorded timestamp, the value of the third recorded timestamp corresponding to a third time at which a third pulse was generated by PWM unit;

storing a value of an intentional offset, the value of the intentional offset comprising an amount of time to delay acquisition of one or more images by the camera;

comparing;

(i) a sum of the value of the third expected timestamp and the value of the intentional offset to (ii) the value of the third recorded timestamp to generate a third error;

calculating a third amount to adjust the nominal frequency based at least in part on the third error; and

configuring the PWM unit to generate pulses at a third adjusted frequency, the third adjusted frequency comprising the nominal frequency adjusted by the third amount.

5. A method comprising:

configuring a trigger unit to generate pulses at a first frequency;

calculating a first expected timestamp based at least in part on the first frequency;

storing first data acquired by a sensor, the sensor acquiring the first data in response to receiving a first pulse;

storing a first recorded timestamp corresponding to a first time at which the first pulse was generated;

associating at least one of the first recorded timestamp or the first expected timestamp with the first data;

analyzing the first recorded timestamp with respect to at least the first expected timestamp;

determining a first error based at least in part on the analyzing; and

configuring the trigger unit to generate pulses at a second frequency based at least in part on the first frequency and the first error.

6. The method as recited in claim 5, wherein:

configuring the trigger unit to generate pulses at the first frequency comprises writing a first value corresponding to the first frequency to a register of a pulse-width-modulation (PWM) unit; and

configuring the trigger unit to generate pulses at the second frequency comprises writing a second value corresponding to the second frequency to the register of the PWM unit.

7. The method as recited in claim 5, further comprising:

calculating a first period, the first period based at least in part on the first frequency; and

determining a time indicated by a clock of a computing device coupled to the trigger unit;

and wherein the calculating the first expected timestamp is based at least in part on the first period and the time indicated by the clock.

8. The method as recited in claim 5, further comprising:

calculating an amount to adjust the first frequency based at least in part on the first error; and

calculating the second frequency, the second frequency comprising the first frequency adjusted by the amount.

9. The method as recited in claim 5, wherein:

the analyzing comprises comparing the first recorded timestamp to a sum of a time indicated by the first expected timestamp and a time indicated by an intentional offset;

the determining the first error comprises determining a difference between the first recorded timestamp and the sum;

and the method further comprises;

calculating an amount to adjust the first frequency based at least in part on the first error; and

calculating the second frequency, the second frequency comprising the first frequency adjusted by the amount.

10. The method as recited in claim 5, further comprising:

calculating a second expected timestamp;

storing second data acquired by the sensor, the sensor acquiring the second data in response to receiving a second pulse;

storing a second recorded timestamp corresponding to a second time at which the second pulse was generated;

associating at least one of the second recorded timestamp or the second expected timestamp with the second data;

analyzing the second recorded timestamp with respect to at least the second expected timestamp;

determining a second error based at least in part on the analyzing of the second recorded timestamp with respect to at least the second expected timestamp; and

configuring the trigger unit to generate pulses at a third frequency based at least in part on the first frequency and the second error.

11. The method as recited in claim 10, further comprising:

calculating an amount to adjust the first frequency based at least in part on the second error; and

calculating the third frequency, the third frequency comprising the first frequency adjusted by the amount.

12. The method as recited in claim 11, wherein the calculating the amount to adjust the first frequency comprises:

calculating a proportional error (P_E) based at least in part on the second error;

calculating an integral error (P_I) based at least in part on the first error and the second error; and

calculating a weighted P_E comprising a first gain applied to the P_E;

calculating a weighted P_I comprising a second gain applied to the P_I; and

summing the weighted P_E and the weighted P_I to determine a total error (P_T).

13. The method as recited in claim 12, wherein the calculating the amount to adjust the first frequency further comprises:

comparing P_T to a maximum adjustment; and

determining that P_T is less than the maximum adjustment; and

and wherein P_T comprises the amount to adjust the first frequency based at least in part on the determining that P_T is less than the maximum adjustment.

14. A system comprising:

a sensor;

a trigger unit;

one or more processors; and

one or more computer-readable media storing computer-executable instructions that, when executed on the one or more processors, cause the one or more processors to perform acts comprising;

configuring the trigger unit to generate pulses at a first frequency;

calculating a first expected timestamp based at least in part on the first frequency;

storing first data acquired by the sensor, the sensor acquiring the first data in response to receiving a first pulse;

storing a first recorded timestamp corresponding to a first time at which the first pulse was generated;

associating at least one of the first recorded timestamp or the first expected timestamp with the first data;

analyzing the first recorded timestamp with respect to at least the first expected timestamp;

determining a first error based at least in part on the analyzing; and

configuring the trigger unit to generate pulses at a second frequency based at least in part on the first frequency and the first error.

15. The system as recited in claim 14, the acts further comprising:

calculating a first period, the first period based at least in part on the first frequency; and

determining a time indicated by a clock of a computing device coupled to the trigger unit;

and wherein the calculating the first expected timestamp is based at least in part on the first period and the time indicated by the clock.

16. The system as recited in claim 14, the acts further comprising:

calculating an amount to adjust the first frequency based at least in part on the first error; and

calculating the second frequency, the second frequency comprising the first frequency adjusted by the amount.

17. The system as recited in claim 14, wherein:

the analyzing comprises comparing the first recorded timestamp to a sum of a time indicated by the first expected timestamp and a time indicated by an intentional offset;

the determining the first error comprises determining the first error as a difference between the first recorded timestamp and the sum;

and the acts further comprise;

calculating an amount to adjust the first frequency based at least in part on the first error; and

calculating the second frequency, the second frequency comprising the first frequency adjusted by the amount.

18. The system as recited in claim 14, the acts further comprising:

calculating a second expected timestamp;

storing second data acquired by the sensor, the sensor acquiring the second data in response to receiving a second pulse;

storing a second recorded timestamp corresponding to a second time at which the second pulse was generated;

associating at least one of the second recorded timestamp or the second expected timestamp with the second data;

analyzing the second recorded timestamp with respect to at least the second expected timestamp;

determining a second error based at least in part on the analyzing of the second recorded timestamp with respect to at least the second expected timestamp; and

configuring the trigger unit to generate pulses at a third frequency based at least in part on the first frequency and the second error.

19. The system as recited in claim 18, the acts further comprising:

calculating an amount to adjust the first frequency based at least in part on the second error; and

calculating the third frequency, the third frequency comprising the first frequency adjusted by the amount.

20. The system as recited in claim 19, wherein the calculating the amount to adjust the first frequency comprises:

calculating a proportional error (P_E) based at least in part on the second error;

calculating an integral error (P_I) based at least in part on the first error and the second error; and

calculating a weighted P_E comprising a first gain applied to the P_E;

calculating a weighted P_I comprising a second gain applied to the P_I; and

summing the weighted P_E and the weighted P_I to determine the amount.