Synchronizing data streams

• US 10,368,057 B1
• Filed: 09/28/2016
• Issued: 07/30/2019
• Est. Priority Date: 09/28/2016
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Saran, Sajesh Kumar, Arnold, Brad Nathaniel, Song, Jack G, Vrooman, Tyler Ogren, Chen, Wei, Schoepflin, Todd Nelson
• Primary Examiner(s): Jones, Heather R
• Application Number:
  US15/279,212
• Time in Patent Office: 1,035 Days
• Field of Search: None
• CPC Class Codes: G11B 27/10 Indexing; Addressing; Timing or synchronising; Measuring tape travel
• CPC Class Codes: H04N 13/257 Colour aspects
• CPC Class Codes: H04N 13/275 from 3D object models, e.g. computer-generated stereoscopic image signals
• CPC Class Codes: H04N 13/296 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 23/10 for generating image signals from different wavelengths
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 5/123 whereby the synchronisation signal directly commands a frequency generator

Abstract

This disclosure describes techniques for synchronizing independent data streams. In some instances, a computing device couples to multiple independent sensors, such as cameras, and applies accurate timestamp information to the individual frames of sensor data from the independent sensors. After aligning these data streams by applying these accurate timestamps, the computing device may, in some instances, encode and transmit these timestamped data streams to one or more entities for further processing. In one example, a first camera (e.g., a depth camera configured to generate a depth map) may capture images of an environment, as may a second camera (e.g., an Red-Green-Blue (RGB) camera configured to generate color images). The resulting images may be temporally aligned with one another via the timestamping, and the resulting aligned images from both the depth sensor and the RGB camera may be used to create a three-dimensional (3D) model of the environment.

Claims

1. A system comprising:

a first camera configured to generate depth data of an environment based on acquired depth images;

a second camera configured to acquire color images of the environment;

a computing device coupled to the first camera and the second camera, the computing device comprising;

a pulse-width-modulation (PWM) unit coupled to the first camera and the second camera and configured to generate a pulse and an interrupt at a first time;

one or more processors to receive the interrupt;

memory, coupled to the one or more processors;

a first driver stored in the memory and executable on the one or more processors to receive a depth image acquired by the first camera, the first camera configured to acquire the first depth image in response to receiving the pulse from the PWM unit;

a second driver stored in the memory and executable on the one or more processors to receive a color image acquired by the second camera, the second camera acquiring the color image in response to receiving the pulse from the PWM unit;

a first timestamp queue for storing, in the memory, timestamps for association with depth images acquired by the first camera;

a second timestamp queue for storing, in the memory, timestamps for association with color images acquired by the second camera;

an interrupt service stored in the memory and executable on the one or more processors to;

receive a call from the one or more processors in response to the one or more processors receiving the interrupt;

store, in response to receiving the call;

(i) a first timestamp corresponding to the first time in the first timestamp queue, and (ii) a second timestamp corresponding to the first time in the second timestamp queue;

an application stored in the memory and executable on the one or more processors to;

receive the depth image from the first driver;

read the first timestamp from the first timestamp queue;

add metadata that is based on the first timestamp to the depth image;

receive the color image from the second driver;

read the second timestamp from the second timestamp queue; and

add metadata that based on the second timestamp to the color image.

2. The system as recited in claim 1, wherein:

the pulse comprises a first pulse;

the interrupt comprises a first interrupt;

the PWM unit is further configured to generate a second pulse and a second interrupt at a second time;

the depth image comprises a first depth image;

the first driver is further to receive a second depth image acquired by the first camera, the first camera configured to acquire the second depth image in response to receiving the second pulse;

the color image comprises a first color image;

the second driver is further to receive a second color image acquired by the second camera, the second camera configured to acquire the second color image in response to receiving the second pulse;

the call comprises a first call;

the interrupt service is further executable on the one or more processors to;

receive a second call from the one or more processors in response to the one or more processors receiving second interrupt; and

store, in response to receiving the second call;

(i) a third timestamp corresponding to the second time in the first timestamp queue, and (ii) a fourth timestamp corresponding to the second time in the second timestamp queue; and

the application is further executable on the one or more processors to;

receive the second depth image from the first driver;

read the third timestamp from the first timestamp queue;

add metadata that is based on the third timestamp to the second depth image;

receive the second color image from the second driver;

read the fourth timestamp from the second timestamp queue; and

add metadata that is based on the fourth timestamp to the second color image.

3. The system as recited in claim 1, wherein the application is further executable on the one or more processors to:

determine a first state of the first timestamp queue;

determine a second state of the second timestamp queue;

compare the first state to the second state;

based at least in part the comparing;

send a first instruction to the PWM unit to cease generating pulses;

empty the first timestamp queue;

empty the second timestamp queue; and

send a second instruction to the PWM unit to begin generating pulses.

4. A method comprising:

generating, at a first time, a pulse by a trigger unit configured to generate periodic pluses, the trigger unit coupled to a first camera and to a second camera;

receiving, by the first camera, the pulse generated at the first time;

receiving, by the second camera, the pulse generated at the first time;

acquiring first image data by the first camera at least partly in response to receiving the pulse;

acquiring second image data by the second camera at least partly in response to receiving the pulse;

storing, by one or more processors of a computing device, the first image data in a first location;

storing, by the one or more processors, the second image data in a second location;

storing, in a first timestamp queue and by the one or more processors, a first timestamp that is based at least in part on the first time;

storing, in a second timestamp queue and by the one or more processors, a second timestamp that is based at least in part on the first time;

receiving the first image data from the first location;

reading the first timestamp from the first timestamp queue;

associating the first timestamp with the first image data;

receiving the second image data from the second location;

reading the second timestamp from the second timestamp queue; and

associating the second timestamp with the second image data.

5. The method as recited in claim 4, wherein the pulse comprises a first pulse, and further comprising:

generating, at a second time, a second pulse by the trigger unit;

acquiring third image data by the first camera;

acquiring fourth image data by the second camera;

storing the third image data in the first location;

storing the fourth image data in the second location;

storing, in the first timestamp queue, a third timestamp that is based at least in part on the second time;

storing, in the second timestamp queue, a fourth timestamp that is based at least in part on the second time;

receiving the third image data from the first location;

reading the third timestamp from the first timestamp queue;

associating the third timestamp with the third image data;

receiving the fourth image data from the second location;

reading the fourth timestamp from the second timestamp queue; and

associating the fourth timestamp with the fourth image data.

6. The method as recited in claim 4, further comprising:

receiving, by one or more processors, an interrupt generated by the trigger unit substantially contemporaneously with the generating of the pulse; and

calling, by the one or more processors, an interrupt service at least partly in response to the receiving of the interrupt;

and wherein;

the storing the first timestamp comprises storing the first timestamp in the first timestamp queue by the interrupt service at least partly in response to the calling; and

the storing the second timestamp comprises storing the second timestamp in the second timestamp queue by the interrupt service at least partly in response to the calling.

7. The method as recited in claim 4, wherein:

the coupling the first camera comprises coupling the first camera to the trigger unit via a first communication channel;

the coupling the second camera comprises coupling the second camera the trigger unit via the first communication channel; and

the generating the pulse comprises generating the pulse by the trigger unit such that the first camera receives the pulse via the first communication channel substantially contemporaneously with the second camera receiving the pulse via the first communication channel.

8. The method as recited in claim 4, wherein the associating the first timestamp with the first image data comprises storing the first timestamp as metadata associated with the first image data.

9. The method as recited in claim 4, wherein the acquiring the first image data comprises acquiring the first image data of an environment by a depth sensor, the acquiring the second image data comprises acquiring the second image data of the environment by an RGB camera, and the method further comprising:

determining, based at least in part on the first and second timestamps, that the first image data temporally aligns with the second image data; and

generating a three-dimensional (3D) model of the environment using at least the first image data and the second image data.

10. The method as recited in claim 4, further comprising:

determining a first state of the first timestamp queue;

determining a second state of the second timestamp queue;

comparing the first state to the second state;

based at least in part the comparing;

sending a first instruction to cause the first camera and the second camera to cease acquiring image data;

emptying the first timestamp queue;

emptying the second timestamp queue; and

sending a second instruction to cause the first camera and the second camera to begin acquiring image data.

11. A system comprising:

a first sensor;

a second sensor;

a trigger unit coupled to the first sensor and the second sensor and configured to generate a pulse;

one or more processors; and

one or more computer-readable media storing computer-executable instructions that, when executed on the one or more processors, cause the one or more processors to perform acts comprising;

acquiring, at a first time, first data using the first sensor at least partly in response to the first sensor receiving the pulse;

acquiring, at or approximately at the first time, second data using the second sensor at least partly in response to the second sensor receiving the pulse;

storing the first data in a first location;

storing the second data in a second location;

storing, in a first timestamp queue, a first timestamp specifying the first time;

storing, in a second timestamp queue, a second timestamp specifying the first time or approximately the first time;

receiving the first data from the first location;

reading the first timestamp from the first timestamp queue;

associating the first timestamp with the first data;

receiving the second data from the second location;

reading the second timestamp from the second timestamp queue; and

associating the second timestamp with the second data.

12. The system as recited in claim 11, wherein the trigger unit comprises a pulse-width-modulation (PWM) unit.

13. The system as recited in claim 12, wherein the pulse comprises a first pulse, the PWM unit is further configured to generate a second pulse at a second time, and the acts further comprise:

receiving, by the first sensor, the second pulse generated at the second time;

receiving, by the second sensor, the second pulse generated at the second time;

acquiring third data by the first sensor;

acquiring fourth data by the second sensor;

storing the third data in the first location;

storing the fourth data in the second location;

storing, in the first timestamp queue, a third timestamp that is based at least in part on the second time;

storing, in the second timestamp queue, a fourth timestamp that is based at least in part on the second time;

receiving the third data from the first location;

reading the third timestamp from the first timestamp queue;

associating the third timestamp with the third data;

receiving the fourth data from the second location;

reading the fourth timestamp from the second timestamp queue; and

associating the fourth timestamp with the fourth data.

14. The system as recited in claim 12, the acts further comprising:

receiving, by one or more processors, an interrupt generated by the PWM unit substantially contemporaneously with the generating of the pulse; and

calling, by the one or more processors, an interrupt service at least partly in response to the receiving of the interrupt;

and wherein;

the storing the first timestamp comprises storing the first timestamp in the first timestamp queue by the interrupt service at least partly in response to the calling; and

the storing the second timestamp comprises storing the second timestamp in the second timestamp queue by the interrupt service at least partly in response to the calling.

15. The system as recited in claim 12, wherein the PWM unit couples to the first and second sensors via a common output pin of the PWM unit such that the first sensor receives the pulse substantially contemporaneously with the second sensor receiving the pulse.

16. The system as recited in claim 11, wherein the first sensor comprises a first camera, and the second sensor comprise a second camera.

17. The system as recited in claim 11, wherein the first sensor comprises a depth sensor and the second sensor comprises an RGB camera, and the acts further comprising:

determining, based at least in part on the first and second timestamps, that the first data temporally aligns with the second data; and

generating a three-dimensional (3D) model of the environment using at least the first data and the second data.

18. The system as recited in claim 11, the acts further comprising:

determining a first state of the first timestamp queue;

determining a second state of the second timestamp queue;

comparing the first state to the second state;

based at least in part the comparing;

sending a first instruction to cause the first sensor and the second sensor to cease acquiring data;

emptying the first timestamp queue;

emptying the second timestamp queue; and

sending a second instruction to cause the first sensor and the second sensor to begin acquiring data.