Time synchronization of sensor data

• US 10,771,669 B1
• Filed: 09/10/2018
• Issued: 09/08/2020
• Est. Priority Date: 09/26/2017
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Balasubramanian, Koussalya, Baker, Paul A., Ockerse, Harold C., Tweddle, Brent E.
• Primary Examiner(s): Wong, Allen C
• Application Number:
  US16/126,030
• Time in Patent Office: 729 Days
• Field of Search: 348135
• CPC Class Codes: G01S 13/865 Combination of radar systems with lidar systems
• CPC Class Codes: G01S 17/87 Combinations of systems using electromagnetic waves other than radio waves
• CPC Class Codes: G01S 17/89 for mapping or imaging
• CPC Class Codes: G01S 7/003 Transmission of data between radar, sonar or lidar systems and remote stations
• CPC Class Codes: G01S 7/497 Means for monitoring or calibrating
• CPC Class Codes: H04N 23/66 Remote control of cameras or camera parts, e.g. by remote control devices
• CPC Class Codes: H04N 23/73 by influencing the exposure time
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Abstract

Sensor data produced in a multi-sensor environment is time synchronized, such as using a master time. A component that derives the master time transmits a signal to a first sensor of the multi-sensor environment, which signal indicates the master time. The component subsequently receives a signal from the first sensor, which signal indicates a first time determined by the first sensor based on the master time and a latency associated with the first sensor. The first time indicates a time of an event associated with the first sensor data. A determination is then made that the first sensor data corresponds to second sensor data produced using a second sensor of the multi-sensor environment based on the first time and a second time indicating a time of an event associated with the second sensor data.

Claims

1. A method for time synchronization of sensor data in a multi-sensor environment using a master time, the method comprising:

transmitting, from a component that derives the master time, a first signal to a first sensor of the multi-sensor environment and to a second sensor of the multi-sensor environment, the first signal including a master time stamp;

receiving, by the component, a second signal from the first sensor, the second signal including first sensor data produced using the first sensor, the second signal including a first time stamp determined by the first sensor based on the master time and a latency associated with the first sensor, the first time stamp indicating a time of an event associated with the first sensor data;

receiving, by the component, a third signal from the second sensor, the third signal including second sensor data produced using the second sensor, the third signal including a second time stamp determined by the second sensor based on the master time stamp and a latency associated with the second sensor, the second time stamp indicating a time of an event associated with the second sensor data; and

determining that the first sensor data corresponds to the second sensor data based on the first time stamp and the second time stamp.

2. The method of claim 1, wherein the first signal includes a trigger for causing the first sensor to produce the first sensor data.

3. The method of claim 1, further comprising:

subsequent to determining that the first sensor data corresponds to the second sensor data, associating the event associated with the first sensor data and the event associated with the second sensor data, wherein determining that the first sensor data corresponds to the second sensor data includes determining that a difference between the first time and the second time is less than a threshold.

4. The method of claim 1, wherein the latency associated with the first sensor changes based on at least one of a condition associated with the multi-sensor environment or a configuration of the first sensor.

5. A method for time synchronization of sensor data in a multi-sensor environment using a master time, the method comprising:

initializing, by a component that derives the master time, the master time according to a geolocation time obtained using a geolocation component of the multi-sensor environment, the geolocation time indicating a time according to a geolocation system clock external to the multi-sensor environment;

transmitting, from the component, a first signal to a first sensor of the multi-sensor environment, the first signal including a master time stamp;

receiving, by the component, a second signal from the first sensor, the second signal including first sensor data produced using the first sensor, the second signal including a first time stamp determined by the first sensor based on the master time and a latency associated with the first sensor, the first time stamp indicating a time of an event associated with the first sensor data; and

determining that the first sensor data corresponds to second sensor data produced using a second sensor of the multi-sensor environment, the determining based on the first time stamp and a second time stamp indicating a time of an event associated with the second sensor data.

6. The method of claim 5, wherein the first signal includes a trigger for causing the first sensor to produce the first sensor data.

7. The method of claim 5, further comprising:

subsequent to determining that the first sensor data corresponds to the second sensor data, associating the event associated with the first sensor data and the event associated with the second sensor data, wherein determining that the first sensor data corresponds to the second sensor data includes determining that a difference between the first time stamp and the second time stamp is less than a threshold.

8. The method of claim 5, wherein the latency associated with the first sensor changes based on at least one of a condition associated with the multi-sensor environment or a configuration of the first sensor.

9. A non-transitory computer-readable storage device including program instructions executable by one or more processors that, when executed, cause the one or more processors to perform operations for time synchronization of sensor data in a multi-sensor environment, the operations comprising:

transmitting a first signal to a lidar sensor to cause the lidar sensor to produce first sensor data at a first time;

transmitting a second signal to a second sensor to cause the second sensor to produce second sensor data at a second time, wherein the second time is determined based on the first time and a phase offset; and

associating the first sensor data with the second sensor data based on the first time and the second time.

10. The non-transitory computer-readable storage device of claim 9, the operations further comprising:

determining the phase offset based on an angle of the lidar sensor.

11. The non-transitory computer-readable storage device of claim 9, wherein the second sensor is a camera.

12. The non-transitory computer-readable storage device of claim 11, wherein a mid-exposure of the camera is timed according to the phase offset.

13. The non-transitory computer-readable storage device of claim 9, wherein the first signal includes a pulse-per-second signal defining a periodic basis for producing the first sensor data using the lidar sensor and the second signal includes a trigger for causing the second sensor to produce the second sensor data, and the trigger is synchronized to the pulse-per-second signal.

14. The non-transitory computer-readable storage device of claim 9, wherein the phase offset is a first phase offset, the operations further comprising:

transmitting a third signal to a third sensor to cause the third sensor to produce third sensor data at a third time, wherein the third time is determined based on the first time and a second phase offset; and

associating the first sensor data with the third sensor data based on the first time and the third time.

15. A system for time synchronization of sensor data in a multi-sensor environment, comprising:

a control component that is configured to;

transmit a first signal to a lidar sensor to cause the lidar sensor to produce first sensor data at a first time,

transmit a second signal to a second sensor to cause the second sensor to produce second sensor data at a second time, wherein the second time is determined based on the first time and a phase offset, and

associate the first sensor data with the second sensor data based on the first time and the second time.

16. The system of claim 15, wherein the control component is further configured to determine the phase offset based on an angle of the lidar sensor.

17. The system of claim 15, wherein the second sensor is a camera.

18. The system of claim 17, wherein a mid-exposure of the camera is timed according to the phase offset.

19. The system of claim 15, wherein the first signal includes a pulse-per-second signal defining a periodic basis for producing the first sensor data using the lidar sensor and the second signal includes a trigger for causing the second sensor to produce the second sensor data, and the trigger is synchronized to the pulse-per-second signal.

20. The system of claim 15, wherein the phase offset is a first phase offset, and the control component is further configured to:

transmit a third signal to a third sensor to cause the third sensor to produce third sensor data at a third time, wherein the third time is determined based on the first time and a second phase offset, and

associate the first sensor data with the third sensor data based on the first time and the third time.

21. A system for time synchronization of sensor data in a multi-sensor environment using a master time, comprising:

a component that derives the master time, wherein the component is configured to;

transmit a first signal to a first sensor of the multi-sensor environment and to a second sensor of the multi-sensor environment, the first signal including a master time stamp,

receive a second signal from the first sensor, the second signal including first sensor data produced using the first sensor, the second signal including a first time stamp determined by the first sensor based on the master time and a latency associated with the first sensor, the first time stamp indicating a time of an event associated with the first sensor data,

receive a third signal from the second sensor, the third signal including second sensor data produced using the second sensor, the third signal including a second time stamp determined by the second sensor based on the master time stamp and a latency associated with the second sensor, the second time stamp indicating a time of an event associated with the second sensor data, and

determine that the first sensor data corresponds to the second sensor data based on the first time stamp and the second time stamp.

22. The system of claim 21, wherein the first signal includes a trigger for causing the first sensor to produce the first sensor data.

23. The system of claim 21, wherein the component is configured to associate the event associated with the first sensor data and the event associated with the second sensor data subsequent to a determination that the first sensor data corresponds to the second sensor data, wherein the component is configured to determine that the first sensor data corresponds to the second sensor data when a difference between the first time and the second time is less than a threshold.

24. The system of claim 21, wherein the latency associated with the first sensor changes based on at least one of a condition associated with the multi-sensor environment or a configuration of the first sensor.

25. The system of claim 21, wherein the component is further configured to:

receive a geolocation time using a geolocation component, the geolocation time indicating a time according to a geolocation system clock external to the multi-sensor environment, and

store a difference between the master time and the geolocation time.

26. A system for time synchronization of sensor data in a multi-sensor environment using a master time, comprising:

a component that derives the master time, wherein the component is configured to;

initialize the master time according to a geolocation time obtained using a geolocation component of the multi-sensor environment, the geolocation time indicating a time according to a geolocation system clock external to the multi-sensor environment,

transmit a first signal to a first sensor of the multi-sensor environment, the first signal including a master time stamp,

receive a second signal from the first sensor, the second signal including first sensor data produced using the first sensor, the second signal including a first time stamp determined by the first sensor based on the master time and a latency associated with the first sensor, the first time stamp indicating a time of an event associated with the first sensor data, and

determine that the first sensor data corresponds to second sensor data produced using a second sensor of the multi-sensor environment based on the first time stamp and a second time stamp indicating a time of an event associated with the second sensor data.

27. The system of claim 26, wherein the first signal includes a trigger for causing the first sensor to produce the first sensor data.

28. The system of claim 26, wherein the component is configured to associate the event associated with the first sensor data and the event associated with the second sensor data subsequent to a determination that the first sensor data corresponds to the second sensor data.

29. The system of claim 28, wherein the component is configured to determine that the first sensor data corresponds to the second sensor data when a difference between the first time stamp and the second time stamp is less than a threshold.

30. The system of claim 26, wherein the latency associated with the first sensor changes based on at least one of a condition associated with the multi-sensor environment or a configuration of the first sensor.