HETEROGENEOUS IMAGE SENSOR SYNCHRONIZATION

• US 20110267269A1
• Filed: 05/03/2010
• Published: 11/03/2011
• Est. Priority Date: 05/03/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Cao, Yuqun, Tardif, John, Bronder, Matthew
• Granted Patent: US 9,084,002 B2
• Time in Patent Office: Days
• US Class Current: 345/158
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/64 for computing dynamical parameters of game objects, e.g. motion determination or computation of frictional forces for a virtual car
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: H04N 21/4112 having fewer capabilities than the client, e.g. thin client having less processing power or no tuning capabilities
• CPC Class Codes: H04N 21/422 Input-only peripherals , i.e. input devices connected to specially adapted client devices, e.g. global positioning system [GPS] input devices also receiving signals from specially adapted client devices H04N21/4104

Abstract

A computer implemented method for synchronizing information from a scene using two heterogeneous sensing devices. Scene capture information is provided by a first sensor and a second sensor. The information comprises video streams including successive frames provided at different frequencies. Each frame is separated by a vertical blanking interval. A video output comprising a stream of successive frames each separated by a vertical blanking interval is rendered based on information in the scene. The method determines whether an adjustment of the first and second video stream relative to the video output stream is required by reference to the video output stream. A correction is then generated to at least one of said vertical blanking intervals.

Claims

1. A computer implemented method for correlating information from a scene using two or more heterogeneous image sensing devices, comprising:

receiving scene capture information from a first image sensor, the information comprising a first data stream including successive frames provided at a first frequency separated by a first wait interval;

receiving scene capture information from a second image sensor, the information comprising a second data stream including successive frames provided at a second, different frequency and being separated by a second wait interval;

processing the scene capture information from the first and the second image sensor to produce an output, the processing having a processing rate;

determining whether an adjustment of the first and second data streams relative to the processing rate is required by reference to the processing rate; and

generating a correction to at least one of said wait intervals.

2. The method of claim 1 wherein each data stream comprises a video stream, the wait interval comprises a vertical blanking interval, and the processing includes rendering an output comprising a stream of successive frames each separated by a third vertical blanking interval.

3. The method of claim 1 wherein the first image sensor and the second image sensor are provided in a capture device and generating a correction comprises providing an instruction to the capture device to insert the correction into the first or second wait interval.

4. The method of claim 1 wherein generating a correction comprises substituting a single longer or shorter wait interval for one of the first or the second wait intervals.

5. The method of claim 2 wherein generating a correction comprises substituting a single longer or shorter vertical blanking interval for the third vertical blanking interval.

6. The method of claim 2 wherein each of the first wait interval, second wait interval and third wait intervals includes a default interval, and generating a correction comprises inserting a correction by changing the vertical blanking interval at a first frame in the stream and resetting the vertical blanking interval to the default interval at a successive frame in the stream.

7. The method of claim 1 wherein determining whether an adjustment is required comprises applying timing information based on the processing rate to each of the first data stream and second data stream, and comparing said timing information to timing information applied to each stream by the first or the second sensor.

8. The method of claim 1 wherein determining whether an adjustment is required comprises detecting a deviation in timing information between the output and one or both of the first and second data streams, and determining whether the adjustment is required based on whether a deviation in the timing will correct itself.

9. A computer readable medium including instructions for programming a processing device to perform a series of steps, comprising:

receiving a first series of image capture frames from a first image sensor, each frame having first timing information associated with each of the first series, and a second series of image capture frames from a second image sensor, each frame having second timing information associated with each of the second series;

rendering a video output comprising a series of output frames, each frame in the output series having third timing information and being correlated one or more of the first series of image capture frames and one or more of the second series of image capture frames;

determining whether the first timing information, the second timing information, and third timing information require an adjustment between the first sensor, the second sensor and the video output;

determining a correction for said adjustment; and

applying a one time correction to one or more vertical blanking intervals between successive frames in one or more of the first series, the second series or the video output.

10. The computer readable medium of claim 9 wherein the first series of image capture frames includes a vertical blanking interval, the second series of image capture frames includes a vertical blanking interval and the video output includes a third vertical blanking interval, each vertical blanking interval includes a default interval, and generating a correction comprises inserting a correction by changing the vertical blanking interval at a first frame in the stream and resetting the vertical blanking interval to the default interval at a successive frame in the stream.

11. The computer readable medium of claim 10 wherein the first sensor and the second sensor are provided in a capture device and generating a correction comprises providing an instruction to the capture device to insert the correction into the vertical blanking interval in the successive frames of the first or second series.

12. The computer readable medium of claim 11 wherein applying a one time correction comprises inserting a single longer vertical blanking interval between successive frames of the first or the second series.

13. The computer readable medium of claim 11 wherein applying a one time correction comprises inserting a single shorter vertical blanking interval between successive frames of the first or the second series.

14. The computer readable medium of claim 11 wherein applying a one time correction comprises inserting a single longer or shorter vertical blanking interval between successive frames of the video output.

15. The computer readable medium of claim 11 wherein determining whether an adjustment is required comprises applying timing information based on the video output to each of the first video stream and second video stream, and comparing said timing information to timing information applied to each stream by the first or the second sensor.

16. The computer readable medium of claim 11 wherein determining whether an adjustment is required comprises detecting a deviation in timing information between the video output and one or both of the first video stream and second video stream, and determining whether the adjustment is required based on whether a deviation in the timing will correct itself.

17. An image capture system including a capture device including a first capture device and a second capture device coupled to a host device, the host device providing a video output, the system including a processor performing the steps of:

receiving image capture information from the first capture device, the information comprising a video stream including successive frames provided at a first frequency separated by a first vertical blanking interval, each frame having timing data;

receiving image capture information from the second capture device, the information comprising a video stream including successive frames provided at a second frequency and being separated by a second vertical blanking interval, each frame having timing data;

rendering an video output stream including successive frames each separated by a third vertical blanking interval, each frame in the video stream having timing information associated therewith;

determining whether the video streams from the first capture device and the second capture device are synchronous with the video output stream by matching the timing data of the first capture device, the second capture device and the video output; and

applying a correction to at least one of the first, second or third vertical blanking interval to synchronize the first capture device, the second capture device and the video output stream.

18. The system of claim 17 wherein the first frequency is different than the second frequency.

19. The system of claim 18 wherein applying a correction comprises inserting a single longer vertical blanking interval or a single shorter vertical blanking interval in one of the first or the second vertical blanking interval.

20. The system of claim 19 wherein each of the first, second and third vertical blanking intervals includes a default interval, and generating a correction comprises inserting a correction by changing the vertical blanking interval at a first frame in the stream and resetting the vertical blanking interval to the default interval at a successive frame in the stream.