Composite Video Generation

• US 20100296571A1
• Filed: 05/22/2009
• Published: 11/25/2010
• Est. Priority Date: 05/22/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Hamid, Ahmed Mohamed Abdul, Refaat, Mahmoud Ahmed, El-Saban, Motaz Ahmed, Kaheel, Ayman Malek
• Granted Patent: US 8,605,783 B2
• Time in Patent Office: Days
• US Class Current: 375/240.01
• CPC Class Codes: H04N 21/21805 enabling multiple viewpoints, e.g. using a plurality of cameras
• CPC Class Codes: H04N 21/23412 for generating or manipulating the scene composition of objects, e.g. MPEG-4 objects
• CPC Class Codes: H04N 21/234363 by altering the spatial resolution, e.g. for clients with a lower screen resolution
• CPC Class Codes: H04N 21/2665 Gathering content from different sources, e.g. Internet and satellite
• CPC Class Codes: H04N 21/41407 embedded in a portable device, e.g. video client on a mobile phone, PDA, laptop constructional details of equipment or arrangements specially adapted for portable computer application G06F1/1626; arrangements specially adapted for mobile receivers in broadcast systems H04H20/57
• CPC Class Codes: H04N 21/44012 involving rendering scenes according to scene graphs, e.g. MPEG-4 scene graphs
• CPC Class Codes: H04N 21/8547 involving timestamps for synchronizing content

Abstract

Composite video generation is described. In an embodiment, mobile video capture devices, such as mobile telephones, capture video streams of a common event. A network node receives the video streams and time-synchronizes them. Frames from each of the video streams are then stitched together to form a composite frame, and these are added to a composite video sequence. In embodiments, the composite video sequence is encoded and streamed to a user terminal over a communications network. In embodiments, the common event is a live event and the composite video sequence is generated in real-time. In some embodiments, the stitching of the video streams is performed by geometrically aligning the frames. In some embodiments, three or more mobile video capture devices provide video streams.

Claims

1. A computer-implemented method of generating a composite video sequence at a node in a communications network comprising a plurality of mobile video capture devices, the method comprising:

receiving a plurality of video streams relating to a common event, each of the video streams comprising a sequence of frames and originating from a different one of the mobile video capture devices;

time-synchronizing the video streams and selecting a frame set comprising a frame from each of the video streams, each frame having a common capture time;

forming a composite frame by stitching at least two frames in the frame set together; and

adding the composite frame to the composite video sequence.

2. A method according to claim 1, further comprising the step of encoding the composite video sequence and streaming the encoded composite video sequence to a user terminal over the communications network.

3. A method according to claim 1, wherein the video streams are of a live event and wherein the method is carried out in real-time.

4. A method according to claim 1, wherein each frame comprises a timestamp recording the capture time of the frame, and the step of time-synchronizing comprises reading the timestamp and queuing the frames from each video stream according to the capture time.

5. A method according to claim 4, wherein each of the plurality of video streams is encoded such that it comprises the timestamp for each frame.

6. A method according to claim 4, wherein the step of queuing comprises buffering the frames from each video stream in one of a plurality of buffers.

7. A method according to claim 1, wherein the step of stitching comprises geometrically aligning each frame in the frame set.

8. A method according to claim 7, wherein the step of geometrically aligning comprises determining whether alignment parameters for a previous frame set can be utilized, and, if so, geometrically aligning according to the alignment parameters for the previous frame set, and, if not, calculating new alignment parameters and geometrically aligning according to the new alignment parameters.

9. A method according to claim 8, wherein the step of determining whether alignment parameters for a previous frame set can be utilized comprises determining whether the alignment parameters for the previous frame set were calculated within a predetermined time interval.

10. A method according to claim 8, wherein the step of determining whether alignment parameters for a previous frame set can be utilized comprises determining whether at least one of the mobile video capture devices is moving more than a threshold movement level.

11. A method according to claim 1, wherein the step of stitching comprises selecting a pixel value for a pixel in the composite frame in an overlap region of the at least two frames.

12. A method according to claim 11, wherein the step of selecting a pixel value comprises determining a distance from the pixel to a boundary of each of the at least two frames.

13. A method according to claim 12, wherein the step of selecting a pixel value further comprises calculating the pixel value as a weighted average of pixel values from each of the at least two frames based on the distance from the pixel to the boundary of each of the at least two frames.

14. A method according to claim 1, further comprising the step of determining that the plurality of video streams relate to a common event.

15. A method according to claim 1, further comprising the step of decoding each of the plurality of video streams prior to time-synchronizing the video streams.

16. A method according to claim 1, wherein the plurality of mobile video capture devices comprises three or more mobile video capture devices, and the step of forming a composite frame comprises:

searching the frame set for frames that are able to be stitched together;

allocating each frame in the frame set to one of a plurality of clusters, such that frames that are able to be stitched together are allocated to the same cluster;

selecting from the plurality of clusters a cluster comprising the largest number of allocated frames; and

stitching the frames in the selected cluster to form the composite frame.

17. A method according to claim 1, further comprising the step of transmitting a time synchronization signal to each of the mobile video capture devices over the communication network.

19. A network node according to claim 18, wherein the network interface receives a plurality of video streams originating from a different one of a plurality of mobile telephones.

18. A network node for generating a composite video sequence, comprising:

a network interface arranged to connect to a communications network and receive a plurality of video streams relating to a common event, each of the video streams comprising a sequence of frames and originating from a different one of a plurality of mobile video capture devices;

a plurality of buffers, each arranged to queue the sequence of frames from one of the video streams according to a frame capture time; and

a processor arranged to select a frame set comprising a frame from each of the plurality of buffers, each frame having a common capture time, stitch at least part of each frame in the frame set together to form a composite frame, add the composite frame to a composite video sequence and transmit the composite video sequence from the network interface to a user terminal connected to the communications network.

20. A computer-implemented method of generating a composite video sequence at a node in a communications network comprising a three or more mobile video capture devices, the method comprising:

receiving a plurality of video streams relating to a common event, each of the video streams comprising a sequence of frames and originating from a different one of the mobile video capture devices;

time-synchronizing the video streams and selecting a frame set comprising a frame from each of the video streams, each frame having a common capture time;

forming a composite frame by searching the frame set for frames that are able to be stitched together, allocating each frame in the frame set to one of a plurality of clusters, such that frames that are able to be stitched together are allocated to the same cluster, selecting from the plurality of clusters a cluster comprising the largest number of allocated frames, and stitching the frames in the selected cluster to form the composite frame; and

adding the composite frame to the composite video sequence.