System and method to combine multiple video streams

• US 8,369,399 B2
• Filed: 02/13/2007
• Issued: 02/05/2013
• Est. Priority Date: 02/13/2006
• Status: Active Grant

• Current Assignee: Sony Semiconductor Solutions Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Feldman, Rodney, Kim, Kyungnam, Egnal, Geoffrey, Wilburn, Bennett
• Primary Examiner(s): Gergiso, Techane
• Application Number:
  US11/674,561
• Publication Number:
  US 20080060034A1
• Time in Patent Office: 2,184 Days
• Field of Search: 375/240, 375/240.08
• US Class Current: 375/240.08
• CPC Class Codes: G01C 11/02 Picture taking arrangements specially adapted for photogrammetry or photographic surveying, e.g. controlling overlapping of pictures
• CPC Class Codes: G01C 11/06 by comparison of two or more pictures of the same area
• CPC Class Codes: G03B 15/006 Apparatus mounted on flying objects
• CPC Class Codes: G03B 37/04 with cameras or projectors providing touching or overlapping fields of view
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G08B 13/19641 Multiple cameras having overlapping views on a single scene
• CPC Class Codes: G08B 13/1965 the vehicle being an aircraft
• CPC Class Codes: G08B 13/19656 Network used to communicate with a camera, e.g. WAN, LAN, Internet
• CPC Class Codes: G08B 13/19693 using multiple video sources viewed on a single or compound screen
• CPC Class Codes: H04N 19/527 Global motion vector estimation
• CPC Class Codes: H04N 21/2146 located in mass transportation means, e.g. aircraft, train or bus flight-deck installations for entertainment or communications B64D11/0015; arrangements specially adapted for transportation systems in broadcast systems H04H20/62; moving wireless networks H04W84/005
• CPC Class Codes: H04N 21/2365 Multiplexing of several video streams
• CPC Class Codes: H04N 21/2665 Gathering content from different sources, e.g. Internet and satellite
• CPC Class Codes: H04N 21/4312 involving specific graphical features, e.g. screen layout, special fonts or colors, blinking icons, highlights or animations
• CPC Class Codes: H04N 21/4314 for fitting data in a restricted space on the screen, e.g. EPG data in a rectangular grid
• CPC Class Codes: H04N 21/4347 Demultiplexing of several video streams
• CPC Class Codes: H04N 21/4348 Demultiplexing of additional data and video streams
• CPC Class Codes: H04N 23/61 based on recognised objects
• CPC Class Codes: H04N 23/62 Control of parameters via user interfaces
• CPC Class Codes: H04N 23/632 for displaying or modifying preview images prior to image capturing, e.g. variety of image resolutions or capturing parameters
• CPC Class Codes: H04N 23/635 : Region indicators; Field of view indicators
• CPC Class Codes: H04N 23/66 : Remote control of cameras or camera parts, e.g. by remote control devices
• CPC Class Codes: H04N 23/68 : for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/683 : performed by a processor, e.g. controlling the readout of an image memory
• CPC Class Codes: H04N 23/695 : Control of camera direction for changing a field of view, e.g. pan, tilt or based on tracking of objects
• CPC Class Codes: H04N 23/698 : for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 23/90 : Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Abstract

Methods and systems for combining multiple video streams are provided. Video feeds are received from multiple optical sensors, and homography information and/or corner metadata is calculated for each frame from each video stream. This data is used to mosaic the separate frames into a single video frame. Local translation of each image may also be used to synchronize the video frames. The optical sensors can be provided by an airborne platform, such as a manned or unmanned surveillance vehicle. Image data can be requested from a ground operator, and transmitted from the airborne platform to the user in real time or at a later time. Various data arrangements may be used by an aggregation system to serialize and/or multiplex image data received from multiple sensor modules. Fixed-size record arrangement and variable-size record arrangement systems are provided.

Claims

1. An apparatus comprising:

a plurality of optical sensors each outputting a video frame corresponding to that sensor;

a plurality of sensor processing modules, each sensor processing module to calculate metadata for a video frame received from at least one of the plurality of optical sensors; and

an aggregation system to serialize video data representing the output of said optical sensors and metadata received from the plurality of sensor processing modules to form a composite image.

2. The apparatus of claim 1, wherein the metadata comprises a motion estimation calculated from the video frame and at least one previous video frame received from the same at least one of the plurality of sensors.

3. The apparatus of claim 1, wherein the metadata comprises a description of at least one feature detected in the video frame.

4. The apparatus of claim 1, wherein the metadata comprises information describing at least one corner detected in the video frame.

5. The apparatus of claim 1, further comprising a storage system to store video data and metadata received from the aggregation system.

6. The apparatus of claim 1, wherein the composite image is a gigapixel image.

7. The apparatus of claim 1, wherein the optical sensors, sensor processing modules, and aggregation system are contained in an airborne observation platform.

8. The apparatus of claim 1, wherein each of the plurality of sensor modules captures a 4 megapixel image.

9. The apparatus of claim 1, wherein the aggregation system provides image data to the storage system in a fixed-size record arrangement.

10. The apparatus of claim 1, wherein the aggregation system provides image data to the storage system in a variable-size record arrangement.

11. The apparatus of claim 1, further comprising a signal generator to generate a frame synchronization signal.

12. A method of generating a video stream, comprising:

receiving a plurality of video frames generated by a plurality of optical sensors;

using a plurality of sensor processing modules, calculating metadata for each video frame received from at least one of the plurality of optical sensors; and

serializing the video frames and metadata received from the plurality of sensor processing modules.

13. The method of claim 12, further comprising calculating a motion estimation for each of the plurality of video frames and at least one previous video frame received from the same at least one of the plurality of sensors.

14. The method of claim 12, wherein the video stream contains image data depicting a sub-window of the area captured by the plurality of optical sensors, the sub-window selected in response to a pan-tilt-zoom request.

15. A method of combining multiple video streams, comprising:

receiving a plurality of video frames, each video frame corresponding to image data obtained concurrently by one of a plurality of optical sensors;

calculating mosaic estimation metadata for each of the plurality of video frames; and

creating a composite video frame from the plurality of video frames and mosaic estimation metadata.

16. The method of claim 15, wherein the mosaic estimation metadata is calculated for a sub-window of the area being imaged by the plurality of optical sensors, the sub-window corresponding to a pan-tilt-zoom request.

17. The method of claim 15, wherein the mosaic estimation metadata includes feature metadata calculated for at least two of the plurality of video frames, the at least two video frames captured by different optical sensors at about the same time.

18. The method of claim 15, wherein the mosaic estimation metadata includes motion estimation metadata calculated based on at least two of the plurality of video frames, the at least two video frames captured by the same optical sensor at different times.

19. The method of claim 15, further comprising:

assigning a fixed position to a video frame selected from the plurality of video frames;

measuring pairwise translation for each of the plurality of video frames neighboring the selected video frame; and

aligning each of the neighboring video frames to the selected video frame based on the pairwise translation measured for that video frame.

20. The method of claim 19, further comprising:

iterating the steps of assigning a fixed position to a selected frame and measuring pairwise translation for each of the neighboring frames; and

calculating an average position for each of the plurality of video frames from the set of positions in which each video frame was fixed based on the measured pairwise translations.

21. The method of claim 15, wherein creating the composite video frame further comprises:

estimating the translation of a feature displayed in each of the plurality of video frames; and

performing a global adjustment on the plurality of video frames to correct for movement of one or more of the plurality of optical sensors based on the estimated translation of the feature.

22. The method of claim 15, wherein the plurality of video frames are captured by a mobile sensor platform.

23. The method of claim 15, wherein the composite video frame is a gigapixel image.

24. The method of claim 15, wherein each of the plurality of video frames is a 4 megapixel image.

25. The method of claim 15, wherein the optical sensors are provided by an airborne observation unit.

26. The method of claim 15, wherein the plurality of optical sensors are configured to concurrently obtain video frames by a frame synchronization signal.

27. A method of aggregating video streams, comprising:

assigning a channel to each of a plurality of video sources;

for each video source, encoding video data from the video source into a series of packets using an electronic apparatus;

encoding each packet with metadata describing the video data encoded in the packet using an electronic apparatus, the metadata containing image transformation information; and

receiving packets from the plurality of video sources, wherein each video source electronically transmits a packet to the channel assigned to the video source.

28. The method of claim 27, wherein the image transformation information includes motion estimation metadata.

29. The method of claim 27, wherein the image transformation information includes feature detection metadata.

30. The method of claim 27, wherein each packet is a size corresponding to the amount of video data encoded by each video source.

31. The method of claim 27, wherein each packet is a size corresponding to a physical storage location in an aggregation system.

32. The method of claim 31, wherein each packet is the same size.

33. The method of claim 32, wherein the packet size is equal to a disk stripe in a multi-disk array.

34. The method of claim 27, wherein one channel is assigned for each optical sensor in a multi-sensor array.

35. The method of claim 27, wherein the plurality of video sources obtain video data concurrently using a synchronization signal.