VISUALIZING VIDEO WITHIN EXISTING STILL IMAGES

• US 20120062748A1
• Filed: 09/14/2010
• Published: 03/15/2012
• Est. Priority Date: 09/14/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Ofek, Eyal, Chen, Billy
• Granted Patent: US 9,594,960 B2
• Time in Patent Office: Days
• US Class Current: 348/149
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 2207/30236 Traffic on road, railway or crossing
• CPC Class Codes: G06T 2207/30256 Lane; Road marking
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: G06V 20/182 Network patterns, e.g. roads or rivers
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 20/54 of traffic, e.g. cars on the road, trains or boats
• CPC Class Codes: G08G 1/04 using optical or ultrasonic detectors
• CPC Class Codes: H04N 23/6845 by combination of a plurality of images sequentially taken
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay

Abstract

Video from a video camera can be integrated into a still image, with which it shares common elements, to provide greater context and understandability. Pre-processing can derive transformation parameters for transforming and aligning the video to be integrated into the still image in a visually fluid manner. The transformation parameters can then be utilized to transform and align the video in real-time and display it within the still image. Pre-processing can comprise stabilization of video, if the video camera is moveable, and can comprise identification of areas of motion and of static elements. Transformation parameters can be derived by fitting the static elements of the video to portions of one or more existing images. Display of the video in real-time in the still image can include display of the entire transformed and aligned video image, or of only selected sections, to provide for a smoother visual integration.

Claims

1. One or more computer-readable media comprising computer-executable instructions for visually integrating a video into at least one existing still image that provides greater image scope than the video, the computer-executable instructions performing steps comprising:

obtaining a sample of the video;

selecting one or more existing still images whose image scope comprises at least a portion of an image scope of the video, the selecting being informed by location metadata associated with both the video and the still images;

identifying image features, common to both the sampled video and the selected one or more still images, that are to be utilized in transforming and aligning the sampled video to the selected one or more still images; and

deriving, with reference to the identified image features, transformation parameters for transforming and aligning the video for integration with the at least one existing still image such that the video appears within the at least one still image with image features of the video being overlaid over equivalent image features of the at least one still image.

2. The computer-readable media of claim 1, wherein the video is from a traffic camera and wherein further the existing still images are map panoramas utilized to provide a street-level view.

3. The computer-readable media of claim 1, wherein the computer-executable instructions for identifying image features that are to be utilized in transforming and aligning the sampled video comprise computer-executable instructions for:

deriving an average image comprising those elements that remain static throughout the sample of the video;

deriving a motion mask identifying areas in which elements move throughout the sample of the video; and

identifying image features in the average image that are along the motion mask.

4. The computer-readable media of claim 1, wherein the computer-executable instructions for identifying image features that are to be utilized in transforming and aligning the sampled video comprise computer-executable instructions for identifying image features in the selected one or more still images that are along overlaid map data of a roadway, associated with the selected one or more still images, along which a traffic camera is oriented, wherein the video is from the traffic camera, and wherein the still images are map panoramas having overlaid map data.

5. The computer-readable media of claim 1, wherein the computer-executable instructions for deriving the transformation parameters comprise computer-executable instructions for:

selecting a first set of image features from among the identified image features;

deriving transformation parameters to transform and align the sample of the video with at least one of the selected still images such that image features from the sample of the video, that are part of the first set of image features, are aligned with corresponding image features from the at least one of the selected still images;

scoring an overall fit between the transformed and aligned sample of the video and the at least one of the selected still images, the scoring being based on an accuracy of alignment of others of the identified image features, that are not part of the first set of image features;

repeating the deriving and the scoring for subsequent sets of image features, selected from among the identified image features; and

selecting, as the transformation parameters, those transformation parameters that yielded a best score of the overall fit.

6. The computer-readable media of claim 1, wherein the image features are lines and wherein further the computer-executable instructions for deriving the transformation parameters comprise computer-executable instructions for performing homography utilizing the lines.

7. The computer-readable media of claim 1, comprising further computer-executable instructions for:

stabilizing the sample of the video into a composite with individual frames of the sample of the video aligned with one another such that common image elements of each frame overlap.

8. The computer-readable media of claim 1, comprising further computer-executable instructions for:

receiving the video;

transforming and aligning the received video utilizing the transformation parameters; and

generating an amalgamated image comprising at least a portion of the transformed and aligned video overlaid over at least one of the selected still images.

9. One or more computer-readable media comprising computer-executable instructions for visually integrating a video into at least one existing still image that provides greater image scope than the video, the computer-executable instructions performing steps comprising:

receiving the video;

transforming and aligning the received video utilizing transformation parameters; and

generating an amalgamated image comprising at least a portion of the transformed and aligned video overlaid over the least one of existing still images.

10. The computer-readable media of claim 9, wherein the video is from a traffic camera and wherein further the at least one existing still image is a map panorama utilized to provide a street-level view.

11. The computer-readable media of claim 10, wherein the computer-executable instructions for generating an amalgamated image comprise computer-executable instructions for:

filtering out portions of the video other than a roadway and vehicular traffic thereon and overlaying only the roadway and the vehicular traffic onto the at least one existing still image.

12. The computer-readable media of claim 9, wherein the computer-executable instructions for generating an amalgamated image comprise computer-executable instructions for:

filtering out static image elements from the video and overlaying only moving image elements from the video onto the at least one of the selected still images.

13. The computer-readable media of claim 9, wherein the at least one existing still image has associated with it location metadata identifying a location proximate to a location identified by location metadata associated with the video.

14. A method for generating an amalgamated image comprising a video visually integrated into an existing still image that provides greater image scope than the video, the method comprising the steps of:

obtaining a sample of the video;

selecting an existing still image whose image scope comprises at least a portion of an image scope of the video, the selecting being informed by location metadata associated with both the video and the still image;

identifying image features, common to both the sampled video and the selected still image, that are to be utilized in transforming and aligning the sampled video to the selected still image;

deriving, with reference to the identified image features, transformation parameters for transforming and aligning the video for integration with the selected still image such that the video appears within the selected still image with image features of the video being overlaid over equivalent image features of the selected still image;

receiving the video;

transforming and aligning the received video utilizing the transformation parameters; and

generating the amalgamated image comprising at least a portion of the transformed and aligned video overlaid over the selected still image.

15. The method of claim 14, wherein the video is from a traffic camera and wherein further the existing still image is a map panorama utilized to provide a street-level view.

16. The method of claim 14, wherein the identifying image features that are to be utilized in transforming and aligning the sampled video comprises:

deriving an average image comprising those elements that remain static throughout the sample of the video;

deriving a motion mask identifying areas in which elements move throughout the sample of the video; and

identifying image features in the average image that are along the motion mask.

17. The method of claim 14, wherein identifying image features that are to be utilized in transforming and aligning the sampled video comprises identifying image features in the selected still image that are along overlaid map data of a roadway, associated with the selected still image, along which a traffic camera is oriented, wherein the video is from the traffic camera, and wherein the selected still image is a map panorama having overlaid map data.

18. The method of claim 14, wherein the deriving the transformation parameters comprises:

selecting a first set of image features from among the identified image features;

deriving transformation parameters to transform and align the sample of the video with the selected still image such that image features from the sample of the video, that are part of the first set of image features, are aligned with corresponding image features from the selected still image;

scoring an overall fit between the transformed and aligned sample of the video and the selected still image, the scoring being based on an accuracy of alignment of others of the identified image features, that are not part of the first set of image features;

repeating the deriving and the scoring for subsequent sets of image features, selected from among the identified image features; and

selecting, as the transformation parameters, those transformation parameters that yielded a best score of the overall fit.

19. The method of claim 14, wherein the image features are lines and wherein further the deriving the transformation parameters comprises performing homography utilizing the lines.

20. The method of claim 14, further comprising the steps of:

stabilizing the sample of the video into a composite with individual frames of the sample of the video aligned with one another such that common image elements of each frame overlap.