Visualizing video within existing still images

• US 9,594,960 B2
• Filed: 09/14/2010
• Issued: 03/14/2017
• Est. Priority Date: 09/14/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Ofek, Eyal, Chen, Billy
• Primary Examiner(s): Vaughn, Jr., William C
• Assistant Examiner(s): JEAN BAPTISTE, JERRY T
• Application Number:
  US12/881,704
• Publication Number:
  US 20120062748A1
• Time in Patent Office: 2,373 Days
• Field of Search: 348/149
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 2207/30236 Traffic on road, railway or crossing
• CPC Class Codes: G06T 2207/30256 Lane; Road marking
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: G06V 20/182 Network patterns, e.g. roads or rivers
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 20/54 of traffic, e.g. cars on the road, trains or boats
• CPC Class Codes: G08G 1/04 using optical or ultrasonic detectors
• CPC Class Codes: H04N 23/6845 by combination of a plurality of images sequentially taken
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay

Abstract

Video from a video camera can be integrated into a still image, with which it shares common elements, to provide greater context and understandability. Pre-processing can derive transformation parameters for transforming and aligning the video to be integrated into the still image in a visually fluid manner. The transformation parameters can then be utilized to transform and align the video in real-time and display it within the still image. Pre-processing can comprise stabilization of video, if the video camera is moveable, and can comprise identification of areas of motion and of static elements. Transformation parameters can be derived by fitting the static elements of the video to portions of one or more existing images. Display of the video in real-time in the still image can include display of the entire transformed and aligned video image, or of only selected sections, to provide for a smoother visual integration.

Claims

1. A computing device comprising:

one or more processing units; and

one or more computer-readable storage media comprising computer-executable instructions which, when executed by the one or more processing units, cause the computing device to;

obtain a sample of a video image feed;

select one or more pre-existing still images whose image scope comprises at least a portion of an image scope of the obtained sample of the video image feed, the selecting being informed by location metadata associated with both the video image feed and the still images;

derive an average image comprising those elements that remain static throughout the sample of the video image feed;

derive a motion mask identifying areas in which elements move throughout the sample of the video image feed;

identifying image features in the average image that are along the motion mask, having corresponding image features in the selected one or more still images; and

derive transformation parameters to transform and align the video image feed such that the identified image features in the sample of the video image feed, after transformation and alignment, have a visual size and visual appearance equivalent to that of the corresponding image features in the selected one or more still images, and can be equivalently overlaid over the corresponding image features in the selected one or more still images.

2. The computing device of claim 1, wherein the video image feed is from a traffic camera and wherein further the pre-existing still images are panoramas providing a street-level view.

3. The computing device of claim 1, wherein the computer-executable instructions for identifying the image features in the video image feed comprise computer-executable instructions for identifying image features in the selected one or more still images that are along a roadway, associated with the selected one or more still images, along which a traffic camera is oriented, wherein the video image feed is from the traffic camera, and wherein the still images are panoramas.

4. The computing device of claim 1, wherein the computer-executable instructions for deriving the transformation parameters comprise computer-executable instructions for:

selecting a first set of image features from among the identified image features in the sample of the video image feed;

deriving preliminary transformation parameters;

transforming and aligning the sample of the video image feed with the derived preliminary transformation parameters;

scoring an overall fit between the transformed and aligned sample of the video image feed and the at least one of the selected still images, the scoring being based on an accuracy of alignment of others of the identified image features that are not part of the first set of image features;

repeating the deriving and the scoring for subsequent sets of image features, selected from among the identified image features in the sample of the video image feed; and

selecting, as the transformation parameters, those preliminary transformation parameters that yielded a best score of the overall fit.

5. The computing device of claim 1, wherein the identified image features are traffic lane marking lines and wherein further the computer-executable instructions for deriving the transformation parameters comprise computer-executable instructions for performing homography utilizing the traffic lane marking lines.

6. The computing device of claim 1, wherein the one or more computer-readable storage media comprise further computer-executable instructions, which, when executed by the one or more processing units, cause the computing device to:

stabilize the sample of the video image feed into a composite with individual frames of the sample of the video image feed aligned with one another such that common image elements of each frame overlap.

7. The computing device of claim 1, wherein the one or more computer-readable storage media comprise further computer-executable instructions, which, when executed by the one or more processing units, cause the computing device to:

transform and align the video image feed while it is being received utilizing the transformation parameters; and

generate a continuously updated amalgamated image comprising at least a portion of an image scope of the transformed and aligned video image feed overlaid over at least one of the selected still images such that the portion of the image scope of the transformed and aligned video image feed is equivalent to an image scope of that portion of the least one of the selected still images over which the transformed and aligned video image feed is overlaid.

8. A computing device comprising:

one or more processing units; and

one or more computer-readable storage media comprising computer-executable instructions which, when executed by the one or more processing units, cause the computing device to;

transform and align a video image feed, while it is being received, utilizing transformation parameters; and

generate a continuously updated amalgamated image comprising at least a portion of an image scope of the transformed and aligned video image feed overlaid over at least one pre-existing still image, which was selected based on location metadata associated with both the video image feed and the at least one pre-existing still image, such that the portion of the image scope of the transformed and aligned video image feed is equivalent to an image scope of that portion of the least one pre-existing still image over which the transformed and aligned video image feed is overlaid, wherein the transformed and aligned video image feed comprises image features having a visual size and appearance equivalent to that of corresponding images features in the least one pre-existing still image;

wherein the transformation parameters are derived based on image features in an average image that are along a motion mask, the image features having corresponding image features in the at least one pre-existing still image, wherein the average image comprises those elements that remain static throughout a sample of the video image feed, and wherein further the motion mask identifies areas in which elements move through the sample of the video image feed.

9. The computing device of claim 8, wherein the video image feed is from a traffic camera and wherein further the at least one pre-existing still image is a panorama.

10. The computing device of claim 9, wherein the computer-executable instructions for generating the amalgamated image comprise computer-executable instructions for:

filtering out portions of the video image feed other than a roadway and vehicular traffic thereon and overlaying only the roadway and the vehicular traffic onto the at least one pre-existing still image.

11. The computing device of claim 8, wherein the computer-executable instructions for generating the amalgamated image comprise computer-executable instructions for:

filtering out static image elements from the video image feed and overlaying only moving image elements from the video image feed onto the at least one pre-existing still image.

12. The computing device of claim 8, wherein the at least one pre-existing still image has associated with it location metadata identifying a location proximate to a location identified by location metadata associated with the video image feed.

19. The computing device of claim 8, wherein the generating the continuously updated amalgamated image comprises generating the continuously updated amalgamated image as part of a street-level view presented through a map-based website.

13. A method for generating an amalgamated image comprising a video image feed visually integrated into a pre-existing still image, the method comprising the steps of:

obtaining a sample of the video image feed;

selecting the pre-existing still image based upon its image scope comprising at least a portion of an image scope of the obtained sample of the video image feed, the selecting being informed by location metadata associated with both the video image feed and the still image;

deriving an average image comprising those elements that remain static throughout the sample of the video image feed;

deriving a motion mask identifying areas in which elements move throughout the sample of the video image feed;

identifying image features in the average image that are along the motion mask, having corresponding image features in the pre-existing still image; and

deriving transformation parameters to transform and align the video image feed such that the identified image features in the sample of the video image feed, after transformation and alignment, have a visual size and appearance equivalent to that of the corresponding image features in the pre-existing still image and can be equivalently overlaid over the corresponding image features in the selected still image.

14. The method of claim 13, wherein the video image feed is from a traffic camera and wherein further the pre-existing still image is a map panorama utilized to provide a street-level view.

15. The method of claim 13, wherein identifying image features in the video image feed comprises identifying image features in the pre-existing still image that are along a roadway, associated with the pre-existing still image, along which a traffic camera is oriented, wherein the video image feed is from the traffic camera, and wherein the pre-existing still image is a panorama.

16. The method of claim 13, wherein the deriving the transformation parameters comprises:

selecting a first set of image features from among the identified image features in the sample of the video image feed;

deriving preliminary transformation parameters;

transforming and aligning the sample of the video image feed with the derived preliminary transformation parameters;

scoring an overall fit between the transformed and aligned sample of the video image feed and the selected still image, the scoring being based on an accuracy of alignment of others of the identified image features that are not part of the first set of image features;

repeating the deriving and the scoring for subsequent sets of image features, selected from among the identified image features in the sample of the video image feed; and

selecting, as the transformation parameters, those preliminary transformation parameters that yielded a best score of the overall fit.

17. The method of claim 13, wherein the identified image features are traffic lane marking lines and wherein further the deriving the transformation parameters comprises performing homography utilizing the traffic lane marking lines.

18. The method of claim 13, further comprising the steps of:

stabilizing the sample of the video image feed into a composite with individual frames of the sample of the video image feed aligned with one another such that common image elements of each frame overlap.

20. The method of claim 13, further comprising the steps of:

transforming and aligning the video image feed while it is being received utilizing the transformation parameters; and

generating a continuously updated amalgamated image comprising at least a portion of an image scope of the transformed and aligned video image feed overlaid over at least one of the selected still images such that the portion of the image scope of the transformed and aligned video image feed is equivalent to an image scope of that portion of the least one of the selected still images over which the transformed and aligned video image feed is overlaid.