Systems, Methods and Devices for Augmenting Video Content

• US 20100067865A1
• Filed: 07/09/2009
• Published: 03/18/2010
• Est. Priority Date: 07/11/2008
• Status: Active Grant

• Current Assignee: Board of Trustees of the Leland Stanford Junior University (Stanford Management Co.)
• Original Assignee: Board of Trustees of the Leland Stanford Junior University (Stanford Management Co.)
• Inventors: Saxena, Ashutosh, Ng, Andrew Y., Batra, Siddharth
• Granted Patent: US 8,477,246 B2
• Time in Patent Office: Days
• US Class Current: 386/278
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20016 Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
• CPC Class Codes: G06T 7/207 for motion estimation over a hierarchy of resolutions multi-resolution motion estimation or hierarchical motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/53
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G11B 27/034 on discs G11B27/036, G11B27/038 take precedence
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay

Abstract

Methods, systems, products and devices are implemented for editing video image frames. According to one such method, image content is embedded into video. A selection input is received for a candidate location in a video frame of the video. The candidate location is traced in subsequent video frames of the video by approximating three-dimensional camera motion between two frames using a model that compensates for camera rotations, camera translations and zooming, and by optimizing the approximation using statistical modeling of three-dimensional camera motion between video frames. Image content is embedded in the candidate location in the subsequent video frames of the video based upon the tracking thereof.

Claims

1. A method for generating video with embedding image content, said method comprising:

receiving a selection input for a candidate location in a video frame of the video;

tracking the candidate location in subsequent video frames of the video by

approximating three-dimensional camera motion between two frames using a model that compensates for camera rotations, camera translations and zooming, and

optimizing the approximation using statistical modeling of three-dimensional camera motion between video frames; and

embedding image content in the candidate location in the subsequent video frames of the video based upon the tracking thereof.

2. The method of claim 1, wherein the statistical modeling includes directed probabilities that a camera position is correct, the directed probabilities being directed from a previous video frame, a current video frame and a previous camera position.

3. The method of claim 1, wherein the statistical modeling includes directed probabilities that a camera position is correct, the directed probabilities being directed from a current video frame and at least one of a subsequent camera position and a subsequent video frame.

4. The method of claim 1, further including the steps of

repeating the step of tracking the candidate location in subsequent video frames of the video at different video resolutions;

and

combining results of the tracking steps at different resolutions to determine the candidate location.

5. The method of claim 1, further including the steps of

determining a mask for the embedded image content as a function of pixel differences between temporally disparate image frames, the pixels being located within the candidate location;

and

obscuring portions of the embedded image content in response to the determined mask.

6. The method of claim 1, further including the steps of matching an appearance of the subsequent video frames by blending one or more global scene properties onto the embedded image content and blending one or more local scene properties onto the embedded image content.

7. The method of claim 1, wherein the statistical modeling is based on machine learning and projective transforms.

8. The method of claim 1, wherein said rendering comprises blending one or more global scene properties onto said image content and blending one or more local scene properties onto said image content.

9. The method of claim 1, wherein the step of approximating three-dimensional camera motion between two frames includes tracking SIFT features between video frames.

10. The method of claim 1, wherein tracking of the candidate location is implemented without prior knowledge of physical markers placed within the video images.

11. The method of claim 1, wherein the embedding image content is a video image-sequence.

12. The method of claim 1, wherein the image content is accessed from a remote file location that can be dynamically modified and wherein modified versions of the image content can be embedded using results of the tracked candidate location that are independent of the dynamic modification.

13. An electronic circuit configured and arranged to:

receive a selection input for a candidate location in a first video frame of the video;

track the candidate location in subsequent video frames of the video by

approximate three-dimensional camera motion between two frames, and

optimize the approximation using statistical modeling of three-dimensional camera motion between video frames; and

embed image content in the candidate location in the subsequent video frames of the video.

14. The circuit of claim 13, wherein the circuit is further configured and arranged to track users that view the video frames with embedded image content and track purchases made for goods or services advertised by the embedded image content.

15. The circuit of claim 13, wherein the circuit is further configured and arranged to generate a projective transform that updates the candidate location for a subsequent video frame, the generation including the steps of

initializing the projective transform based upon a comparison of image featured common to the first video frame and the subsequent video frame;

optimize the projective transform as a function of a normalized cross correlation and a relative deviation from the initial projective transform;

increasing the resolution of the first video frame and the subsequent video frame; and

repeating the step of optimizing using the frames at the increased resolution.

16. The circuit of claim 13, wherein the circuit is further configured and arranged to generate an occlusion mask by

correlating pixels in the first video frame and the subsequent video frame as a function of the tracked candidate location;

calculating, in a color space, a distance between the correlated pixels; and

determining occluded pixels based upon a threshold distance.

17. The circuit of claim 13, further including a display circuit for providing a visual display of the video with the embedded image content.

18. A computer product comprising:

computer readable medium storing instructions that when executed perform the steps of;

receiving a selection input for a candidate location in a video frame of the video;

tracking the candidate location in subsequent video frames of the video by

approximating three-dimensional camera motion between two frames using a model that compensates for camera rotations, camera translations and zooming, and

optimizing the approximation using statistical modeling of three-dimensional camera motion between video frames; and

embedding image content in the candidate location in the subsequent video frames of the video based upon the tracking thereof.

19. The product of claim 18, wherein the computer readable medium further includes instructions for rendering the embedding image content by

determining a mean and variance of color intensity in the candidate location without the embedded image content;

adjusting the color intensity of the embedded image content towards the determined mean and variance; and

multiplicatively combining the luminosities of the embedded image content and in the candidate location without the embedded image content.

20. The product of claim 18, wherein the computer readable medium is a non-volatile memory circuit.