Scene recognition methods for virtual insertions

• US 20100238351A1
• Filed: 03/15/2010
• Published: 09/23/2010
• Est. Priority Date: 03/13/2009
• Status: Active Grant

• Current Assignee: Disney Enterprises Incorporated (The Walt Disney Company)
• Original Assignee: Disney Enterprises Incorporated (The Walt Disney Company)
• Inventors: Shamur, Eyal, House, Gregory, Gong, Ximin, Katz, Sheldon
• Granted Patent: US 8,599,317 B2
• Time in Patent Office: Days
• US Class Current: 348/598
• CPC Class Codes: G11B 27/036 Insert-editing
• CPC Class Codes: H04H 60/04 Studio equipment; Interconnection of studios
• CPC Class Codes: H04H 60/37 for identifying segments of broadcast information, e.g. scenes or extracting programme ID
• CPC Class Codes: H04H 60/59 of video recognising characters or patterns in general G06F18/00, G06V20/00
• CPC Class Codes: H04H 60/66 for using the result on distributors' side

Abstract

A method of adding a virtual insertion to an image, according to an embodiment, includes extracting dynamic features from an input image, associating the dynamic extracted features with dynamic reference features in a reference feature database, generating a camera model based on the associations, mixing a virtual insertion into the input image based on the camera model, and outputting an image containing both the input image and the virtual insertion. According to another embodiment, a method of adding a virtual insertion to an image includes generating a biased camera model using a statistically selected subset of a plurality of non-fixed regions of the image, locating fixed reference objects in the image, using the biased camera model as an entry point for a search, generating a corrected camera model using the fixed reference objects in the image, and adding a virtual insertion to the image using the corrected camera model.

Claims

1. A method of adding a virtual insertion to an input video sequence, comprising:

extracting features from an input image, a plurality of extracted features being dynamic and the input image being contained in the input video sequence;

associating the extracted features with reference features in a reference feature database, a plurality of the reference features being dynamic;

generating a camera model based on the associations between the extracted features and the reference features;

mixing a virtual insertion into the video sequence based on the camera model; and

outputting an output video sequence containing the virtual insertion.

2. The method of claim 1, further comprising:

iteratively repeating the extracting, associating and generating steps to refine the camera model.

3. The method of claim 2, wherein iteratively repeating comprises:

identifying one or more fixed reference features based on a camera model generated in a previous iteration; and

generating an updated camera model using the one or more fixed reference features.

4. The method of claim 1, wherein the extracting step is automated.

5. The method of claim 1, further comprising:

calibrating a scene prior to extracting features from an input image.

6. The method of claim 5, wherein calibrating comprises:

mapping locations in a reference image to locations in a unifying reference space;

generating a reference camera model based on the association between the reference image locations and the reference space locations;

extracting reference features from the reference camera model, the plurality of reference features including a dynamic object; and

updating the reference feature database with the extracted reference features.

7. The method of claim 6, wherein mapping comprises determining reference point world coordinates relative to known landmarks in a scene.

8. The method of claim 5, wherein calibrating a scene occurs prior to an event.

9. The method of claim 5, further comprising repeating the calibrating step when the features to be extracted from the input image have changed substantially from the reference features in the database.

10. The method of claim 5, further comprising repeating the extracting step when the camera model based on the associations between the extracted features and the reference features cannot be generated within a given amount of time.

11. The method of claim 1, wherein the extracting, associating, generating, mixing, and outputting steps are performed substantially in real time.

12. The method of claim 1, wherein the dynamic features include spectators at a sporting event.

13. A method of adding a virtual insertion to a video sequence, comprising:

generating a biased camera model using a statistically selected subset of a plurality of non-fixed regions of an image contained in the video sequence, wherein the biased camera model provides approximate information regarding locations of reference objects;

locating a fixed reference object in the image using the biased camera model as an entry point for a restricted location search;

generating a corrected camera model using the fixed reference object in the image to refine the biased camera model;

adding a virtual insertion to the video sequence using the corrected camera model; and

outputting the video sequence having the virtual insertion.

14. The method of claim 13, wherein the virtual insertion is an advertisement.

15. The method of claim 13, wherein the non-fixed regions of an image comprise spectators at an event.

16. The method of claim 13, wherein the fixed reference object is a boundary marking.

17. The method of claim 13, wherein the method is performed substantially in real time.

18. The method of claim 13, wherein generating a biased camera model comprises inputting dynamic reference points into one or more of a random sample consensus algorithm, a reference point culling algorithm, or an error minimization algorithm.

19. The method of claim 13, wherein generating a corrected camera model comprises calculating a homography using a pairs of points technique.

20. The method of claim 13, wherein generating a corrected camera model comprises minimizing the distances between fixed reference points in the image and corresponding reference points in a world coordinate system.

21. A method of tracking multiple virtual insertions in a video scene, comprising:

tracking a first virtual insertion based on a first camera model obtained in a first search process;

positioning a second virtual insertion based on a second camera model obtained in a second search process; and

tracking the second virtual insertion based on the first camera model.

22. The method of claim 21, wherein tracking a first virtual insertion comprises using a camera model obtained in a first search process and world coordinates, and tracking the second virtual insertion comprises translating image coordinates obtained in the second search process into the world coordinates.