System and method for managing the visual effects insertion in a video stream

• US 8,483,549 B2
• Filed: 08/17/2006
• Issued: 07/09/2013
• Est. Priority Date: 08/17/2005
• Status: Active Grant

• Current Assignee: Orad Hi-Tec Systems Ltd. (Avid Technology Incorporated)
• Original Assignee: Orad Hi-Tec Systems Ltd. (Avid Technology Incorporated)
• Inventors: Steinberg, Alexander, Sharir, Avi, Benovici, Ofir, Livshits, Zinovy, Godet, Guillaume, Wojdala, Andrzej
• Primary Examiner(s): Kim, Jung
• Assistant Examiner(s): Ho, Thomas
• Application Number:
  US12/064,149
• Publication Number:
  US 20080226253A1
• Time in Patent Office: 2,518 Days
• Field of Search: 386/280, 386/242
• US Class Current: 386/280
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay

Abstract

A method and system are presented for use in real-time insertion of visual effects in a video stream. At least one reference image is used that contains a region of interest, which is to be involved in the video stream. the reference image is processed by defining a set of at least a few reference features therein, and inserting the visual effects into the reference image with a desired orientation of the visual effects with respect to the region of interest in the reference image. Successive video frames are grabbed in a sequence of a few video frames, and at least one of video frames is used as at least one snapshot image. The snapshot image(s) is/are process by selecting therein at least two from the reference features and determining a reference transformation between the snapshot image and the reference image. Independently of this processing of the snapshot image, the snapshot image and other video frames are processed to determine a sequence of transformations between them. These data are utilized to determine an operative transformation between the reference image and either one of the video frames, thereby enabling real-time management of the appearance of the visual effects in the video stream by using the operative transformation for the insertion of the visual effects in other video frames.

Claims

1. A method for use in real-time insertion of visual effects in a video stream, the method comprising:

i. processing at least one reference image that contains a region of interest, which is to be involved in the video stream, said processing comprising receiving user input for inserting visual effects into said at least one reference image with a desired orientation of said visual effects with respect to the region of interest to thereby form an augmented reference image, and defining a set of at least a few reference features in said at least one reference image;

ii. grabbing successive sequence of video frames from said video stream, and processing said successive sequence to generate a corresponding augmented sequence of video frames with said visual effects appearing therein;

said processing comprising;

selecting at least one frame of said video frames, in which at least two reference features from said few reference features appear, for used as at least one snapshot image, and processing said snapshot image to establish correspondence between at least two reference features in the snapshot image and the reference image and determine a reference transformation between the snapshot image and the reference image;

independent of said processing of the snapshot image with said at least two features, applying an independent processing to the snapshot image and to other frames of said successive sequence of the video frames and determining a sequence of transformations between the frames of said successive sequence; and

utilizing said determined sequence of transformations and said determined reference transformation to determine an operative transformation between the reference image and either one of said frames in the successive sequence;

thereby enabling real-time management of the appearance of the visual effects in the video stream by using said operative transformation for the insertion of the visual effects in other video frames of said successive sequence.

2. The method of claim 1, wherein said determining of the sequence of transformations between said at least some of the video frames comprises applying pattern recognition to the video frames.

3. The method of claim 2, wherein said pattern recognition utilizes estimation of a motion of background windows at a video rate.

4. The method of claim 1, wherein the video frames are grabbed using an imager configured for rotation while at a fixed position thereof with respect to the region of interest.

5. The method of claim 1, wherein said processing of said one or more snapshot images comprises selecting and marking of only two of said reference features therein.

6. The method of claim 1, wherein said set of reference features includes at least four reference features.

7. The method of claim 1, comprising selecting in at least some of the video frames at least one moving object in the region of interest, a current position of said at least one moving object determining the appearance of the visual effects in the current video frame.

8. The method of claim 1, wherein said set of the reference features in the reference image and said at least two reference features in said one or more snapshot images are selected and marked by user.

9. The method of claim 1, for use in the insertion of visual effects in real time broadcasting or in post production.

10. The method of claim 1, for use in the insertion of the visual effects and graphic advertisements, during live broadcasting or post production of a sport game.

11. The method of claim 10, wherein said sport game includes at least one of the following:

soccer, basketball, volleyball, beach volleyball, handball, cricket, rugby, American football, golf, track and field sports, skiing, ice skating, hockey, badminton, curling, tennis, table tennis, biking, racing, boxing, judo, sumo, wrestling, and swimming.

12. A system for use in real-time insertion of visual effects in a video stream, the system comprising:

(a) a frame grabber utility configured for receiving data indicative of the video stream;

(b) a manager utility connectable to the frame grabber and configured and operable to carry out the following;

receive user input with regard to certain visual effects for inserting the visual effects into at least one reference image with a desired orientation of said visual effects with respect to a region of interest in the at least one reference image, process the reference image and inserting said visual effects into said reference image with said desired orientation of said visual effects with respect to the region of interest, to thereby form an augmented reference image, and define a set of at least a few reference features in said at least one reference image; and

select at least one video frame of a successive sequence of video frames in which at least two of said reference features appear, for use as at least one snapshot image;

(c) a processor utility configured and operative to be responsive to data indicative of the successive sequence of video frames of said video stream, data indicative of the reference image with said few reference features therein, and data indicative of said at least one snapshot image with said at least two reference features therein;

said processor utility being configured for processing said successive sequence of video frames and generating a corresponding augmented sequence of video frames with said visual effects appearing therein by independently carrying out a first processing of said data indicative of the video frames sequence and determining a sequence of transformations between at least some of said video frames including that of the at least one snapshot image, and a second processing of said data indicative of the reference and snapshot images and determining a reference transformation between the snapshot image and the reference image based on correspondence between at least two reference features in the reference image and snapshot image; and

being configured for analyzing the determined reference transformation and the determined sequence of transformations to determine an operative transformation between the reference image and either one of said video frames in the successive sequence, and use this operative transformation for real-time management of the appearance of the visual effects in the video stream.

13. The system of claim 12, wherein said processor utility is configured and operable to perform said determining of the sequence of transformations between said at least some of the video frames by applying pattern recognition to the video frames.

14. The system of claim 13, wherein said pattern recognition comprises estimation of a motion of background windows at a video rate.

15. The system of claim 12, wherein the frame grabber is configured and operable for grabbing the video frames using an imager configured for rotation while at a fixed position thereof with respect to the region of interest.

16. The system of claim 12, wherein said processor utility is configured for processing of said one or more snapshot images by selecting and marking of only two of said reference features therein.

17. The system of claim 12, wherein said set of reference features includes at least four reference features.

18. The system of claim 12, wherein said processor utility is configured and operable for selecting in at least some of the video frames at least one moving object in the region of interest, a current position of said at least one moving object determining the appearance of the visual effects in the current video frame.

19. The system of claim 12, for use in the insertion of visual effects in real time broadcasting or in post production.

20. The system of claim 12, for use in the insertion of the visual effects and graphic advertisements, during live broadcasting or post production of a sport game.