Foreground detection

• US 7,894,669 B2
• Filed: 03/27/2007
• Issued: 02/22/2011
• Est. Priority Date: 12/18/2000
• Status: Active Grant

• Current Assignee: Sportsmedia Technology Corporation
• Original Assignee: Sportvision, Inc. (Sportsmedia Technology Corporation)
• Inventors: Gloudemans, James R. II, Hsiao, Walter
• Primary Examiner(s): Do; Anh Hong
• Application Number:
  US11/691,758
• Publication Number:
  US 20070285575A1
• Time in Patent Office: 1,428 Days
• Field of Search: 382/176, 382/190, 382/110, 382/164, 382/218, 382/106, 358/450, 352/63, 235/383, 386/52, 348/47, 348/51, 348/159, 348/169, 348/E13.008, 348/E13.011
• US Class Current: 382/176
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/174 involving the use of two or more images
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: H04N 5/2627 for providing spin image effect, 3D stop motion effect or temporal freeze effect 2D image animation in general G06T13/80

Abstract

A system is disclosed that can find an image of a foreground object in a still image or video image. Finding the image of the foreground object can be used to reduce errors and reduce the time needed when creating morphs of an image. One implementation uses the detection of the image of the foreground object to create virtual camera movement, which is the illusion that a camera is moving around a scene that is frozen in time.

Claims

1. A machine implemented method for producing virtual camera motion, comprising the steps of:

receiving a set of two or more images of a scene from two or more cameras not physically connected to each other;

identifying foreground for at least a subset of said images of said scene, said step of identifying foreground includes determining a difference field for each of the two or more images with respect to a corresponding adjacent image in time for each image, combining the difference fields for each of the two or more images to generate a combined image, and determining a foreground in the combined image; and

creating a video of said scene conveying an illusion of a camera moving around said scene, said step of creating a video is based on said set of two or more images and said step of identifying foreground, wherein said scene appears frozen in time in said illusion of said camera moving around said scene.

2. The machine implemented method of claim 1, wherein said step of identifying foreground includes:

generating a foreground mask.

3. The machine implemented method of claim 1, wherein said step of identifying foreground includes:

identifying stationary objects in a foreground in the set of two or more images.

4. The machine implemented method of claim 1, wherein said step of identifying foreground includes:

identifying moving objects in a foreground in the set of two or more images.

5. A machine implemented method for producing virtual camera motion, comprising the steps of:

receiving a set of two or more images of a scene from two or more cameras not physically connected to each other;

identifying foreground for at least a subset of said images of said scene; and

creating a video of said scene conveying an illusion of a camera moving around said scene, said step of creating a video is based on said set of two or more images and said step of identifying foreground, wherein said scene appears frozen in time in said illusion of said camera moving around said scene, said step of creating a video includes warping a first image to a second image, both the first image and second image associated with a point in time, the first image associated with a first camera and the second image associated with a second camera of the two or more cameras.

6. The machine implemented method of claim 5, further comprising:

removing the foreground pixels from two selected images of the two or more images;

filling in the foreground pixels of the selected images with background pixels to create background images; and

creating new fields as different blends of the background images.

7. One or more processor readable storage devices having processor readable code embodied on said processor readable storage devices, said processor readable code for programming one or more processors to perform a method comprising the steps of:

accessing a set of two or more images of a scene;

identifying foreground for at least a subset of said images of said scene, said step of identifying foreground includes performing a first frame difference between an intermediate image of said set of two or more images and a previous image of said set of two or more images and performing a second frame difference between said intermediate image and a subsequent image of said set of two or more images, said previous image captured at a time prior to that of said intermediate image, said subsequent image captured at a time subsequent to that of said intermediate image;

generating one or more new images of the scene from the set of two or more images; and

creating a video of said scene conveying an illusion of a camera moving around said scene, said step of creating a video is based on said set of two or more images and said step of identifying foreground, said video including the set of two or more images and the one or more new images.

8. The one or more processor readable storage devices of claim 7, wherein said step of accessing includes:

accessing a first image captured by a first camera and a second image captured by a second camera.

9. The one or more processor readable storage devices of claim 7, wherein said step of identifying foreground includes:

generating a foreground mask.

10. The one or more processor readable storage devices of claim 7, wherein said step of identifying foreground includes:

identifying stationary objects in a foreground in the set of two or more images.

11. The one or more processor readable storage devices of claim 7, wherein said step of identifying foreground includes:

identifying moving objects in a foreground in the set of two or more images.

12. One or more processor readable storage devices having processor readable code embodied on said processor readable storage devices, said processor readable code for programming one or more processors to perform a method comprising the steps of:

accessing a set of two or more images of a scene;

identifying foreground for at least a subset of said images of said scene, said step of identifying foreground includes performing a frame difference for the set of two or more images to generate difference fields, said step of identifying foreground further includes creating pixel clusters in the difference fields, removing one or more selected pixel clusters having on pixels from the difference fields, and filling in one or more selected pixel clusters having off pixels from the difference fields;

generating one or more new images of the scene from the set of two or more images; and

creating a video of said scene conveying an illusion of a camera moving around said scene, said step of creating a video is based on said set of two or more images and said step of identifying foreground, said video including the set of two or more images and the one or more new images.

13. One or more processor readable storage devices having processor readable code embodied on said processor readable storage devices, said processor readable code for programming one or more processors to perform a method comprising the steps of:

accessing a set of two or more images of a scene;

identifying foreground for at least a subset of said images of said scene;

generating one or more new images of the scene from the set of two or more images, said step of generating one or more new images includes generating a background from the set of two or more images of said scene and inserting the identified foreground over said background; and

creating a video of said scene conveying an illusion of a camera moving around said scene, said step of creating a video is based on said set of two or more images and said step of identifying foreground, said video including the set of two or more images and the one or more new images.

14. The one or more processor readable storage devices of claim 13, wherein said step of generating the background includes:

removing the foreground from each of two or more images of said scene; and

filling the removed foreground portion of each image with background pixels.

15. The one or more processor readable storage devices of claim 13, wherein said step of generating the background includes blurring the background.

16. The one or more processor readable storage devices of claim 13, wherein said step of generating the background includes:

blending the background of two images from the set of two or more images.

17. A method for producing virtual camera motion, comprising the steps of:

inputting into an image processing system a first plurality of images of a scene from a first image sensor, said first image sensor captures said scene from a first angle, said first plurality of images includes an intermediate image, a previous image, and a subsequent image;

creating a foreground mask for said intermediate image, said step of creating a foreground mask includes performing a first comparison of said intermediate image with said previous image and performing a second comparison of said intermediate image with said subsequent image, said previous image captured at a time prior to that of said intermediate image, said subsequent image captured at a time subsequent to that of said intermediate image; and

generating a new image based on said foreground mask, said step of generating a new image includes inserting a portion of said intermediate image into said new image; and

outputting said new image.

18. The method of claim 17, wherein:

said step of creating a foreground mask includes combining results from said first comparison and said second comparison.

19. The method of claim 18, wherein:

said step of performing a first comparison includes performing a frame difference between said intermediate image and said previous image;

said step of performing a second comparison includes performing a frame difference between said intermediate image and said subsequent image; and

said step of combining results from said first comparison and said second comparison includes creating a logical AND of said first comparison and said second comparison.

20. The method of claim 17, wherein said step of creating a foreground mask includes:

creating a moving foreground mask; and

creating a stationary foreground mask, said foreground mask includes said stationary foreground mask and said moving foreground mask.

21. A machine implemented method for producing virtual camera motion, comprising the steps of:

receiving a first plurality of images of a scene from a first image sensor, said first image sensor captures said scene from a first angle;

receiving a second plurality of images of said scene from a second image sensor, said second image sensor captures said scene from a second angle;

identifying foreground for an intermediate image of said first plurality of images, said step of identifying foreground includes performing a first comparison of said intermediate image with a second image of said first plurality of images, said second image captured at a time different from that of said intermediate image;

creating a new image, said new image includes a foreground that appears to have been captured between said first angle and said second angle; and

creating a video of said scene conveying an illusion of a camera moving around said scene, said video includes said new image.

22. The machine implemented method of claim 21, wherein:

said step of identifying foreground includes performing a second comparison of said intermediate image with a subsequent image of said first plurality of images, said second image captured at a time prior to that of said intermediate image, said subsequent image captured at a time subsequent to that of said intermediate image.

23. The machine implemented method of claim 22, wherein:

said step of identifying foreground includes combining results from said first comparison and said second comparison.

24. The machine implemented method of claim 23, wherein:

said step of performing a first comparison includes performing a frame difference between said intermediate image and said previous image;

said step of performing a second comparison includes performing a frame difference between said intermediate image and said subsequent image; and

said step of combining results from said first comparison and said second comparison includes creating a logical AND of said first comparison and said second comparison.

25. The machine implemented method of claim 21, wherein said step of identifying foreground includes:

creating a moving foreground mask;

creating a stationary foreground mask; and

adding said stationary foreground mask to said moving foreground mask.

26. The machine implemented method of claim 25, wherein said step of identifying foreground includes:

filtering said moving foreground mask to remove noise.

27. The machine implemented method of claim 21, wherein:

said scene appears frozen in time in said illusion of said camera moving around said scene.

28. The machine implemented method of claim 21, further comprising:

removing foreground pixels from said intermediate image based on said step of identifying foreground; and

filling in said foreground pixels with background pixels.

29. The machine implemented method of claim 21, wherein:

said new image includes a background that appears to have been captured between said first angle and said second angle.

30. The machine implemented method of claim 21, wherein said step of creating a new image includes:

blending the background of said intermediate image with the background of a contemporaneous image of said second plurality of images to create a new background that appears to have been captured between said first angle and said second angle, said contemporaneous image and said intermediate image were captured at substantially the same time.