Foreground detection

• US 7,221,794 B1
• Filed: 08/13/2001
• Issued: 05/22/2007
• Est. Priority Date: 12/18/2000
• Status: Active Grant

• Current Assignee: Sportsmedia Technology Corporation
• Original Assignee: Sportsvision, Inc.
• Inventors: Gloudemans, James R. II, Hsiao, Walter
• Primary Examiner(s): Do; Anh Hong
• Application Number:
  US09/929,598
• Time in Patent Office: 2,108 Days
• Field of Search: 382/110, 382/190, 382/236, 382/283, 382/141, 382/164, 382/294, 382/176, 382/154, 382/173, 345/419, 345/422, 345/592, 345/427, 359/478, 352/45, 352/133, 348/159
• US Class Current: 382/176
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/174 involving the use of two or more images
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: H04N 5/2627 for providing spin image effect, 3D stop motion effect or temporal freeze effect 2D image animation in general G06T13/80

Abstract

A system is disclosed that can find an image of a foreground object in a still image or video image. Finding the image of the foreground object can be used to reduce errors and reduce the time needed when creating morphs of an image. One implementation uses the detection of the image of the foreground object to create virtual camera movement, which is the illusion that a camera is moving around a scene that is frozen in time.

Claims

1. A machine implemented method for producing virtual camera motion, comprising:

receiving a set of two or more images of a scene;

identifying foreground for at least a subset of said images of said scene; and

creating a video of said scene conveying an illusion of a camera moving around said scene, said step of creating a video is based on said set of two or more images and said step of identifying foreground, wherein said step of identifying foreground comprises the steps of,

subtracting a first image of said scene from a second image of said scene to create a first difference, said first image and said second image are from a first camera,

subtracting a third image of said scene from said second image to create a second difference, said third image is from said first camera, and

creating a union of said first difference and said second difference, said union identifies said foreground.

2. A method according to claim 1, wherein said step of identifying further comprises the steps of:

creating clusters of on pixels in said first difference;

creating clusters of off pixels in said first difference;

removing small clusters of on pixels in said first difference;

filling in small clusters of off pixels in said first difference;

creating clusters of on pixels in said second difference;

creating clusters of off pixels in said second difference;

removing small clusters of on pixels in said second difference; and

filling in small clusters of off pixels in said second difference.

3. A method according to claim 2, wherein said step of identifying further comprises the steps of:

filtering said first difference;

filtering said second difference; and

filtering said union.

4. A method according to claim 2, wherein said step of identifying further comprises the steps of:

creating clusters of on pixels in said union;

creating clusters of off pixels in said union;

removing small clusters of on pixels in said union; and

filling in small clusters of off pixels in said union.

5. A machine implemented method for producing virtual camera motion, comprising:

receiving a set of two or more images of a scene;

identifying foreground for at least a subset of said images of said scene; and

creating a video of said scene conveying an illusion of a camera moving around said scene, said step of creating a video is based on said set of two or more images and said step of identifying foreground, wherein,

said step of receiving includes receiving a first video image, a second video image and a third video image,

said first video image views a foreground object at a first angle,

said second video image views said foreground object at a second angle,

said third video image views said foreground object at a third angle,

said step of creating a video includes creating a first set of one or more new video images and a second set of one or more video images,

said first set of one or more video images appear to view said foreground object at angles between said first angle and said second angle,

said second set of one or more video images appear to view said foreground object at angles between said second angle and said third angle, and

said video includes said first set of one or more video images and said second set of one or more video images.

6. A method according to claim 5, wherein:

said video further includes at least a portion of said first video image prior to said first set of one or more video images, at least a portion of said second video image prior to said second set of one or more video images and at least a portion of said third video image subsequent to said second set of one or more video images.

7. A method according to claim 5, wherein said step of creating a video includes the step of:

creating one or more blurred backgrounds based on said step of identifying foreground, said first set of one or more video images and second set of one or more video images include said blurred backgrounds.

8. A method according to claim 5, wherein said step of creating a video includes the step of:

creating one or more solid backgrounds based on said step of identifying foreground, said first set of one or more video images and second set of one or more video images include said solid backgrounds.

9. A method according to claim 5, wherein said step of creating a first set of one or more video images comprises the steps of:

blending said first video image with said second image using different blending factors to create different backgrounds for said first set of one or more video images; and

blurring said backgrounds of said first set one or more video images using different blurring factors for at least a subset of said one or more video images.

10. A method according to claim 9, wherein said step of identifying foreground comprises the steps of:

subtracting a fourth video image from said first video image to create a first difference, said fourth video image and said second video image are from a first camera;

subtracting a fifth video image from said second video image to create a second difference, said fifth image is from said first camera; and

creating a union of said first difference and said second difference, said union identifies said foreground.

11. A method according to claim 5, wherein creating a first set of one or more video images comprises the steps of:

finding edges in said first video image;

finding edges in said second video image;

finding edges in said third video image;

creating matches of at least portions of edges in said first video image, said second video image and said third video image;

creating chains of said matches;

discarding bad chains; and

creating morphs of at least a portion of said first video image and said second video image based on said chains.

12. A method according to claim 5, wherein creating a first set of one or more video images comprises the steps of:

identifying edges in said first video image and said second video image;

interpolating positions for said edges in said first set of one or more video images; and

blending regions of said first video image between said edges with regions of said second video image between said edges according to different blending factors for each of said first set of one or more new video images, said blending factors correlate to said interpolated positions.

13. A method for producing virtual camera motion, comprising the steps of:

receiving a set of two or more images of a scene;

identifying foreground for at least a subset of said images of said scene; and

creating a video of said scene conveying an illusion of a camera moving around said scene, said step of creating a video is based on said set of two or more images and said step of identifying foreground, wherein said step of creating a video comprises the steps of,

warping a first image from a first camera to a second image from a second camera;

warping a third image from a third camera to said second image from said second camera;

removing foregrounds from said first image, said second image and said third image;

filling in background for said removed foreground in said first image, said second

image and said third image;

creating new images by blending backgrounds of said first image with said second image and said second image with said third image;

blurring said new images;

morphing said removed foregrounds;

adding said morphed foregrounds to said new images; and

unwarping said new images.

14. A method for producing virtual camera motion, comprising the steps of:

receiving a set of two or more images of a scene;

identifying foreground for at least a subset of said images of said scene; and

creating a video of said scene conveying an illusion of a camera moving around said scene, said step of creating a video is based on said set of two or more images and said step of identifying foreground, wherein said step of creating includes the steps of,

creating one or more video images having blurred backgrounds based on said step of identifying foreground; and

assembling said created one or more video images into said video.

15. A method for producing virtual camera motion, comprising the steps of:

receiving a set of two or more images of a scene;

identifying foreground for at least a subset of said images of said scene; and

creating a video of said scene conveying an illusion of a camera moving around said scene, said step of creating a video is based on said set of two or more images and said step of identifying foreground, wherein said step of creating includes the steps of,

creating one or more video images by blending a first image from a first camera with a second image from a second camera using different blending factors to create different backgrounds for said one or more video images;

blurring said backgrounds of said one or more video images using different blurring factors for at least a subset of said one or more video images; and

assembling said created one or more video images into said video.

16. A method for producing virtual camera motion, comprising the steps of:

receiving a set of two or more images of a scene;

identifying foreground for at least a subset of said images of said scene; and

creating a video of said scene conveying an illusion of a camera moving around said scene, said step of creating a video is based on said set of two or more images and said step of identifying foreground, wherein said step of creating includes the steps of,

creating one or more video images having solid backgrounds based on said step of identifying foreground, said video includes said video images having solid backgrounds; and

assembling said created one or more video images into said video.

17. A method for producing virtual camera motion, comprising the steps of:

receiving a set of two or more images of a scene;

identifying foreground for at least a subset of said images of said scene; and

creating a video of said scene conveying an illusion of a camera moving around said scene, said step of creating a video is based on said set of two or more images and said step of identifying foreground, wherein said step of creating comprises the steps of,

finding edges in a first video image from a first camera;

finding edges in a second video image from a second camera;

finding edges in a third video image from a third camera;

creating matches of at least portions edges in said first video image, said second video image and said third video image;

creating chains of said matches;

discarding bad chains;

creating morphs of at least a portion of said first video image and said second video image based on said chains;

creating a new set of video images, adding said morphs to said new set of video images; and

assembling said new set of images into said video.

18. A machine implemented method for producing virtual camera motion, comprising the steps of:

receiving two or more images of a scene which view a foreground object from a first set of different angles;

identifying foreground for said two or more images of said scene; and

creating one or more new images of said scene based on said two or more images and step of identifying, said new images appear to view said foreground object from new angles different than said first set of different angles.

19. A method according to claim 18, wherein said step of identifying foreground comprises the steps of:

subtracting a first image of said scene from a second image of said scene to create a first difference, said first image and said second image are from a first camera;

subtracting a third image of said scene from said second image to create a second difference, said third image is from said first camera; and

creating a union of said first difference and said second difference, said union identifies said foreground.

20. A method according to claim 18, further comprising the step of:

assembling said new images of said scene into a movie that conveys the illusion of a camera moving around said scene as said scene appears frozen in time.

21. A method according to claim 18, wherein:

said one or more video images are created with blurred backgrounds based on said step of identifying foreground.

22. A method according to claim 18, wherein said step of creating one or more new images comprises the steps of:

creating said one or more new images by blending a first image from a first camera with a second image from a second camera using different blending factors to create different backgrounds for said one or more new images;

blurring said backgrounds of said one or more new images using different blurring factors for at least a subset of said one or more new images; and

assembling said created one or more video images into a video.

23. A method according to claim 18, wherein said step of creating one or more new images comprises the steps of:

creating said one or more new images having solid backgrounds based on said step of identifying foreground; and

assembling said created one or more video images into said video.

24. A method according to claim 18, wherein said step of creating one or more new images comprises the steps of:

finding edges in a first video image from a first camera, said two or more images of a scene include said first video image;

finding edges in a second video image from a second camera, said two or more images of a scene include said second video image;

finding edges in a third video image from a third camera, said two or more images of a scene include said third video image;

creating matches of at least portions edges in said first video image, said second video image and said third video image;

creating chains of said matches;

discarding bad chains;

creating morphs of at least a portion of said first video image and said second video image based on said chains;

creating a new set of video images, adding said morphs to said new set of video images; and

assembling said new set of images into a video.

25. A machine implemented method for identifying foreground, comprising the steps of:

receiving a first image from a first camera;

receiving a second image from said first camera;

receiving a third image from said first camera;

subtracting said second image from said first image to create a first difference

subtracting said third image from said first image to create a second difference; and

creating a union of said first difference and said second difference, said union identifies said foreground.

26. A method according to claim 25, further comprising the steps of:

creating clusters of on pixels in said first difference;

creating clusters of off pixels in said first difference;

removing small clusters of on pixels in said first difference;

filling in small clusters of off pixels in said first difference;

creating clusters of on pixels in said second difference;

creating clusters of off pixels in said second difference;

removing small clusters of on pixels in said second difference; and

filling in small clusters of off pixels in said second difference.

27. A method according to claim 26, further comprising the steps of:

filtering said first difference;

filtering said second difference; and

filtering said union.

28. A method according to claim 26, further comprising the steps of:

creating clusters of on pixels in said union;

creating clusters of off pixels in said union;

removing small clusters of on pixels in said union; and

filling in small clusters of off pixels in said union.

29. One or more processor readable storage devices having processor readable code embodied on said processor readable storage devices, said processor readable code for programming one or more processors to perform a method comprising the steps of:

receiving a set of two or more images of a scene;

identifying foreground for at least a subset of said images of said scene; and

creating a video of said scene conveying an illusion of a camera moving around said scene, said step of creating a video is based on said set of two or more images and said step of identifying foreground, wherein said step of identifying foreground comprises the steps of,

subtracting a first image of said scene from a second image of said scene to create a first difference, said first image and said second image are from a first camera;

subtracting a third image of said scene from said second image to create a second difference, said third image is from said first camera; and

creating a union of said first difference and said second difference, said union identifies said foreground.

30. One or more processor readable storage devices having processor readable code embodied on said processor readable storage devices, said processor readable code for programming one or more processors to perform a method comprising the steps of:

receiving a set of two or more images of a scene;

identifying foreground for at least a subset of said images of said scene; and

creating a video of said scene conveying an illusion of a camera moving around said scene, said step of creating a video is based on said set of two or more images and said step of identifying foreground, wherein said step of creating includes the steps of,

creating one or more video images having blurred backgrounds based on said step of identifying foreground; and

assembling said created one or more video images into said video.

31. One or more processor readable storage devices having processor readable code embodied on said processor readable storage devices, said processor readable code for programming one or more processors to perform a method comprising the steps of:

receiving a set of two or more images of a scene;

identifying foreground for at least a subset of said images of said scene; and

creating a video of said scene conveying an illusion of a camera moving around said scene, said step of creating a video is based on said set of two or more images and said step of identifying foreground, wherein said step of creating includes the steps of,

creating one or more video images by blending a first image from a first camera with a second image from a second camera using different blending factors to create different backgrounds for said one or more video images;

blurring said backgrounds of said one or more video images using different blurring factors for at least a subset of said one or more video images; and

assembling said created one or more video images into said video.

32. One or more processor readable storage devices having processor readable code embodied on said processor readable storage devices, said processor readable code for programming one or more processors to perform a method comprising the steps of:

receiving a set of two or more images of a scene;

identifying foreground for at least a subset of said images of said scene; and

creating a video of said scene conveying an illusion of a camera moving around said scene, said step of creating a video is based on said set of two or more images and said step of identifying foreground, wherein said step of creating comprises the steps of,

finding edges in a first video image from a first camera;

finding edges in a second video image from a second camera;

finding edges in a third video image from a third camera;

creating matches of at least portions edges in said first video image, said second video image and said third video image;

creating chains of said matches;

discarding bad chains;

creating morphs of at least a portion of said first video image and said second video image based on said chains;

creating a new set of video images, adding said morphs to said new set of video images; and

assembling said new set of images into said video.

33. One or more processor readable storage devices having processor readable code embodied on said processor readable storage devices, said processor readable code for programming one or more processors to perform a method comprising the steps of:

receiving a set of two or more images of a scene;

identifying foreground for at least a subset of said images of said scene; and

creating a video of said scene conveying an illusion of a camera moving around said scene, said step of creating a video is based on said set of two or more images and said step of identifying foreground, wherein said step of creating a video comprises the steps of,

warping a first image from a first camera to a second image from a second camera;

warping a third image from a third camera to said second image from said second camera;

removing foregrounds from said first image, said second image and said third image;

filling in background for said removed foreground in said first image, said second image and said third image;

creating new images by blending backgrounds of said first image with said second image and said second image with said third image;

blurring said new images;

morphing said removed foregrounds;

adding said morphed foregrounds to said new images; and

unwarping said new images.

34. An apparatus, comprising:

a communication interface;

one or more storage devices; and

one or more processors in communication with said one or more storage devices and said communication interface, said one or more processors receive a set of two or more images of a scene, identify foreground for at least a subset of said images of said scene, and create a video of said scene conveying an illusion of a camera moving around said scene frozen in time, said creating said video is based on said set of two or more images and said step of identifying foreground.

35. An apparatus according to claim 34, wherein said one or more processors subtract a first image of said scene from a second image of said scene to create a first difference, said first image and said second image are from a first camera, subtract a third image of said scene from said second image to create a second difference, said third image is from said first camera, and create a union of said first difference and said second difference, said union identifies said foreground.

36. An apparatus according to claim 34, wherein said one or more processors create one or more video images by blending a first image from a first camera with a second image from a second camera using different blending factors to create different backgrounds for said one or more video images, blur said backgrounds of said one or more video images using different blurring factors for at least a subset of said one or more video images, and assemble said created one or more video images into said video.

37. An apparatus according to claim 36, further comprising:

three cameras;

camera control electronics connected to said camera;

synchronization electronics in communication with said camera control electronics; and

one or more time code inserters in communication with said camera control electronics and said processor, said cameras capture said two or more images of said scene, said time code inserters insert time codes into said two or more images of said scene including said first image and said second image, said step of creating one or more video images by blending uses said time codes in said first image and said second image.

38. An apparatus according to claim 34, wherein said one or more processors create one or more video images having blurred backgrounds based on said identified foreground, and assemble said created one or more video images into said video.

39. One or more processor readable storage devices having processor readable code embodied on said processor readable storage devices, said processor readable code for programming one or more processors to perform a method comprising the steps of:

receiving two or more images of a scene which view a foreground object from a first set of different angles;

identifying foreground for said two or more images of said scene; and

creating one or more new images of said scene based on said two or more images and step of identifying, said new images appear to view said foreground object from new angles different than said first set of different angles.

40. One or more processor readable storage devices according to claim 39, wherein said step of identifying foreground comprises the steps of:

subtracting a first image of said scene from a second image of said scene to create a first difference, said first image and said second image are from a first camera;

subtracting a third image of said scene from said second image to create a second difference, said third image is from said first camera; and

creating a union of said first difference and said second difference, said union identifies said foreground.

41. One or more processor readable storage devices according to claim 39, wherein said method further comprises the step of:

assembling said new images of said scene into a movie that conveys the illusion of a camera moving around said scene as said scene appears frozen in time.

42. One or more processor readable storage devices according to claim 39, wherein said step of creating one or more new images comprises the steps of:

creating said one or more new images by blending a first image from a first camera with a second image from a second camera using different blending factors to create different backgrounds for said one or more new images;

blurring said backgrounds of said one or more new images using different blurring factors for at least a subset of said one or more new images; and

assembling said created one or more video images into a video.

43. One or more processor readable storage devices according to claim 39, wherein said step of creating one or more new images comprises the steps of:

finding edges in a first video image from a first camera, said two or more images of a scene include said first video image;

finding edges in a second video image from a second camera, said two or more images of a scene include said second video image;

finding edges in a third video image from a third camera, said two or more images of a scene include said third video image;

creating matches of at least portions edges in said first video image, said second video image and said third video image;

creating chains of said matches;

discarding bad chains;

creating morphs of at least a portion of said first video image and said second video image based on said chains;

creating a new set of video images, adding said morphs to said new set of video images; and

assembling said new set of images into a video.

44. An apparatus, comprising:

a communication interface;

one or more storage devices; and

one or more processors in communication with said one or more storage devices and said communication interface, said one or more processors receive two or more images of a scene which view a foreground object from a first set of different angles, identify foreground for said two or more images of said scene, and create one or more new images of said scene based on said two or more images and said identify, said new images appear to view said foreground object from new angles different than said first set of different angles.

45. An apparatus according to claim 44, wherein said one or more processors subtract a first image of said scene from a second image of said scene to create a first difference, said first image and said second image are from a first camera, subtract a third image of said scene from said second image to create a second difference, said third image is from said first camera, and create a union of said first difference and said second difference, said union identifies said foreground.

46. An apparatus according to claim 44, wherein said one or more processors create said one or more new images by blending a first image from a first camera with a second image from a second camera using different blending factors to create different backgrounds for said one or more new images, blur said backgrounds of said one or more new images using different blurring factors for at least a subset of said one or more new images, and assemble said created one or more video images into a video.

47. An apparatus according to claim 46, further comprising:

three cameras;

camera control electronics connected to said camera;

synchronization electronics in communication with said camera control electronics; and

one or more time code inserters in communication with said camera control electronics and said processor, said cameras capture said two or more images of said scene, said time code inserters insert time codes into said two or more images of said scene including said first image and said second image, said step of creating one or more video images by blending uses said time codes in said first image and said second image.

48. An apparatus according to claim 44, wherein said one or more processors assemble said new images of said scene into a movie that conveys the illusion of a camera moving around said scene as said scene appears frozen in time.

49. An apparatus according to claim 48, wherein said one or more processors edges in a first video image from a first camera, said two or more images of a scene include said first video image, find edges in a second video image from a second camera, said two or more images of a scene include said second video image, find edges in a third video image from a third camera, said two or more images of a scene include said third video image, create matches of at least portions edges in said first video image, said second video image and said third video image, create chains of said matches, discard bad chains, create morphs of at least a portion of said first video image and said second video image based on said chains, create a new set of video images, adding said morphs to said new set of video images, and assemble said new set of images into a video.

50. One or more processor readable storage devices having processor readable code embodied on said processor readable storage devices, said processor readable code for programming one or more processors to perform a method comprising the steps of:

receiving a first image from a first camera;

receiving a second image from said first camera;

receiving a third image from said first camera;

subtracting said second image from said first image to create a first difference

subtracting said third image from said first image to create a second difference; and

creating a union of said first difference and said second difference, said union identifies said foreground.

51. One or more processor readable storage devices according to claim 50, wherein said method further comprises the steps of:

creating clusters of on pixels in said first difference;

creating clusters of off pixels in said first difference;

removing small clusters of on pixels in said first difference;

filling in small clusters of off pixels in said first difference;

creating clusters of on pixels in said second difference;

creating clusters of off pixels in said second difference;

removing small clusters of on pixels in said second difference; and

filling in small clusters of off pixels in said second difference.

52. One or more processor readable storage devices according to claim 51, wherein said method further comprises the steps of:

filtering said first difference;

filtering said second difference; and

filtering said union.

53. One or more processor readable storage devices according to claim 50, wherein said method further comprises the steps of:

creating clusters of on pixels in said union;

creating clusters of off pixels in said union;

removing small clusters of on pixels in said union; and

filling in small clusters of off pixels in said union.

54. An apparatus, comprising:

a communication interface;

one or more storage devices; and

one or more processors in communication with said one or more storage devices and said communication interface, said one or more processors receive a first image from a first camera, receive a second image from said first camera, receive a third image from said first camera, subtract said second image from said first image to create a first difference, subtract said third image from said first image to create a second difference, and create a union of said first difference and said second difference, said union identifies said foreground.

55. An apparatus according to claim 54, wherein said one or more processors create clusters of on pixels in said first difference, create clusters of off pixels in said first difference, remove small clusters of on pixels in said first difference, fill in small clusters of off pixels in said first difference, create clusters of on pixels in said second difference, create clusters of off pixels in said second difference, remove small clusters of on pixels in said second difference, and fill in small clusters of off pixels in said second difference.

56. An apparatus according to claim 55, wherein said one or more processors filter said first difference, filter said second difference, and filter said union.

57. An apparatus according to claim 54, wherein said one or more processors create clusters of on pixels in said union, create clusters of off pixels in said union, remove small clusters of on pixels in said union, and fill in small clusters of off pixels in said union.