Apparatus and method for producing images using digitally stored dynamic background sets

• US 6,731,825 B1
• Filed: 07/11/2000
• Issued: 05/04/2004
• Est. Priority Date: 07/11/2000
• Status: Expired due to Term

• Current Assignee: Mind Fusion, LLC (Ascend Innovation Management, LLC)
• Original Assignee: Sarnoff Corporation (SRI International, Inc.)
• Inventors: Hashfield, Paul, Acampora, Alfonse Anthony, Fedele, Nicola John, Bortfeld, David Paul
• Primary Examiner(s): Couso, Yon J.
• Application Number:
  US09/613,926
• Time in Patent Office: 1,393 Days
• Field of Search: 382/284, 382/294, 382/300, 358/464, 358/450, 348/584, 348/586, 348/598, 345/589, 345/592, 345/630, 345/632-641
• US Class Current: 382/284
• CPC Class Codes: H04N 5/2224 related to virtual studio applications
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay

Abstract

A method and apparatus for providing a composite of stored dynamic background activities with newly filmed or captured live foreground activities, thereby providing a real-time and on-site guide to actors and directors.

Claims

1. Method for providing a composite image sequence, said method comprising the steps of:

a) retrieving a background image sequence along with ancillary data representative of a scene condition when said background image sequence was captured, wherein said ancillary data for said background image sequence is generated when said background image sequence is captured;

b) retrieving a foreground image sequence along with ancillary data representative of the a scene condition when said foreground image sequence was captured, wherein said ancillary data for said foreground image sequence is generated when said foreground image sequence is captured; and

c) compositing said background image sequence with said foreground image sequence in accordance with said ancillary data of said background image sequence and said ancillary data of said foreground image sequence to form the composite image sequence.

2. The method of claim 1, wherein said compositing step c) composites said background image sequence with said foreground image sequence in low resolution.

3. The method of claim 1, further comprising the step of:

b1) performing an image processing operation on said retrieved background image sequence to produce a modified background image sequence, wherein said compositing step c) composites said modified background image sequence with said foreground image sequence.

4. The method of claim 3, wherein said image processing operation is a mosaic operation that interpolates said retrieved background image sequence into said modified background image sequence having a different view point.

5. The method of claim 3, wherein said image processing operation decimates said retrieved background image sequence into said modified background image sequence having a lower resolution.

6. The method of claim 1, wherein said ancillary data comprises information associated with a camera parameter.

7. The method of claim 1, wherein said ancillary data comprises information associated with a lighting condition.

8. The method of claim 1, further comprising the step of:

d) distributing said composite image sequence to a viewfinder of a camera.

9. The method of claim 1, further comprising the step of:

d) distributing said composite image sequence to a monitor.

10. The method of claim 1, wherein said ancillary data is stored within a video field or frame of said background image sequence.

11. The method of claim 10, wherein said ancillary data is stored within a vertical blanking interval of said video field or frame.

12. The method of claim 1, wherein said ancillary data is stored within a video field or frame of said foreground image sequence.

13. The method of claim 12, wherein said ancillary data is stored within a vertical blanking interval of said video field or frame.

14. An apparatus for providing a composite image sequence, said apparatus comprising:

a first storage for storing a background image sequence along with ancillary data representative of a scene condition when said background image sequence was captured, wherein said ancillary data for said background image sequence is generated when said background image sequence is captured;

a second storage for storing a foreground image sequence along with ancillary data representative of then a scene condition when said foreground image sequence was captured, wherein said ancillary data for said foreground image sequence is generated when said foreground image sequence is captured; and

a controller, coupled to said first and second storages, for compositing said background image sequence with said foreground image sequence in accordance with said ancillary data of said background image sequence and said ancillary data of said foreground image sequence to form the composite image sequence.

15. The apparatus of claim 14, wherein said controller composites said background image sequence with said foreground image sequence in low resolution.

16. The apparatus of claim 14, further comprising:

a pyramid generator, coupled to said controller, for performing an image processing operation on said retrieved background image sequence to produce a modified background image sequence, wherein said controller composites said modified background image sequence with said foreground image sequence.

17. The apparatus of claim 16, wherein said pyramid generator interpolates said retrieved background image sequence into said modified background image sequence having a different view point.

18. The apparatus of claim 16, wherein said pyramid generator decimates said retrieved background image sequence into said modified background image sequence having a lower resolution.

19. The apparatus of claim 14, further comprising:

a camera, coupled to said controller, wherein said composite image sequence is distributed to said viewfinder of said camera.

20. A computer-readable medium having stored thereon a plurality of instructions, the plurality of instructions including instructions which, when executed by a processor, cause the processor to perform the steps comprising of:

a) retrieving a background image sequence along with ancillary data representative of a scene condition when said background image sequence was captured, wherein said ancillary data for said background image sequence is generated when said background image sequence is captured;

b) retrieving a foreground image sequence along with ancillary data representative of a scene condition when said foreground image sequence was captured, wherein said ancillary data for said foreground image sequence is generated when said foreground image sequence is captured; and

c) compositing said background image sequence with said foreground image sequence in accordance with said ancillary data of said background image sequence and said ancillary data of said foreground image sequence to form the composite image sequence.