Object-oriented video system

• US 6,078,619 A
• Filed: 03/10/1999
• Issued: 06/20/2000
• Est. Priority Date: 09/12/1996
• Status: Expired due to Term

• Current Assignee: Hanger Solutions, LLC (IP Investments Group LLC), University of Bath
• Original Assignee: University of Bath
• Inventors: Nicholls, Jeremy Andrew, Monro, Donald Martin
• Primary Examiner(s): LIN, BRYAN T
• Application Number:
  US09/267,365
• Time in Patent Office: 468 Days
• Field of Search: 348/384, 348/390, 348/409, 348/415, 348/416, 348/420, 348/700, 348/701, 382/232, 382/233, 382/236, 382/238
• US Class Current: 375/240
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/152 by measuring the fullness of the transmission buffer
• CPC Class Codes: H04N 19/176 the region being a block, e.g. a macroblock
• CPC Class Codes: H04N 19/18 the unit being a set of transform coefficients
• CPC Class Codes: H04N 19/23 with coding of regions that are present throughout a whole video segment, e.g. sprites, background or mosaic
• CPC Class Codes: H04N 19/37 with arrangements for assigning different transmission priorities to video input data or to video coded data
• CPC Class Codes: H04N 19/50 using predictive coding H04N19/61 takes precedence
• CPC Class Codes: H04N 19/503 involving temporal prediction adaptive coding with adaptive selection between spatial and temporal predictive coding H04N19/107; adaptive coding with adaptive selection among a plurality of temporal predictive coding modes H04N19/109
• CPC Class Codes: H04N 19/507 using conditional replenishment
• CPC Class Codes: H04N 19/59 involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution
• CPC Class Codes: H04N 19/88 involving rearrangement of data among different coding units, e.g. shuffling, interleaving, scrambling or permutation of pixel data or permutation of transform coefficient data among different blocks
• CPC Class Codes: H04N 19/96 Tree coding, e.g. quad-tree coding
• CPC Class Codes: H04N 19/99 involving fractal coding

Abstract

An object-oriented video system is implemented as a two layer object model in a software-only video compressor. Quadtree decomposition on an error metric between the input and transmitted images directs the coder towards a foreground layer of active image fragments. A rate buffering system limits the bandwidth by transmitting only the foreground blocks which most improve the image and are above some error threshold. A high fidelity background layer is identified and communicated to the decoder, which can be used to redraw background fragments as foreground objects move across them. Blocks can be coded by various methods, such as fractal transforms or truncated DCTs. The system can be implemented in RISC processors without the need for dedicated hardware. It is suitable for low bit rate applications with slowly varying backgrounds, such as personal video communications over packet networks, or closed circuit TV surveillance.

Claims

1. A coder for an object-oriented video system, the coder comprising

(a) Input image means for receiving a sequence of input images, together defining a moving image, for coding;

(b) An object separator arranged to separate out from the moving image a foreground object and a background object, the background object being relatively motionless with respect to the foreground object, the object separator utilizing

(i) a potential background store, within which is built up, over a plurality of input images a potential background, and

(ii) an accepted background store within which is stored at least some parts of the potential background which have persisted for a desired period; and

(c) A block coder arranged to code and classify blocks of the image and to produce an output date stream for storage or onward transmission, each coded block within the data stream being identified as relating either to the foreground or to the background object.

2. A coder as claimed in claim 1 in which those blocks in the output data stream that relate to the foreground object are flagged as such.

3. A coder as claimed in claim 1 in which those blocks in the output data stream that relate to the background object are flagged as such.

4. A coder as claimed in claim 1 in which the output data stream is arranged to include background object blocks and foreground object blocks in a predefined order and sequence.

5. A coder as claimed in claim 1 in which the moving image is partitioned into a plurality of input blocks which are compared, block by block, with corresponding blocks in similar partitions of the stored accepted background object and the stored potential background object.

6. A coder as claimed in claim 5 in which each block in the accepted background object is overwritten by the corresponding block in the potential background object when the potential background object block is repeatedly found to be more similar to the corresponding input block than to the accepted background block.

7. A coder as claimed in claim 1 in which each block in the output data stream is identified as one of three types:

(a) a newly-sent block from the background object;

(b) a previously-sent block from the background object; and

(c) a block from the foreground object.

8. A coder as claimed in claim 7 in which block types (a) and (c) have associated with them data representative of the image fragment covered by the block.

9. A coder as claimed in claim 7 or claim 8 in which block type (b) has associated with it an instruction, comprehensible to a compatible decoder, to redraw the image fragment covered by the block from the known background.

10. A coder as claimed in claim 1, wherein the coder is, arranged to maintain the background object according to its Y-luminance values.

11. A coder as claimed in claim 1 in which the stored accepted background object is representative of image fragments which have previously been coded and placed in the output stream as blocks identified as relating to the background object.

12. A coder as claimed in claim 1, further including update means for forcing a re-send of selected portions of the moving image, whether or not previously sent.

13. A coder as claimed in claim 12 in which the update means repeatedly re-sends the selected portions of the moving image in pseudo-random order.

14. A coder as claimed in claim 12 or claim 13 in which the update means re-sends the selected portions of the image at a variable frequency.

15. A coder as claimed in claim 14 in which the variable frequency of the update means is adjustable, based on the actual or expected noise to which the output data stream is subject.

16. A coder as claimed in claim 1, wherein the coder is implemented by means of a programmed computer processor.

25. A method as claimed in claim 26, wherein the medium comprises a computer disk.

26. A coder as claimed in claim 1 in which the block coder is arranged to sort as well as classsify blocks of the image.

17. An object-oriented video system comprising:

(1) A coder having;

(a) Input image means for receiving a sequence of input images, together defining a moving image, for coding;

(b) An object separator arranged to separate out from the moving image a foreground object and a background object, the background object being relatively motionless with respect to the foreground object, the object separator utilizing

(i) a potential background object store, within which is built up, over a plurality of input images, a potential background, and

(ii) an accepted background object store within which is stored at least some parts of the potential background which have persisted for a desired period;

(c) A block coder arranged to code and sort blocks of the image and to produce an output data stream for storage or onward transmission, each coded block within the data stream being identified as relating either to the foreground or to the background object; and

(2) A decoder adapted to receive the output data stream and to create a video signal therefrom.

18. A system as claimed in claim 17 in which the decoder includes memory means for storing the background object insofar as it is known to the decoder.

19. A system as claimed in claim 18 in which the decoder includes background-reuse means arranged to incorporate within the video signal a fragment of the stored background object on receipt of an instruction to do so from the coder.

20. A system as claimed in claim 19, further including a video camera, adapted to provide the sequence of input images, the camera having a camera body.

21. A system as claimed in claim 20 in which the coder resides inside the camera body.

22. A system as claimed in claim 17 in which the coder comprises a programmed digital computer.

23. A method of operating a video system comprising:

(a) receiving a sequence of input images, together defining a moving image;

(b) separating out from the moving image a foreground object and a background object, the background object being relatively motionless with respect to the foreground object, including maintaining a potential background object store, within which is built up, over a plurality of input images, a potential background, and an accepted background object store within which is stored at least some parts of the potential background which have persisted for a desired period; and

(c) coding and sorting blocks of the image and producing an output datastream for storage or onward transmission, each coded block within the datastream being identified as relating either to the foreground or to the background object.

24. A method as claimed in claim 23, further including a computer storage medium holding a computer program.