Real-time video object generation for smart cameras

• US 7,167,519 B2
• Filed: 12/20/2002
• Issued: 01/23/2007
• Est. Priority Date: 12/20/2001
• Status: Active Grant

• Current Assignee: Verkada Inc.
• Original Assignee: Siemens Corporate Research Incorporated (Siemens AG)
• Inventors: Comaniciu, Dorin, Del Bue, Alessio, Ramesh, Visvanathan
• Primary Examiner(s): PHILIPPE, GIMS S
• Application Number:
  US10/325,413
• Publication Number:
  US 20030174773A1
• Time in Patent Office: 1,495 Days
• Field of Search: 375/240.08, 375/240.09, 375/240.24, 386/165, 386/173, 386/236, 386/242, 386/251, 386/243, 386/249, 382/165, 382/173, 382/236, 382/242, 382/251, 382/243, 382/249, 382/177
• US Class Current: 375/240.08
• CPC Class Codes: H04N 19/124 Quantisation
• CPC Class Codes: H04N 19/126 Details of normalisation or weighting functions, e.g. normalisation matrices or variable uniform quantisers
• CPC Class Codes: H04N 19/14 Coding unit complexity, e.g. amount of activity or edge presence estimation H04N19/146 takes precedence
• CPC Class Codes: H04N 19/146 Data rate or code amount at the encoder output
• CPC Class Codes: H04N 19/149 by estimating the code amount by means of a model, e.g. mathematical model or statistical model
• CPC Class Codes: H04N 19/17 the unit being an image region, e.g. an object
• CPC Class Codes: H04N 19/172 the region being a picture, frame or field
• CPC Class Codes: H04N 19/174 the region being a slice, e.g. a line of blocks or a group of blocks
• CPC Class Codes: H04N 19/196 being specially adapted for the computation of encoding parameters, e.g. by averaging previously computed encoding parameters processing of motion vectors H04N19/513
• CPC Class Codes: H04N 19/23 with coding of regions that are present throughout a whole video segment, e.g. sprites, background or mosaic
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 7/141 between two video terminals, e.g. videophone telephonic communication systems combined with television receiver for reception of entertainment or information matter H04M11/085
• CPC Class Codes: H04N 7/183 for receiving images from a single remote source

Abstract

An apparatus and method for video object generation and selective encoding is provided. The apparatus includes a detection module for detecting a first object in at least one image frame of a series of image frames; a tracking module for tracking the first object in successive image frames and segmenting the first object from a background, the background being a second object; and an encoder for encoding the first and second objects to be transmitted to a receiver, wherein the first object is compressed at a high compression rate and the second object is compressed at a low compression rate. The receiver merges the first and second object to form a composite image frame. The method provides for detecting, tracking and segmenting one or more objects, such as a face, from a background to be encoded at the same or different compression rates to conserve bandwidth.

Claims

1. An apparatus for video object generation and selective encoding, the apparatus comprising:

a detection module for detecting a first object in at least one image frame of a series of image frames, wherein the detection module detects the first object by initializing a plurality of regions in the at least one image frame, for each initialization computes a degree of similarity between a model and a candidate object in the at least one image frame, and applies an iterative comparative procedure to the degrees of similarity computed, the iterations being based on a gradient vector to shift the location of candidate object in the at least one frame, to derive as the location of the candidate object in the at least one frame that location which has characteristics most similar to the characteristics of the model;

a tracking module for tracking the first object in successive image frames of the series of image frames and segmenting the first object from a background, the background being a second object; and

an encoder for encoding the first and second objects to be transmitted to a receiver, wherein the first object is compressed at a high compression rate and the second object is compressed at a low compression rate.

2. The apparatus of claim 1, further comprising a camera for acquiring the series of image frames.

3. The apparatus of claim 2, further comprising a frame grabber for grabbing image frames from the camera and outputting the image frames to the detection module and tracking module.

4. The apparatus as in claim 2, further comprising a camera control module for controlling a position of the camera to ensure the first object is centered in an image frame.

5. The apparatus as in claim 1, further comprising a modeling module for modeling the first object by computing a statistical characterization of the first object.

6. The apparatus as in claim 1, wherein the receiver merges the first and second object to form a composite image frame.

7. The apparatus as in claim 1, wherein the detection module causes the iterations to be repeated until the shift in locations is less than a given first threshold.

8. The apparatus as in claim 7, wherein the detection module uses a mean shift iteration to compute the gradient vector along which the location of the candidate object is shifted.

9. The apparatus as in claim 7, wherein the tracking module computes a degree of similarity between the detected object and a candidate object in a successive frame, and applies an iterative comparative procedure to the degrees of similarity computed, the iterations being based on a gradient vector to shift the location of candidate object in the successive frame, to derive as the location of the candidate object in the successive frame that location which has characteristics most similar to the characteristics of the detected object in the initial frame.

10. The apparatus as in claim 9, wherein the tracking module causes the iterations to be repeated until the shift in locations is less than a given second threshold.

11. The apparatus as in claim 10, wherein the degree of similarity is expressed by a metric derived from the Bhattacharyya coefficient.

12. The apparatus as in claim 1, wherein the encoding module is MPEG-4 complaint.

13. A method for video object generation and selective encoding, the method comprising the steps of:

detecting a first object from at least one of a plurality of successive image frames, wherein detecting further includes initializing a plurality of regions in the at least one image frame, for each initialization, computing a degree of similarity between a model and a candidate object in the at least one image frame, and applying an iterative comparative procedure to the degrees of similarity computed, the iterations being based on a gradient vector to shift the location of candidate object in the at least one frame, to derive as the location of the candidate object in the at least one frame that location which has characteristics most similar to the characteristics of the model;

tracking the first object through the plurality of image frames;

segmenting the first object from a background of the image frame, the background being a second object; and

encoding the first and second objects to be transmitted to a receiver, wherein the first object is compressed at a high compression rate and the second object is compressed at a low compression rate.

14. The method as in claim 13, further comprising the step of acquiring the plurality of successive image frames by a camera.

15. The method as in claim 14, further comprising the step of controlling a position of the camera to ensure the first detected object is centered in an image frame.

16. The method as in claim 13, further comprising the step of modeling the first object by computing a statistical characterization of the first object.

17. The method as in claim 13, further comprising the steps of receiving the first compressed object and the second compressed object and decoding the first and second object to form a composite image frame.

18. The method as in claim 13, further comprising the step of repeating the iterations until the shift in locations is less than a given first threshold.

19. The method as in claim 18, wherein the detection step uses a mean shift iteration to compute the gradient vector along which the location of the candidate object is shifted.

20. The method as in claim 18, wherein the tracking step further includes:

computing a degree of similarity between the detected object and a candidate object in a successive frame; and

applying an iterative comparative procedure to the degrees of similarity computed, the iterations being based on a gradient vector to shift the location of candidate object in the successive frame, to derive as the location of the candidate object in the successive frame that location which has characteristics most similar to the characteristics of the detected object in the initial frame.

21. The method as in claim 20, further comprising the step of repeating the iterations until the shift in locations is less than a given second threshold.

22. The apparatus as in claim 21, wherein the degree of similarity is expressed by a metric derived from the Bhattacharyya coefficient.

23. The method as in claim 13, wherein the segmenting step includes applying a segmentation mask to the first object defining an area to be segmented.

24. The method as in claim 23, wherein the segmentation mask is of a shape resembling the first object.

25. The method as in claim 13, wherein the tracking, segmenting and encoding steps are continuously repeated only for the first object.

26. A program storage device readable by a machine, tangibly embodying a program of instructions executable by the machine to perform method steps for video object generation and selective encoding, the method steps comprising:

detecting a first object from at least one of a plurality of successive image frames, wherein detecting further includes initializing a plurality of regions in the at least one image frame, for each initialization, computing a degree of similarity between a model and a candidate object in the at least one image frame, and applying an iterative comparative procedure to the degrees of similarity computed, the iterations being based on a gradient vector to shift the location of candidate object in the at least one frame, to derive as the location of the candidate object in the at least one frame that location which has characteristics most similar to the characteristics of the model;

tracking the first object through the plurality of image frames;

segmenting the first object from a background of the image frame, the background being a second object; and

encoding the first and second objects to be transmitted to a receiver, wherein the first object is compressed at a high compression rate and the second object is compressed at a low compression rate.

27. A method for video object generation and selective encoding, the method comprising the steps of:

detecting a plurality of objects from at least one of a plurality of successive image frames, wherein detecting includes initializing multiple regions in the at least one image frame, for each initialization, computing a degree of similarity between a plurality of models and candidate objects in the at least one frame, and applying an iterative comparative procedure to the degrees of similarity computed, the iterations being based on a gradient vector to shift the location of candidate objects in the at least one frame, to derive as the location of the candidate objects in the at least one frame those locations which have characteristics most similar to the characteristics of the plurality of models;

tracking the plurality of objects through the plurality of image frames;

segmenting the plurality of objects from the at least one image frame; and

encoding the plurality of objects to be transmitted to a receiver, wherein each of the plurality of objects is compressed at a different compression rate.

28. The method as in claim 27, further comprising the step of modeling the plurality of objects by computing a statistical characterization of each of the plurality of objects.

29. The method as in claim 27, further comprising the steps of receiving the plurality of compressed objects and decoding the plurality of compressed objects to form a composite image frame.

30. The method as in claim 27, further comprising the step of repeating the iterations until the shift in locations is less than a given first threshold.

31. The method as in claim 30, wherein the detection step uses a mean shift iteration to compute the gradient vector along which the location of the candidate objects is shifted.

32. The method as in claim 30, wherein the tracking step further includes:

computing a degree of similarity between the detected objects and candidate objects in a successive frame; and

applying an iterative comparative procedure to the degrees of similarity computed, the iterations being based on a gradient vector to shift the location of candidate objects in the successive frame, to derive as the location of the candidate objects in the successive frame that location which has characteristics most similar to the characteristics of the detected objects in the initial frame.

33. The method as in claim 32, further comprising the step of repeating the iterations until the shift in locations is less than a given second threshold.

34. The method as in claim 33, wherein the degree of similarity is expressed by a metric derived from the Bhattacharyya coefficient.

35. The method of claim 34, wherein said gradient vector corresponds to a maximization of said Bhattacharyya coefficient.