Background Detection As An Optimization For Gesture Recognition

• US 20130308856A1
• Filed: 08/20/2012
• Published: 11/21/2013
• Est. Priority Date: 01/12/2012
• Status: Abandoned Application

• Current Assignee: Google Inc. (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Carpenter, Thor, Petterson, Frank, Vivekanandan, Janahan
• Application Number:
  US13/589,209
• Publication Number:
  US 20130308856A1
• Time in Patent Office: Days
• US Class Current: 382/164
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20016 Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/143 involving probabilistic approaches, e.g. Markov random field [MRF] modelling
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

Methods and systems are provided allowing for background identification and gesture recognition in video images. A computer-implemented image processing method includes: receiving, using at least one processing circuit, a plurality of image frames of a video; constructing, using at least one processing circuit, a plurality of statistical models of the plurality of image frames at a plurality of pixel granularity levels; constructing, using at least one processing circuit, a plurality of probabilistic models of an input image frame at a plurality of channel granularity levels based on the plurality of statistical models; merging at least some of the plurality of probabilistic models based on a weighted average to form a single probability image; determining background pixels, based on a probability threshold value, from the single probability image; and determining whether the plurality of image frames, when examined in a particular sequence, conveys a gesture by the object.

Claims

1. A computer-implemented image processing method for recognizing a gesture made by an object, the method comprising:

receiving, using at least one processing circuit, a plurality of image frames of a video, wherein each pixel of each of the plurality of image frames has a blue channel, a green channel, a red channel, and an alpha channel;

constructing, using at least one processing circuit, a plurality of statistical models of the plurality of image frames at a plurality of pixel granularity levels, the plurality of statistical models including;

at a first pixel granularity level, a spatio-temporal (S-T) histogram for each of the pixels from the plurality of image frames, wherein a first axis of the S-T histogram represents channel value bins, and wherein a second axis of the S-T histogram represents counts of image frames per bin;

at a second pixel granularity level higher than the first pixel granularity level, aggregate histograms for the blue, green, and red channels, respectively, based on aggregated pixel values at the second pixel granularity level;

constructing, using at least one processing circuit, a plurality of probabilistic models of an input image frame at a plurality of channel granularity levels based on the plurality of statistical models, the plurality of probabilistic models including;

at a first channel granularity level, a probability image from each of the S-T histogram and the aggregate histograms, wherein each of the probability images comprises a plurality of pixels each indicating a probability of a corresponding pixel in the input image being a background pixel;

at a second channel granularity level higher than the first channel granularity level, compact probability images from the probability images at the first channel granularity level;

merging the compact probability images based on a weighted average to form a single probability image;

subsampling pixels in the single probability image;

determining background pixels, based on a probability threshold value, from the subsampled single probability image; and

determining whether the plurality of image frames, when examined in a particular sequence, conveys a gesture by the object.

2. The method of claim 1, further comprising:

determining whether a gesture by the object resembles a predetermined gesture.

3. The method of claim 1, further comprising:

determining foreground pixels, based on a probability threshold value, from the subsampled single probability image.

4. The method of claim 1, further comprising:

automatically replacing the determined background pixels with desired pixel values.

5. The method of claim 1, further comprising:

automatically removing the determined background pixels.

6. A computer-implemented image processing method for recognizing a gesture made by an object, the method comprising:

receiving, using at least one processing circuit, a plurality of image frames of a video;

constructing, using at least one processing circuit, a plurality of statistical models of the plurality of image frames at a plurality of pixel granularity levels;

constructing, using at least one processing circuit, a plurality of probabilistic models of an input image frame at a plurality of channel granularity levels based on the plurality of statistical models;

merging at least some of the plurality of probabilistic models based on a weighted average to form a single probability image;

determining background pixels, based on a probability threshold value, from the single probability image; and

determining whether the plurality of image frames, when examined in a particular sequence, conveys a gesture by the object.

7. The method of claim 6, wherein the plurality of statistical models comprise:

at a first pixel granularity level, a spatio-temporal (S-T) histogram for each of the pixels from the plurality of image frames, wherein a horizontal axis of the S-T histogram represents channel value bins, and wherein a vertical axis of the S-T histogram represents counts of image frames per bin; and

at a second pixel granularity level higher than the first pixel granularity level, aggregate histograms for the blue, green, and red channels, respectively, based on aggregated pixel values at a the higher pixel granularity level.

8. The method of claim 6, wherein the plurality of probabilistic models comprise:

at a first channel granularity level, a probability image from each of the S-T histogram and the aggregate histograms, wherein each of the probability images comprises a plurality of pixels each indicating a probability of a corresponding pixel in the input image being a background pixel; and

at a second channel granularity level higher than the first channel granularity level, compact probability images from the probability images at the single-channel granularity level.

9. The method of claim 8, wherein the compact probability images are obtained from one of a mean, a median, or a minimum operation over the probability images at the first channel granularity level for a plurality of channels, wherein the first channel granularity level is a single-channel granularity level.

10. The method of claim 8, wherein the compact probability images include:

a compact S-T probability image at the first pixel granularity level and the second channel granularity level;

a compact aggregate background probability image at the second pixel granularity level across a first-order approximation of a background region and at the second channel granularity level; and

a compact aggregate foreground probability image at the second pixel granularity level across a first-order approximation of a foreground region and at the second channel granularity level,

wherein the compact S-T probability image is given a higher weight in the weighted average.

11. The method of claim 6, wherein the weighted average gives a higher weight to the probabilistic models at a lower pixel granularity level.

12. The method of claim 6, further comprising:

subsampling pixels in the single probability image,

wherein the background pixels are determined from the subsampled single probability image.

13. The method of claim 6, further comprising automatically replacing the determined background pixels with desired pixel values.

14. The method of claim 6, further comprising alpha-blending the determined background pixels with foreground pixels.

15. The method of claim 6, wherein each pixel of each of the plurality of image frames has a blue channel, a green channel, a red channel, and an alpha channel.

16. The method of claim 6, wherein each pixel of each of the plurality of image frames has a blue channel, a green channel, and a red channel.

17. The method of claim 6, further comprising:

adding and subtracting image frames to the plurality of image frames; and

updating the plurality of statistical models and the plurality of probabilistic models based on the plurality of image frames with the added and subtracted image frames.

18. The method of claim 6, further comprising:

sampling one of the plurality of image frames at a probability equal to a desired statistics update frequency.

19. An image processing system comprising at least one processing circuit configured to:

receive a plurality of image frames of a video;

construct a plurality of statistical models of the plurality of image frames at a plurality of pixel granularity levels;

construct a plurality of probabilistic models of an input image frame at a plurality of channel granularity levels based on the plurality of statistical models;

merge at least some of the plurality of probabilistic models based on a weighted average to form a single probability image;

determine background pixels, based on a probability threshold value, from the single probability image; and

determine whether the plurality of image frames, when examined in a particular sequence, conveys a gesture by an object within the frames.

20. The system of claim 19, wherein the plurality of statistical models comprise:

at a first pixel granularity level, a spatio-temporal (S-T) histogram for each of the pixels from the plurality of image frames, wherein a first axis of the S-T histogram represents channel value bins, and wherein a second axis of the S-T histogram represents counts of image frames per bin; and

at a second pixel granularity level higher than the first pixel granularity level, aggregate histograms for the blue, green, and red channels, respectively, based on aggregated pixel values at a the second pixel granularity level.

21. The system of claim 19, wherein the plurality of probabilistic models comprise:

at a first channel granularity level, a probability image from each of the S-T histogram and the aggregate histograms, wherein each of the probability images comprises a plurality of pixels each indicating a probability of a corresponding pixel in the input image being a background pixel; and

at a second channel granularity level higher than the first channel granularity level, compact probability images from the probability images at the single-channel granularity level.

22. The system of claim 21, wherein the compact probability images are obtained from one of a mean, a median, or a minimum operation over the probability images at the first channel granularity level for a plurality of channels.

23. The system of claim 21, wherein the compact probability images include:

a compact S-T probability image at the first pixel granularity level and the second channel granularity level;

a compact aggregate background probability image at the second pixel granularity level across a first-order approximation of a background region and the second channel granularity level; and

a compact aggregate foreground probability image at the second pixel granularity level across a first-order approximation of a foreground region and the second channel granularity level,

wherein the compact S-T probability image is given a higher weight in the weighted average.

24. The system of claim 19, wherein the system is configured to determine whether a gesture by the object resembles a predetermined gesture.

25. The system of claim 19, wherein the system is configured to determine foreground pixels, based on a probability threshold value, from the subsampled single probability image.

26. The system of claim 19, wherein the system is configured to replace the determined background pixels with desired pixel values.

27. The system of claim 19, wherein the system is configured to remove the determined background pixels.