Object tracking using background and foreground models

• US 9,152,243 B2
• Filed: 02/26/2013
• Issued: 10/06/2015
• Est. Priority Date: 08/23/2012
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Gu, Jin
• Primary Examiner(s): NAKHJAVAN, SHERVIN K
• Application Number:
  US13/777,897
• Publication Number:
  US 20140056471A1
• Time in Patent Office: 952 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 18/21342 using statistical independence, i.e. minimising mutual information or maximising non-gaussianity
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06T 7/277 involving stochastic approaches, e.g. using Kalman filters
• CPC Class Codes: G06V 10/26 Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
• CPC Class Codes: G06V 10/7715 Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods
• CPC Class Codes: G06V 20/647 by matching two-dimensional images to three-dimensional objects
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

Various arrangements for modeling a scene are presented. A plurality of images of the scene captured over a period of time may be received, each image comprising a plurality of pixels. A plurality of background models may be created using the plurality of images. At least one background model may be created for each pixel of the plurality of pixels. A plurality of foreground models may be created using the plurality of images. A foreground model may be created for each pixel of at least a first subset of pixels of the plurality of pixels. The background models and the foreground models may be indicative of the scene over the period of time.

Claims

1. A method for modeling a scene, the method comprising:

receiving, by a computer system, a plurality of images of the scene captured over a period of time, each image comprising a plurality of pixels;

creating, by the computer system, a plurality of background models using the plurality of images, wherein at least one background model is created for each pixel of the plurality of pixels;

creating, by the computer system, a plurality of foreground models using the plurality of images, wherein a foreground model is created for each pixel of at least a first subset of pixels of the plurality of pixels, wherein the plurality of background models and the plurality of foreground models are indicative of the scene over the period of time;

determining, by the computer system, a second subset of the plurality of pixels at least partially based on the plurality of background models and the plurality of foreground models, wherein the second subset of the plurality of pixels contains only some of the plurality of pixels, and the determining the second subset of the plurality of pixels comprises, for a pixel having a foreground model and a background model, determining;

if a probability the pixel is likely to be background is greater than a first threshold value based on the background model for that pixel, and

if a probability the pixel is likely to be foreground is greater than a second threshold value based on the foreground model for that pixel;

determining, by the computer system, that the second subset of the plurality of pixels is representative of at least a portion of a control object; and

controlling, by the computer system, an application based upon a detected movement of the control object.

2. The method for modeling the scene of claim 1, further comprising

outputting only the second subset of the plurality of pixels.

3. The method for modeling the scene of claim 2, further comprising

performing a depth segmentation process on the second subset of the plurality of pixels.

4. The method for modeling the scene of claim 1, wherein each foreground model of the plurality of foreground models is created if a pixel corresponding to the foreground model represents a portion of a person.

5. The method for modeling the scene of claim 1, wherein each pixel of the image comprises a depth value and an intensity value.

6. The method for modeling the scene of claim 5, wherein creating each background model of the plurality of background models comprises:

determining for each pixel at least a first and second statistical measure established over a first portion of the period of time based at least in part on the intensity value and the depth value for the pixel in multiple images of the plurality of images.

7. The method for modeling the scene of claim 6, further comprising:

creating a second background model for a pixel of the plurality of pixels, wherein the second background model comprises at least a third and fourth statistical measure established for the pixel over a second portion of the period of time based at least in part on the intensity value and the depth value for the pixel in multiple images of the plurality of images.

8. The method for modeling the scene of claim 1, further comprising:

receiving an indication for each pixel of the first subset of pixels and a depth value for each pixel of the first subset of pixels, the indication being indicative that the pixel corresponds to a person.

9. The method for modeling the scene of claim 1, further comprising:

receiving a second plurality of images of the scene captured over a second period of time; and

updating one or more background models of the plurality of background models and foreground models of the plurality of foreground models based on the second plurality of images.

10. The method for modeling the scene of claim 1, wherein the plurality of background models and the plurality of foreground models are used by an application for determining whether a user is present.

11. The method for modeling the scene of claim 1, wherein the plurality of background models and the plurality of foreground models are used by an application for determining that an input has occurred.

12. The method for modeling the scene of claim 11, further comprising:

determining that an object in a current image is in a foreground of the scene based at least in part on the plurality of background models and the plurality of foreground models, wherein the application determines that the object has provided the input in response to determining that the object is in the foreground.

13. A system for modeling a scene, the system comprising:

an image acquisition module configured to receive a plurality of images of the scene captured over a period of time, each image comprising a plurality of pixels;

a background modeling module configured to create a plurality of background models using the plurality of images, wherein at least one background model is created for each pixel of the plurality of pixels;

a foreground modeling module configured to create a plurality of foreground models using the plurality of images, wherein a foreground model is created for each pixel of at least a first subset of pixels of the plurality of pixels, wherein the plurality of background models and the plurality of foreground models are indicative of the scene over the period of time;

an extraction module configured to;

determine a second subset of the plurality of pixels at least partially based on the plurality of background models and the plurality of foreground models, wherein the second subset of the plurality of pixels contains only some of the plurality of pixels, and, for a pixel having a foreground model and a background model, determine if a probability the pixel is likely to be background is greater than a first threshold value based on the background model for that pixel, and if a probability the pixel is likely to be foreground is greater than a second threshold value based on the foreground model for that pixel; and

at least one module configured to determine that the second subset of the plurality of pixels is representative of at least a portion of a control object, and to control an application based upon a detected movement of the control object.

14. The system for modeling the scene of claim 13, further comprising:

an output module configured to create an output of the second subset of the plurality of pixels.

15. The system for modeling the scene of claim 14, wherein the extraction module is further configured to determine, for a pixel having a foreground model and a background model, if the pixel more likely corresponds to the foreground model or the background model.

16. The system for modeling the scene of claim 14, further comprising:

a depth segmentation module configured to perform a depth segmentation process on the second subset of the plurality of pixels.

17. The system for modeling the scene of claim 13, wherein each foreground model of the plurality of foreground models is created if a pixel corresponding to the foreground model represents a portion of a person.

18. The system for modeling the scene of claim 13, wherein each pixel of the image comprises a depth value and an intensity value.

19. The system for modeling the scene of claim 18, wherein the background modeling module comprises:

a submodule configured to determine for each pixel at least a first and second statistical measure established over a first portion of the period of time based at least in part on the intensity value and the depth value for the pixel in multiple images of the plurality of images.

20. The system for modeling the scene of claim 19, wherein the background modeling module is further configured to create a second background model for a pixel of the plurality of pixels, wherein the second background model comprises at least a third and fourth statistical measure established for the pixel over a second portion of the period of time based at least in part on the intensity value and the depth value for the pixel in multiple images of the plurality of images.

21. The system for modeling the scene of claim 13, wherein the extraction module is configured to receive an indication for each pixel of the first subset of pixels and a depth value for each pixel of the first subset of pixels, the indication being indicative that the pixel corresponds to a person.

22. The system for modeling the scene of claim 13, wherein:

the image acquisition module is configured to receive a second plurality of images of the scene captured over a second period of time; and

the background modeling module is configured to update one or more background models of the plurality of background models based on the second plurality of images; and

the foreground modeling module is configured to update one or more foreground models of the plurality of foreground models based on the second plurality of images.

23. The system for modeling the scene of claim 13, wherein the plurality of background models and the plurality of foreground models are used by an application for determining whether a user is present.

24. The system for modeling the scene of claim 13, wherein the plurality of background models and the plurality of foreground models are used by an application for determining that an input has occurred.

25. A non-transitory computer-readable medium having computer-readable instructions stored thereon, the computer-readable instructions being configured to cause a computer to:

receive a plurality of images of the scene captured over a period of time, each image comprising a plurality of pixels;

create a plurality of background models using the plurality of images, wherein at least one background model is created for each pixel of the plurality of pixels;

create a plurality of foreground models using the plurality of images, wherein a foreground model is created for each pixel of at least a first subset of pixels of the plurality of pixels, wherein the plurality of background models and the plurality of foreground models are indicative of the scene over the period of time;

determine a second subset of the plurality of pixels at least partially based on the plurality of background models and the plurality of foreground models wherein the second subset of the plurality of pixels contains only some of the plurality of pixels, and, for a pixel having a foreground model and a background model, determine if a probability the pixel is likely to be background is greater than a first threshold value based on the background model for that pixel, and if a probability the pixel is likely to be foreground is greater than a second threshold value based on the foreground model for that pixel.

26. An apparatus comprising:

means for receiving a plurality of images of the scene captured over a period of time, each image comprising a plurality of pixels;

means for creating a plurality of background models using the plurality of images, wherein at least one background model is created for each pixel of the plurality of pixels;

means for creating a plurality of foreground models using the plurality of images, wherein a foreground model is created for each pixel of at least a first subset of pixels of the plurality of pixels, wherein the plurality of background models and the plurality of foreground models are indicative of the scene over the period of time;

means for determining a second subset of the plurality of pixels at least partially based on the plurality of background models and the plurality of foreground models, wherein the second subset of the plurality of pixels contains only some of the plurality of pixels, and, for a pixel having a foreground model and a background model, if a probability the pixel is likely to be background is greater than a first threshold value based on the background model for that pixel, and if a probability the pixel is likely to be foreground is greater than a second threshold value based on the foreground model for that pixel; and

means for determining that the second subset of the plurality of pixels is representative of at least a portion of a control object, and controlling an application based upon a detected movement of the control object.

27. The apparatus for modeling the scene of claim 26, further comprising:

means for determining a second subset of the plurality of pixels at least partially based on the plurality of background models and the plurality of foreground models, wherein;

the second subset of the plurality of pixels contains only some of the plurality of pixels; and

means for outputting the second subset of the plurality of pixels.

28. The apparatus for modeling the scene of claim 26, wherein each foreground model of the plurality of foreground models is created if a pixel corresponding to the foreground model represents a portion of a person.

29. The apparatus for modeling the scene of claim 26, wherein each pixel of the image comprises a depth value and an intensity value.