Method and system for processing captured image information in an interactive video display system

• US 20050089194A1
• Filed: 10/25/2004
• Published: 04/28/2005
• Est. Priority Date: 10/24/2003
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Intellectual Ventures Holding 67 LLC (Intellectual Ventures LLC)
• Inventors: Bell, Matthew
• Granted Patent: US 7,536,032 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/174 involving the use of two or more images
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 10/28 Quantising the image, e.g. histogram thresholding for discrimination between background and foreground patterns

Abstract

A method and system for processing captured image information in an interactive video display system. In one embodiment, a special learning condition of a captured camera image is detected. The captured camera image is compared to a normal background model image and to a second background model image, wherein the second background model is learned at a faster rate than the normal background model. A vision image is generated based on the comparisons. In another embodiment, an object in the captured image information that does not move for a predetermined time period is detected. A burn-in image comprising the object is generated, wherein the burn-in image is operable to allow a vision system of the interactive video display system to classify the object as background.

Claims

1. A method for processing captured image information in an interactive video display system, said method comprising:

detecting a special learning condition of a captured camera image;

comparing said captured camera image to a normal background model image;

comparing said captured camera image to a second background model image, wherein said second background model is learned at a faster rate than said normal background model; and

generating a vision image based on said comparing said captured camera image to said normal background model image and said comparing said captured camera image to said second background model image.

2. The method as recited in claim 1 wherein said special learning condition is a rapid learning condition.

3. The method as recited in claim 1, wherein a plurality of second background models are used.

4. The method as recited in claim 1 wherein said special learning condition is detected in response to determining that a predetermined percentage of pixels of said captured camera image are foreground pixels.

5. The method as recited in claim 1 wherein said special learning condition is detected in response to determining that at least a pre-defined portion of pixels of said captured camera image differ from said normal background model by a value exceeding a threshold for a particular length of time.

6. The method as recited in claim 1 further comprising generating said second background model by updating a history data structure of said second background model at a faster rate than a history data structure of said normal background model.

7. The method as recited in claim 1 wherein said comparing said captured camera image to said normal background model image comprises generating a first output image and wherein said comparing said captured camera image to said second background model image comprises generating a second output image.

8. The method as recited in claim 7 wherein said first output image is an image identifying a foreground portion and a background portion of said captured camera image.

9. The method as recited in claim 7 wherein said second output image is an image identifying a foreground portion and a background portion of said captured camera image.

10. The method as recited in claim 7 wherein said generating said vision image comprises performing a logical AND operation on said first output image and said second output image, such that a pixel must be foreground in both output images for it to be foreground in said vision image.

11. A computer-usable medium having computer-readable program code embodied therein for causing a computer system to perform a method for processing captured image information from an interactive video display system, said method comprising:

detecting a special learning condition of a captured camera image;

comparing said captured camera image to a normal background model image;

comparing said captured camera image to a second background model image, wherein said second background model is learned at a faster rate than said normal background model; and

generating a vision image based on said comparing said captured camera image to said normal background model image and said comparing said captured camera image to said second background model image.

12. The computer-usable medium as recited in claim 11 wherein said special learning condition is a rapid learning condition.

13. The computer-usable medium as recited in claim 11, wherein a plurality of second background models are used.

14. The computer-usable medium as recited in claim 11 wherein said special learning condition is detected in response to determining that a predetermined percentage of pixels of said captured camera image are foreground pixels.

15. The computer-usable medium as recited in claim 11 wherein said special learning condition is detected in response to determining that at least a pre-defined portion of pixels of said captured camera image differ from said normal background model by a value exceeding a threshold for a particular length of time.

16. The computer-usable medium as recited in claim 11 further comprising generating said second background model by updating a history data structure of said second background model at a faster rate than a history data structure of said normal background model.

17. The computer-usable medium as recited in claim 11 wherein said comparing said captured camera image to said normal background model image comprises generating a first output image and wherein said comparing said captured camera image to said second background model image comprises generating a second output image.

18. The computer-usable medium as recited in claim 17 wherein said first output image is an image identifying a foreground portion and a background portion of said captured camera image.

19. The computer-usable medium as recited in claim 17 wherein said second output image is an image identifying a foreground portion and a background portion of said captured camera image.

20. The computer-usable medium as recited in claim 17 wherein said generating said vision image comprises performing a logical AND operation on said first output image and said second output image, such that a pixel must be foreground in both output images for it to be foreground in said vision image.

21. A method for processing captured image information in an interactive video display system, said method comprising:

detecting an object in said captured image information that remains substantially fixed in place for a predetermined time period; and

generating a burn-in image comprising said object, wherein said burn-in image is operable to allow a vision system of said interactive video display system to classify said object as background.

22. The method as recited in claim 21 wherein said detecting said object comprises:

classifying at least one pixel corresponding to said object as a burned-in pixel if said pixel is a foreground pixel for said predetermined time period.

23. The method as recited in claim 21 wherein said detecting said object comprises:

classifying at least one pixel corresponding to said object as a burned-in pixel if said pixel is a foreground pixel for a particular portion of said predetermined time period.

24. The method as recited in claim 21 wherein said detecting said object comprises:

classifying at least one pixel corresponding to said object as a burned-in pixel if said pixel is never continuously a background pixel for longer than a particular length of time during said predetermined time period.

25. The method as recited in claim 21 wherein said generating said burn-in image comprises:

representing burned-in pixels of said burn-in image with a binary first value mask, wherein said burn-in pixels correspond to said object; and

representing remaining pixels of said burn-in image with a binary second value mask.

26. The method as recited in claim 21 further comprising:

generating a modified vision image in which pixels in the vision image created by said vision system that are burned-in in said burn-in image are set to background in the modified vision image.

27. The method as recited in claim 21 wherein said generating said modified vision image further comprises:

performing a logical AND operation on said burn-in image and said image to generate said vision image.

28. The method as recited in claim 27 wherein said detecting said object comprises:

updating a memory image for each vision image, wherein a foreground pixel is stored as a non-zero value;

updating an accumulation image with said memory image, wherein said memory image is added to said accumulation image; and

identifying a pixel as a burn-in pixel if a value of said pixel exceeds a threshold.

29. A computer-usable medium having computer-readable program code embodied therein for causing a computer system to perform a method for processing captured image information in an interactive video display system, said method comprising:

detecting an object in said captured image information that remains substantially fixed in place for a predetermined time period; and

generating a burn-in image comprising said object, wherein said burn-in image is operable to allow a vision system of said interactive video display system to classify said object as background.

30. The computer-usable medium as recited in claim 29 wherein said detecting said object comprises:

classifying at least one pixel corresponding to said object as a burned-in pixel if said pixel is a foreground pixel for said predetermined time period.

31. The computer-usable medium as recited in claim 29 wherein said detecting said object comprises:

classifying at least one pixel corresponding to said object as a burned-in pixel if said pixel is a foreground pixel for a particular portion of said predetermined time period.

32. The computer-usable medium as recited in claim 29 wherein said detecting said object comprises:

classifying at least one pixel corresponding to said object as a burned-in pixel if said pixel is never continuously a background pixel for longer than a particular length of time during said predetermined time period.

33. The computer-usable medium as recited in claim 29 wherein said generating said burn-in image comprises:

representing burn-in pixels of said burn-in image with a binary first value mask, wherein said burn-in pixels correspond to said object; and

representing remaining pixels of said burn-in image with a binary second value mask.

34. The computer-usable medium as recited in claim 29 wherein said method further comprises:

generating a modified vision image in which pixels in the vision image created by said vision system that are burned-in in said burn-in image are set to background in the modified vision image.

35. The computer-usable medium as recited in claim 34 wherein said generating said modified vision image further comprises:

performing a logical AND operation on said burn-in image and said image to generate said vision image.

36. The computer-usable medium as recited in claim 35 wherein said detecting said object comprises:

updating a memory image for each vision image, wherein a foreground pixel is stored as a non-zero value;

updating an accumulation image with said memory image, wherein said memory image is added to said accumulation image; and

identifying a pixel as a burn-in pixel if a value of said pixel exceeds a threshold.

37. A method for computing an interaction of an object with a video item, said method comprising:

using a processor to determine a gradient for said object;

using a processor to determine a boundary for said video item, wherein said video item is a virtual button; and

identifying an interaction by using said gradient and said boundary if calculations based on said gradient exceeds a threshold, wherein said interaction is a person pushing said virtual button.