System, method and article of manufacture for detecting collisions between video images generated by a camera and an object depicted on a display

• US 6,738,066 B1
• Filed: 07/30/1999
• Issued: 05/18/2004
• Est. Priority Date: 07/30/1999
• Status: Expired due to Term

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Planet Electric Incorporated
• Inventors: Nguyen, Katerina H.
• Primary Examiner(s): Zimmerman, Mark
• Assistant Examiner(s): Nguyen, Kimbinh T.
• Application Number:
  US09/364,629
• Time in Patent Office: 1,754 Days
• Field of Search: 345/421, 345/473, 345/422, 345/958, 345/419, 345/474, 345/629, 382/103, 382/154
• US Class Current: 345/474
• CPC Class Codes: A63F 2300/1012 involving biosensors worn by the player, e.g. for measuring heart beat, limb activity
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/643 by determining the impact between objects, e.g. collision detection
• CPC Class Codes: A63F 2300/69 Involving elements of the real world in the game world, e.g. measurement in live races, real video
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: Y10S 345/958 Collision avoidance

Abstract

A system, method and article of manufacture are provided for detecting collisions or any other type of interactions between video images and an object depicted on a display. First, video images generated by a camera are received. Upon receipt, a first collision detection operation is executed for generating a first confidence value representative of a confidence that the received video images have collided with an object depicted on a display. Also executed is a second collision detection operation for generating a second confidence value representative of a confidence that the received video images have collided with the object depicted on the display. The first confidence value and the second confidence value are then made available for various applications. Such applications may depict an interaction between the video images and the object depicted on the display based on the first confidence value and the second confidence value.

Claims

1. A method of detecting interactions between video images and an object depicted on a display, comprising:

receiving video images having a person image;

executing a first interaction detection operation for generating a first confidence value representative of a confidence that a portion of the received video images has interacted with an object depicted on a display, wherein the first interaction detection operation is based at least in part on where the portion of the received video images first appeared and includes generating at least one of a speed and a direction of the object depicted on the display based on the interaction between at least one body part of the person image in the video images and the object depicted on the display;

executing a second interaction detection operation for generating a second confidence value representative of a confidence that a portion of the received video images has interacted with the object depicted on the display, and determning valid ranges of motion based on an estimated location of a head and an estimated location of a torso of the person image in the video images; and

outputting the first confidence value and the second confidence value, wherein the depiction of the interaction between the video images and the object depicted on the display is based on the first confidence value and the second confidence value.

2. The method of claim 1, wherein the first interaction detection operation includes a background subtraction operation and the second interaction detection operation includes an operation other than a background subtraction operation.

3. The method of claim 1, wherein the interaction includes the object depicted on the display reacting to an interaction with the video images.

4. The method of claim 1, wherein the object depicted on the display includes an animated object.

5. The method of claim 1, wherein the first interaction detection operation includes:

subtracting a background image of the video images in order to extract the person image; and

recognizing body parts of the person image.

6. The method of claim 1, wherein the speed is generated based on an overlap between the at least one body part of the person image in the video images and the object depicted on the display.

7. The method of claim 1, wherein the direction is generated based on a relative position between the at least one body part of the person image in the video images and a center of the object depicted on the display.

8. The method of claim 5, wherein recognizing the body parts of the person image includes:

identifying a location and a number of person images in the video images;

tracking a head of the person image in the video images;

tracking a torso of the person image in the video images; and

tracking limbs of the person image in the video images.

9. The method of claim 8, wherein the location and the number of the person images in the video images are identified using a history of the location and the person images in the video images.

10. The method of claim 8, wherein the location and the number of the person images in the video images are identified using a mass distribution.

11. The method of claim 10, wherein the location and the number of the person images in the video images are identified by determining a mass distribution confidence indicative of a confidence that a portion of the mass distribution represents the person image in the video images, wherein the first confidence value is based at least in part on the mass distribution confidence.

12. The method of claim 10, wherein the location and the number of the person images in the video images are identified by determining a position confidence indicative of a confidence that a portion of the mass distribution represents a person image based on a position of the portion of the mass distribution, wherein the first confidence value is based at least in part on the position confidence.

13. The method of claim 10, wherein the location and the number of the person images in the video images are identified by determining a threshold confidence indicative of a confidence that a portion of the mass distribution represents a person image based on a magnitude of a portion of the mass distribution relative to a threshold, wherein the first confidence value is based at least in part on the threshold confidence.

14. The method of claim 9, wherein the head of the at least one person image in the video images is tracked using a history of the head.

15. The method of claim 9, wherein the head of the at least one person image in the video images is tracked using a mass distribution.

16. The method of claim 9, wherein recognizing the body parts of the person image further includes determining a confidence of a bounding box of the head of the person image, wherein the first confidence value is based at least in part on the bounding box confidence.

17. The method of claim 9, wherein the torso of the at least one person image in the video images is tracked using information relating to the tracking of the head of the at least one person image in the video images.

18. The method of claim 1, wherein the second interaction detection operation includes:

generating a motion distribution of a person image in the video images by utilizing frame differencing;

filtering the motion distribution;

estimating a location of a head of the person image in the video images using head tracking;

estimating a location of a torso of the person image in the video images based on the estimated location of the head of the person image in the video images; and

eliminating motion outside of the valid ranges of the motion.

19. The method of claim 18, wherein the second interaction detection operation includes assuming a location of the head is in a center of the motion distribution if there is no head tracking.

20. The method of claim 18, wherein the second interaction operation further includes determining a confidence of a bounding box of the head of the person image, wherein the second confidence value is based at least in part on the bounding box confidence.

21. The method of claim 1, wherein the video images are generated by a camera.

22. A computer program embodied on a computer readable medium for detecting interactions between video images and an object depicted on a display, comprising:

a code segment for receiving video images having a person image;

a code segment for executing a first interaction detection operation for generating a first confidence value representative of a confidence that a portion of the received video images has interacted with an object depicted on a display, wherein the first interaction detection operation is based at least in part on where the portion of the received video images first appeared and includes generating at least one of a speed and a direction of the object depicted on the display based on the interaction between at least one body part of the person image in the video images and the object depicted on the display;

a code segment for executing a second interaction detection operation for generating a second confidence value representative of a confidence that a portion of the received video images has interacted with the object depicted on the display, and determining valid ranges of motion based on the estimated location of the head and the estimated location of the torso of the person image in the video images; and

a code segment for depicting the interaction between the video images and the object depicted on the display based on the first confidence value and the second confidence value.

23. The computer program of claim 22, wherein the first interaction detection operation includes a background subtraction operation and the second interaction detection operation includes an operation other than a background subtraction operation.

24. The computer program of claim 22, wherein the interaction includes the object depicted on the display reacting to an interaction with the video images.

25. The computer program of claim 22, wherein the object depicted on the display includes an animated object.

26. The computer program of claim 22, wherein the first interaction detection operation is executed by:

a code segment for subtracting a background image of the video images in order to extract a person image; and

a code segment for recognizing body parts of the person image.

27. The computer program of claim 22, wherein the second interaction detection operation is executed by:

a code segment for generating a motion distribution of a person image in the video images by utilizing frame differencing;

a code segment for filtering the motion distribution;

a code segment for estimating a location of a head of the person image in the video images using head tracking;

a code segment for estimating a location of a torso of the person image in the video images based on the estimated location of the head of the person image in the video images;

a code segment for eliminating motion outside of the valid ranges of the motion; and

a code segment for generating at least one of a speed and a direction of the object depicted on the display based on an interaction between at least one body part of the person image of the video images and the object depicted on the display.

28. The computer program of claim 27, wherein the video images are generated by a camera.

29. A system for detecting interactions between video images and an object, comprising:

an input device for receiving video images;

logic for executing a first interaction detection operation for generating a first confidence value representative of a confidence that a portion of the received video images has interacted with an object depicted on a display, wherein the first interaction detection operation is based at least in part on where the portion of the received video images first appeared and includes generating at least one of a speed and a direction of the object depicted on the display based on the interaction between at least one body part of the person image in the video images and the object depicted on the display;

logic for executing a second interaction detection operation for generating a second confidence value representative of a confidence that a portion of the received video images has interacted with the object depicted on the display, and determining valid ranges of motion based on the estimated location of the head and the estimated location of the torso of the person image in the video images; and

a display for depicting the interaction between the video images and the object based on the first confidence value and the second confidence value.