Method and system for facilitating wireless, full-body, real-time user interaction with a digitally represented visual environment

• US 5,563,988 A
• Filed: 08/01/1994
• Issued: 10/08/1996
• Est. Priority Date: 08/01/1994
• Status: Expired due to Term

• Current Assignee: Massachusetts Institute of Technology
• Original Assignee: Massachusetts Institute of Technology
• Inventors: Darrell, Trevor J., Johnson, Michael P., Russell, Kenneth B., Starner, Thad E., Maes, Pattie E., Blumberg, Bruce M., Pentland, Alex P.
• Primary Examiner(s): Zimmerman, Mark K.
• Application Number:
  US08/284,038
• Time in Patent Office: 799 Days
• Field of Search: 395/121, 395/152
• US Class Current: 345/421
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: H04N 7/15 Conference systems
• CPC Class Codes: H04N 7/157 defining a virtual conference space and using avatars or agents computer conference optimisation or adaptation H04L12/1827

Abstract

An electronic system for integrating a user'"'"'s moving image into a computer-generated pictorial environment, and allowing the user to interact with the environment and objects therein. The environment is rendered on a screen display in a manner that simulates three dimensions, including respect for occlusion relationships among the user and objects in the environment. The resulting image, which changes as the user gestures and interacts with the environment, is projected onto a large screen that the user faces. The environment includes a series of mobile entities, or "agents," which respond to one or more of the user'"'"'s changing position, gestures and voice, exhibiting context-dependent behavior that appears semi-intelligent.

Claims

1. A system for facilitating real-time interaction between a user and a digitally represented visual environment within which the user'"'"'s moving image is integrated, the system comprising:

a. imaging means for digitizing, into an array of multichromatic pixels, the full-body pictorial image of a user and a background, the user being physically displaced from the imaging means by a depthwise distance;

b. an isolator for separating the user'"'"'s image from the background;

c. wireless means for ascertaining the depthwise distance between the user and the imaging means;

d. means for generating a visual display showing objects;

e. an integrator for introducing the user'"'"'s digitized image into the visual display such that the user'"'"'s image occludes any overlapping objects having depth parameters greater than the user'"'"'s depthwise distance, and is occluded by any overlapping objects having depth parameters less than the user'"'"'s depthwise distance.

2. The system of claim 1 wherein the visual-display generation means comprises means for rendering objects on the visual display such that objects having depth parameters less than those of overlapping objects occlude the overlapping objects.

3. The system of claim 1 wherein the user'"'"'s side-to-side position relative to the imaging means determines the side-to-side location, within the visual display, at which the integrator introduces the user'"'"'s image.

4. The system of claim 1 further comprising:

a. a two-dimensional display screen; and

b. a localization module for ascertaining the user'"'"'s three-dimensional location relative to the imaging means;

wherein

c. the visual-display generation means renders a graphical environment on the display screen in a manner that simulates three dimensions; and

d. the integrator introduces the user'"'"'s image at a location within the graphical environment determined by the user'"'"'s three-dimensional location relative to the imaging means.

5. The system of claim 4 wherein the localization module determines the user'"'"'s three-dimensional location based on recording, by the imaging means, of a ground plane and the user'"'"'s image thereon, and transformation of the recording based on calibration of the imaging means.

6. The system of claim 1 wherein the isolator operates by means of vision-based background subtraction.

7. The system of claim 6 wherein the isolator comprises a processor for:

a. analyzing digitized images of the background, without the user, to determine mean values for each background pixel; and

b. scanning a current digitized image that includes the user in order to designate, as pixels representative of the user, pixels having values that differ significantly from the mean values of corresponding background pixels.

8. The system of claim 1 further comprising means for analyzing the user'"'"'s image to identify feature regions corresponding at least to the user'"'"'s hands, feet, head and center of mass.

9. The system of claim 7 wherein the user'"'"'s image includes an edge contour and the feature-analysis means associates feature classifications with regions of high curvature along the contour.

10. The system of claim 8 wherein the feature-analysis means classifies features, each having a high-curvature region that itself has a centroid location positioned within a two-dimensional coordinate plane, based on the position of the centroid location.

11. The system of claim 10 wherein the feature-analysis means analyzes the centroid position relative to other portions of the edge contour.

12. The system of claim 10 wherein the feature-analysis means analyzes the centroid position relative to a bounding box.

13. The system of claim 1 further comprising a gestural decoder for analyzing the user'"'"'s image to detect gestures.

14. The system of claim 13 further comprising means for analyzing the user'"'"'s image to identify feature regions corresponding at least to the user'"'"'s hands, feet and head, and wherein the gestural decoder compares successively digitized user images to detect specific movement patterns of at least one feature region.

15. The system of claim 1 further comprising means for generating, for presentation on the visual display by means of the integrator, agents having controllable positions and appearances, the generating means including means for updating the position and appearance of each agent based on (i) the location in the visual display where the user is introduced, and (ii) the location in the visual display where the agent is presented.

16. The system of claim 15 further comprising a location-tracking module that specifies, for each agent, the location in the visual display of the agent and its three-dimensional distance from the user and each displayed object.

17. The system of claim 15 further comprising a gestural decoder for analyzing the user'"'"'s image to detect gestures, and wherein an agent is updated based further on detected gestures.

18. The system of claim 15 further comprising:

a. a modeling database containing parameters specifying (i) a repertoire of activities, (ii) a range of permitted movements and appearances, (iii) an activity-selection model, and (iv) parameters each specifying an instantaneous importance associated with the activities, the parameters being variable over time; and

b. an activity generator for controlling the appearance and position of agent based on (i) the location in the visual display where the user is introduced, (ii) the location in the visual display where the agent is presented and (iii) the contents of the modeling database.

19. The system of claim 18 wherein the modeling database further includes parameters specifying a set of motivational variables that themselves specify a numeric value associated with an agent activity and which may vary over time.

20. The system of claim 18 wherein the activity generator updates an agent based also on user gestures.

21. The system of claim 18 wherein the activity generator updates an agent based also on sounds made by the user.

22. The system of claim 18 wherein the modeling database further includes numeric values associated with each activity, each numeric value specifying an instantaneous importance associated with the activity.

23. The system of claim 22 wherein the modeling database further includes parameters specifying fatigue factors that reduce, over time, the numeric activity values associated with current activities.

24. The system of claim 22 wherein the activity generator periodically updates the values associated with each activity.

25. The system of claim 18 wherein the activity generator also produces sound.

26. The system of claim 1 wherein the image of a user digitized by the imaging means includes an edge contour, and further comprising:

a. a modeling module for identifying regions of high curvature along the contour, and classifying at least some of the high-curvature regions as body features; and

b. a gestural decoder for analyzing the classified features over time to detect gestures.

27. The system of claim 26 wherein the modeling module identifies a feature classification based on the position of a centroid location of the high-curvature region within a two-dimensional coordinate plane.

28. The system of claim 27 wherein the modeling module analyzes centroid position relative to other portions of the edge contour.

29. The system of claim 27 wherein the modeling module analyzes centroid position relative to a bounding box.

30. The system of claim 26 wherein the body features classified by the modeling module include hands, feet, head and center of mass, and wherein the gestural decoder compares successively digitized user images to detect movement of at least one feature region.

31. A method for facilitating real-time interaction between a user and a digitally represented visual environment within which the user'"'"'s moving image is integrated, the method comprising the steps of:

a. digitizing, into an array of multichromatic pixels, the full-body pictorial image of a user and a background;

b. isolating the user'"'"'s image from the background;

c. wirelessly ascertaining a depthwise distance between the user and a reference point and associating with the user a depth parameter representative of the depthwise distance;

d. generating, on a digital output device, a visual display showing objects having depth parameters associated therewith; and

e. introducing the user'"'"'s digitized image into the visual display such that the user'"'"'s image occludes any overlapping objects having depth parameters greater than that of the user, and is occluded by any overlapping objects having depth parameters less than that of the user.

32. The method of claim 31 wherein the generation step further comprises the substep of rendering objects on the visual display such that objects having depth parameters less than those of overlapping objects occlude the overlapping objects.

33. The method of claim 31 wherein the user'"'"'s side-to-side position relative to the imaging means determines the side-to-side location, within the visual display, at which the user'"'"'s image is introduced.

34. The method of claim 31 wherein:

a. the user is associated with a two-dimensional location as well as the depth parameter, thereby specifying an actual three-dimensional user location;

b. the visual display is generated in a manner that simulates three dimensions; and

c. the user'"'"'s image is introduced at a location within the graphical environment determined by the the specified three-dimensional user location.

35. The method of claim 31 wherein the user is isolated by means of vision-based background subtraction.

36. The method of claim 35 wherein the user is isolated according to steps comprising:

a. analyzing digitized images of the background, without the user, to determine mean values for each background pixel;

b. scanning a current digitized image that includes the user to designate, as current pixels representative of the isolated user, pixels whose values differ significantly from the mean values of corresponding background pixels.

37. The method of claim 36 wherein each pixel has at least one color associated therewith and further comprising the steps of:

a. for each background pixel, determining a mean luminance value of the at least one color associated with the background pixel;

b. for each current pixel, determining a luminance value of the at least one color associated with the current pixel; and

c. designating a current pixel as representative of the isolated user if any color luminance value associated with the current pixel differs from the corresponding mean color luminance value of the corresponding background pixel by at least a fixed threshold.

38. The method of claim 36 further comprising the step of designating a pixel as representative of the isolated user if its luminance value differs from a mean luminance value of the corresponding background pixel by at least a fixed luma threshold.

39. The method of claim 35 wherein the user is isolated according to steps comprising:

a. identifying at least one foreground pixel of a current image based on difference in luminance between the foreground pixel and a mean luminance value derived from corresponding pixels from a plurality of previous images;

b. examining eight pixels neighboring the foreground pixel to identify, based on a luminance difference between the neighboring pixel and a mean luminance value derived from corresponding pixels from a plurality of previous images, each of the eight pixels as a foreground pixel or a background pixel;

c. repeating step (b) for each neighboring pixel identified as foreground; and

d. repeating steps (b) and (c) until no further foreground pixels are identified.

40. The method of claim 31 further comprising the step of analyzing the user'"'"'s image to identify feature regions corresponding at least to the user'"'"'s hands, feet, head and center of mass.

41. The method of claim 40 further comprising the steps of identifying an edge contour around the user'"'"'s image and associating feature classifications with regions of high curvature along the contour.

42. The method of claim 41 wherein the feature regions each have a high-curvature region that itself has a centroid location positioned within a two-dimensional coordinate plane, and the step of associating feature classifications is carried out based on the position of the centroid location.

43. The method of claim 42 further comprising the step of analyzing centroid position relative to other portions of the edge contour.

44. The method of claim 42 further comprising the step of analyzing centroid position relative to a bounding box.

45. The method of claim 31 further comprising the steps of:

a. generating, for presentation on the visual display, agents having controllable positions and appearances;

b. analyzing the user'"'"'s image to identify feature regions corresponding at least to the user'"'"'s hands and feet;

c. comparing successively digitized user images to detect movement of at least one feature region;

d. recognizing gestures based on the detected movements; and

e. responding in real time to the recognized gestures by updating at least one of the position and appearance of at least one agent on the display based on (i) the location in the visual display where the user is introduced, and (ii) current agent location.

46. The method of claim 45 further comprising the step of analyzing the user'"'"'s image to detect gestures, and wherein an agent is updated based further on the detected gestures.

47. The method of claim 45 wherein an agent is updated based further on sounds made by the user.

48. The method of claim 45 wherein an agent is updated based further on an agent activity-selection model.

49. The method of claim 45 wherein an agent is updated based on a behavior model specifying a set of motivational variables that themselves specify a numeric value associated with an agent activity and which may vary over time.