Real time perception of and response to the actions of an unencumbered participant/user

• US 4,843,568 A
• Filed: 04/11/1986
• Issued: 06/27/1989
• Est. Priority Date: 04/11/1986
• Status: Expired due to Fees

• Current Assignee: Katrin Hinrichsen, Myron W. Krueger, Thomas S. Gionfriddo
• Original Assignee: Katrin Hinrichsen, Myron W. Krueger, Thomas S. Gionfriddo
• Inventors: Krueger, Myron W., Hinrichsen, Katrin, Gionfriddo, Thomas S.
• Primary Examiner(s): Harkcom, Gary V.
• Assistant Examiner(s): HERNDON, HEATHER R
• Application Number:
  US06/850,770
• Time in Patent Office: 1,173 Days
• Field of Search: 364/518-522, 358/93, 358/107, 358/108, 358/81
• US Class Current: 382/100
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/69 Involving elements of the real world in the game world, e.g. measurement in live races, real video
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/033 Pointing devices displaced or positioned by the user , e.g. mice, trackballs, pens or joysticks; Accessories therefor digitisers characterised by the transducing means G06F3/041
• CPC Class Codes: G06F 3/042 by opto-electronic means
• CPC Class Codes: G06T 7/00 Image analysis
• CPC Class Codes: G09B 21/00 Teaching, or communicating with, the blind, deaf or mute audible presentation of material to be studied G09B5/04; devices or methods for replacing direct visual or auditory perception by another kind of perception A61F9/08, A61F11/04; audible indication of meter readings or of colour G01D7/12; watches for blind persons G04B25/02; speech analysis, speech recognition G10L; sound-recording or reproducing, per se G11B

Abstract

Apparatus and method for using an image of the human body to control real time computer events wherein data regarding a participant are acquired through any means of imaging without marking the participant or requiring that he or she use or wear a stimulus source, sensing device or special clothing. Perception and feature analysis are carried out by specialized circuitry and computer software, and response to perception is expressed using any device controllable by a computer. Participants generally are unaware of processing delays between action and reaction, and their perception of an event is similar to dealing with a human being or animate creature. Perception and response occur in real time with action and reaction closely coupled.

Claims

1. A device for perceiving, processing and responding in real time to a demonstrative behavior of at least one unmarked participant, said device comprising:

means spaced from at least one unmarked participant for distinguishing the participant from a background;

means for imaging each distinguished participant and providing an image thereof;

means for digitizing said image provided by said means for imaging;

means connected to said means for digitizing for ascertaining features of the digitized image;

means connected to said means for ascertaining for identifying at least one characteristic of at least one of said ascertained features, each of the identified characteristics being a selected one of a location and a movement of a respective ascertained feature, said means for identifying providing a control parameter in response to the at least one identified characteristic;

means responsive to said control parameter for processing said ascertained features of said digitized image to provide a signal indicative of a demonstrative behavior of said participant; and

means for responding in real time to said signal indicative of a demonstrative behavior of said participant.

2. The device as defined in claim 1, wherein the means for imaging includes a video camera.

3. The device as defined in claim 1, wherein the means for processing includes control means for controlling said means for ascertaining features of said digitized image.

4. The device as defined in claim 1, wherein the means for digitizing includes an analog to digital convertor receiving said image from said means for imaging, means for digitally thresholding an output of said analog to digital convertor with hysteresis, and means for filtering a digital noise from an output of said means for digitally thresholding.

5. The device as defined in claim 1, wherein said digitized image includes boundary information corresponding to a boundary between the distinguished participant and said background, and the means for imaging and the means for ascertaining are constituted in part by multiple processor means for operating in parallel, said multiple processor means including means for extracting various types of information from said boundary information in said digitized image.

6. The device as defined in claim 5, wherein said at least one unmarked participant includes a left-most participant and a right-most participant and said means for imaging produces a plurality of horizontal scan lines, and wherein the means for ascertaining includes means for performing an analysis of said left-most and said right-most participant as distinguished from said background on each said horizontal scan line.

7. The device as defined in claim 1, wherein said at least one unmarked participant includes a left-most participant and a right-most participant and said means for imaging produces a plurality of horizontal scan lines, and wherein said means for ascertaining includes a micro-coded hardware processor means having means for extracting information from a boundary between said background and said left-most and right-most participants on each said horizontal scan line.

8. The device as defined in claim 1, wherein the means for ascertaining includes means for performing a directed search in a bit-map image for a participant versus background transition and means for capturing and storing said bit-map image from said digitized image.

9. The device as defined in claim 1, wherein the means for ascertaining includes hardware processor means for capturing a bit-map image from said digitized image and means for extracting requested participant versus background transition points from said bit-map image.

10. The device as defined in claim 1, wherein the means for ascertaining includes means for tracing an image contour of said digitized image and means for detecting features of various size by point to point measurements along said image contour.

11. The device as defined in claim 1, wherein the means for ascertaining includes micro-coded hardware processor means including means for capturing a bit-map image from said digitized image, means for tracing contours of said participant in an image area in said bit-map image, and means for finding features of a specified size along said contours.

12. The device as defined in claim 1, wherein said means for ascertaining includes dedicated hardware means for detecting a finger feature of said participant.

13. The device as defined in claim 1, wherein said means for imaging includes a plurality of means for vision processing adapted to accumulate data and the means for ascertaining includes means for processing data accumulated by said plurality of means for vision processing.

14. The device as defined in claim 1, wherein the means for identifying includes means for controlling predetermined perception parameters and has means for processing said perception parameters.

15. The device as defined in claim 1, wherein the means for identifying includes means for controlling a parameter and means responsive to said parameter for placing said device in a selected one of an alternative interactive mode and an alternative interactive function.

16. The device as defined in claim 1, wherein said means for responding includes means for color graphics display.

17. The device as defined in claim 1, wherein the means for responding includes means for generating a video image of said participant, means for creating and displaying graphic objects, and means for real time interaction of said participant'"'"'s video image with said graphic objects.

18. The device as defined in claim 1, wherein the means for responding includes means for projecting an image of said participant, means for projecting a controllable graphic object and means for real time replacement of said participant'"'"'s projected image by said controllable graphic object.

19. A device as defined in claim 1, wherein the means for responding includes means for synthesizing sound.

20. A process for controlling a device adapted to perform a plurality of operations to perceive, process and respond in real time to a plurality of demonstrative behaviors of at least one unmarked participant, said process comprising:

associating each of a plurality of demonstrative behaviors of at least one unmarked participant with a respective operation of said device;

generating low level code associated with said operations, said low level code including at least one independent primitive module;

controlling a plurality of special purpose processors of said device in accordance with said low level code; and

sequencing the control of said special purpose processors in accordance with a performed sequence of said demonstrative behaviors so as to realize real time performance of a corresponding sequence of said operations.

21. A device for representing graphically, in real time, at least one unmarked participant in a demonstrative activity associated with said participant, said device comprising:

means for obtaining a contrasting image of said unmarked participant;

means for digitizing said contrasting image to provide a digitized image;

means for timing control of said means for obtaining said contrasting image;

means for ascertaining features of said digitized image;

means connected to said means for timing control for storing ascertained features of said digitized image and for processing said stored ascertained features of said digitized image;

means for identifying at least one characteristic of at least one of said ascertained features, each of the identified characteristics being a selected one of a location and a movement of a respective ascertained feature of said digitized image;

means responsive to said means for identifying for controlling said means for storing and for processing so as to provide information indicative of a demonstrative activity of said participant; and

means responsive to said information for updating a selected one of a graphic and a manipulatable image representation, in real time, of a demonstrative behavior of said participant.

22. The device as defined in claim 21, wherein the means for updating includes means for representing said participant graphically conjointly with a simultaneous image of said participant and is constituted in part by said means for obtaining a contrasting image of said participant.

23. The device as defined in claim 22, wherein said means for updating includes means for providing a simultaneous digitized image representation.

24. The device as defined in claim 22, wherein said means for updating includes means for providing a simultaneous video image representation.

25. The device as defined in claim 21, wherein the means for updating includes means for representing said participant graphically and means for obtaining synthesized sound, said means for obtaining synthesized sound being connected to said means for storing and processing said digitized image.

26. The device as defined in claim 21, wherein the means for updating includes means for representing graphically, in real time, said unmarked participant conjointly and simultaneously with a representation of a pre-selected environment for said participant.

27. The device as defined in claim 26, wherein the pre-selected environment is a graphic environment.

28. The device as defined in claim 27, wherein the pre-selected environment is a fractal graphic environment.

29. The device as defined in claim 26, wherein the pre-selected environment is a synthesized sound environment.

30. The device as defined in claim 26, wherein the pre-selected environment is a synthesized color environment.

31. The device as defined in claim 26, wherein the pre-selected environment is a synthesized sound and color environment.

32. The device as defined in claim 21, wherein said at least one participant is represented graphically on screen means.

33. The device as defined in claim 21, wherein the means for updating further includes means for special effect processing.

34. A method for perceiving, processing and responding, in real time, to a demonstrative behavior of at least one unmarked participant, the method comprising the steps of:

imaging said unmarked participant to obtain an analog image of the same;

converting said image from analog to digital form to obtain a digitized form of said image;

first processing said digitized form of said image to obtain desired features from said digitized form of said image;

storing said obtained desired features;

synchronizing said first processing and storing by establishing at least one characteristic of said desired features for identification and comparison with said stored desired features, each of the characteristics being a selected one of a location and a movement of a respective feature;

further processing, in real time, said stored desired features and said at least one characteristic to provide processed features; and

projecting, in an imaged environment, a processed image of said processed features, in real time.

35. The method as defined in claim 34, wherein the environment in which said processed image is projected is a graphic environment.

36. The method as defined in claim 34, wherein said processed image is projected with said processed features as graphic features.

37. The method as defined in claim 34, wherein said processed image is projected with said processed features as color.

38. The method as defined in claim 34, wherein said processed image is projected with said processed features as sound.

39. The method as defined in claim 34, wherein the further processing includes obtaining vision processor features;

storing the obtained vision processor features;

graphically processing said at least one characteristic with the stored vision processor features;

adding graphic effects to the graphically processed vision processor features and using the result for further processing, in real time, said at least one characteristic.