User input device and method for interaction with graphic images

• US 7,071,914 B1
• Filed: 09/01/2000
• Issued: 07/04/2006
• Est. Priority Date: 09/01/2000
• Status: Expired due to Term

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment Incorporated (Sony Group Corp.)
• Inventors: Marks, Richard
• Primary Examiner(s): Eisen, Alexander
• Application Number:
  US09/654,141
• Time in Patent Office: 2,132 Days
• Field of Search: 345/156, 345/158, 345/473, 345/474, 345/157, 463 36- 39, 700/15, 700/17, 700/240, 700/264, 715/863, 382/118, 382/103, 382/168, 250/221
• US Class Current: 345/156
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883

Abstract

A device and method for effecting movement, responsive to user input, of an object on a graphical display are disclosed. An input device comprises a component for capturing video images, an input image processor that generates an output signal responsive to motion from the video images, and an output image processor that is programmed to effect movement of the object on the graphical display in response to signals received from the input image processor. Various algorithms are employed within the input image processor to determine initial and derivative data that controls the movement of the object on the graphical display. In a preferred embodiment, video images are captured and processed to isolate a human form from a background, arm position and movement data are calculated from the human form and a signal is generated responsive to this data for controlling the movement of an object, such as a bird, on a graphical display. The movement controlled on the graphical display can take the form of a moving object, or of the change of perspective that such an object might undergo, for example, a bird'"'"'s eye view.

Claims

1. A method for generating signals to effect one of translational movement, rotational movement, and both translational and rotational movements of an object on a graphical display using one of human arm position data, human arm movement data, and both human arm position and movement data, comprising:

providing an image processor and a device for capturing a video sequence;

capturing, from the video sequence, a frame that does not include a person;

isolating a view comprising a foreground subject image view by performing an algorithm on the video sequence and the frame that does not include the person, wherein the algorithm includes subtracting the frame that does not include the person from individual frames in the video sequence;

determining whether the isolated view includes an image of a person by ensuring that a number of nonzero pixels in the foreground image is within a range indicative of a presence of the image of the person;

determining a horizontal extent of a torso in the image of the person so as to isolate arm portions of the person in frames of the captured video sequence;

computing arm angles by calculating angles of principle moment of nonzero pixels in the arm portions of the image of the person; and

generating an arm position data signal responsive to arm angles for effecting one of translational movement, rotational movement, and both translational and rotational movement of an object on a graphical display.

2. The method of claim 1 wherein ensuring that the number of nonzero pixels in the foreground image is within the range indicative of the presence of the image of the person includes,

counting a total number of nonzero pixels in the foreground image;

and

verifying that the total number of nonzero pixels in the foreground image is within a range extending from a minimum threshold number of pixels to a maximum threshold number of pixels.

3. The method of claim 1 wherein the following algorithm is used in the isolating step:

(a) obtain static background Y0 U0 V0 frames;

(b) smooth images Y0 U0 V0 using a 5×

5 Gaussian convolution;

(c) obtain current Y U V video frames;

(d) smooth images Y U V using a 5×

5 Gaussian convolution;

(e) for each pixel in Y, compute Ydif=abs(Y−

Y0);

(f) for each pixel in U, compute Udif=abs(U−

U0);

(g) for each pixel in V, compute Vdif=abs(V−

V0);

(h) for each pixel in Ydif Udif Vdif, compute Sum=Ydif+Udif*8+Vdif*8;

(i) for each pixel in Sum, compute Foreground=1 if Sum>

Threshold, 0 otherwise;

(j) erode Foreground using standard erosion morphological filter (to remove any single-pixel erroneous measurements, such as caused by salt-and-pepper noise).

4. The method of claim 1 wherein the arm position data signal generated in the generating step is selected from the group consisting of signals related to object airspeed acceleration, bank angle, and pitch angle.

5. The method of claim 1 wherein the arm position data signal generated in the generating step is determined with the inclusion of smoothing constants.

6. A method for generating signals for use in a flight simulator graphical display using human arm position data to effect one of translational movement, rotational movement, and both translational and rotational movement, comprising:

providing a device for capturing video images and an image processor;

capturing video images with the device, the video images including an image of a background without a human form and an image of a background with a human form;

using the image processor to process the captured video images to isolate the human form from the background;

isolating arm portions of the human form from a captured video image using the image processor;

calculating arm position and movement data using the image processor;

generating a signal responsive to the arm position and movement data using the image processor for use in generating a state of a flight simulator graphical display, wherein the flight simulator graphical display includes as an object a flying creature that moves wings in response to the generated signal; and

generating flapping noise corresponding to movement of the wings of the flying creature.

7. The method of claim 6 wherein the flight simulator graphical display depicts a change in perspective of what a flying creature would see.

8. The method of claim 6 wherein a volume of the flapping noises increases with an increased rate of arm motion.

9. The method of claim 6 wherein the flapping noises are triggered when a signed time rate of change of an average of arm angles exceeds a pre-determined threshold.

10. A method for generating signals for use in a flight simulator graphical display using human arm position data to effect one of translational movement, rotational movement, and both translational and rotational movement, wherein the flight simulator graphical display includes as an object a flying creature that moves wings, comprising:

providing a device for capturing video images and an image processor;

capturing video images with the device;

using the image processor to process the captured video images to isolate a human form from a background;

isolating arm portions of the human form from a captured video image using the image processor;

calculating arm position and movement data using the image processor;

generating a signal responsive to the arm position and movement data using the image processor, the signal to be used in generating a state of the flight simulator graphical display; and

generating flapping noises corresponding to a movement of the wings of the flying creature.

11. The method of claim 10 wherein a volume of the flapping noises increases with an increased rate of arm motion.

12. The method of claim 10 wherein the flapping noises are triggered when a signed time rate of change of an average of calculated arm angles exceeds a predetermined threshold.