Motion-based command generation technology

• US 5,704,836 A
• Filed: 02/21/1996
• Issued: 01/06/1998
• Est. Priority Date: 03/23/1995
• Status: Expired due to Fees

• Current Assignee: Perception Systems, Inc.
• Original Assignee: Perception Systems, Inc.
• Inventors: Norton, John Mark, Brozsek, Bela Laszlo
• Primary Examiner(s): Harrison, Jessica
• Assistant Examiner(s): SCHAAF, JAMES CHARLES
• Application Number:
  US08/604,186
• Time in Patent Office: 685 Days
• Field of Search: 463/36, 463/29, 463/47, 395/156, 395/158, 250/202, 250/221
• US Class Current: 463/36
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/215 comprising means for detecting acoustic signals, e.g. using a microphone
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 13/533 for prompting the player, e.g. by displaying a game menu
• CPC Class Codes: A63F 13/65 automatically by game devices or servers from real world data, e.g. measurement in live racing competition
• CPC Class Codes: A63F 13/67 adaptively or by learning from player actions, e.g. skill level adjustment or by storing successful combat sequences for re-use
• CPC Class Codes: A63F 13/833 Hand-to-hand fighting, e.g. martial arts competition A63F13/837 takes precedence
• CPC Class Codes: A63F 2300/1012 involving biosensors worn by the player, e.g. for measuring heart beat, limb activity
• CPC Class Codes: A63F 2300/1081 Input via voice recognition
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/308 Details of the user interface
• CPC Class Codes: A63F 2300/6027 using adaptive systems learning from user actions, e.g. for skill level adjustment
• CPC Class Codes: A63F 2300/69 Involving elements of the real world in the game world, e.g. measurement in live races, real video
• CPC Class Codes: A63F 2300/8029 Fighting without shooting
• CPC Class Codes: A63F 2300/8082 Virtual reality
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0421 by interrupting or reflecting a light beam, e.g. optical touch-screen
• CPC Class Codes: G06T 7/254 involving subtraction of images

Abstract

A command system optically monitors the movement of a subject and provides command signals to a computer system to control a graphic of a graphical user interface displayed on a monitor such as an animated character in a video game. The command system has an optical detector unit which continuously scans a subject frame in which the subject is positioned. The subject frame is divided into sub-regions around the subject in which movement of the subject is detected, for example, an arm moving. A command unit receives and processes frame data from the optical detector unit. The command unit determines the velocity of any movement detected in any of the sub-regions and generates an appropriate command signal based on the velocity of the movement. The command unit may also use the direction of the movement in generating command signals. The command signals are then provided to the computer system to move and control the graphical user interface accordingly.

Claims

1. A command system for monitoring movement of a subject and for controlling in accordance with movement of the subject a graphical user interface, the command system comprising:

a) an optical detector unit having an aperture through which light from a field of view is receivable, and an output for frame data indicative of the light received from the field of view the subject being positionable in the field of view, the frame data defining a subject frame, the output providing the frame data on a subject frame-by-subject frame basis; and

b) a command unit having an input in communication with the output of the optical detector unit for receiving the frame data on the subject frame-by-subject frame basis, and an output for providing a command signal to control the graphic user interface the command unit including processing circuitry in communication with the input and the output of the command unit for;

(1) determining position of the subject in a previous subject frame from the frame data of the previous subject frame, and position of the subject in a present subject frame from the frame data of the present subject frame;

(2) calculating velocity of the subject based upon the position of the subject in the previous subject frame and upon the position of the subject in the present subject frame, the velocity being a vector having a magnitude and a direction;

(3) generating a command signal based upon the velocity of the subject;

(4) determining whether the subject is positioned in a proper position within the field of view, the proper position being defined as a position where the subject is substantially centered within the field of view;

(5) determining which direction the subject needs to move to be in the proper position in the field of view; and

(6) generating a positioning signal based upon the direction the subject needs to move to be in the proper position, with the output of the command unit providing the positioning signal; and

c) a subject positioning unit having a input in communication with the output of the command unit for receiving the positioning signal, and a subject interface including an array of light-emitting diodes (LEDs) configured such that when the LEDs illuminate based upon the positioning signal, the LEDs indicate to the subject which direction the subject needs to move to be in the proper position.

2. The command system of claim 1 wherein the array of LEDs includes an up LED, a down LED, a left LED, and a right LED;

the subject interface illuminating the LED corresponding to the direction in which the subject needs to move to be in the proper position as indicated by the positioning signal.

3. A command system for monitoring movement of a subject and for controlling in accordance with movement of the subject a graphical user interface, the command system comprising:

an optical detector unit including an aperture through which from a field of view is receivable a charge-coupled device CCD having a plurality of pixels on which light received through the aperture is incident, and an output for frame data indicative of the light received from the field of view, the subject being positionable in the field of view, the frame data defining a subject frame, the output providing the frame data on a subject frame-by-subject frame basis, each of the pixels having a value indicative of the amount of light incident thereon, the value of the pixels changing from one subject frame to the next subject frame when the amount of light incident thereon changes, the frame data for a subject frame being comprised of the values of the pixels for the subject frame; and

a command unit having an input in communication with the output of the optical detector unit for receiving the frame data on the subject frame-by-subject frame basis, and an output for providing a command signal to control the graphic user interface, the command unit including processing circuitry in communication with the input and the output of the command unit for;

determining position of the subject in a previous subject frame from the frame data of the previous subject frame, and position of the subject in a present subject frame from the frame data of the present subject frame;

calculating velocity of the subject based upon the position of the subject in the previous subject frame and upon the position of the subject in the present subject frame, the velocity being a vector having a magnitude and a direction; and

generating a command signal based upon the velocity of the subject;

wherein the processing circuitry of the command unit further;

a) compares the value of each of the pixels of the present subject frame with the value of the corresponding pixel of the previous subject frame to obtain a difference value for each of the pixels;

b) assigns an ON difference bit to each of the pixels for which the difference value is greater than a predetermined threshold value;

c) determines an edge of the subject in the subject frame by locating at least two adjacent pixels to which an ON difference bit was assigned;

d) records the position of the adjacent pixels;

e) calculates the velocity of the subject based upon the position of the adjacent pixels in the previous subject frame and upon the position of the adjacent pixels in the present subject frame; and

f) generates a command signal based upon the velocity of the subject.

4. The command system of claim 3 wherein:

a) the CCD of the optical detector unit provides the frame data on the subject frame-by-subject frame basis at a predetermined frequency, whereby there is a predetermined period of time between the previous subject frame and the present subject frame; and

b) the processing circuitry of the command unit calculates velocity by;

(1) calculating distance between the position of the adjacent pixels in the previous subject frame and the position of the adjacent pixels in the present subject frame; and

(2) dividing the distance by the predetermined period of time between the previous subject frame and the present subject frame.

5. The command system of claim 3 wherein the processing circuitry of the command unit:

a) defines a command vector extending from the position of the adjacent pixels in the previous subject frame to the position of the adjacent pixels in the present subject frame;

b) calculates a magnitude of the command vector; and

c) generates a command signal based upon the magnitude of the command vector.

6. The command system of claim 5 wherein the processing circuitry of the command unit:

a) determines a horizontal direction vector and a vertical direction vector of the command vector;

b) calculates a magnitude of the horizontal direction vector and a magnitude of the vertical direction vector;

c) compares the magnitude of the horizontal direction vector with the magnitude of the vertical direction vector to determine which of the two direction vectors has the greater magnitude; and

d) generates a command signal based upon the direction vector which has the greater magnitude.

7. The command system of claim 3 wherein the processing circuitry of the command unit:

a) determines whether the velocity of the subject is within a predetermined range;

b) generates a command signal based upon the velocity of the subject when the velocity of the subject is within the predetermined range.

8. The command system of claim 3 wherein the processing circuitry of the command unit:

a) calculates a weighted average of the velocity of the subject in the subject frames previous to the present subject frame;

b) adjusts the velocity of the subject in the present frame based upon the weighted average.

9. The command system of claim 3 wherein:

a) the processing circuitry of the command unit includes comparison circuitry for comparing the value of each of the pixels of the present subject frame with the value of the corresponding pixel of the previous subject frame to obtain a difference value for each of the pixels; and

b) the comparison circuitry of the command unit and the CCD of the optical detector unit are fabricated on a single chip.

10. A command system for monitoring movement of a subject and for controlling in accordance with movement of the subject a graphical user interface, the command system comprising:

an optical detector unit including aperture through which from a field of view is receivable a charge-coupled device CCD having a plurality of pixels on which light received through the aperture is incident, and an output for providing frame data indicative of the light received from the field of view, the subject being positionable in the field of view, the frame data defining a subject frame, the output providing the frame data on a subject frame-by-subject frame basis, each of the pixels having a value indicative of the amount of light incident thereon, the value of the pixels changing from one subject frame to the next subject frame when the amount of light incident thereon changes, the frame data for a subject frame being comprised of the values of the pixels for the subject frame; and

a command unit having an input in communication with the output of the optical detector unit for receiving the frame data on the subject frame-by-subject frame basis, and an output for providing a command signal to control the graphic user interface, the command unit including processing circuitry in communication with the input and the output of the command unit for;

determining position of the subject in a previous subject frame from the frame data of the previous subject frame, and position of the subject in a present subject frame from the frame data of the present subject frame;

calculating velocity of the subject based upon the position of the subject in the previous subject frame and upon the position of the subject in the present subject frame, the velocity being a vector having a magnitude and a direction; and

generating a command signal based upon the velocity of the subject;

wherein the processing circuitry of the command unit further;

a) compares the value of each of the pixels of the present subject frame with the value of the corresponding pixel of the previous subject frame to obtain a difference value for each of the pixels;

b) assigns a difference bit to each of the pixels, the difference bit being ON when the difference value is greater than a predetermined threshold value or OFF when the difference value is less than the predetermined threshold value;

c) combines into bytes the difference bits from the pixels, each of the bytes containing the difference bits from eight adjacent pixels;

d) determines whether the number of difference bits being ON in each of the bytes exceeds a predetermined number;

e) assigns a flag to each of the bytes for which the number of difference bits being ON exceeds the predetermined number;

f) determines an edge of the subject in the subject frame by locating at least two bytes to which a flag was assigned;

g) records the position of the adjacent bytes;

h) calculates the velocity of the subject based upon the position of the adjacent bytes in the previous subject frame and upon the position of the adjacent bytes in the present subject frame; and

i) generates a command signal based upon the velocity of the subject.

11. A command system for monitoring movement of a subject and for controlling in accordance with movement of the subject a graphical user interface, the command system comprising:

an optical detector unit including an aperture through which light from a field of view is receivable, a charge-coupled device (CCD) having a plurality of pixels on which light received through the aperture is incident, and an output for providing frame data indicative of the light received from the field of view, the subject being positionable in the field of view, the frame data defining a subject frame, the output providing the frame data on a subject frame-by-subject frame basis, each of the pixels having a value indicative of the amount of light incident thereon, the value of the pixels changing from one subject frame to the next subject frame when the amount of light incident thereon changes, the frame data for a subject frame being comprised of the values of the pixels for the subject frame, the CCD having a sensitivity to the amount of light incident on the pixels, the sensitivity being adjustable; and

a command unit having an input in communication with the output of the optical detector unit for receiving the frame data on the subject frame-by-subject frame basis, and an output for providing a command signal to control the graphic user interface, the command unit including processing circuitry in communication with the input and the output of the command unit for;

determining position of the subject in a previous subject frame from the frame data of the previous subject frame, and position of the subject in a present subject frame from the frame data of the present subject frame;

calculating velocity of the subject based upon the position of the subject in the previous subject frame and upon the position of the subject in the present subject frame, the velocity being a vector having a magnitude and a direction; and

generating a command signal based upon the velocity of the subject;

wherein the processing circuitry of the command unit further;

a) compares the value of each of the pixels of the present subject frame with the value of the corresponding pixel of the previous subject frame to obtain a difference value for each of the pixels;

b) assigns a difference bit to each of the pixels for which the difference value is greater than a predetermined threshold value;

c) counts the number of pixels to which a difference bit was assigned; and

d) adjusts the sensitivity of the CCD when the number of pixels to which a difference bit was assigned exceeds a predetermined number.

12. A command system for monitoring movement of a subject and for controlling in accordance with movement of the subject a graphical user interface, the command system comprising:

an optical detector unit having an aperture through which light from a field of view is receivable, and an output for providing frame data indicative of the light received from the field of view, the subject being positionable in the field of view, the frame data defining a subject frame, the output providing the frame data on a subject frame-by-subject frame basis; and

a command unit having an input in communication with the output of the optical detector unit for receiving the frame data on the subject frame-by-subject frame basis, and an output for providing a command signal to control the graphic user interface, the command unit including processing circuitry in communication with the input and the output of the command unit for;

determining position of the subject in a previous subject frame from the frame data of the previous subject frame, and position of the subject in a present subject frame from the frame data of the present subject frame;

calculating velocity of the subject based upon the position of the subject in the previous subject frame and upon the position of the subject in the present subject frame, the velocity being a vector having a magnitude and a direction; and

generating a command signal based upon the velocity of the subject;

wherein the processing circuitry of the command unit further;

a) defines a plurality of sub-regions within the subject frame;

b) determines the position of the subject in each of the sub-regions of the previous subject frame from the frame data of the previous subject frame, and the position of the subject in each of the sub-regions of the present subject frame from the frame data of the present subject frame;

c) calculates velocity of the subject in each of the sub-regions based upon the position of the subject in each of the sub-regions of the previous subject frame, and upon the position of the subject in the sub-regions of the present subject frame corresponding to each of the sub-regions of the previous subject frame; and

d) generates a command signal based upon the velocity of the subject in each of the sub-regions, the number of command signals generated being up to the number of sub-regions defined within the subject frame.

13. The command system of claim 12 wherein the processing circuitry of the command unit defines each of the sub-regions such that the velocity of the subject in each of the sub-regions results in a command signal for each of the sub-regions which is different from the command signals for the other sub-regions, each of the command signals controlling the graphical user interface differently.

14. The command system of claim 12 wherein the processing circuitry of the command unit defines an area within the subject frame in which the velocity of the subject is not calculated.

15. A method for providing a command signal indicative of movement of a subject comprising the steps of:

receiving light from a field of view in which a subject is positionable with an array of pixels, each of the pixels having a value indicative of the amount of light incident thereon;

generating sets of data from the values of the pixels, each of the sets of data being indicative of light received from the field of view at an instant of time different from the other sets of data;

calculating velocity of the subject from the sets of data, the velocity being a vector having a magnitude and a direction, with the steps of;

a) comparing the value of each of the pixels from a present set of data with the value of the corresponding pixel from a previous set of data to obtain a difference value for each of the pixels;

b) assigning an ON difference bit to each of the pixels for which the difference value is greater than a predetermined threshold value;

c) determining an edge of the subject in the field of view by locating at least two adjacent pixels to which an ON difference bit was assigned;

d) recording the position of the adjacent pixels; and

e) calculating velocity of the subject based upon the position of the adjacent pixels from the previous set of data and upon the position of the pixels in the present set of data; and

generating a command signal based on the velocity of the subject.

16. The method of claim 15 wherein said step of generating sets of data comprises the step of generating sets of data from the values of the pixels at a predetermined frequency, each of the sets of data being indicative of light received from the field of view at an instant of time different from the other sets of data and having a predetermined period of time between the set of data and a previous set of data;

wherein said step of calculating velocity comprises the steps of;

a) calculating distance between the position of the adjacent pixels from the previous set of data and the position of the adjacent pixels from the present set of data; and

b) dividing the distance by the predetermined period of time.

17. The method of claim 15 wherein said step of calculating velocity comprises the step of:

a) defining a command vector extending from the position of the adjacent pixels from the previous set of data to the position of the adjacent pixels from the present set of data; and

b) calculating a magnitude of the command vector;

wherein said step of generating a command signal comprises the step of generating a command signal based upon the magnitude of the command vector.

18. The method of claim 15 further comprising the step of determining whether the velocity of the subject is within a predetermined range by being greater than a minimum threshold and less than a maximum threshold;

wherein said step of generating a command signal comprises the step of generating a command signal based upon the velocity of the subject when the velocity of the subject is within the predetermined range.

19. The method of claim 15 further comprising the steps of:

a) calculating a weighted average of the velocity of the subject from the previous sets of data; and

b) adjusting the velocity of the subject based upon the weighted average.

20. A method for providing a command signal indicative of movement of a subject comprising the steps of:

receiving light from a field of view in which a subject is positionable with an array of pixels, each of the pixels having a value indicative of the amount of light incident thereon;

generating sets of data from the values of the pixels, each of the sets of data being indicative of light received from the field of view at an instant of time different from the other sets of data;

calculating velocity of the subject from the sets of data, the velocity being a vector having a magnitude and a direction, with the steps of;

a) comparing the value of each of the pixels from a present set of data with the value of the corresponding pixel from a previous set of data to obtain a difference value for each of the pixels;

b) assigning an ON difference bit to each of the pixels, the difference bit being ON when the difference value is greater than a predetermined threshold value or OFF when the difference value is less than the predetermined threshold value;

c) combining into difference bytes the difference bits of the pixels, each of the difference bytes containing the difference bits from eight adjacent pixels;

d) determining whether the number of difference bits being ON in each of the difference bytes exceeds a predetermined number;

e) assigning a flag to each of the difference bytes for which the number of difference bits being ON exceeds the predetermined number;

f) determining an edge of the subject in the field of view by locating at least two difference bytes to which a flag was assigned; and

g) recording the position of the adjacent difference bytes;

wherein said step of calculating velocity comprises the step of calculating velocity of the subject based upon the position of the adjacent difference bytes from the previous set of data and upon the position of the adjacent difference bytes from the present set of data; and

generating a command signal based on the velocity of the subject.

21. A method for providing a command signal indicative of movement of a subject comprising the steps of:

receiving light from a field of view in which a subject is positionable with a charge-coupled device (CCD) having an array of pixels, each of the pixels having a value indicative of the amount of light incident thereon, the CCD having an adjustable sensitivity; and

adjusting the sensitivity of the CCD with the steps of;

a) comparing the value of each of the pixels from a present set of data with the value of the corresponding pixels from a previous set of data to obtain a difference value for each of the pixels;

b) assigning a difference bit to each of the pixels for which the difference value is greater than a predetermined threshold value;

c) counting the number of pixels to which a difference bit was assigned; and

d) adjusting the sensitivity of the CCD when the number of pixels to which a difference bit was assigned exceeds a predetermined number;

generating sets of data from the values of the pixels, each of the sets of data being indicative of light received from the field of view at an instant of time different from the other sets of data;

calculating velocity of the subject from the sets of data, the velocity being a vector having a magnitude and a direction; and

generating a command signal based on the velocity of the subject.

26. The method of claim 21 further comprising the step of positioning the subject in the field of view when the subject is not substantially centrally positioned in the field of view.

22. A method for providing a command signal indicative of movement of a subject comprising the steps of:

receiving light from a field of view in which a subject is positionable;

generating sets of data from the received light, each of the sets of data being indicative of the amount of light received from the field of view at an instant of time different from the other sets of data and defining a subject frame for that instant of time;

defining a plurality of sub-regions within the subject frames;

determining a previous position of the subject in each of the sub-regions of the subject frame from a previous set of data and a present position of the subject in each of the sub-regions of the subject frame from a present set of data;

calculating velocity of the subject in each of the sub-regions based upon the previous position of the subject and the present position of the subject, the velocity being a vector having a magnitude and a direction; and

generating a command signal based upon the velocity of the subject in each of the sub-regions, the number of command signals being up to the number of sub-regions defined within the subject frame, each of the command signals being different from the other command signals.

23. The method of claim 22 further comprising the step of defining an area within the subject frame in which the subject is positionable and in which the velocity of the subject is not calculated.

24. A method for controlling a character displayed on a video display unit comprising the steps of:

a) positioning a subject within a field of view of an optical detector unit;

b) defining a plurality of sub-regions around the subject within the field of view;

c) detecting movement of the subject within the sub-regions;

d) generating a command signal based on the detected movement; and

e) animating the character based upon the command signal.

25. The method of claim 24:

wherein the detected movement of the subject has a velocity with a magnitude and a direction; and

further comprising the steps of;

a) calculating the magnitude of the velocity of the detected movement of the subject; and

b) calculating the direction of the velocity of the detected movement of the subject;

wherein said step of generating a command signal comprises the step of generating a command signal based upon the magnitude and the direction of the velocity of the detected movement of the subject.

27. A command system for monitoring movement of a subject and for controlling in accordance with movement of the subject a graphical user interface, the command system comprising:

a) an optical detector unit having an aperture through which light from a field of view is receivable, and an output for providing frame data indicative of the light received from the field of view, the subject being positionable in the field of view, the frame data defining a subject frame, the output providing the frame data on a subject frame-by-subject frame basis; and

b) a command unit having an input in communication with the output of the optical detector unit for receiving the frame data on the subject frame-by-subject frame basis, and an output for providing a command signal to control the graphic user interface, the command unit including processing circuitry in communication with the input and the output of the command unit for;

(1) determining position of the subject in a previous subject frame from the frame data of the previous subject frame, and position of the subject in a present subject frame from the frame data of the present subject frame;

(2) calculating velocity of the subject as a vector having a magnitude and a direction, the velocity being determined based upon the position of the subject in the previous subject frame and upon the position of the subject in the present subject frame, the direction of the velocity having a horizontal component and a vertical component;

(3) calculating the magnitude of the horizontal component and the magnitude of the vertical component of the velocity of the subject;

(4) comparing the magnitude of the velocity of the horizontal component with the magnitude of the velocity of the vertical component to determine which of the two magnitudes of velocity is greater; and

(5) generating a command signal based upon the greater of the two magnitudes of velocity.

28. A method for providing a command signal indicative of movement of a subject comprising the steps of:

a) receiving light from a field of view in which a subject is positionable;

b) generating sets of data from the received light, each of the sets of data being indicative of light received from the field of view at an instant of time different from the other sets of data;

c) determining a previous position of the subject in the field of view from a previous set of data;

d) determining a present position of the subject in the field of view from a present set of data;

e) determining velocity of the subject from the previous position of the subject and the present position of the subject, the velocity being a vector having a magnitude and a direction with a horizontal component and a vertical component;

f) calculating the magnitude of the velocity of the horizontal component and the magnitude of the velocity of the vertical component of the subject based upon the previous position and the present position of the subject;

g) comparing the magnitude of the velocity of the horizontal component with the magnitude of the velocity of the vertical component to determine which of the two magnitudes of velocity is greater; and

f) generating a command signal based upon the greater of the two magnitudes of velocity.