Interactive games with prediction and plan with assisted learning method

• US 20110256914A1
• Filed: 04/02/2010
• Published: 10/20/2011
• Est. Priority Date: 07/25/2005
• Status: Abandoned Application

• Current Assignee: Ned M. Ahdoot
• Original Assignee: Ned M. Ahdoot
• Inventors: Ahdoot, Ned M.
• Application Number:
  US12/798,335
• Publication Number:
  US 20110256914A1
• Time in Patent Office: Days
• US Class Current: 463/8
• CPC Class Codes: A63B 2220/806 Video cameras
• CPC Class Codes: A63B 2244/10 Combat sports
• CPC Class Codes: A63B 24/0003 Analysing the course of a movement or motion sequences during an exercise or trainings sequence, e.g. swing for golf or tennis
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/22 Setup operations, e.g. calibration, key configuration or button assignment
• CPC Class Codes: A63F 13/26 having at least one additional display device, e.g. on the game controller or outside a game booth
• CPC Class Codes: A63F 13/42 by mapping the input signals into game commands, e.g. mapping the displacement of a stylus on a touch screen to the steering angle of a virtual vehicle
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 13/655 by importing photos, e.g. of the player
• CPC Class Codes: A63F 13/67 adaptively or by learning from player actions, e.g. skill level adjustment or by storing successful combat sequences for re-use
• CPC Class Codes: A63F 13/833 Hand-to-hand fighting, e.g. martial arts competition A63F13/837 takes precedence
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/301 using an additional display connected to the game console, e.g. on the controller
• CPC Class Codes: A63F 2300/308 Details of the user interface
• CPC Class Codes: A63F 2300/69 Involving elements of the real world in the game world, e.g. measurement in live races, real video
• CPC Class Codes: A63F 2300/8029 Fighting without shooting
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386

Abstract

A method for engaging a player or a pair of players in a motion related game including the steps of attaching plural geometrical colored elements onto selected portions of the player(s) garments and processing a video stream of each of the players to separately identify the positions, velocities an accelerations of the colored elements. The method further comprises generation of a combatant competitor image and moving the image in a manor to overcome the player. In a further approach, two players are recorded and their video images are presented one screens frontal to the other of the players. The same colored elements are used to enable controller calculations of fighting proficiency of the players and enable assisted learning.

Claims

1. An apparatus and a method of playing real time interactive motion related hand-to-hand combat involving a player wearing 3D glasses and 3D colored geometric shapes on his/her moving bodily parts such as head, hands and feet to get engaged with virtual image of a competitor player;

the apparatus receiving the video frames of the player, initially checking for an offense or defense motion that takes place within a short time, in relation to the time that it takes for an effective single stroke of player'"'"'s offense or defense, and reading a prediction for continuation (trajectory) of same offense or defense of the player along with a plan, for displaying an appropriate offense or defense video motions of the player;

the apparatus compares the actual player motion to the prediction of the player;

through an establishment of quantizing the player'"'"'s body motions based upon the initial prediction and comparison to the actual trajectories of the play, thus an assisted learning is provided;

the apparatus comprising the following summarized steps;

a) the apparatus captures the received video frames from the player and identifies portions of the player with individual 3D colored geometric shape elements thus identifying body parts of the player while in motion;

b) determining positions in 3-space of the portions of the player on each video frame, thus calculating changes in 3-D position from one frame to another frame;

c) the positional changes in a frames, in conjunction with the associated frame timing (period between frames) allows the derivation of velocity, acceleration from previous frame to the next;

d) the initial trajectory of a motion, during a short time at the beginning of each stroke, including location, velocity and acceleration are established for a typical player motion in a period of time ('"'"'b′

number of frames set during the initialization);

e) identifying each player'"'"'s early moves that is consistent within the arbitrary period of time, “

b”

to represent an early offense, defense, or no motion, the early detections of each player motions that is identified as a beginning of the player motion such as a hit, a stroke, or a dodge are hereafter called an “

event”

;

f) each event is further associated with a continuation of the same offense or defense motion by the player, the association is a link between a controller generated trajectories, of a pre recorded play of a pair of pro players for offense and defense;

g) using the information derived from the above, recognizing the early moves of a player as an offense or defense and predict a continuation of the same motion of the player towards a goal;

this prediction or expectation is in a form of upcoming player'"'"'s trajectory hereafter called a “

prediction”

;

h) each offense or defense event will be associated with a prediction and a plan, the prediction will predict that the player will continue with the same event for the rest of the intended motion;

a plan is a controller generated video image of a pro player that reacts and responds to the player'"'"'s detected initial event;

i) for each game, the predictions and the plans will rely on a system of dividing player specifications and measurement motions magnitudes as a numerator divided by a quantization number “

n”

used as a divisor;

comparing result of the division with the remainder, for the game be further refined and categorized into different quantization levels that is used for the process of further modifications and degrees of player'"'"'s desired expertise and styles of play;

j) after the detection of an event, the player'"'"'s motions are further received and analyzed to the end of the predicted motion;

k) the detection and the plan is continued unless a new event is detected due to player'"'"'s discontinuation of initial movement and restart of a new event;

l) the electronic quantization (divide) circuit and a memory address lookup table is used to translate physical attributes of a player including its motion strengths to generate new memory bank addresses within a memory bank to write and read prediction trajectories and plan scenarios;

m) programmer assisted learning is accomplished during and follow up of a player'"'"'s real time motions through quantization method and compared to an existing predicted trajectories that is stored in the memory banks, thus new entries in the memory banks are created by the programmers;

n) by adjusting different variables that signifies different thresholds of motions, and utilizing methods in this application, the program is instructed to change the quantization numbers, thus detect more refined levels of player motions during detection, for assisted learning purposes;

o) store new refined values in the predictions data banks for more accurate prediction process;

p) at the end of the prediction or plan, the trajectory of the player and the image'"'"'s motion, are compared to evaluate scores and awarding points to each of the players for successful offensive and defensive actions;

q) using the capability provided in above steps, the player is provided the option to choose the degree of skill and different styles of a play (by choosing offense or defense from a menu of different players famous in that game).

2. The method of claim 1 wherein, utilizing a digital video camera interfaced to a distributed controller to capture real time 3-D motions of a player comprising the further steps of:

a) calibrating the system initially by placing the player(s) at a fixed distance from the camera and having the colored elements, and bodily signatures of the player to be calibrated with a video gray scale for real time 3-D motion detection;

b) continually receiving the camera'"'"'s real time electro-optical, auto focus, and zooming control information along with video camera signals measuring the 3 dimensional positions of the player(s) at motions;

c) while in motion, the depth (z) is calculated by the ratio of the of the total pixel count of the colored elements worn by the player(s) to the total video pixels of the colored elements measured during initial calibration;

d) utilizing a camera that could be commanded to perform auto focus or controller controlled focus;

e) adjusting the pixel count information of the multi colored geometric elements, and player(s) bodily signature based upon the received camera'"'"'s auto-focus or controller controlled focus;

f) trajectory of motion, speed, and acceleration of the players body parts is measured upon the differential changes of recent frame to the previous frame, provide filtering of images to provide a sharp image and eliminate background noises;

g) differential changes are measured from frame to frame by following the periphery of each colored element and measuring pixel changes;

h) utilize a controller controlled camera that is commanded to focus and stay focus on a specific moving colored element;

i) utilize a controller controlled camera that its zooming is controller controlled;

j) placing the digital camera on a controller controlled gimbal to follow the player'"'"'s motions. the pixel count derived from step c will be further adjusted based upon the 2-d gimbal motions;

k) utilizing the digital camera with inferred sensors to monitor the bodily temperature of the player.

3. Provisions of claim 1 for addressing a Mass Memory, wherein physical attributes of a player and detected motion are used to generate the Memory Bank address;

the physical attributes are input to a plurality of logic dividers within a module to quantize the data;

a Feedback Controller receiving the quantized information to check and decide if the quantized level “

n”

(result of the divisor) need to be changed;

an address lookup table to translate quantized physical data to a physical memory address;

a crossbar switches to enable reading relevant data module in the Mass Memory, and a video data output controller looking at the Mass Memory;

a video and data controller, interfaced to the output of the Mass Memory to distribute data to various registers including Feedback Registers;

as follows;

a) a mathematical block consisting of plurality of logic dividers each having a input holding register;

each register is set to different variable values of a(1), a(2), a(3), to a(x), wherein “

a”

denotes the input variables data of player'"'"'s body parts including personal specification such as weight, height, degree of expertise, and measured motional variables including distances, velocity and accelerations that are used as the numerators to the dividers;

registers, receiving its data from different sources, of sensors or computations or feedbacks from the mass memory;

mathematical calculators coupled to each one of the “

n”

register to perform mathematical operation on the content of the registers;

divider circuits divides personal specifications and motional variables of a player used a numerator divides them by an integer “

n”

, provided during initialization and dynamically changed during a play;

each physical motion variable will have its own quantization number “

n”

;

the integer results and the remainder are sent to the Feedback Controller to decide on the next level of divisor “

n”

for assisted learning purposes;

to quantize physical attributes of a player such as physical specifications (weight and others) and derived motion activities of players body parts such as location, velocity and acceleration;

b) the Feedback Controller receiving divisor, results and the remainder, examines the result and the remainder to performs the following;

if the magnitude of the remainder fall within the result keep the existing quantize level “

n”

;

if the magnitude of the remainder is larger than result, change “

n”

till remainder is less or equal the result;

report the new quantize level “

n”

to the arithmetic divider block;

c) an Address Lookup Table memory or a cascaded address lookup table receives plurality of quantized variable data for each one of the player'"'"'s physical and motion attributes from the feedback controller;

the data of the address look up table is preloaded by the controller to translates the physical attributes of the player, to a the physical logical address of the mass memory;

part of the output address from the feedback controller is set as an input address to a Crossbar Switch to provide enabling of memory modules and memory units within the mass memory;

the feedback controller to set crossbar switch controls for connection of one of the inputs to output of the crossbar switch;

d) a data controller interfaced to the mass memory output to transfer video data to the video terminals, read prediction and plan trajectories and read the feedback information;

feedback information are stored in registers to be used as another address to the feedback controller;

e) the feedback address could get bypassed by a signal from controller feedback controller;

f) Mass Memory data includes a pre established predictions of a player based upon initial detection of an event, predictions of the image player, plans for future actions of the image and 3D video data pertaining to the image'"'"'s motion trajectories, and its corresponding 3D offense or defense motion;

initially, when the program is being developed, the mass memory will be used to store data from video of two player'"'"'s, being engaged in a combat with one another and their motions captured by each one wearing a camera (and other cameras monitoring the play);

using the same apparatus, an appropriate number “

n”

is chosen as a devisor of the magnitudes of physical and motion data of the players such weight, distance, velocities, and accelerations;

the divisor “

n”

is adjusted by the Feedback Controller such that the remainder is less than the magnitude of the result;

when the play is in progress, the Feedback Controller checks the remainder and the magnitude of the result;

if the remainder is equal or within the result magnitude, it does nothing;

if the remainder is higher or lower than the quantized magnitude, it provides a list of umber of changes of “

n”

for each one of the data entries, such as distance velocity and accelerations for the operator (programmer) to check the changes and generate new addresses to the mass memory (from one of the existing “

not used”

memory addresses);

the operator will then provide new prediction to the players trajectory, new plan of action including the trajectory and the video of the image and stores it in the relevant new address;

this is very similar to a child being taught new skills.

4. The method of claim 1 wherein the controller'"'"'s further actions are synchronized to the start of a player'"'"'s motions, or verbal commands on a frame by frame basis, further comprising the steps of:

a) each incoming frame is compared to the previous frame to detect the magnitude of change compared to the previous frame. changes in the incoming frames surpassing a threshold are lead to further processing, changes in the incoming frames not surpassing the threshold are counted, discarded, and led to further processing;

b) continuous incoming frames not surpassing a threshold for a certain period of time (“

c”

number of frames) are counted, discarded, and led to an offense motion by the controller generated image;

c) the trajectory of a motion, including location, velocity and acceleration are established for a player motion in real time;

d) generate an imaginary x, y, and z positional distances of player with respect to the CRT position as a reference;

generate an imaginary x, y, and z positional distances of image with respect to the CRT position as a reference;

a diagonal distance is generated from two points of the image and the player'"'"'s positions offense and defense parts hereafter called “

penetration distance”

;

when the penetration distance is reached by the designated offense and defense parts, a score is made;

e) the voice activated command or other commands are analyzed and led to different processing stages, depending upon the nature of the commands;

5. The apparatus of claims 1, 2, 3 and 4 wherein an Event Detection and Prediction distributed digital image controller, continually monitors the offense and defense movement of the player to detect offense and defense motions that are consistent within certain time period (“

b”

number of frames) called an event and is defined as offense or a defense motion by the player;

comprising of the steps;

a) consecutive frames that have passed the threshold, each are compared to the previous frame to detect the magnitude of change, changes are added to the previous trajectory of the player'"'"'s motions;

b) if received frame number is less than b number of frames, repeat previous step, otherwise go to the next step;

c) set address to the mass memory and check (at the end of “

b”

number of frames), the results of the player'"'"'s motions with that of the image at the end of “

b”

number for frames;

the Mass memory is used for comparison of trajectories the player and the image during each frame by setting the trajectory of the image to the feedback registers and reading the result from the pre loaded memory data for each body parts;

d) if at the end of “

b”

number of frames, the player'"'"'s motions indicate an offense aimed at the image'"'"'s defensive trajectory, continue processing player'"'"'s offensive by informing an Offense Event Follower Controller otherwise inform the an Event Defense Controller.

6. The apparatus of claims 1 wherein the Offense Event Follower Controller receives player'"'"'s offensive event from the Event Controller, it performs the following steps:

a) read the image'"'"'s defense trajectory plan from Mass Memory and initialize “

n”

that is dependent upon degree of expertise initially chosen by the player;

b) from the next incoming video frame, calculate player'"'"'s actual new motion coordinates, amend it to the previous calculated trajectory;

continually display the planned offense motions video of the image and transition to next step;

c) compare image'"'"'s prediction trajectory with player'"'"'s actual trajectory of motion;

if the measured player'"'"'s offense 3D trajectory and predicted image'"'"'s defense trajectories are off (based upon the value of “

n”

as explained in claim 5, go to step g), otherwise go to next step;

d) checks if the player'"'"'s offense penetrates or hits the image'"'"'s defense;

if it does not go to step e), otherwise go to step f);

players motion characteristics are updated in mass memory input registers and image'"'"'s motion characteristics are updated in the feedback registers of memory bank addressing;

the distance from player to image and the decision on the hit or no hit is provided by the memory bank data;

if it is not a hit is considered a miss (dodge) otherwise it is considered a hit and scores are made;

the process then goes to next step;

e) check for the end of image'"'"'s planned defense;

if it is not the end of planned defense, go to step b), otherwise go next step;

f) calculate players offense compared to the image'"'"'s defense, show scores, and go to step a);

g) inform the Feedback Controller for it to analyze the player'"'"'s actual player trajectory with the previous prediction and make a new entry “

n”

as a new address to the memory and a new levels of prediction and plan.

7. The apparatus of claims 1 wherein the Defense Event Follower Controller receives player'"'"'s defensive event from the Event Controller, it performs the following steps:

a) read the image'"'"'s offense trajectory plan from Mass Memory and initialize “

n”

that is dependent upon degree of expertise initially chosen by the player;

b) from the next incoming video, frame, calculate player'"'"'s actual new motion coordinates, amend it to the previous calculated trajectory;

continually display the planned offense motions video of the image and transition to next step;

c) compare image'"'"'s prediction trajectory with player'"'"'s actual trajectory of motion;

if the measured player'"'"'s offense 3D trajectory and predicted image'"'"'s offense trajectories are off (based upon the value of “

n”

as explained in claim 5, go to step g), otherwise go to next step;

d) checks if the player'"'"'s offense penetrates or hits the image'"'"'s defense;

if it does not go to step e), otherwise go to step f);

players motion characteristics are updated in mass memory input registers and image'"'"'s motion characteristics are updated in the feedback registers of memory bank addressing;

the distance from player to image and the decision on the hit or no hit is provided by the memory bank data;

if it is not a hit is considered a miss (dodge) otherwise it is considered a hit and scores are made;

the process then goes to next step;

e) check for the end of image'"'"'s planned offense;

If it is not the end of planned offense, go to step b), otherwise go to next step;

f) calculate players offense compared to the image'"'"'s defense, show scores, and go to step a);

g) inform the Feedback Controller for it to analyze the player'"'"'s actual player trajectory with the previous prediction and make a new entry “

n”

as a new address to the memory and a new levels of prediction and plan.

8. A method claim 1 for playing a motion related hand-to-hand combat type game, between a player and the image of a competitor player;

each player is provided with its own sets of cameras and the said apparatus that detects and calculates both players'"'"' motions;

the method comprising the steps of;

a) identifying portions of the players with individual colored elements and thus identifying player'"'"'s initial calibration measurements;

b) recording the players as video images and filtering the images into separate signals according to the colored elements for both of the players;

c) calculating rotational changes of colored elements;

d) all the offense and defense bodily parts (colored elements) are initially calibrated with respect to the distance to the camera;

e) transmit the positional calibration measurements of the player 1 to player 2 controller;

f) determining real time positions in 3-space of the portions of the players on each video frame of each of the recordings of the relevant player, and calculating changes in position between each of the frames, and further generating 3D trajectory of relevant player including x, y, z, velocity, and acceleration of each of the portions movements;

g) the real time video or real time trajectory changes of the player 1 is transmitted to the player 2 controller;

h) continually transmitting the player'"'"'s video or changes in the motion trajectory of the player 1 to the player 2;

i) each controller will generate an imaginary x, y, and z positional boundaries of their respective player. The imaginary boundary creates a circular boundary around each player'"'"'s real time positions in all three dimensions that is updated on frame by frame basis;

j) the boundary for each player'"'"'s bodily part is a variable number that its value is based upon each body part and that is adjusted to signify degree of player'"'"'s skill

k) the received video or the trajectory of player 2 is subtracted (or normalized) from the initially received calibration of player 2;

l) the real time trajectory and circular boundaries of the player 1 is compared to the normalized trajectories player 2;

m) when the sensitive (head or belly, or others) body parts of the player 1 trajectory boundaries are penetrated with the normalized offensive body parts of the player 2, a score is made;

n) velocity, and acceleration of the player 2'"'"'s offensive part penetrating the positional circular boundaries of player 1 is the degree of score.

9. A method of claim 1, claim 4, and claim 5 wherein assisted learning is accomplished during the detection and follow up of a player'"'"'s real time motions when compared to the predicted value;

if the real time detected trajectories are deviant from the projected by a pre assigned positive or negative value, the player motions are captured and stored for later analysis and additions to the predictions memory bank by methods comprising the steps of

a) a camera is attached to a pair of players, each camera capturing the opponent'"'"'s real time motions;

the trajectory of the motions later analyzed for initial baseline programming of the offense and defense predictions memory data;

b) conventional video data generation is utilized for the initial and baseline programming of the offense and defense predictions memory data;

c) the deviations from the baseline are captured for later analysis of offense and defense predictions and setting up different thresholds of motions needed for detection and assisted learning;

d) based upon the initial baseline memory prediction bank and the said apparatus, the captured motions of a pro players is fed to this program to learn and store a more refined levels of real time player motions for prediction purposes;

e) captured motions that are deviated from the baseline are further analyzed by the programmers for a realistic additions to the initial memory prediction process;

f) the program can be set to a assisted learning mode by allowing it to select a higher levels of quantization of the captured real time motions variables, and thresholds outlined in above claims 1, for more refined additions and entries to the memory bank;

g) the degree for player'"'"'s speed, is decided by adjustment of “

b”

number of frames (claim 4) for player speed selection;

h) degree of expertise is chosen by selecting lower or higher quantization levels of prediction memory bank.

10. The method of claim 1 further comprising the step of increasing the number of cameras and display monitors to assist the player'"'"'s view of the image at different angles while turning and facing from one camera to another, wherein:

a) the Controller to provide an image 3d field of play for the player to use as a visual guidelines for his/hers movements in a field of play, while the image is moved around from one side of the field of play to the other;

the Controller to detect player positions from different cameras and decide which camera provides the best detection angle and display the image in a relevant field of play to be viewed by the player.