Trajectory detection and feedback system for tennis

• US 9,370,704 B2
• Filed: 01/29/2014
• Issued: 06/21/2016
• Est. Priority Date: 08/21/2006
• Status: Active Grant

• Current Assignee: Pillar Vision Incorporated
• Original Assignee: Pillar Vision Incorporated
• Inventors: Marty, Alan W
• Primary Examiner(s): D'Agostino, Paul A
• Application Number:
  US14/167,876
• Publication Number:
  US 20140180451A1
• Time in Patent Office: 874 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: A63B 15/00 Clubs for gymnastics or the like , e.g. for swinging exercises
• CPC Class Codes: A63B 2024/0009 Computerised real time comparison with previous movements or motion sequences of the user
• CPC Class Codes: A63B 2024/0031 at the starting point
• CPC Class Codes: A63B 2024/0034 during flight
• CPC Class Codes: A63B 2024/0037 on a target surface or at impact on the ground
• CPC Class Codes: A63B 2071/0625 Emitting sound, noise or music
• CPC Class Codes: A63B 2071/0655 Tactile feedback
• CPC Class Codes: A63B 2071/0663 worn on the wrist, e.g. wrist bands
• CPC Class Codes: A63B 2102/02 Tennis
• CPC Class Codes: A63B 2102/06 Squash
• CPC Class Codes: A63B 2102/16 Table tennis
• CPC Class Codes: A63B 2209/10 with adhesive type surfaces, i.e. hook and loop-type fastener
• CPC Class Codes: A63B 2220/00 Measuring of physical parameters relating to sporting activity
• CPC Class Codes: A63B 2220/10 Positions
• CPC Class Codes: A63B 2220/12 Absolute positions, e.g. by using GPS
• CPC Class Codes: A63B 2220/16 Angular positions
• CPC Class Codes: A63B 2220/30 Speed
• CPC Class Codes: A63B 2220/40 Acceleration
• CPC Class Codes: A63B 2220/72 Temperature
• CPC Class Codes: A63B 2220/75 Humidity
• CPC Class Codes: A63B 2220/806 : Video cameras
• CPC Class Codes: A63B 2220/83 : characterised by the position of the sensor
• CPC Class Codes: A63B 2220/833 : Sensors arranged on the exercise apparatus or sports implement
• CPC Class Codes: A63B 2225/20 : with means for remote communication, e.g. internet or the like
• CPC Class Codes: A63B 2225/50 : Wireless data transmission, e.g. by radio transmitters or telemetry
• CPC Class Codes: A63B 24/0003 : Analysing the course of a movement or motion sequences during an exercise or trainings sequence, e.g. swing for golf or tennis
• CPC Class Codes: A63B 24/0021 : Tracking a path or terminating locations
• CPC Class Codes: A63B 24/0062 : Monitoring athletic performances, e.g. for determining the work of a user on an exercise apparatus, the completed jogging or cycling distance
• CPC Class Codes: A63B 24/0084 : Exercising apparatus with means for competitions, e.g. virtual races
• CPC Class Codes: A63B 60/16 : Caps; Ferrules
• CPC Class Codes: A63B 69/38 : for tennis A63B61/006, A63B69/0073 and A63B69/0097 take precedence
• CPC Class Codes: G06F 3/016 : Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06V 20/20 : in augmented reality scenes
• CPC Class Codes: G06V 40/23 : Recognition of whole body movements, e.g. for sport training
• CPC Class Codes: G09B 19/0038 : Sports

Abstract

A system for capturing and analyzing a trajectory of a tennis ball or other object associated with a play of a game of tennis and providing feedback is described. The system may be designed to capture and analyze a trajectory of a tennis ball during various activities related to the play of a game of tennis. The system may be configured to provide immediate feedback that may be utilized by a player to improve their performance as well as provide entertainment value above and beyond what is normally associated with the play of a game of tennis. Further, the system may be operable for use in an area where tennis is normally played during normal playing activities, such as player playing a game or practicing on an outdoor tennis court. The system may be operable to account for factors associated with its ambient environment, such as wind, temperature and humidity.

Claims

1. A non-transitory computer readable medium configured to store a computer program used by a computer, the computer readable medium comprising:

computer code configured to control generation of images of a virtual tennis ball as a function of time which are output to a display worn on a user'"'"'s head which delivers images into at least one of a user'"'"'s eyes to mimic a position of a real tennis ball as the function of time approaching a user;

computer code configured to determine (1) speed, position, and orientation of a hand-held device as the function of time based upon sensor data from one or more sensors, (2) when the virtual tennis ball is hit by the user based upon the position and the orientation of the hand-held device as the function of time and a virtual position of the tennis ball as the function of time, and (3) an orientation of the hand-held device relative to the virtual tennis ball when the virtual tennis ball is determined to be hit by the user;

computer code configured to determine a spin of the virtual tennis ball based upon the speed of the hand-held device as the function of time and the orientation of the hand-held device relative to the virtual tennis ball when the virtual tennis ball is determined to be hit by the user;

computer code configured to calculate a trajectory for the virtual tennis ball based upon the speed of the hand-held device, the spin of the virtual tennis ball, and the orientation of the hand-held device relative to the virtual tennis ball when the virtual tennis ball is determined to be hit by the user;

when the virtual tennis ball is determined to be hit by the user, computer code configured to trigger a haptic sensation generator in the hand-held device wherein the haptic sensation generator transmits one or more haptic sensations to the user which mimic a feel of the real tennis ball being hit; and

computer code configured to control, based upon the calculated trajectory, generation of second images of the virtual tennis ball which are output to the display worn on the user'"'"'s head and which show the virtual position of the virtual tennis ball as the function of time after the virtual tennis ball is determined to be hit moving away from the user according to the calculated trajectory.

2. The computer readable medium of claim 1, wherein the user, when viewing the display, is also able to view real objects around the user to provide a mixed reality experience wherein the images of the virtual tennis ball are generated to blend with the real objects.

3. The computer readable medium of claim 2, wherein the objects are a real net and real court markings on a real tennis court.

4. The computer readable medium of claim 1, wherein the one or more haptic sensations include vibrations.

5. The computer readable medium of claim 1, wherein the one or more haptic sensations include force feedback.

6. The computer readable medium of claim 5, further comprising computer code configured to control angular velocities of one or more rotating mass wherein the angular velocities are selected to generate a selected amount of force feedback based upon a velocity of the virtual tennis ball.

7. The computer readable medium of claim 1, further comprising computer code, based upon the position and the orientation of the hand-held device as the function of time, configured to determine a location where the virtual tennis ball impacts a face of a virtual tennis racquet.

8. The computer readable medium of claim 7, wherein the one or more haptic sensations are generated based upon the location where the virtual tennis ball is determined to impact the face of the virtual tennis racquet.

9. The computer readable medium of claim 7, further comprising computer code configured to determine whether a 3-D trajectory associated with the virtual tennis ball intersects the virtual face.

10. The computer readable medium of claim 1, further comprising computer code configured to determine a position of a virtual face of a virtual tennis racquet as the function of time based upon the position and the orientation of the hand-held device as the function of time.

11. The computer readable medium of claim 1, wherein the hand-held device is a real tennis racquet further comprising computer code configured to, based upon the position and the orientation of the real tennis racquet as it is swung by the user, determine a location where the virtual tennis ball would impact a face of the real tennis racquet if the virtual tennis ball were the real tennis ball.

12. The computer readable medium of claim 1, further comprising computer code configured to determine a position of the user as the function of time prior to and while the user is swinging the hand-held device at the virtual tennis ball.

13. The computer readable medium of claim 12, further comprising computer code configured to determine whether the virtual tennis ball is hit based upon the position of the user as the function of time prior to and while the use is swinging the hand-held device.

14. The computer readable medium of claim 1, further comprising computer code configured to generate the images including one or more of a virtual net, virtual court markings, a virtual opponent or combinations thereof as the function of time.

15. The computer readable medium of claim 1, further comprising computer code configured to control generation of third images of the virtual tennis ball to the display which show the virtual position of the virtual tennis ball as the function of time after the virtual tennis ball is determined to be hit by the user such that the virtual tennis ball is moving towards the user from the perspective of the real tennis ball as viewed through the user'"'"'s eyes on the tennis court to allow the user to attempt to return the virtual ball which was previously hit by the user.

16. The computer readable medium of claim 1, wherein after the second images are generated, the display is worn by a second user and the hand-held device is held by the second user further comprising computer code configured to control generation of third images of the virtual tennis ball to the display which show the virtual position of the virtual tennis ball as the function of time after the virtual tennis ball is determined to be hit by the user such that the virtual tennis ball is moving towards the second user from the perspective of the real tennis ball as viewed through a second user'"'"'s eyes on the tennis court to allow the second user to attempt to return the virtual ball which was previously hit by the user.

17. The computer readable medium of claim 1, further comprising computer code configured to determine a position of the user as the function of time relative to the position of the virtual tennis ball as the function of time.

18. The computer readable medium of claim 17, wherein the images of the virtual tennis ball approaching the user are based upon the position of the user as the function of time relative to the position of the virtual tennis ball as the function of time.

19. The computer readable medium of claim 1, wherein the hand-held device is configured to detect whether it is being held with one or two hands further comprising computer code configured to generate the one or more haptic sensations based upon whether the one or the two hands are holding the hand-held device.

30. The computer readable medium of claim 1, wherein the one or more haptic sensations are based the speed of the hand-held device as the function of time and the orientation of the hand-held device relative to the virtual tennis ball when the virtual tennis ball is determined to be hit by the user.

20. A non-transitory computer readable medium configured to store a computer program used by a computer, the computer readable medium comprising:

computer code configured to receive image data from one or more sensors worn on a user'"'"'s head wherein the image data includes images of a real tennis ball being hit towards the user on a real tennis court;

based upon the image data, computer code configured to determine a position of the real tennis ball as a first function of time;

computer code configured to control generation of images of a virtual tennis ball as a second function of time which are output to a display worn on the user'"'"'s head which projects images into at least one of a user'"'"'s eyes to mimic a position of the real tennis ball as the function of time approaching the user wherein the images of the virtual tennis ball approaching the user are generated based upon the image data received by the one or more sensors;

computer code configured to determine (1) speed, position, and orientation of a hand-held device as the second function of time based upon sensor data from one or more sensors, (2) when the virtual tennis ball is hit by the user based upon the position and the orientation of the hand-held device as the second function of time and a virtual position of the tennis ball as the second function of time, and (3) an orientation of the hand-held device relative to the virtual tennis ball when the virtual tennis ball is determined to be hit by the user;

computer code configured to determine a spin of the virtual tennis ball based the speed of the hand-held device as the second function of time and the orientation of the hand-held device relative to the virtual tennis ball when the virtual tennis ball is determined to be hit by the user;

computer code configured to calculate a trajectory for the virtual tennis ball based upon the speed of the hand-held device as the second function of time, the spin of the virtual tennis ball, and the orientation of the hand-held device relative to the virtual tennis ball when the virtual tennis ball is determined to be hit by the user;

when the virtual tennis ball is determined to be hit by the user, computer code configured to trigger a haptic sensation generator in the hand-held device wherein the haptic sensation generator transmits one or more haptic sensations to the user which mimic a feel of the real tennis ball being hit; and

computer code configured to control, based upon the trajectory, generation of second images of the virtual tennis ball which are output to the display worn on the user'"'"'s head and which show the virtual position of the virtual tennis ball as the second function of time after the virtual tennis ball is determined to be hit moving away from the user according to the trajectory.

21. The computer readable medium of claim 20, wherein the display and the one or more sensors worn on the user'"'"'s head are mounted to glasses worn by the user.

22. The computer readable medium of claim 20, wherein received image data is from a single camera further comprising computer code configured to determine a distance from the single camera to the real tennis ball based upon the image data as the first function of time.

23. The computer readable medium of claim 20, wherein the image data is from two cameras worn by the user further comprising computer code configured to determine a distance of real tennis ball from the user as the first function of time based upon the image data.

24. The computer readable medium of claim 20, further comprising computer code configured to determine a position of the user as the first function of time when the real tennis ball is hit towards the user.

25. The computer readable medium of claim 24, wherein the position of the user as the first function of time is determined using one or more accelerometers worn by the user, the image data or the one or more accelerometers and the image data.

26. The computer readable medium of claim 20, further comprising computer code for receiving sound data from a microphone when the user hits the real tennis ball with a real tennis racquet;

and

computer code for outputting a sound when the virtual tennis ball is hit wherein the sound is based upon the sound data.

27. The computer readable medium of claim 20, wherein the one or more sensors from which the sensor data is received are coupled to a real tennis racquet.

28. The computer readable medium of claim 27, wherein the sensor data is used to determine whether the real tennis racquet is being held with one or two hands.

29. The computer readable medium of claim 27, wherein the sensor data is used to determine forces and vibrations which the user feels while holding the real tennis racquet.

31. A system, comprising:

one or more sensors configured to sense movement of a hand-held device and to provide sensor data indicative of the sensed movement;

a display worn on a head of a user;

a haptic sensor generator coupled to the hand-held device; and

a logic device configured to control the display to deliver images of a virtual tennis ball into at least one eye of the user, the logic device configured to determine a speed and a position of the hand-held device based upon the sensor data as the hand-held device is swung by the user, the logic device configured to determine when the virtual tennis ball is hit by the user based upon the determined position and to determine an orientation of the hand-held device based upon the sensor data when the virtual tennis ball is determined to be hit by the user, wherein the logic device is configured to determine a spin of the virtual tennis ball based upon the speed of the hand-held device and the orientation of the hand-held device when the virtual tennis ball is determined to be hit by the user, wherein the logic device is configured to a calculate a trajectory for the virtual tennis ball based upon the speed of the hand-held device, the spin of the virtual tennis ball, and the orientation of the hand-held device relative to the virtual tennis ball when the virtual tennis ball is determined to be hit by the user, wherein the logic device is configured to trigger the haptic sensation generator to transmit one or more haptic sensations to the user which mimic a feel of a real tennis ball being hit when the virtual tennis ball is determined to be hit by the user, wherein the logic device is further configured to control the display based upon the calculated trajectory to deliver images of the virtual tennis ball into at least one eye of the user such that the virtual tennis ball appears to be moving away from the user according to the calculated trajectory.

32. The system of claim 31, wherein the one or more haptic sensations are based upon the speed of the hand-held device.

33. The system of claim 32, wherein the one or more haptic sensations are based upon the orientation of the hand-held device relative to the virtual tennis ball when the virtual tennis ball is determined to be hit by the user.

34. The system of claim 33, wherein the one or more haptic sensations are based upon a position of the hand-held device relative to the virtual tennis ball when the virtual tennis ball is determined to be hit by the user.

35. A method, comprising:

sensing movement of a hand-held device by one or more sensors thereby providing sensor data indicative of the sensed movement;

controlling a display worn on a head of user to deliver images of a virtual tennis ball into at least one eye of the user;

determining with a logic device a speed and a position of the hand-held device based upon the sensor data as the hand-held device is swung by the user;

determining with the logic device when the virtual tennis ball is hit by the user based upon the determined position of the hand-held device;

determining with the logic device an orientation of the hand-held device based upon the sensor data when the virtual tennis ball is determined to be hit by the user;

determining with the logic device a spin of the virtual tennis ball based upon the speed of the hand-held device and the orientation of the hand-held device when the virtual tennis ball is determined to be hit by the user;

calculating with the logic device a trajectory for the virtual tennis ball based upon the speed of the hand-held device, the spin of the virtual tennis ball, and the orientation of the hand-held device when the virtual tennis ball is determined to be hit by the user;

triggering with the logic device a haptic sensation generator to transmit one or more haptic sensations to the user which mimic a feel of a real tennis ball being hit when the virtual tennis ball is determined to be hit by the user; and

controlling the display based upon the calculated trajectory to deliver images of the virtual tennis ball into at least one eye of the user such that the virtual tennis ball appears to be moving away from the user according to the calculated trajectory.

36. The system of claim 35, wherein the one or more haptic sensations are based upon the speed of the hand-held device.

37. The system of claim 36, wherein the one or more haptic sensations are based upon the orientation of the hand-held device relative to the virtual tennis ball when the virtual tennis ball is determined to be hit by the user.

38. The system of claim 37, wherein the one or more haptic sensations are based upon a position of the hand-held device relative to the virtual tennis ball when the virtual tennis ball is determined to be hit by the user.