Motion attributes recognition system and methods

• US 10,213,645 B1
• Filed: 09/02/2015
• Issued: 02/26/2019
• Est. Priority Date: 10/03/2011
• Status: Active Grant

• Current Assignee: Swingbyte Incorporated
• Original Assignee: Swingbyte Incorporated
• Inventors: Wu, Ying, Jiang, Nan, Pedenko, Alex
• Primary Examiner(s): Cuff, Michael
• Application Number:
  US14/843,526
• Time in Patent Office: 1,273 Days
• Field of Search: 700 91, 473131, 473223
• CPC Class Codes: A63B 2024/0012 Comparing movements or motion sequences with a registered reference
• CPC Class Codes: A63B 2071/063 Spoken or verbal instructions
• CPC Class Codes: A63B 2220/13 Relative positions
• CPC Class Codes: A63B 2220/16 Angular positions
• CPC Class Codes: A63B 2220/34 Angular speed
• CPC Class Codes: A63B 2220/40 Acceleration
• CPC Class Codes: A63B 2220/833 Sensors arranged on the exercise apparatus or sports implement
• CPC Class Codes: A63B 2225/20 with means for remote communication, e.g. internet or the like
• CPC Class Codes: A63B 2225/50 Wireless data transmission, e.g. by radio transmitters or telemetry
• CPC Class Codes: A63B 24/0003 Analysing the course of a movement or motion sequences during an exercise or trainings sequence, e.g. swing for golf or tennis
• CPC Class Codes: A63B 24/0006 Computerised comparison for qualitative assessment of motion sequences or the course of a movement
• CPC Class Codes: A63B 60/46 Measurement devices associated with golf clubs, bats, rackets or the like for measuring physical parameters relating to sporting activity, e.g. baseball bats with impact indicators or bracelets for measuring the golf swing
• CPC Class Codes: A63B 69/36 for golf
• CPC Class Codes: A63B 69/3632 Clubs or attachments on clubs, e.g. for measuring, aligning A63B69/3614 takes precedence; clubs for swinging exercises in general A63B15/00
• CPC Class Codes: A63B 71/0619 Displays, user interfaces and indicating devices, specially adapted for sport equipment, e.g. display mounted on treadmills
• CPC Class Codes: A63B 71/0622 Visual, audio or audio-visual systems for entertaining, instructing or motivating the user
• CPC Class Codes: G06F 18/217 Validation; Performance evaluation; Active pattern learning techniques
• CPC Class Codes: G06F 18/2411 based on the proximity to a decision surface, e.g. support vector machines
• CPC Class Codes: G06F 18/24137 Distances to cluster centroïds
• CPC Class Codes: G06F 18/2415 based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
• CPC Class Codes: G06F 2218/16 : by matching signal segments
• CPC Class Codes: G06T 2207/10016 : Video; Image sequence
• CPC Class Codes: G06T 2207/30221 : Sports video; Sports image
• CPC Class Codes: G06T 7/215 : Motion-based segmentation
• CPC Class Codes: G06V 10/764 : using classification, e.g. of video objects
• CPC Class Codes: G06V 40/23 : Recognition of whole body movements, e.g. for sport training
• CPC Class Codes: G09B 19/0038 : Sports
• CPC Class Codes: G16H 20/30 : relating to physical therapies or activities, e.g. physiotherapy, acupressure or exercising
• CPC Class Codes: G16H 50/70 : for mining of medical data, e.g. analysing previous cases of other patients
• CPC Class Codes: H04M 1/72403 : with means for local support of applications that increase the functionality
• CPC Class Codes: H04M 1/72412 : using two-way short-range wireless interfaces

Abstract

Systems and methods for motion attribute recognition are defined using data stream pre-processing to orient, align and segment motion data before using non-parametric classification recognition to search a motion data exemplar database or using parametric classification recognition to find attributes by comparing pre-processed motion data with support vector machines. Results from the non-parametric classification recognition and the parametric classification recognition may be fused to produce a single result. Active learning and metric learning are used to improve searches of the database and comparisons to the support vector machines.

Claims

1. A motion attribute recognition system comprising:

a pre-processing component configured to receive a data stream including motion data and to output action exemplars comprising;

an orientation matrix conversion process configured to transform raw motion data into a quaternion suitable to measure the distance between two rotations by comparing the quaternion of a one rotation with the quaternion of the other rotation;

a spatial alignment process configured to align the starting plane of the motion data; and

a temporal segmentation process configured to locate the actual segment of the motion data within the data stream;

a non-parametric classification recognition component in communication with the pre-processing component comprising;

a database of stored action exemplars searchable to and configured to output motion attributes from a query comprising the motion data;

a database search engine configured to search the database of action exemplars for comparison with a received action exemplar from the pre-processing component; and

a non-parametric classification recognition process configured to receive action exemplars from the pre-processing component and the database search engine and to output attributes of the action exemplar received from the pre-processing component; and

a parametric classification recognition component in communication with the pre-processing component comprising;

one or more support vector machines wherein each support vector machine is configured to find one of the presence and absence of an attribute in an action exemplar received from the pre-processing component; and

a parametric classification recognition process configured to receive an action exemplar from the pre-processing component and to query the one or more support vector machines for the presence of an attribute in the action exemplar and to output one of the presence and absence of an attribute in the action exemplar;

a display; and

an user interface in communication with the database and the display and configured to receive the motion data, to query the database with the motion data, to receive attributes from the database in response to the query, and to output attribute information on the display in order to present a virtual coach.

2. The motion attribute recognition system of claim 1, further comprising a classification fusion process to integrate the non-parametric classification recognition process and the parametric classification recognition process by fusing the results from the classification recognition processes based on linear weighting.

3. The motion attribute recognition system of claim 1, wherein the non-parametric classification recognition process further comprises the database search engine configured to compare the received action exemplar from the pre-processing component with the action exemplars in the database of stored action exemplars by measuring the distance between the two action exemplars with dynamic time warping based matching.

4. The motion attribute recognition system of claim 3, further comprising a metric learning component configured to optimize the distance between the two action exemplars.

5. The motion attribute recognition system of claim 1, further comprising an active learning component configured to receive human annotations marking attributes tagged to the stored action exemplars and in communication with the database search engine and further configured to spot inconsistencies in the human annotation tags marking attributes, to identify typical patterns in stored action exemplars with data clustering and to identify stored action exemplars at data cluster boundaries to suggest for further annotation tagging.

6. The motion attribute recognition system of claim 1, further comprising the database search engine configured to use lower bound estimation to estimate the lower bound distance for all exemplars with respect to the received action exemplar and to safely skip searching the database for action exemplars with lower bounds larger than the current estimated the lower bound distance met in the searching process.

7. The motion attribute recognition system of claim 6, further comprising the database search engine configured to use branch bound and bound based sub-linear searching.

8. The motion attribute recognition system of claim 1, further comprising an active learning component configured to receive human annotations marking attributes in parametric training case data and in communication with the parametric classification recognition process and the one or more support vector machines respectively, to receive parametric training case data, to add human annotations, to save the annotated training case data, and to spot inconsistencies in the human annotations.

9. The motion attribute recognition system of claim 1, further comprising a metric learning component configured to find and to assign different weight factors for different attributes present in the support vector machines.

10. The motion attribute recognition system of claim 1, further comprising a convolutional feature computing process in communication with the one or more support vector machines and the parametric classification recognition process and configured to extract attributes from action exemplars using convolutional feature computing.

11. A virtual golf coaching system comprising:

a first electronic device attachable to a golf club that transmits motion data from a swinging golf club describing the golf club movement; and

a second electronic device comprising;

a display;

a motion attribute recognition system comprising a database of stored golf swing exemplars searchable to and configured to output golf swing attributes from a query comprising the motion data; and

an application in communication with the first electronic device, the database and the display and configured to receive the motion data transmitted from the first electronic device, to query the database with the motion data, to receive attributes from the database in response to the query, and to output attribute information on the display;

a pre-processing component configured to receive a data stream including the motion data and to output golf swing exemplars comprising;

an orientation matrix conversion process configured to transform raw motion data into a quaternion suitable to measure the distance between two rotations by comparing the quaternion of a one rotation with the quaternion of the other rotation;

a spatial alignment process configured to align the starting plane of the motion data; and

a temporal segmentation process configured to locate the actual segment of the motion data within the data stream;

a non-parametric classification recognition component in communication with the pre-processing component comprising;

the database of stored golf swing exemplars;

a database search engine configured to search the database of stored golf swing exemplars for comparison with a received golf swing exemplar from the pre-processing component; and

a non-parametric classification recognition process configured to receive golf swing exemplars from the pre-processing component and the database search engine and to output attributes of the golf swing exemplar received from the pre-processing component; and

a parametric classification recognition component in communication with the pre-processing component comprising;

one or more support vector machines wherein each support vector machine is configured to find one of the presence and absence of an attribute in a golf swing exemplar received from the pre-processing component; and

a parametric classification recognition process configured to receive a golf swing exemplar from the pre-processing component and to query the one or more support vector machines for the presence of an attribute in the golf swing exemplar and to output one of the presence and absence of an attribute in the golf swing exemplar.

12. The virtual golf coaching system of claim 11, further comprising a classification fusion process to integrate the non-parametric classification recognition process and the parametric classification recognition process by fusing the results from the classification recognition processes based on linear weighting.

13. The virtual golf coaching system of claim 11, further comprising:

the non-parametric classification recognition process further comprising the database search engine configured to compare the received golf swing exemplar from the pre-processing component with the golf swing exemplars in the database of stored golf swing exemplars by measuring the distance between the two golf swing exemplars with dynamic time warping based matching;

a metric learning component configured to optimize the distance between the two golf swing exemplars; and

an active learning component configured to receive human annotations marking attributes tagged to the stored golf swing exemplars and in communication with the database search engine and further configured to spot inconsistencies in the human annotation tags marking attributes, to identify typical patterns in stored golf swing exemplars with data clustering and to identify stored golf swing exemplars at data cluster boundaries to suggest further annotation tagging.

14. The virtual golf coaching system of claim 11, further comprising the database search engine configured to use lower bound estimation to estimate the lower bound distance for all exemplars with respect to the received golf swing exemplar and to safely skip searching the database for golf swing exemplars with lower bounds larger than the current estimated the lower bound distance met in the searching process.

15. The virtual golf coaching system of claim 14, further comprising the database search engine configured to use branch bound and bound based sub-linear searching.

16. The virtual golf coaching system of claim 11, further comprising an active learning component configured to receive human annotations marking attributes in parametric training case data and in communication with the parametric classification recognition process and the one or more support vector machines respectively, to receive parametric training case data, to add human annotations, to save the annotated training case data, and to spot inconsistencies in the human annotations.

17. The virtual golf coaching system of claim 11, further comprising a metric learning component configured to find and to assign different weight factors for different attributes present in the support vector machines.

18. The virtual golf coaching system of claim 11, further comprising a metric learning component configured to find and to assign different weight factors for different attributes present in the support vector machines.

19. A method for virtual golf coaching comprising:

attaching a first electronic device to a golf club that transmits motion data from a swinging golf club describing the golf club movement;

swinging the golf club;

providing a second electronic device comprising;

providing a display;

providing a database of stored golf swing exemplars searchable to and configured to output golf swing attributes from a query comprising the motion data; and

providing a motion attribute recognition system comprising;

pre-processing a received a data stream including the motion data and outputting golf swing exemplars comprising;

transforming raw motion data into a quaternion suitable to measure the distance between two rotations by comparing the quaternion of a one rotation with the quaternion of the other rotation using an orientation matrix conversion process;

aligning the starting plane of the motion data using a spatial alignment process; and

locating the actual segment of the motion data within the data stream using a temporal segmentation process;

providing a non-parametric classification recognition component in communication with the pre-processing component comprising;

storing golf swing exemplars in the database;

searching the database of stored golf swing exemplars for comparison with a received golf swing exemplar from the pre-processing component using a database search engine; and

receiving golf swing exemplars from the pre-processing component and the database search engine and outputting attributes of the golf swing exemplar received from the pre-processing component using a non-parametric classification recognition process; and

providing a parametric classification recognition component in communication with the pre-processing component comprising;

finding one of the presence and absence of an attribute in a golf swing exemplar received from the pre-processing component using one or more support vector machines; and

receiving a golf swing exemplar from the pre-processing component and querying the one or more support vector machines for the presence of an attribute in the golf swing exemplar and outputting one of the presence and absence of an attribute in the golf swing exemplar using a parametric classification recognition process; and

configuring an application in communication with the first electronic device, the database and the display to receive the motion data transmitted from the first electronic device, to query the database with the motion data, to receive attributes from the database in response to the query, and to output attribute information on the display.