Method and device for associating frames in a video of an activity of a person with an event

• US 9,978,425 B2
• Filed: 04/25/2016
• Issued: 05/22/2018
• Est. Priority Date: 04/23/2015
• Status: Active Grant

• Current Assignee: Adidas AG
• Original Assignee: Adidas AG
• Inventors: Kirk, Robert Frank, Zwick, Constantin, Schuldhaus, Dominik, Eskofier, Bjoern Michael, Krger, Harald, Stve, Maike Petra
• Primary Examiner(s): Zhao, Daquan
• Application Number:
  US15/137,092
• Publication Number:
  US 20160314818A1
• Time in Patent Office: 757 Days
• Field of Search: 386228, 386239, 386248, 386200, 386201
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 1/1694 the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer
• CPC Class Codes: G06F 18/2178 based on feedback of a supervisor
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06V 20/42 of sport video content
• CPC Class Codes: G06V 20/44 Event detection
• CPC Class Codes: G06V 20/47 Detecting features for summarising video content
• CPC Class Codes: G11B 27/022 Electronic editing of analogue information signals, e.g. audio or video signals
• CPC Class Codes: G11B 27/031 Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: G11B 27/10 Indexing; Addressing; Timing or synchronising; Measuring tape travel
• CPC Class Codes: G11B 27/11 by using information not detectable on the record carrier
• CPC Class Codes: G11B 27/19 by using information detectable on the record carrier
• CPC Class Codes: G11B 27/3036 Time code signal
• CPC Class Codes: H04N 5/77 between a recording apparatus and a television camera

Abstract

Described are methods and systems for associating frames in a video of an activity of a person with an event. The methods include recording a video of an activity of a person; storing a time-series of a plurality of sensor data (82) obtained from a sensor assembly (12) of at least one sensor (31, 32, 33, 34, 35) coupled to the person while the person is performing the activity; synchronizing the video with the sensor data (82); detecting an event in the time-series; and associating the event with at least one corresponding frame in the video showing the event.

Claims

1. A method comprising:

recording a video of an activity of a person;

storing a time-series comprising a plurality of sensor data obtained from a sensor assembly, wherein the sensor assembly comprises at least one sensor coupled to the person while the person is performing the activity;

synchronizing the video with the plurality of sensor data;

detecting an event in the time-series by at least;

preprocessing the time-series;

segmenting the time-series into a plurality of windows;

detecting each of the plurality of windows comprising at least one outlier;

extracting a plurality of features from the time-series in each of the plurality of windows; and

estimating an event class for each of the plurality of windows based on the plurality of features extracted from the time-series in each of the plurality of windows; and

associating the event with at least one corresponding frame in the video showing the event.

2. The method according to claim 1, further comprising:

generating a second video showing the event of the activity of the person, wherein the second video comprises the at least one corresponding frame.

3. The method according to claim 1, wherein synchronizing the video with the plurality of sensor data comprises:

detecting a predetermined synchronization event in the time-series; and

detecting the predetermined synchronization event in at least one frame in the video.

4. The method according to claim 1, wherein the activity comprises a sports activity.

5. The method according to claim 1, wherein the activity is soccer.

6. The method according to claim 1, wherein the event comprises at least one of:

a kick, a short pass, a long pass, a shot, or an act of controlling a ball.

7. The method according to claim 1, wherein the at least one sensor comprises at least one of:

an accelerometer, a gyroscope, or a magnetic field sensor.

8. The method according to claim 1, wherein the sensor assembly comprises at least two sensors, and wherein the plurality of sensor data from each of the at least two sensors is combined in the time-series.

9. The method according to claim 1, wherein recording the video of the activity of the person comprises recording the video with at least two cameras.

10. The method according to claim 1, further comprising:

saving the video in a database.

11. The method according to claim 1, wherein preprocessing the time-series comprises low pass filtering and downsampling.

12. The method according to claim 1, wherein each of the plurality of windows comprises a fixed sized window.

13. The method according to claim 12, further comprising:

centering each of the plurality of windows at a center of the fixed size window, wherein the center corresponds to a maximum peak in the time-series.

14. The method according to claim 13, further comprising:

removing each of the plurality of windows in which the maximum peak at the center is below a threshold.

15. The method according to claim 1, further comprising:

adapting a window size of the plurality of windows to at least one of;

a shape or an extent of the event.

16. The method according to claim 1, wherein detecting each of the plurality of windows comprising at least one outlier comprises using at least one of:

a Rule based system or a one-class Support Vector Machines.

17. The method according to claim 1, wherein the plurality of features are extracted from the plurality of windows which do not comprise the at least one outlier.

18. The method according to claim 1, wherein the plurality of features are based on at least one of temporal statistics, spatio-temporal statistics, spectral statistics, or ensemble statistics by applying at least one of a wavelet analysis, a principal component analysis, or a Fast Fourier Transform.

19. The method according to claim 1, wherein the plurality of features comprise at least one of a simple mean, a normalized signal energy, a movement intensity, a signal magnitude area, a correlation between axes, a maximum value in a window, a minimum value in a window, a maximum detail coefficient of a wavelet transform, a correlation with a template, a projection onto a principal component of a template, or a distance to an eigenspace of a template, a spectral centroid, a bandwidth, or a dominant frequency.

20. The method according to claim 1, further comprising:

reducing a number of the plurality of features using at least one feature selection procedure.

21. The method according to claim 1, wherein estimating the event class is based on at least one classifier, wherein the at least one classifier comprises at least one of:

a Bayesian classifier, a Naï

ve Bayes classifier, a maximum margin classifier, an AdaBoost classifier, a Random Forest classifier, a Nearest Neighbor classifier, a Neural Network classifier, a Rule based classifier, or a Tree based classifier.

22. The method according to claim 21, further comprising:

fusing at least two decisions of the at least one classifier to estimate a final prediction of the event class.

23. The method according to claim 1, wherein detecting the event comprises matching the event to a template based on correlation, Matched Filtering, Dynamic Time Warping, or Longest Common Subsequence (LCS) and its sliding window variant, warping LCS.

24. The method according to claim 1, wherein estimating the event class comprises:

discriminating between at least two merged event groups containing at least two events that are similar in shape; and

discriminating between the at least two events belonging to one specific group.

25. The method according to claim 1, wherein estimating the event class is based on at least one classifier that has been trained based on a supervised training.

26. The method according to claim 1, further comprising:

providing a plurality of summary statistics, wherein at least a portion of the plurality of summary statistics is based on the event.

27. The method according to claim 1, further comprising:

estimating a speed of a ball based on the time-series if the event is associated with a kick.

28. A system comprising:

at least one camera configured to record a video of an activity of a person;

at least one sensor assembly comprising at least one sensor, the sensor assembly configured to be coupled to the person while the person is performing the activity;

a non-transitory memory configured to store a time-series comprising a plurality of sensor data obtained from the at least one sensor; and

a processor configured to;

synchronize the video with the plurality of sensor data;

detect an event in the time-series by at least;

preprocessing the time-series;

segmenting the time-series into a plurality of windows;

detecting each of the plurality of windows comprising at least one outlier;

extracting a plurality of features from the time-series in each of the plurality of windows; and

estimating an event class for each of the plurality of windows based on the plurality of features extracted from the time-series in each of the plurality of windows; and

associate the event with at least one corresponding frame in the video showing the event.

29. The system according to claim 28, wherein the sensor assembly is configured to be coupled to a body part of the person.

30. The system according to claim 28, wherein the sensor assembly is configured to be attached to at least one of:

a strap or a bracelet.

31. The system according to claim 28, further comprising:

a wearable device comprising the sensor assembly.

32. The system according to claim 28, wherein the sensor assembly comprises the non-transitory memory.

33. The system according to claim 28, wherein the at least one camera comprises at least two cameras.

34. A non-transitory computer-readable medium comprising one or more software applications configured to be executed by a processor, the one or more software applications configured to:

record a video of an activity of a person;

store a time-series comprising a plurality of sensor data obtained from a sensor assembly, wherein the sensor assembly comprises at least one sensor coupled to the person while the person is performing the activity;

synchronize the video with the plurality of sensor data;

detect an event in the time-series by at least;

preprocessing the time-series;

segmenting the time-series into a plurality of windows;

detecting each of the plurality of windows comprising at least one outlier;

extracting a plurality of features from the time-series in each of the plurality of windows; and

estimating an event class for each of the plurality of windows based on the plurality of features extracted from the time-series in each of the plurality of window; and

associate the event with at least one corresponding frame in the video showing the event.

35. The non-transitory computer-readable medium according to claim 34, wherein the one or more software applications is further configured to:

generate a second video showing the event of the activity of the person, wherein the second video comprises the at least one corresponding frame.

36. The non-transitory computer-readable medium according to claim 34, wherein synchronize the video with the plurality of sensor data comprises:

detecting a predetermined synchronization event in the time-series; and

detecting the predetermined synchronization event in at least one frame in the video.