EVENT ANALYSIS SYSTEM

• US 20150317801A1
• Filed: 07/16/2015
• Published: 11/05/2015
• Est. Priority Date: 08/26/2010
• Status: Active Grant

• Current Assignee: Newlight Capital, LLC
• Original Assignee: Blast Motion Incorporated
• Inventors: BENTLEY, Michael, BOSE, Bhaskar, Kaps, Ryan, GUPTA, Piyush, HAAS, Juergen, ESTREM, Brian
• Granted Patent: US 9,401,178 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: A63F 13/212 using sensors worn by the player, e.g. for measuring heart beat or leg activity
• CPC Class Codes: A63F 13/355 Performing operations on behalf of clients with restricted processing capabilities, e.g. servers transform changing game scene into an encoded video stream for transmitting to a mobile phone or a thin client
• CPC Class Codes: A63F 2300/105 using inertial sensors, e.g. accelerometers, gyroscopes
• CPC Class Codes: A63F 2300/69 Involving elements of the real world in the game world, e.g. measurement in live races, real video
• CPC Class Codes: G06T 2207/30221 Sports video; Sports image
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06V 20/42 of sport video content
• CPC Class Codes: G06V 20/47 Detecting features for summarising video content
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training
• CPC Class Codes: G08B 21/043 detecting an emergency event, e.g. a fall
• CPC Class Codes: G08B 21/0492 Sensor dual technology, i.e. two or more technologies collaborate to extract unsafe condition, e.g. video tracking and RFID tracking
• CPC Class Codes: G11B 27/031 Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: H04N 21/42201 biosensors, e.g. heat sensor for presence detection, EEG sensors or any limb activity sensors worn by the user input arrangements for interaction with the human body based on nervous system activity detection G06F3/015
• CPC Class Codes: H04N 21/42202 environmental sensors, e.g. for detecting temperature, luminosity, pressure, earthquakes
• CPC Class Codes: H04N 21/42203 sound input device, e.g. microphone
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

Enables event analysis from sensors including environmental, physiological and motion capture sensors. Also enables displaying information based on events recognized using sensor data associated with a user, piece of equipment or based on previous motion analysis data from the user or other user(s) or other sensors. Enables intelligent analysis, synchronization, and transfer of generally concise event videos synchronized with motion data from motion capture sensor(s) coupled with a user or piece of equipment. Enables creating, transferring, obtaining, and storing concise event videos generally without non-event video. Events stored in the database identifies trends, correlations, models, and patterns in event data. Greatly saves storage and increases upload speed by uploading event videos and avoiding upload of non-pertinent portions of large videos. Creates highlight and fail reels filtered by metrics and can sort by metric. Compares motion trajectories of users and objects to optimally efficient trajectories, and to desired trajectories.

Claims

1. An event analysis system comprising:

at least one motion capture element configured to couple with a user or piece of equipment or mobile device coupled with the user, wherein said at least one motion capture element comprises

a sensor data memory;

a sensor configured to capture one or more values associated with an orientation, position, velocity, acceleration, angular velocity, and angular acceleration of said at least one motion capture element;

a first communication interface;

a microprocessor coupled with said sensor data memory, said sensor and said radio, wherein said microprocessor is configured to

collect data that comprises sensor values from said sensor;

store said data in said sensor data memory;

optionally analyze said data and recognize an event within said data to determine event data;

transmit said data or said event data associated with said event or both said data and said event data via said first communication interface;

a database;

a computer comprising

a computer memory;

a second communication interface configured to communicate with said first communication interface to obtain said data or said event data associated with said event or both said data and said event data and to communicate with said database;

wherein said computer is coupled with said computer memory and is coupled with said second communication interface, wherein said computer is configured to

receive

said data from said second communication interface and analyze said data and recognize an event within said data to determine event data,

or

said event data from said second communication interface,

or

both said data and said event data from said second communication interface;

analyze said event data to form motion analysis data;

store said event data, or said motion analysis data, or both said event data and said motion analysis data in said computer memory;

obtain an event start time and an event stop time from said event data;

obtain at least one video start time and at least one video stop time associated with at least one video;

synchronize said event data, said motion analysis data or any combination thereof with said at least one video based on

a first time associated with said data or said event data obtained from said at least one motion capture element coupled with said user or said piece of equipment or said mobile device coupled with the user and

at least one time associated said at least one video to

obtain at least one synchronized event video without at least a portion of said at least one video that occurs outside of a timespan between said event start time to said event stop time; and

,

store said at least one synchronized event video in said database without a store of said at least said portion of said at least one video that occurs outside of said timespan between said event start time to said event stop time.

2. The system of claim 1 wherein said computer comprises at least one processor in said mobile device, a mobile phone, a smart phone, a smart watch, a camera, a laptop computer, a notebook computer, a tablet computer, a desktop computer, a server computer or any combination of any number of said mobile device, said mobile phone, said smart phone, said smart watch, said camera, said laptop computer, said notebook computer, said tablet computer, said desktop computer and said server computer.

3. The system of claim 1 wherein said computer is further configured to

display a synchronized event video comprising both of

said event data, said motion analysis data or any combination thereof that occurs during a timespan from said event start time to said event stop time;

and,

said at least one video captured during said timespan from said event start time to said event stop time.

4. The system of claim 1 wherein

said computer is further configured to

discard or

instruct another computer to discard or

instruct a camera to discard

at least a portion of said at least one video outside of said event start time to said event stop time.

5. The system of claim 1 wherein said computer is further configured to

send a control message

locally to at least one camera coupled with said computer or

externally said at least one camera

to modify video recording parameters of said at least one video based on said data or said event data or said motion analysis data;

wherein said video recording parameters comprise one or more of frame rate, resolution, color depth, color or grayscale, compression method, compression quality, and recording on or off.

6. The system of claim 4, wherein

said sensor comprises an accelerometer that measures acceleration in one or more axes;

said recognize an event within said data comprises one or both of

compare the magnitude of said acceleration to an acceleration threshold value;

compare the change in said acceleration over a time interval to an acceleration change threshold value.

7. The system of claim 1 wherein said computer is further configured to transmit said at least one synchronized event video or a portion of said at least one synchronized event video to one or more of a repository, a viewer, a server, another computer, a social media site, said mobile device, a network, and an emergency service without a transmit of said at least said portion of said at least one video that occurs outside of said timespan between said event start time to said event stop time.

8. The system of claim 1 wherein said computer is further configured to

accept a metric associated with said at least one synchronized event video;

accept selection criteria for said metric;

determine a matching set of synchronized event videos that have values associated with said metric that pass said selection criteria;

create a list of, or display, said matching set of synchronized event videos or corresponding thumbnails thereof along with said value associated with said metric for each of said matching set of synchronized event videos or said corresponding thumbnails;

or

generate a highlight reel or fail reel or both, of said matching set of synchronized event,

or

both create said list of, or display said matching set of synchronized event videos or corresponding thumbnails, and said generate said highlight reel or said fail reel or both.

9. The system of claim 1 wherein said computer is further configured to

select a motion variable that is a function of time, having as its domain an event time interval from said event start time to said event stop time;

select an efficiency function that maps trajectories of said motion variable to an efficiency value;

calculate an actual trajectory of said motion variable over said event time interval from one or more of said data, said event data, and said motion analysis data;

determine an optimal trajectory of said motion variable over said event time interval, wherein

said optimal trajectory has the same values of said motion variable as said actual trajectory at the endpoints of said event time interval;

said optimal trajectory maximizes said efficiency function over the set of possible trajectories having the same values of said motion variable as said actual trajectory at the endpoints of said event time interval; and

,

calculate an efficiency index by comparing said actual trajectory to said optimal trajectory.

10. The system of claim 9, wherein said determine an optimal trajectory comprises

select one or more high efficiency synchronized event videos from said database;

calculate said optimal trajectory from said one or more high efficiency synchronized event videos.

11. The system of claim 9, further comprising wherein said determine an optimal trajectory comprises

create or obtaining a model of the mechanics of motion for said event;

derive said optimal trajectory from said model of the mechanics of motion for said event.

12. The system of claim 1, wherein said computer is further configured to

obtain a description of an object of interest;

calculate an observed trajectory of said object of interest from said synchronized event video;

calculate a desired trajectory of said object of interest;

compare said observed trajectory of said object of interest to said desired trajectory of said object of interest;

calculate a desired change in the initial conditions of said observed trajectory of said object of interest that would result in said desired trajectory of said object of interest;

add said desired change in the initial conditions of said observed trajectory to said motion analysis data associated with said synchronized event video;

display said desired change in the initial condition of said observed trajectory and said synchronized event video.

13. The system of claim 12, further wherein

said computer is further configured to obtain a model of the area of activity where said observed trajectory of said object of interest occurred, wherein said model of the area of activity comprises one or more of

the topography of said area of activity;

a coefficient of friction at one or more points of said area of activity;

the forces between said area of activity and objects moving through said area of activity; and

,

one or more physical properties at one or more points of said area of activity;

said calculate a desired change in the initial conditions of said observed trajectory comprises using said model of said area of activity to calculate a predicted trajectory from said desired change in the initial conditions of said observed trajectory.

14. The system of claim 1, wherein said computer is further configured to

obtain an area of activity;

select one or more synchronized event videos from said database that occurred in said area of activity;

analyze the selected synchronized event videos to develop a model of said area of activity, wherein said model of said area of activity comprises one or more of

the topography of said area of activity;

a coefficient of friction at one or more points of said area of activity;

the forces between said area of activity and objects moving through said area of activity; and

,

one or more physical properties at one or more points of said area of activity.

15. The system of claim 14, wherein said computer is further configured to obtain a description of an object of interest;

calculate an observed trajectory of said object of interest from said synchronized event video, wherein said observed trajectory of said object of interest occurred in said area of activity;

calculate a desired trajectory of said object of interest;

compare said observed trajectory of said object of interest to said desired trajectory of said object of interest;

calculate a desired change in the initial conditions of said observed trajectory of said object of interest that would result in said desired trajectory of said object of interest using said model of said area of activity to calculate a predicted trajectory from said desired change in the initial conditions of said observed trajectory;

add said desired change in the initial conditions of said observed trajectory to said motion analysis data associated with said synchronized event video;

display said desired change in the initial condition of said observed trajectory and said synchronized event video.

16. The system of claim 1 wherein said computer is further configured to

analyze said database to identify one or more accidents recorded in said at least one synchronized event video;

transmit accident reports comprising data selected from the synchronized event videos associated with said one or more accidents to one or more of an emergency service, a government agency, a safety agency, a quality control organization, and a group of persons potentially at risk of additional accidents similar to said one or more accidents.

17. The system of claim 1 wherein said computer is further configured to

analyze said database to identify locations where events in said at least one synchronized video have occurred;

generate text or graphics or both text and graphics identifying said locations;

display said text or graphics or both text and graphics and one or more of

said at least one synchronized event video;

a map of a region that includes one or more of said locations;

an image of a region that includes one or more of said locations.

18. The system of claim 1, wherein

said piece of equipment is configured to impact other objects during legitimate use;

said computer is further configured to

obtain one or more normal impact signatures that describe said motion analysis data for said event wherein said piece of equipment impacts said other objects during said legitimate use;

select one or more synchronized event videos from said database wherein said event for said synchronized event video is an impact event for said piece of equipment;

compare the selected synchronized event videos to said one or more normal impact signatures to determine whether said piece of equipment has been used for said legitimate use.

19. The system of claim 1, wherein

one or more of said at least one motion capture element are coupled to a joint of said user or to said piece of equipment which is coupled to said joint of said user;

said computer is further configured to

select one or more synchronized event videos from said database wherein said event for said synchronized event video is a rotation event for said joint of said user;

calculate an angle of rotation for each of said rotation events;

compare said angle of rotation for each of said rotation events to one or more of

a target value for said angle of rotation;

a target range for said angle of rotation;

a threshold value for said angle of rotation; and

,

an angle of rotation for a previous rotation event.

20. The system of claim 19, wherein

said one or more of said at least one motion capture element comprise

a first motion capture element coupled to a first side of said joint of said user;

a second motion capture element coupled to a second side of said joint of said user;

wherein said first motion capture element and said second motion capture element comprise

an accelerometer;

a magnetometer; and

,

a rate gyroscope.

21. The system of claim 19, wherein said computer is further configured to

send an alert to another computer or to another user if said angle of rotation exceeds said target value or said threshold value.

22. The system of claim 1, further comprising at least one other sensor coupled with said microprocessor or said computer that is configured to capture said one or more other values associated with said temperature, humidity, wind, elevation, light, sound, heart rate and wherein said recognize said event comprises

determine a prospective event based on said data;

correlate said data with at least one of said other values to determine if said prospective event is a true event or a false positive event.

23. The system of claim 1, further comprising at least one other sensor coupled with said microprocessor or said computer that is configured to capture said one or more other values associated with said temperature, humidity, wind, elevation, light, sound, heart rate and wherein said recognize said event comprises

determine a type of equipment of said piece of equipment or type of activity based on said one or more values captured from said at least one other sensor in combination with said one or more values associated with said orientation, position, velocity, acceleration, angular velocity, and angular acceleration of said at least one motion capture element.

24. The system of claim 1, wherein said microprocessor is further configured to, or said computer is further configured to, or both said microprocessor and said computer are further configured to confirm said event for a particular location and time by analyzing one or more of text, audio, image, and video from a server to create a confirmed event.

25. The system of claim 1, wherein

said sensor or said computer comprises a microphone that records audio signals;

said recognize an event comprises

determine a prospective event based on said data;

correlate said data with said audio signals to determine if said prospective event is a true event or a false positive event; and

,

wherein said computer is further configured to store said audio signals in said database with said at least one synchronized event video if said prospective event is a true event

26. The system of claim 1 wherein said computer is further configured to

synchronize said motion analysis data with said video based on image analysis to more accurately determine a start event frame or stop event frame in said video or both, that is most closely associated with said event start time or said event stop time or both.

27. The system of claim 1 wherein said computer is further configured to

access previously stored event data or motion analysis data associated with said user or piece of equipment;

display information comprising a presentation of said event data associated with said user on a display based on

said event data or motion analysis data associated with said user or piece of equipment;

and,

said previously stored event data or motion analysis data associated with said user or piece of equipment.

28. The system of claim 1 wherein said computer is further configured to

access previously stored event data or motion analysis data associated with at least one other user or at least one other piece of equipment;

display information comprising a presentation of said event data associated with said user on a display based on

said event data or motion analysis data associated with said user or piece of equipment;

and,

said previously stored event data or motion analysis data associated with said at least one other user or said at least one other piece of equipment.

29. The system of claim 1 wherein said microprocessor in said at least one motion capture element is configured to transmit said event to at least one other motion capture element or said computer or any combination thereof, and wherein said at least one other motion capture element or said at least one other mobile device or any combination thereof is configured to save data or transmit data or both save data and transmit data associated with said event even if said at least one other motion capture element has not detected said event.

30. The system of claim 1 wherein said computer is further configured to

request or broadcast a request for camera locations proximal to said event or oriented to view said event or both proximal to and oriented to view said event;

and,

request said at least one video from said at least one camera proximal to said event wherein said video contains said event without at least a portion of said at least one video outside of said event start time to said event stop time.