Event analysis and tagging system

• US 9,911,045 B2
• Filed: 04/10/2017
• Issued: 03/06/2018
• Est. Priority Date: 08/26/2010
• Status: Active Grant

• Current Assignee: Newlight Capital, LLC
• Original Assignee: Blast Motion Incorporated
• Inventors: Bose, Bhaskar, Gupta, Piyush, Haas, Juergen, Estrem, Brian, Bentley, Michael, Kaps, Ryan
• Primary Examiner(s): DEODHAR, OMKAR A
• Application Number:
  US15/482,902
• Publication Number:
  US 20170300755A1
• Time in Patent Office: 330 Days
• CPC Class Codes: A63F 13/00 Video games, i.e. games using an electronically generated display having two or more dimensions gaming systems which provide a financial reward G07F17/32
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/35 Details of game servers
• CPC Class Codes: A63F 13/65 automatically by game devices or servers from real world data, e.g. measurement in live racing competition
• CPC Class Codes: A63F 2300/105 using inertial sensors, e.g. accelerometers, gyroscopes
• CPC Class Codes: A63F 2300/69 Involving elements of the real world in the game world, e.g. measurement in live races, real video
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30221 Sports video; Sports image
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G06V 20/42 of sport video content
• CPC Class Codes: G06V 20/44 Event detection
• CPC Class Codes: G06V 20/47 Detecting features for summarising video content
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training
• CPC Class Codes: G08B 21/043 detecting an emergency event, e.g. a fall
• CPC Class Codes: G08B 21/0492 Sensor dual technology, i.e. two or more technologies collaborate to extract unsafe condition, e.g. video tracking and RFID tracking
• CPC Class Codes: G11B 27/031 Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: G11B 27/10 Indexing; Addressing; Timing or synchronising; Measuring tape travel
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

A system that analyzes data from sensors and video cameras to generated synchronized event videos and to automatically select or generate tags for an event. Enables creating, transferring, obtaining, and storing concise event videos generally without non-event video. Events stored in the database identifies trends, correlations, models, and patterns in event data. Tags may represent for example activity types, players, performance levels, or scoring results. The system may analyze social media postings to confirm or augment event tags. Users may filter and analyze saved events based on the assigned tags. The system may create highlight and fail reels filtered by metrics and by tags.

Claims

1. An event analysis and tagging system comprising:

at least one motion capture element configured to couple with a user or piece of equipment or mobile device coupled with the user, wherein said at least one motion capture element comprises

a sensor data memory;

a sensor configured to capture one or more values associated with an orientation, position, velocity, acceleration, angular velocity, and angular acceleration of said at least one motion capture element;

a first communication interface; and

,

a microprocessor coupled with said sensor data memory, said sensor and said first communication interface, wherein said microprocessor is configured to

collect data that comprises sensor values from said sensor;

store said data in said sensor data memory;

optionally analyze said data and recognize an event within said data to determine event data;

transmit said data or said event data associated with said event or both said data and said event data via said first communication interface;

a second memory comprising a database or computer memory or both;

wherein said microprocessor is configured to transmit said data or said event data to a computer, said computer comprising

a second communication interface configured to communicate with said first communication interface to obtain said data or said event data associated with said event or both said data and said event data;

wherein said computer is coupled with said second memory and is coupled with said second communication interface, wherein said computer is configured to

receive

said data from said second communication interface and analyze said data and recognize said event within said data to determine said event data,

or

said event data from said second communication interface,

or

both said data and said event data from said second communication interface;

optionally analyze said event data to form motion analysis data;

store said event data in said second memory;

obtain an event start time and an event stop time from said event data;

obtain at least one video start time and at least one video stop time associated with at least one video;

synchronize said event data with said at least one video based on

a first time associated with said data or said event data obtained from said at least one motion capture element coupled with said user or said piece of equipment or said mobile device coupled with the user and

at least one time associated with said at least one video to obtain at least one synchronized event video without at least a portion of said at least one video that occurs outside of a timespan between said event start time to said event stop time; and

,

store said at least one synchronized event video in said second memory and discard at least said portion of said at least one video that occurs outside of said timespan between said event start time to said event stop time based on said event data,

or

transmit said at least one synchronized event video to another computer without a transfer of said at least said portion of said at least one video that occurs outside of said timespan between said event start time to said event stop time

or

store said at least one synchronized event video in said second memory, discard at least said portion of said at least one video that occurs outside of said timespan between said event start time to said event stop time as non-event related video based on said event data, and transmit said at least one synchronized event video to another computer without a transfer of said at least said portion of said at least one video that occurs outside of said timespan between said event start time to said event stop time.

2. The event analysis and tagging system of claim 1, wherein said computer is further configured to

analyze one or more of said event data, said at least one video, text from a server, audio from a server, image from a server, and video from a server to determine one or more tags for said event; and,

store said one or more tags in said second memory.

3. The event analysis and tagging system of claim 2, wherein said server comprises one or more of an email server, a social media site, a photo sharing site, a video sharing site, a blog, a wiki, a second database, a newsgroup, an RSS server, a multimedia repository, a document repository, and a text message server.

4. The event analysis and tagging system of claim 2, wherein said computer is further configured to

generate an integrated event record comprising

said at least one synchronized event video, and

one or more media selected from said one or more of said text from a server, said audio from a server, said image from a server, and said video from a server.

5. The event analysis and tagging system of claim 4, wherein said computer is further configured to

publish said integrated event record to a social media site.

6. The event analysis and tagging system of claim 2, wherein said analyze one or more of said event data, said at least one video, text from a server, audio from a server, image from a server, and video from a server comprises search for key words or key phrases related to said event.

7. The event analysis and tagging system of claim 2, wherein said one or more tags represent one or more of

an activity type of said event;

a location of said event;

a timestamp of said event;

a stage of an activity associated with said event;

a player identity associated with said event;

a performance level associated with said event; and

,

a scoring result associated with said event.

8. The event analysis and tagging system of claim 1, wherein said computer is further configured to

publish said at least one synchronized event video to a social media site.

9. The event analysis and tagging system of claim 1, further comprising a manual tagging interface configured to

accept one or more user selected tags for said event;

and,

store said one or more user selected tags in said second memory with said event.

10. The event analysis and tagging system of claim 2, further comprising an event filter coupled to said second memory, wherein said event filter is configured to

accept one or more filter tags;

and,

return said at least one synchronized event video for events in said second memory having tags that match said one or more filter tags.

11. The event analysis and tagging system of claim 10, wherein

said event filter is further configured to create a highlight reel comprising video for said events in said second memory having tags that match said one or more filter tags.

12. The event analysis and tagging system of claim 1 wherein said computer further comprises at least one processor in said mobile device, a mobile phone, a smart phone, a smart watch, a camera, a laptop computer, a notebook computer, a tablet computer, a desktop computer, a server computer or any combination of any number of said mobile device, said mobile phone, said smart phone, said smart watch, said camera, said laptop computer, said notebook computer, said tablet computer, said desktop computer and said server computer and wherein said computer is further configured to transmit said at least one synchronized event video or a portion of said at least one synchronized event video to one or more of a repository, a viewer, a server, a social media site, said mobile device, a network, and an emergency service without said transmit of said at least said portion of said at least one video that occurs outside of said timespan between said event start time to said event stop time.

13. The event analysis and tagging system of claim 1 wherein said computer is further configured to display both of

said event data, said motion analysis data or any combination thereof that occurs during said timespan from said event start time to said event stop time;

and,

said at least one synchronized event video.

14. The event analysis and tagging system of claim 1 wherein

said computer is further configured to

discard or

instruct another computer to discard or

instruct a camera to discard

said at least said portion of said at least one video that occurs outside of said timespan between said event start time to said event stop time.

15. The event analysis and tagging system of claim 1 wherein said computer is further configured to

communicate with at least one camera;

and,

send a control message

locally to said at least one camera coupled with said computer or

externally to said at least one camera,

to modify one or more video recording parameters of said at least one video associated with said at least one camera based on said data or said event data or said motion analysis data;

wherein said video recording parameters comprise one or more of frame rate, resolution, color depth, color or grayscale, compression method, compression quality, and recording on or off.

16. The event analysis and tagging system of claim 1 wherein said computer is further configured to

accept a metric associated with said at least one synchronized event video;

accept selection criteria associated with said metric;

determine a matching set of synchronized event videos that have a value or values associated with said metric that pass said selection criteria; and

,

create a list of, or display, said matching set of synchronized event videos or corresponding thumbnails thereof having said value or values associated with said metric associated with each of said matching set of synchronized event videos or said corresponding thumbnails,

or

generate a highlight reel or fail reel or both, of said matching set of synchronized event videos,

or

both said create said list of, or display said matching set of synchronized event videos or said corresponding thumbnails, and said generate said highlight reel or said fail reel or both.

17. The event analysis and tagging system of claim 1 wherein said computer is further configured to

select a motion variable that is a function of time, having as its domain an event time interval from said event start time to said event stop time;

select an efficiency function that maps trajectories of said motion variable to an efficiency value;

calculate an actual trajectory of said motion variable over said event time interval from one or more of said data, said event data, and said motion analysis data;

determine an optimal trajectory of said motion variable over said event time interval, wherein

said optimal trajectory has a same set of values of said motion variable as said actual trajectory at endpoints of said event time interval, and

said optimal trajectory maximizes said efficiency function over a set of possible trajectories that have the same set of values of said motion variable as said actual trajectory at the endpoints of said event time interval; and

,

calculate an efficiency index by comparing said actual trajectory to said optimal trajectory.

18. The event analysis and tagging system of claim 1, wherein said computer is further configured to obtain a description of an object of interest;

calculate an observed trajectory of said object of interest from said at least one synchronized event video;

calculate a desired trajectory of said object of interest;

compare said observed trajectory of said object of interest to said desired trajectory of said object of interest;

calculate a desired change in initial conditions of said observed trajectory of said object of interest to create said desired trajectory of said object of interest;

add said desired change in the initial conditions of said observed trajectory to said motion analysis data or said event data associated with said at least one synchronized event video; and

,

display said desired change in the initial conditions of said observed trajectory and said at least one synchronized event video.

19. The event analysis and tagging system of claim 1, wherein said computer is further configured to

obtain a description of an object of interest;

obtain a model of an area of activity, wherein said model of said area of activity comprises one or more of

a topography of said area of activity;

a coefficient of friction at one or more points of said area of activity;

forces between said area of activity and objects that move through said area of activity; and

,

one or more physical properties at said one or more points of said area of activity;

calculate an observed trajectory of said object of interest from said at least one synchronized event video, wherein said observed trajectory of said object of interest occurred in said area of activity;

calculate a desired trajectory of said object of interest;

compare said observed trajectory of said object of interest to said desired trajectory of said object of interest;

calculate a desired change in initial conditions of said observed trajectory of said object of interest to create said desired trajectory of said object of interest through use of said model of said area of activity to calculate a predicted trajectory from said desired change in the initial conditions of said observed trajectory;

add said desired change in the initial conditions of said observed trajectory to said data or said event data associated with said at least one synchronized event video; and

,

display said desired change in the initial conditions of said observed trajectory and said at least one synchronized event video.

20. The event analysis and tagging system of claim 1 wherein said computer is further configured to

analyze said second memory to identify one or more accidents recorded in said at least one synchronized event video;

and,

transmit accident reports comprising accident related data selected from synchronized event videos associated with said one or more accidents to one or more of an emergency service, a government agency, a safety agency, a quality control organization, and a group of persons potentially at risk of additional accidents similar to said one or more accidents.

21. The event analysis and tagging system of claim 1 wherein said computer is further configured to

analyze said second memory to identify one or more locations where one or more events in said at least one synchronized video have occurred;

generate text or graphics or both text and graphics identifying said one or more locations; and

,

display said text or graphics or said both text and graphics and one or more of said at least one synchronized event video;

a map of a region that includes said one or more locations; and

,

an image of a region that includes said one or more locations.

22. The event analysis and tagging system of claim 1, further comprising at least one other sensor coupled with said microprocessor directly or indirectly or coupled with said computer directly or indirectly that is configured to capture one or more other values associated with one or more of an environmental variable and a physiological variable, wherein said recognize said event comprises

determine a prospective event based on said data;

and,

correlate said data with at least one of said one or more other values to determine whether said prospective event is a true event or a false positive event.

23. The event analysis and tagging system of claim 1, further comprising at least one other sensor coupled with said microprocessor directly or indirectly or said computer directly or indirectly that is configured to capture one or more other values associated with one or more of an environmental variable and a physiological variable, wherein said recognize said event comprises

determine a type of equipment of said piece of equipment or type of activity based on said one or more other values captured from said at least one other sensor in combination with said one or more values associated with said orientation, position, velocity, acceleration, angular velocity, and angular acceleration of said at least one motion capture element.

24. The event analysis and tagging system of claim 1, wherein said microprocessor is further configured to, or said computer is further configured to, or both said microprocessor and said computer are further configured to confirm said event at a particular location and time by analyzing one or more of text, audio, image, and video from a server to create a confirmed event.

25. The event analysis and tagging system of claim 1, wherein said sensor or said computer comprises a microphone that records audio signals;

said recognize said event comprises

determine a prospective event based on said data; and

,

correlate said data with said audio signals to determine whether said prospective event is a true event or a false positive event; and

,

said computer is further configured to store said audio signals in said second memory with said at least one synchronized event video when said prospective event is said true event.

26. The event analysis and tagging system of claim 1 wherein said computer is further configured to

synchronize said data, said motion analysis data or said event data or any combination thereof with said at least one video based on image analysis to more accurately determine one or both of a start event frame or stop event frame in said at least one video or both said start event frame and said stop event frame, that is most closely associated with said event start time or said event stop time or both said start event frame and said stop event frame.

27. The event analysis and tagging system of claim 1 wherein said computer is further configured to

access previously stored data, event data or motion analysis data or any combination thereof associated with said user or piece of equipment;

and,

display information comprising a presentation of said event data associated with said user on a display based on

said data, said event data or said motion analysis data or any combination thereof associated with said user or piece of equipment;

and,

said previously stored data, event data or motion analysis data or any combination thereof associated with said user or piece of equipment.

28. The event analysis and tagging system of claim 1 wherein said computer is further configured to

access previously stored data, event data or motion analysis data or any combination thereof associated with at least one other user or at least one other piece of equipment;

and,

display information comprising a presentation of said event data associated with said user on a display based on

said data, said event data or said motion analysis data or any combination thereof associated with said user or piece of equipment;

and,

said previously stored data, event data or motion analysis data or any combination thereof associated with said at least one other user or said at least one other piece of equipment.

29. The event analysis and tagging system of claim 1 wherein said microprocessor in said at least one motion capture element is further configured to transmit said event to at least one other motion capture element or at least one other mobile device or any combination thereof, and wherein said at least one other motion capture element or said at least one other mobile device or any combination thereof is configured to save data or transmit data or both save data and transmit data associated with said event even if said at least one other motion capture element has not detected said event.

30. The event analysis and tagging system of claim 1 wherein said computer is further configured to

request or broadcast a request to obtain identification of one or more cameras that each have a location proximal to said event or oriented to view said event or both having locations proximal to and oriented to view said event;

and,

request said at least one video from at least one camera of said one or more cameras, wherein said at least one video contains at least a portion of said event without said at least said portion of said at least one video that occurs outside of said event start time to said event stop time.