Multi-sensor even analysis and tagging system

• US 10,109,061 B2
• Filed: 05/09/2017
• Issued: 10/23/2018
• Est. Priority Date: 08/26/2010
• Status: Active Grant

• Current Assignee: Newlight Capital, LLC
• Original Assignee: Blast Motion Incorporated
• Inventors: Bose, Bhaskar, Gupta, Piyush, Haas, Juergen, Estrem, Brian, Bentley, Michael, Kaps, Ryan
• Primary Examiner(s): Deodhar, Omkar
• Application Number:
  US15/590,398
• Publication Number:
  US 20170243359A1
• Time in Patent Office: 532 Days
• CPC Class Codes: A63F 13/00 Video games, i.e. games using an electronically generated display having two or more dimensions gaming systems which provide a financial reward G07F17/32
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/212 using sensors worn by the player, e.g. for measuring heart beat or leg activity
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/812 Ball games, e.g. soccer or baseball
• CPC Class Codes: A63F 2300/105 using inertial sensors, e.g. accelerometers, gyroscopes
• CPC Class Codes: A63F 2300/69 Involving elements of the real world in the game world, e.g. measurement in live races, real video
• CPC Class Codes: G06T 2207/30221 Sports video; Sports image
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G06V 20/44 Event detection
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training
• CPC Class Codes: G08B 21/043 detecting an emergency event, e.g. a fall
• CPC Class Codes: G08B 21/0492 Sensor dual technology, i.e. two or more technologies collaborate to extract unsafe condition, e.g. video tracking and RFID tracking
• CPC Class Codes: G11B 27/022 Electronic editing of analogue information signals, e.g. audio or video signals
• CPC Class Codes: G11B 27/031 Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: G11B 27/10 Indexing; Addressing; Timing or synchronising; Measuring tape travel
• CPC Class Codes: G11B 27/17 using electrical sensing means see provisionally also G11B27/13
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence
• CPC Class Codes: G11B 31/006 : with video camera or receiver
• CPC Class Codes: H04N 7/181 : for receiving images from a plurality of remote sources

Abstract

A system that analyzes data from multiple sensors, potentially of different types, that track motions of players, equipment, and projectiles such as balls. Data from different sensors is combined to generate integrated metrics for events and activities. Illustrative sensors may include inertial sensors, cameras, radars, and light gates. As an illustrative example, a video camera may track motion of a pitched baseball, and an inertial sensor may track motion of a bat; the system may use the combined data to analyze the effectiveness of the swing in hitting the pitch. The system may also use sensor data to automatically select or generate tags for an event; tags may represent for example activity types, players, performance levels, or scoring results. The system may analyze social media postings to confirm or augment event tags. Users may filter and analyze saved events based on the assigned tags.

Claims

1. A multi-sensor event analysis and tagging system comprising:

a plurality of sensors comprising

a first inertial sensor coupled to a player or a piece of equipment used by the player, and configured to measure one or more of a position, an orientation, a velocity, an angular velocity, an acceleration, and an angular acceleration of the player or said piece of equipment or both said player and said piece of equipment;

a second sensor configured to measure motion of a projectile, wherein said second sensor comprises one or more of a video camera, a light gate, a radar, or a second inertial sensor coupled to said projectile; and

,

an event analysis and tagging system comprising

a communications interface configured to receive sensor data from said plurality of sensors of a swing of said piece of equipment by the player in an attempt to contact said projectile;

an event database;

a processor coupled to said communications interface and configured to

receive said sensor data from said communications interface;

synchronize said sensor data to a common time scale, forming synchronized sensor data;

analyze said synchronized sensor data to

detect an event or confirm said event based on said sensor data received from said plurality of said sensors or analyze one or more of text, audio, image, and video from a computer or server, or any combination of said detect, said confirm or said analyze to

 

determine one or more tags for said event and,

calculate

a projectile trajectory from said synchronized sensor data;

a piece of equipment trajectory from said synchronized sensor data; and

,

one or more metrics for said event from the relative spatial and temporal relationships between said projectile trajectory and said piece of equipment trajectory;

store one or both of said synchronized sensor data and said one or more metrics for said event in said event database;

store said one or more tags in said event database with said event; and

,

publish said event and one or more of said one or more tags to a social media site.

2. The multi-sensor event analysis and tagging system of claim 1, wherein said projectile comprises one or more of a ball, a football, a rugby ball, an Australian rules football, a soccer ball, a volleyball, a water polo ball, a polo ball, a basketball, a lacrosse ball, a field hockey ball, a croquet ball, a billiard ball, a horseshoe, a shuffleboard disc, a tennis ball, a ping pong ball, a racquet ball, a hand ball, a bocce ball, a lawn dart, a squash ball, a shuttlecock, a baseball, a softball, a golf ball, a bowling ball, a hockey puck, a dodgeball, a kick ball, a Wiffle™

ball, a javelin, a shot put, a discus, a marble, a bullet, an arrow, a knife, a throwing star, a bolas, a grenade, a water balloon, a boomerang, a Frisbee™

, a caber, and a curling stone.

3. The multi-sensor event analysis and tagging system of claim 1, wherein said piece of equipment comprises one or more of a bat, a racquet, a paddle, a golf club, a bow, a gun, a slingshot, a sabre, a quarterstaff, a lacrosse stick, a hockey stick, a field hockey stick, a polo mallet, a croquet mallet, a pool cue, a shuffleboard cue, a glove, a shoe, a belt, a watch, a helmet, a cap, a ski, a snowboard, a skateboard, a surfboard, an ice skate, a sled, a luge, a windsurfing board, a hang glider, a roller skate, a roller blade, a vehicle, a snowmobile, a jet ski, a bicycle, a tricycle, a unicycle, a motorcycle, a snowmobile, and a mechanical bull.

4. The multi-sensor event analysis and tagging system of claim 1, wherein

said piece of equipment is a bat;

said projectile is a ball; and

,

said motion of said projectile is a pitch of said ball.

5. The multi-sensor event analysis and tagging system of claim 4, wherein said metrics comprise an elapsed time between a pitch start time when said ball leaves a pitcher and a swing start time when said player starts acceleration of said bat towards said ball for said swing.

6. The multi-sensor event analysis and tagging system of claim 1, wherein said piece of equipment trajectory is a trajectory of a preferred hitting location on said piece of equipment.

7. The multi-sensor event analysis and tagging system of claim 6, wherein

said piece of equipment is a bat;

said preferred hitting location is a sweet spot of said bat.

8. The multi-sensor event analysis and tagging system of claim 1, wherein

said processor is further configured to calculate an optimal hitting point on said projectile trajectory, said optimal hitting point comprising an optimal hitting time and an optimal hitting location;

said one or more metrics comprise one or more of

a swing accuracy metric calculated from a vector difference between said optimal hitting location and a location of said piece of equipment trajectory at said optimal hitting time;

a spatial deviation metric calculated from a vector difference between said optimal hitting location and a closest point on said piece of equipment trajectory to said optimal hitting location; and

,

a temporal deviation metric calculated from a time difference between said optimal hitting time and a time at which said piece of equipment trajectory reaches said closest point on said piece of equipment trajectory to said optimal hitting location.

9. The multi-sensor event analysis and tagging system of claim 1, wherein said one or more of text, audio, image, and video comprise one or more of email messages, voice calls, voicemails, audio recordings, video calls, video messages, video recordings, text messages, chat messages, postings on social media sites, postings on blogs, or postings on wikis.

10. The multi-sensor event analysis and tagging system of claim 1, wherein said server comprises one or more of an email server, a social media site, a photo sharing site, a video sharing site, a blog, a wiki, a database, a newsgroup, an RSS server, a multimedia repository, a document repository, and a text message server.

11. The multi-sensor event analysis and tagging system of claim 1, wherein said analyzing one or more of text, audio, image, and video comprises searching said text for key words or key phrases related to said event.

12. The multi-sensor event analysis and tagging system of claim 1, wherein said one or more tags represent one or more of

an activity type of said event;

a location of said event;

a stage of an activity associated with said event;

a player identity associated with said event;

a performance level associated with said event; and

,

a scoring result associated with said event.

13. The multi-sensor event analysis and tagging system of claim 1, further comprising:

a manual tagging interface configured to

present said event to a user;

accept one or more user selected tags for said event from said user; and

,

store said one or more user selected tags in said event database with said event.