Event detection, confirmation and publication system that integrates sensor data and social media

• US 9,940,508 B2
• Filed: 05/09/2017
• Issued: 04/10/2018
• Est. Priority Date: 08/26/2010
• Status: Active Grant

• Current Assignee: Newlight Capital, LLC
• Original Assignee: Blast Motion Incorporated
• Inventors: Kaps, Ryan, Bentley, Michael, Bose, Bhaskar
• Primary Examiner(s): DEODHAR, OMKAR A
• Application Number:
  US15/590,511
• Publication Number:
  US 20170262697A1
• Time in Patent Office: 336 Days
• CPC Class Codes: A63B 24/0062 Monitoring athletic performances, e.g. for determining the work of a user on an exercise apparatus, the completed jogging or cycling distance
• CPC Class Codes: A63B 71/06 Indicating or scoring devices for games or players , or for other sports activities
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/212 using sensors worn by the player, e.g. for measuring heart beat or leg activity
• CPC Class Codes: A63F 13/65 automatically by game devices or servers from real world data, e.g. measurement in live racing competition
• CPC Class Codes: A63F 13/812 Ball games, e.g. soccer or baseball
• CPC Class Codes: A63F 2300/105 using inertial sensors, e.g. accelerometers, gyroscopes
• CPC Class Codes: A63F 2300/69 Involving elements of the real world in the game world, e.g. measurement in live races, real video
• CPC Class Codes: G06T 2207/30 Subject of image; Context of image processing
• CPC Class Codes: G06T 2207/30221 Sports video; Sports image
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G06V 20/42 of sport video content
• CPC Class Codes: G06V 20/44 Event detection
• CPC Class Codes: G06V 20/47 Detecting features for summarising video content
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training
• CPC Class Codes: G08B 21/043 detecting an emergency event, e.g. a fall
• CPC Class Codes: G08B 27/005 with transmission via computer network
• CPC Class Codes: G11B 27/022 Electronic editing of analogue information signals, e.g. audio or video signals
• CPC Class Codes: G11B 27/031 Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: G11B 27/17 : using electrical sensing means see provisionally also G11B27/13
• CPC Class Codes: G11B 31/006 : with video camera or receiver
• CPC Class Codes: H04N 5/77 : between a recording apparatus and a television camera
• CPC Class Codes: H04N 5/91 : Television signal processing therefor
• CPC Class Codes: H04N 7/18 : Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
• CPC Class Codes: H04N 7/181 : for receiving images from a plurality of remote sources
• CPC Class Codes: H04N 9/8042 : involving data reduction

Abstract

Enables integration of sensor data with other information on servers such as social media sites to detect, confirm and/or publish events. Sensors may measure values such as motion, temperature, humidity, wind, pressure, elevation, light, sound, or heart rate, etc. Sensor data and event tags may be utilized to curate text, images, video, sound and post the results to social networks, for example in a dedicated feed. Event tags generated by the system may represent for example activity types, players, performance levels, or scoring results. The system may analyze social media postings to confirm or augment event tags. Users may filter and analyze saved events based on the assigned tags. The system may create highlight and fail reels filtered by metrics and by tags. Recommendations may be provided to a user based on analysis of sensor data and other information; recommendations may include for example recommended friends, purchases, or activities.

Claims

1. A system that integrates sensor data and social media comprising:

a computer comprising

a computer memory; and

,

a first communication interface configured to obtain data, or event data comprising one or more values from at least one sensor configured to measure a property of an object wherein said one or more values are associated with

one or more of

an inertial sensor value associated with an orientation, position, velocity, acceleration, angular velocity, angular acceleration,

a physical value,

an environmental value,

a physiological value

associated with a user or piece of equipment or mobile device associated with the user;

wherein said computer is coupled with said computer memory and is coupled with said first communication interface, wherein said computer is configured to

receive

said data from said first communication interface and analyze said data and recognize an event within said data to determine said event data,

or

said event data from said first communication interface,

or

both said data and said event data from said first communication interface;

store said event data in said computer memory;

obtain an event start time and an event stop time from said event data;

confirm said event recognized by said computer or obtained from said first communication interface for a particular location and time by analyzing one or more of text, audio, image, and video from a server to create a confirmed event, wherein said one or more of said text, audio, image, and video comprise one or more of email messages, voice calls, voicemails, audio recordings, video calls, video messages, video recordings, text messages, chat messages, postings on social media sites, postings on blogs, and postings on wikis; and

,

wherein said computer is further configured to

generate a suggestion or alert based on analysis of said data, said event data, or both said data and said event data;

transmit said suggestion or said alert to said user;

wherein said suggestion comprises

a second user or a group of users having associated data, associated event data, or both associated data and associated event data that is similar to said data, said event data, or both said data and said event data, or

a recommended product or a recommended piece of equipment that is appropriate for a performance level determined from said data, from said event data, or from both said data and said event data, or

a recommended activity that is appropriate for a performance level determined from said data, from said event data, or from both said data and said event data.

2. The system of claim 1, wherein said computer is further configured to publish said confirmed event to a social media site or a social media service.

3. The system of claim 2, wherein said server comprises said social media site or said social media service.

4. The system of claim 2, wherein said computer is further configured to generate a curated event record of said confirmed event;

and, publish said curated event record to said social media site or said social media service.

5. The system of claim 4, wherein said generate said curated event record comprises:

obtain a media capture;

select a portion of said media capture occurring between said event start time and said event stop time; and

,

include said portion of said media capture in said curated event record.

6. The system of claim 5, wherein said select a portion of said media capture occurring between said event start time and said event stop time comprises:

identify one or more highlights based on one or both of said data and said event data; and

,

select said portion of said media capture to include said one or more highlights.

7. The system of claim 5, wherein said media capture comprises one or more of an image, a video, and an audio.

8. The system of claim 5, wherein said computer is further configured to

discard or

instruct another computer to discard or

instruct a media capture device to discard

at least a second portion of said media capture that occurs outside of a timespan from said event start time to said event stop time.

9. The system of claim 7, wherein said media capture comprises one or more of a panoramic image and a panoramic video.

10. The system of claim 7, wherein said media capture comprises one or more of a virtual reality display, a virtual reality presentation, a virtual reality recording, an augmented reality display, an augmented reality presentation, and an augmented reality recording.

11. The system of claim 2, wherein said computer is further configured to

obtain one or more additional sensor values associated with one or more of an orientation, position, velocity, acceleration, angular velocity, angular acceleration, electromagnetic field, temperature, humidity, wind, pressure, elevation, light, sound, and heart rate from a second sensor or second computer or from a plurality of other sensors or other computers to perform said confirm said event.

12. The system of claim 2, wherein said computer is further configured to

obtain one or more additional sensor values from a second sensor to perform said confirm said event, wherein said second sensor comprises one or more of a video camera, a light gate, a radar, or an inertial sensor.

13. The system of claim 2, wherein

said computer is further configured to

determine one or more tags for said event by analyzing one or more of

said data,

said event data, and

said one or more of said text, audio, image, and video from said server;

and,

publish said one or more tags for said event to said social media site or said social media service; and

,

said one or more tags represent one or more of

an activity type of said event;

a location of said event;

a timestamp of said event;

a stage of an activity associated with said event;

a player identity associated with said event;

a performance level associated with said event; and

,

a scoring result associated with said event.

14. The system of claim 2, wherein said suggestion comprises recommended friends.

15. The system of claim 2, wherein said alert comprises an indication of an emergency.

16. The system of claim 2, wherein said suggestion comprises an indication of fake news.

17. The system of claim 2, wherein said computer is further configured to

obtain a plurality of instances of said data or said event data or both said data and said event data from a plurality of other sensors or other computers;

correlate said plurality of instances of said data or said event data or both said data and said event data based on a time and a location of each of said plurality of instances of said data or said event data or both said data and said event data; and

,

perform said confirm said event when

the number of said plurality of instances of said data or said event data or both said data and said event data occurring within a time range and a location range exceeds an event count threshold, or,

an aggregate metric calculated from said plurality of instances of said data or said event data or both said data and said event data for the plurality of instances of said data or said event data or both said data and said event data occurring within said time range and said location range exceeds an aggregate metric threshold.

18. The system of claim 2, wherein said computer is further configured to

detect a first value from said one or more values having a first threshold value and

detect a second value from said one or more values having a second threshold value within a time window;

signify a prospective event;

compare said prospective event to a characteristic signal associated with a typical event and eliminate any false positive events;

signify a valid event if said prospective event is not a false positive event; and

,

save said valid event in said memory comprising information within an event time window as said data.

19. The system of claim 2, wherein said computer is further configured to

obtain at least one video from one or more of

a camera or smart glasses associated with said user,

said computer,

said mobile device,

said server, and

another server;

obtain at least one video start time and at least one video stop time associated with said at least one video; and

,

synchronize said event data with said at least one video based on

said event start time and said event stop time, and

at least one time associated with said at least one video, to

create at least one synchronized event video comprising both of

said event data that occurs during a timespan from said event start time to said event stop time;

and,

said at least one video captured during said timespan from said event start time to said event stop time.

20. The system of claim 19, wherein said computer is further configured to

accept a metric or one or more tags associated with said at least one synchronized event video;

accept selection criteria for said metric or said one or more tags;

determine a matching set of synchronized event videos that have a value or values associated with said metric or with said one or more tags that pass said selection criteria; and

,

display said matching set of synchronized event videos or corresponding thumbnails thereof along with said value or values associated with said metric or with said one or more tags for each of said matching set of synchronized event videos or said corresponding thumbnails.

24. The system of claim 1, wherein said least one sensor configured to measure said property of said object wherein said one or more values are associated with a chemical value.

21. A sensor and social media integration system comprising:

a computer comprising

a computer memory; and

,

a first communication interface configured to obtain data from at least one sensor configured to measure a property of an object;

wherein said computer is coupled with said computer memory and is coupled with said first communication interface, wherein said computer is configured to

receive

said data from said first communication interface and analyze said data and recognize an event within said data to determine event data,

or

said event data from said first communication interface,

or

both said data and said event data from said first communication interface;

store said event data in said computer memory;

obtain an event start time and an event stop time from said event data;

confirm said event recognized by said computer or obtained from said first communication interface for a particular location and time by analyzing one or more of text, audio, image, and video from a server to create a confirmed event, wherein said one or more of said text, audio, image, and video comprise one or more of email messages, voice calls, voicemails, audio recordings, video calls, video messages, video recordings, text messages, chat messages, postings on social media sites, postings on blogs, and postings on wikis; and

,

publish said confirmed event.

22. The system of claim 21, wherein said object comprises one or more of

a person, a group of people, a body part of said person or of said group of people, a food, a drink, a plant, an animal, a piece of equipment, a machine, an automobile, a vehicle, an engine, a building, a room, an area, and a body of water.

23. The system of claim 21, wherein said property comprises one or more of

acoustic pressure, acoustic power, acoustic frequency, sound, vibration, seismic activity, air flow, fluid flow, mass flow, oxygen level, hydrogen level, ozone level, pH, smoke level, carbon dioxide level, carbon monoxide level, chemical composition, ionization level, chemical reaction rate, radiation level, charge, electric current, electric potential, resistance, conductance, capacitance, inductance, impedance, electromagnetic field, electromagnetic frequency, wavelength, Doppler shift, light level, particle count, photon count, amplitude, temperature, moisture, humidity, barometric pressure, pollution level, precipitation level, tide level, wind velocity, mass, weight, density, position, depth, altitude, displacement, proximity, presence, orientation, angle, inclination, tilt, shock, strain, mileage, velocity, speed, angular velocity, acceleration, angular acceleration, force, torque, momentum, revolutions per minute, heart rate, blood pressure, body temperature, blood composition, body fluid composition, tissue composition, oxygen saturation, and respiration rate.