Systems and methods for identifying activities and/or events represented in a video

• US 10,534,966 B1
• Filed: 02/02/2017
• Issued: 01/14/2020
• Est. Priority Date: 02/02/2017
• Status: Active Grant

• Current Assignee: GoPro, Inc.
• Original Assignee: GoPro, Inc.
• Inventors: Tse, Daniel, Chik, Desmond, Wu, Guanhang
• Primary Examiner(s): Anyikire, Chikaodili E
• Application Number:
  US15/422,852
• Time in Patent Office: 1,076 Days
• Field of Search: None
• CPC Class Codes: G01P 13/00 Indicating or recording presence, absence, or direction, of movement electric switches H01H; counting moving objects G06M7/00
• CPC Class Codes: G06F 2218/12 Classification; Matching
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training

Abstract

Systems and method of identifying activities and/or events represented in a video are presented herein. An activity and/or event may be represented in a video by virtue of one or both of an entity moving with a capture device during capture of the video preforming the activity and/or event, or the video portraying one or more entities performing the activity and/or event. Activity types may be characterized by one or more of common movements, equipment, spatial context, and/or other features. Events may be characterized by one or both of individual movements and/or sets of movements that may routinely occur during performance of an activity. The identification of activities and/or events represented in a video may be based on one or more spectrogram representations of sensor output signals of one or more sensors coupled to a capture device.

Claims

1. A system configured to identify activities and/or events represented in a video, the system comprising:

one or more physical processors configured by machine-readable instructions to;

obtain motion sensor output signals generated from a motion sensor of a capture device configured to capture the video, the motion sensor output signals being generated contemporaneously with capture of the video by the capture device, the motion sensor output signals spanning a time duration, wherein the motion sensor output signals characterize motion of the capture device over time;

generate a set of spectrogram representations of the motion sensor output signals based on a transform of the motion sensor output signals to a frequency domain, the spectrogram representation characterizing the motion of the capture device over time within the frequency domain, individual spectrogram representations in the set of spectrogram representation being generated based on successions of transforms of the motion sensor output signals within time windows along the time duration, the time windows for an individual spectrogram representation having an individual time length that sets an individual time resolution of the individual spectrogram representation, the set of spectrogram representations of the sensor output signals including;

a first spectrogram representation generated based on successions of transforms of the motion sensor output signals within time windows having a first time length;

a second spectrogram representation generated based on successions of transforms of the motion sensor output signals within time windows having a second time length; and

a third spectrogram representation generated based on successions of transforms of the motion sensor output signals within time windows having a third time length;

encode the set of spectrogram representations into an image file, the individual spectrogram representations in the set of spectrogram representations being encoded into individual color channels of the image file, such that;

the first spectrogram representation is encoded into a first color channel of the image file;

the second spectrogram representation is encoded into a second color channel of the image file; and

the third spectrogram representation is encoded into a third color channel of the image file; and

identify, based on the image file, one or more activities and/or events represented in the video.

2. The system of claim 1, wherein the motion of the capture device is related to one or more of speed, acceleration, rate of rotation, or orientation.

3. The system of claim 1, wherein the transforms comprise Fourier transforms.

4. The system of claim 1, wherein individual activities are of one or more activity types, wherein an activity type is characterized by one or more of common movements, equipment, or spatial context.

5. The system of claim 4, wherein the one or more activity types include one or more of a sports type or a leisure type.

6. The system of claim 1, wherein individual events are characterized by one or both of individual movements or sets of movements that routinely occur during performance of an activity.

7. A method of identifying activities and/or events represented in a video, the method being implemented in a computer system comprising one or more physical processors and storage media storing machine-readable instructions, the method comprising:

obtaining motion sensor output signals generated from a motion sensor of a capture device configured to capture the video, the motion sensor output signals being generated contemporaneously with capture of the video by the capture device, the motion sensor output signals spanning a time duration, wherein the motion sensor output signals characterize motion of the capture device over time;

generating a set of spectrogram representations of the motion sensor output signals based on a transform of the motion sensor output signals to a frequency domain, the spectrogram representation characterizing the motion of the capture device over time within the frequency domain, individual spectrogram representations in the set of spectrogram representation being generated based on successions of transforms of the motion sensor output signals within time windows along the time duration, the time windows for an individual spectrogram representation having an individual time length that sets an individual time resolution of the individual spectrogram representation, the set of spectrogram representations of the sensor output signals including;

a first spectrogram representation generated based on successions of transforms of the motion sensor output signals within time windows having a first time length;

a second spectrogram representation generated based on successions of transforms of the motion sensor output signals within time windows having a second time length; and

a third spectrogram representation generated based on successions of transforms of the motion sensor output signals within time windows having a third time length;

encoding the set of spectrogram representations into an image file, the individual spectrogram representations in the set of spectrogram representations being encoded into individual color channels of the image file, including;

encoding the first spectrogram representation into a first color channel of the image file;

encoding the second spectrogram representation into a second color channel of the image file; and

encoding the third spectrogram representation into a third color channel of the image file; and

identifying, based on the image file, one or more activities and/or events represented in the video.

8. The method of claim 7, wherein the motion of the capture device is related to one or more of speed, acceleration, rate of rotation, or orientation.

9. The method of claim 7, wherein the transforms comprise Fourier transforms.

10. The method of claim 7, wherein individual activities are of one or more activity types, wherein an activity type is characterized by one or more of common movements, equipment, or spatial context.

11. The method of claim 10, wherein the one or more activity types include one or more of a sports type or a leisure type.

12. The method of claim 7, wherein individual events are characterized by one or both of individual movements or sets of movements that routinely occur during performance of an activity.