AUTOMATIC DIGITAL CURATION AND TAGGING OF ACTION VIDEOS

• US 20130330054A1
• Filed: 01/04/2013
• Published: 12/12/2013
• Est. Priority Date: 06/11/2012
• Status: Active Grant

• Current Assignee: AlpineReplay, Inc.
• Original Assignee: AlpineReplay, Inc.
• Inventors: Lokshin, David J.
• Granted Patent: US 8,929,709 B2
• Time in Patent Office: Days
• US Class Current: 386/227
• CPC Class Codes: G06F 16/78 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/7867 using information manually generated, e.g. tags, keywords, comments, title and artist information, manually generated time, location and usage information, user ratings
• CPC Class Codes: G06F 16/787 using geographical or spatial information, e.g. location
• CPC Class Codes: G06V 20/42 of sport video content
• CPC Class Codes: G06V 20/70 Labelling scene content, e.g. deriving syntactic or semantic representations
• CPC Class Codes: G11B 20/00 Signal processing not specific to the method of recording or reproducing; Circuits therefor
• CPC Class Codes: G11B 20/10527 Audio or video recording; Data buffering arrangements G11B20/12 - G11B20/18 take precedence
• CPC Class Codes: G11B 2020/10537 Audio or video recording
• CPC Class Codes: G11B 27/031 Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: G11B 27/10 Indexing; Addressing; Timing or synchronising; Measuring tape travel
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence
• CPC Class Codes: G11B 27/32 on separate auxiliary tracks of the same or an auxiliary record carrier
• CPC Class Codes: H04N 21/42202 environmental sensors, e.g. for detecting temperature, luminosity, pressure, earthquakes
• CPC Class Codes: H04N 21/4524 involving the geographical location of the client retrieval from the Internet by querying based on geographical locations G06F16/9537; systems specially adapted for using geographical information in broadcast systems H04H60/70; protocols in which the network application is adapted for the location of the user terminal in communication control or processing H04L67/52; services making use of the location of users or terminals in wireless networks H04W4/02; locating users or terminals in wireless networks H04W64/00
• CPC Class Codes: H04N 21/8133 specifically related to the content, e.g. biography of the actors in a movie, detailed information about an article seen in a video program
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 5/77 between a recording apparatus and a television camera
• CPC Class Codes: H04N 5/772 the recording apparatus and the television camera being placed in the same enclosure
• CPC Class Codes: H04N 9/79 Processing of colour television signals in connection with recording
• CPC Class Codes: H04N 9/8205 involving the multiplexing of an additional signal and the colour video signal

Abstract

A system and method for automatic digital curation, annotation, and tagging of action videos receives, in a processing system, GPS data, and in some embodiments inertial data, from a sensor-based device carried by a sportsman during a sporting activity. The processing system processes the GPS data and the inertial data, if present to identify at least one sportsman event, and stores data identifying the sportsman event in a performance database. Video data from a video camera is stored in a video database, the video data including location, time, and direction information associated with frames of video. Timecode data in the video database is synchronized with timecode data in the performance database, and the data identifying the sportsman event is used to automatically select, annotate, tag or edit said video data.

Claims

1. A method, comprising:

receiving, in a processing system, GPS data from a device carried by a user or attached to sports equipment during a sporting activity, said GPS data comprising at least one of;

position data or time data;

processing said GPS data in said processing system to identify at least one event engaged in by the user;

storing data identifying said at least one event in a performance record;

storing video data in a video record, the video data having meta data comprising at least time and location information associated with frames of video;

synchronizing timecode data in said video records with timecode data in said performance records; and

,

using said data identifying at least one event to automatically select, annotate, tag or edit said video data.

2. The method according to claim 1, wherein said device carried by a user comprises a sensor-based device, wherein said receiving step comprises receiving inertial data from the sensor-based device, and wherein said processing step comprises processing said inertial data.

3. The method according to claim 1, wherein the user is a sportsman and the event is a sporting event.

4. The method according to claim 1, wherein said performance record comprises a performance database.

5. The method according to claim 1, wherein said video record comprises a video database.

6. The method according to claim 1, wherein said metadata further comprises direction information.

7. The method according to claim 1, wherein said step of synchronizing timecode data in said video records with timecode data in said performance records is conducted using GPS Time stamps.

8. The method according to claim 1, wherein said step of synchronizing timecode data in said video records with timecode data in said performance records is conducted using wireless network time stamps

9. The method according to claim 1, wherein said at least one event identified by said processing system comprises an event selected from the set consisting of:

a jump, a flip, a rotation, a high speed portion, a turn, or a fast turn.

10. The method according to claim 1, wherein the device is a portable device comprising at least one GPS sensor and at least one inertial sensor.

11. The method according to claim 1, wherein said processing system is embedded in said device.

12. The method according to claim 1, wherein said processing system is a separate device from said device.

13. The method according to claim 1, wherein said processing system is a server.

14. The method according to claim 2, wherein said GPS data and said inertial data are stored in said performance database.

15. The method according to claim 1, wherein said device is co-located with a camera that generates said video data.

16. The method according to claim 1, wherein said device is separate and independent from a camera that generates said video data.

17. The method according to claim 1, wherein said performance database and said video database are the same database, with separate records for performance data and for video data.

18. The method according to claim 1, further comprising using additional meta data to speed up matching of performance and video data.

19. The method according to claim 16, wherein said step of using additional meta data comprises using a user'"'"'s name or ID to simplify a search.

20. A system, comprising:

a device configured to be carried by a user during a sporting activity, said device having at least one GPS sensor that generates GPS data;

a video camera configured to generate video data; and

,

a processing system configured to;

(a) process said GPS data to identify at least one event engaged in by the user;

(b) store data identifying said at least one event in a performance database;

(c) access video data in a video database, the video data comprising location, time, and direction information associated with frames of video;

(d) synchronize timecode data in said video database with timecode data in said performance database; and

,

(e) use said data identifying at least one event to automatically select, annotate, tag or edit said video data.

21. The system of claim 20, wherein said device to be carried by a user comprises a sensor-based device, wherein said receiving step comprises receiving inertial data from the sensor-based device, and wherein said processing step comprises processing said inertial data.

22. The system of claim 20, wherein the user is a sportsman and the event is a sporting event.

23. The system according to claim 20, wherein said device is configured to be carried attached to the user'"'"'s equipment or clothing.

24. The system according to claim 20, wherein said at least one event identified by said processing system comprises an event selected from the set consisting of:

a jump, a flip, a rotation, a high speed portion, a turn, or a fast turn.

25. The system according to claim 20, wherein the device is a portable device comprising at least one GPS sensor and at least one inertial sensor.

26. The system according to claim 20, wherein said processing system is embedded in said sensor-based device.

27. The system according to claim 20, wherein said processing system is a separate device from said sensor-based device.

28. The system according to claim 20, wherein said processing system is a server.

29. The system according to claim 20, wherein said GPS data and said inertial data are stored in said performance database.

30. The system according to claim 20, wherein said device is co-located with said video camera.

31. The system according to claim 20, wherein said device is separate and independent from said video camera.

32. The system according to claim 20, wherein said performance database and said video database are the same database, with separate records for performance data and for video data.

33. A computer program product comprising a non-transitory computer readable storage medium having computer readable program code embodied in the medium for implementing a method, the computer program product comprising:

program code for receiving, in a processing system, GPS data from a device carried by a user during a sporting activity, said GPS data comprising at least one of;

position data or time data;

program code for processing said GPS data in said processing system to identify at least one event engaged in by the user;

program code for storing data identifying said at least one event in a performance database;

program code for storing video data in a video database, the video data comprising location, time, and direction information associated with frames of video;

program code for synchronizing timecode data in said video database with timecode data in said performance database; and

,

program code for using said data identifying at least one event to automatically select, annotate, tag or edit said video data.