METHOD AND APPARATUS FOR SENSOR AIDED EXTRACTION OF SPATIO-TEMPORTAL FEATURES

• US 20140212002A1
• Filed: 01/15/2014
• Published: 07/31/2014
• Est. Priority Date: 01/30/2013
• Status: Active Grant

• Current Assignee: Nokia Technologies Oy (Nokia Corporation)
• Original Assignee: Nokia Corporation
• Inventors: Curcio, Igor Danilo Diego, Mate, Sujeet Shyamsundar, Cricri, Francesco
• Granted Patent: US 9,471,993 B2
• Time in Patent Office: Days
• US Class Current: 382/107
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors

Abstract

A method, apparatus and computer program product are provided for extracting spatio-temporal features with the aid of sensor information. An exemplary method comprises receiving video data and auxiliary sensor data and associating the two with timestamp information. The method may also include segmenting an input data stream into stable segments and extracting temporal features from the associated video data. The method may further include extracting temporal features either form the whole video or only from the video data where little or no stable segments are detected and performing camera view motion compensation by using information provided by the auxiliary sensors for modifying the feature-descriptors.

Claims

1. A method comprising:

receiving video data captured from a camera device;

receiving sensor data captured from auxiliary sensors associated with the camera device;

causing association of the sensor data and the video data using time data, wherein each of one or more samples of sensor data is associated with a single video frame;

detecting camera view motion in the sensor data;

causing, utilizing a processor, segmentation of the sensor data into stable segments; and

extracting one or more temporal features from the video data associated with the stable segments of the sensor data.

2. The method according to claim 1 further comprising:

performing orientation compensation on the one or more temporal features.

3. The method according to claim 1, wherein sensor data describes both camera motion and ZOOM.

4. The method according to claim 1, further comprising determining a quality level of the sensor data from one or more auxiliary sensors and adapting the extraction of the one or more temporal features according to the detected quality levels.

5. The method according to claim 1, wherein the temporal features extracted from the video data are space-time interest points.

6. The method according to claim 1, wherein temporal features extracted from the video data are one or more dense trajectories.

7. The method according to claim 1 wherein, when less than a predetermined threshold of camera view-motion is detected, or when less than a predetermined threshold of video data associated with the stable segments is present, the method further comprising:

causing extraction of one or more temporal features from the video data;

performing orientation compensation on the one or more temporal features;

causing extraction of a direction and compensation amount from at least a portion of the sensor data; and

performing camera view motion compensation on the orientation compensated temporal features using the direction and compensation amount.

8. The method according claim 7, wherein the orientation compensation may comprise aligning one or more descriptors with respect to a global vertical direction.

9. The method according to claim 7, wherein camera view motion compensation comprises identifying a histogram bin describing an orientation to be modified according to the direction information and modifying the histogram bin according to the compensation amount.

10. An apparatus comprising at least one processor and at least one memory storing computer program code, wherein the at least one memory and stored computer program code are configured, with the at least one processor, to cause the apparatus to:

receive video data captured from a camera device;

receive sensor data captured from auxiliary sensors associated with the camera device;

cause association of the sensor data and the video data use time data, wherein each of one or more samples of sensor data is associated with a single video frame;

detect camera view motion in the sensor data and cause segmentation of the sensor data into stable segments; and

extract one or more temporal features from the video data associated with the stable segments of the sensor data.

11. The apparatus according to claim 10, wherein the at least one memory and stored computer program code are further configured, with the at least one processor, to cause the apparatus to:

perform orientation compensation on the one or more temporal features.

12. The apparatus according to claim 10, wherein sensor data describes both camera motion and zoom.

13. The apparatus according to claim 10, wherein the at least one memory and stored computer program code are further configured, with the at least one processor, to cause the apparatus to determine a quality level of the sensor data from one or more auxiliary sensors and adapt the extraction of the one or more temporal features according to the detected quality levels.

14. The apparatus according to claim 10, wherein the temporal features extracted from the video data are space-time interest points.

15. The apparatus according to claim 10, wherein the temporal features extracted from the video data are one or more dense trajectories.

16. The apparatus according to claim 10,

wherein, when less than a predetermined threshold of camera view motion is detected, or when less than a predetermined threshold of video data associated with the stable segments is present, the at least one memory and stored computer program code are further configured, with the at least one processor, to cause the apparatus to cause extraction of one or more temporal features from the video data;

perform orientation compensation on the one or more temporal features;

cause extraction of a direction and compensation amount from at least a portion of the sensor data; and

perform camera view motion compensation on the compensated temporal features use the direction and compensation amount.

17. The apparatus according to claim 16, wherein the orientation compensation may comprise aligning one or more descriptors with respect to a global vertical direction.

18. The apparatus according to claim 16, wherein camera view motion compensation comprises identifying a histogram bin describing an orientation to be modified according to the direction information and modifying the histogram bin according to the compensation amount.

19. A computer program product comprising at least one non-transitory computer-readable storage medium having computer-readable program instructions stored therein, the computer-readable program instructions includes program instructions configured for:

receiving video data captured from a camera device;

receiving sensor data captured from auxiliary sensors associated with the camera device;

causing association of the sensor data and the video data using time data, wherein each of one or more samples of sensor data is associated with a single video frame;

detecting camera view motion in the sensor data;

causing, utilizing a processor, segmentation of the sensor data into stable segments; and

extracting one or more temporal features from the video data associated with the stable segments of the sensor data.

20. A computer program product according to claim 19, wherein the computer-readable program instructions includes program instructions configured for:

when less than a predetermined threshold of camera view-motion is detected, or when less than a predetermined threshold of video data associated with the stable segments is present,

causing extraction of one or more temporal features from the video data;

performing orientation compensation on the one or more temporal features;

causing extraction of a direction and compensation amount from at least a portion of the sensor data; and

performing camera view motion compensation on the orientation compensated temporal features using the direction and compensation amount.