Video hashing system and method

• US 8,897,512 B1
• Filed: 07/22/2013
• Issued: 11/25/2014
• Est. Priority Date: 03/07/2007
• Status: Expired due to Fees

• Current Assignee: Cubic Corporation
• Original Assignee: MotionDSP, Inc. (Cubic Corporation)
• Inventors: Bozinovic, Nikola, Velickovic, Ivan, Vuckovic, Ivan, Andjelkovic, Sasa, Andjelkovic, Nebojsa, Djordjevic, Danilo
• Primary Examiner(s): Liew, Alex
• Application Number:
  US13/947,366
• Time in Patent Office: 491 Days
• Field of Search: 382124-127
• US Class Current: 382/124
• CPC Class Codes: G06F 16/783 using metadata automatically derived from the content
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames

Abstract

In one embodiment, a method of operating a processing system includes receiving a reference video and calculating global camera motion parameters for a plurality of the reference video'"'"'s frames. The method further includes using the camera motion parameters to identify reference video frames corresponding to motion-based events, generating hash values for the identified and neighboring frames in the reference video; and storing the identified frames and hash values. In one version, the method also includes receiving a subject video, calculating global camera motion parameters for the frames of the subject video, using these parameters to identify frames of motion-based events in the subject video, and generating hash values for the identified and neighboring frames in the subject video. The identified frames and hash values of the subject can be compared with those stored for the reference video to evaluate similarity.

Claims

1. A method of operating a first processing system that includes one or more processors and storage, the method comprising:

providing as input to the processing system a reference video comprising an ordered time sequence of frames;

calculating using at least one of the processors global camera motion parameters for a plurality of the frames of the reference video;

identifying events using at least one of the processors, including identifying frames corresponding to scene-change-based and/or motion-based and/or sub-sampling-based events in the reference video, the identifying the frames of motion-based events using the global camera motion parameters calculated for the reference video;

generating using at least one of the processors hash values for the identified frames and their neighboring frames in the reference video, the hash values comprising, for each identified event, one or more binary descriptors for one or more frames of the frames that correspond to the identified event; and

storing in in the storage the identified frames of the reference video and the generated hash values for the reference video,

wherein the identifying the frames in the reference video includes identifying frames corresponding to one or more of a change-of-motion-based reference-video event identified by detecting a change of motion in the sequence, a sub-sampling-based reference-video event determined by sub-sampling in time the ordered time sequence according to a pre-determined sampling profile, and a detected change-of-scene-based reference-video event,

such that the stored identified frames and stored hash values are retrievable to be used for evaluating a measure of similarity of a video and the reference video.

2. A method as recited in claim 1, further comprising additional steps carried out by hardware comprising the first processing system or a second processing system coupled to the first processing system, or a:

receiving as input a subject video comprising a time sequence of subject-video frames;

calculating global camera motion parameters for the subject video for each frame of the subject video;

identifying subject video frames corresponding to events including at least motion-based subject video events in the subject video, the identifying of the subject video frames corresponding to the motion-based subject video events using the global camera motion parameters calculated for the subject video,

wherein the identifying the frames in the subject video includes identifying frames corresponding to one or more of a change-of-motion-based subject-video event identified by detecting a change of motion in the sequence, a sub-sampling-based subject-video event determined by sub-sampling in time the ordered time sequence according to the pre-determined sampling profile, and a detected change-of-scene-based subject-video event;

generating subject video hash values for the identified subject video frames and their neighboring frames in the subject video, using the global camera motion parameters calculated for the subject video; and

evaluating a measure of similarity of the subject video and the reference video by a process that includes comparing identified subject video frames and the generated subject video hash values with the stored identified reference-video frames and stored reference-video hash values.

3. A method of operating a processing system, comprising:

providing as input to the processing system a first fingerprint generated by a process comprising video hashing a first video data set, the set comprising a first ordered time sequence of frames of a first video, the first fingerprint comprising one or more video hash values of the first video data set,

providing as input to the processing system a second video data set comprising a second ordered time sequence of frames of a second video;

generating in the processing system a second fingerprint of the second video data set by a process comprising video hashing the second video data set, the second fingerprint comprising one or more video hash values of the second video data set; and

comparing on the processing system the first and second fingerprints of the first and second video data sets,

wherein the video hashing of a video data set that comprises an ordered time sequence of frames of video includes;

analyzing the video data set to identify one or more events in the ordered time sequence, an identified event comprising one or more of a change-of-motion-based event identified by detecting a change of motion in the sequence, a sub-sampling-based event determined by sub-sampling in time the ordered time sequence according to a pre-determined sampling profile, and a detected change-of-scene-based event; and

generating, for each identified event, one or more binary descriptors for one or more frames of the video data set that correspond to the identified event.

4. A method as recited in claim 3 wherein analyzing the video data set comprises calculating global camera motion parameters for a plurality of frames of the video data set, and identifying frames corresponding to change-in-motion-based events including detecting change in motion using the global camera motion parameters calculated for the video data set.

5. A method as recited in claim 3 wherein analyzing the video data set comprises sub-sampling in time the ordered time sequence according to a pre-determined sampling profile, such that the generating includes calculating binary descriptors for a plurality of frames of the first video data set corresponding to the sub-sampling.

6. A method as recited in claim 3 wherein analyzing the video data set comprises identifying scene-cut events and frames corresponding to the scene-cut events by detecting changes of scene in the video data set.

7. A method as recited in claim 3 wherein generating one or more binary descriptors for one or more frames of the video data set that correspond to an identified event comprises:

dividing a video frame corresponding to the identified event into tiles;

calculating the average luminance for each tile; and

applying a global threshold across the tiles to form a binary word comprising a plurality of bits, wherein each bit corresponds to one of the tiles, and wherein a first bit value indicates an average luminance of the tile above the global threshold, while the opposite bit value to the first bit value indicates an average luminance of the tile below or equal to the global threshold.

8. A method as recited in claim 3 wherein comparing the first and second fingerprints comprises correlating a first time line of events and a second time line of events indicative of the times of identified events of the first and second fingerprints, respectively.

9. A method as recited in claim 3 wherein comparing the first and second fingerprints comprises correlating a first time line of events and a second time line of events indicative of the times of identified events of the first and second fingerprints, respectively, matching identified events from the first time line to identified events of the second time line, and comparing binary descriptors corresponding to frames associated with the matched identified events.

10. A method as recited in claim 3 wherein comparing the first and second fingerprints comprises generating a score characterizing a degree to which the first and second fingerprints match.

11. One or more non-transitory computer-readable media comprising computer-readable instructions operative, when executed, to cause one or more processors of a processing system to:

accept as input to the processing system a first fingerprint generated by a process comprising video hashing a first video data set, the set comprising a first ordered time sequence of frames of a first video, the first fingerprint comprising one or more video hash values—

of the first video data set;

accept as input to the processing system a second video data set comprising a second ordered time sequence of frames of a second video;

generate a second fingerprint of the second video data set by a process comprising video hashing the second video data set, the second fingerprint comprising one or more video hash values of the second video data set; and

compare the first and second fingerprints of the first and second video data sets,

wherein the video hashing of a video data set that comprises an ordered time sequence of frames of video includes;

analyzing the video data set to identify one or more events in the ordered time sequence, an identified event comprising one or more of a change-of-motion-based event identified by detecting a change of motion in the sequence, a sub-sampling-based event determined by sub-sampling in time the ordered time sequence according to a pre-determined sampling profile, and a detected change-of-scene-based event; and

generating, for each identified event, one or more binary descriptors for one or more frames of the video data set that correspond to the identified event.

12. One or more non-transitory computer-readable media as recited in claim 11 wherein analyzing the video data set comprises calculating global camera motion parameters for a plurality of frames of the video data set, and identifying frames corresponding to change-in-motion-based events including detecting change in motion using the global camera motion parameters calculated for the video data set.

13. One or more non-transitory computer-readable media as recited in claim 11 wherein analyzing the video data set comprises sub-sampling in time the ordered time sequence according to a pre-determined sampling profile, such that the generating includes calculating binary descriptors for a plurality of frames of the first video data set corresponding to the sub-sampling.

14. One or more non-transitory computer-readable media as recited in claim 11 wherein analyzing the video data set comprises identifying scene-cut events and frames corresponding to the scene-cut events by detecting changes of scene in the video data set.

15. One or more non-transitory computer-readable media as recited in claim 11 wherein generating one or more binary descriptors for one or more frames of the video data set that correspond to an identified event comprises:

dividing a video frame corresponding to the identified event into tiles;

calculating the average luminance for each tile; and

applying a global threshold across the tiles to form a binary word comprising a plurality of bits, wherein each bit corresponds to one of the tiles, and wherein a first bit value indicates an average luminance of the tile above the global threshold, while the opposite bit value to the first bit value indicates an average luminance of the tile below or equal to the global threshold.

16. One or more non-transitory computer-readable media as recited in claim 11 wherein comparing the first and second fingerprints comprises correlating a first time line and a second time line indicative of the times of identified events of the first and second fingerprints, respectively.

17. One or more non-transitory computer-readable media as recited in claim 11 wherein comparing the first and second fingerprints comprises correlating a first time line and a second time line indicative of the times of the identified events of the first and second fingerprints, respectively, matching identified events from the first time line to identified events of the second time line, and comparing binary descriptors corresponding to frames associated with the matched identified events.

18. One or more non-transitory computer-readable media as recited in claim 11 wherein comparing the first and second fingerprints comprises generating a score characterizing a degree to which the first and second video fingerprints match.

19. A system comprising:

a hardware system comprising one or more processors and one or more non-transitory storage media;

wherein the one or more storage media are configured with instructions that, when executed, cause the hardware system to be operative to;

receive as input a reference video comprising a time sequence of frames;

calculate global camera motion parameters for a plurality of the frames of the reference video;

identify events using at least one of the processors, the identifying including including—

frames corresponding to scene-change-based and/or motion-based an/or sub-sampling-based events in the reference video, the identifying of the motion-based events using the global camera motion parameters calculated for the reference video;

generate hash values for the identified frames and their neighboring frames in the reference video, the hash values comprising, for each identified event, one or more binary descriptors for one or more frames of the frames that correspond to the identified event; and

store the identified frames of the reference video in and the generated hash values for the reference video, the storing in at least one of the one or more storage media,

wherein the identifying the frames in the reference video includes identifying frames corresponding to one or more of a change-of-motion-based reference-video event identified by detecting a change of motion in the sequence, a sub-sampling-based reference-video event determined by sub-sampling in time the ordered time sequence according to a pre-determined sampling profile, and a detected change-of-scene-based reference-video event,

such that the stored identified frames and stored hash values are retrievable to be used for evaluating a measure of similarity of a video and the reference video.

20. A system as recited in claim 19, wherein the instructions when executed, further cause the hardware system to be operative to:

receive as input a subject video comprising a time sequence of subject-video frames;

calculate global camera motion parameters for the subject video for each frame of the subject video;

identify subject video frames corresponding to scene-change-based subject video events and/or motion-based subject video events in the subject video, the identifying subject video frames of the motion-based subject video events using the global camera motion parameters calculated for the subject video,

wherein the identifying the frames in the subject video includes identifying frames corresponding to one or more of a change-of-motion-based subject-video event identified by detecting a change of motion in the sequence, a sub-sampling-based subject-video event determined by sub-sampling in time the ordered time sequence according to the pre-determined sampling profile, and a detected change-of-scene-based subject-video event;

generate subject video hash values for the identified subject video frames and their neighboring frames in the subject video, using the global camera motion parameters for the subject video; and

evaluate a measure of similarity of the subject video and the reference video by a process that includes comparing identified subject video frames and the generated subject video hash values with the stored identified reference-video frames and stored reference-video hash values.

21. A system as recited in claim 19, wherein the one or more storage media include at least one memory, and at least one mass storage device, local or remote coupled to the hardware system via a network.