Efficient video data representation and content based video retrieval framework

• US 10,462,490 B2
• Filed: 11/06/2015
• Issued: 10/29/2019
• Est. Priority Date: 11/06/2015
• Status: Active Grant

• Current Assignee: Raytheon Company (Rtx Corporation)
• Original Assignee: Raytheon Company (Rtx Corporation)
• Inventors: Pan, Jingning
• Primary Examiner(s): Perungvoor, Sathyanaraya V
• Assistant Examiner(s): Dang, Philip P.
• Application Number:
  US14/934,618
• Publication Number:
  US 20170134754A1
• Time in Patent Office: 1,453 Days
• CPC Class Codes: G06F 16/7837 using objects detected or recognised in the video content
• CPC Class Codes: H04N 19/179 the unit being a scene or a shot
• CPC Class Codes: H04N 19/23 with coding of regions that are present throughout a whole video segment, e.g. sprites, background or mosaic
• CPC Class Codes: H04N 19/63 using sub-band based transform, e.g. wavelets

Abstract

An efficient object-detection-driven video data representation along with a unified content based video compression and retrieval framework and wavelet based searching engine is described to encode the disclosed video data representation. The video data representation and unified compression and retrieval and searching engine together facilitate rapid retrieval of desired video information from large stores of video data.

Claims

1. A method, executed by one or more processors, for processing a series of image frames in a video data to compress, store, search and/or retrieve images of interest in the video data, the method comprising:

performing a segmenting process on the video data, wherein the segmenting process distinguishes different portions of the video data representing different objects and background images;

performing a wavelet transform process on each of the different portions to compress the video data, wherein the wavelet transform process encodes each of the different objects and background images in a respective wavelet pyramid representation;

identifying one or more features associated with each of the different objects and background images; and

encoding the one or more features in association with the respective encoded wavelet pyramid representation of their respective different objects and background images to generate a feature indexed wavelet pyramid representation of the video data, wherein the feature indexed wavelet period representation comprises the respective wavelet pyramid representations and a separate encoded index of attributes associated with the objects encoded in the respective wavelet pyramid representation, wherein the attributes are in the group consisting of object detection ID, scheme, value, tracking ID and tracking information of the objects.

2. The method of claim 1, further comprising:

detecting one or more shot boundaries using the feature indexed wavelet pyramid representation of the video data without decoding the feature indexed wavelet pyramid representation of the video data, wherein each of the shot boundaries separates different shots in the video data, and wherein each of the different shots represents a respective sequence of similar frames of the video data.

3. The method of claim 2, comprising:

partitioning the video data into one or more sets of the different shots.

4. The method of claim 3, comprising:

detecting the one or more shot boundaries in the video data by identifying abrupt transitions of wavelet coefficients from frame to frame in the feature indexed wavelet pyramid representation of the video data.

5. The method of claim 4, wherein the wavelet coefficients are associated with a key object in the respective sequence of similar frames.

6. The method of claim 3 comprising:

detecting the one or more shot boundaries in the video data by identifying abrupt transitions of the one or more features from frame to frame in the wavelet pyramid representation of the video data.

7. The method of claim 6, wherein the one or more features are associated with a key object in the respective sequence of similar frames.

8. The method of claim 6, further comprising:

identifying a key frame in each of the different shots that includes one or more key objects and/or key features that are common to the respective sequence of similar frames within the respective shot; and

identifying an association between each of the key frames and their corresponding shots.

9. The method of claim 8, further comprising:

storing the partitioned video data based on the association.

10. The method of claim 9, further comprising:

locating a portion of the video data associated with a selected one or more of the features by searching for the selected one or more of the feature in only the key frames for portions of the video data associated with the feature without decoding the feature indexed wavelet pyramid representation of the video data;

identifying one or more of the key frames that are associated with the selected one or more features; and

retrieving a respective shot represented by the one or more of the key frames that are associated with the one or more features.

11. The method of claim 9, further comprising:

locating a portion of the video data associated with a selected one or more of the objects by searching the wavelet pyramid representation of only the key frames for video data associated with the selected one or more of the objects;

identifying one or more of the key frames that are associated with the selected one or more of the objects; and

retrieving a respective shot represented by the one or more of the key frames that are associated with the one or more of the objects.

12. A system for processing a sequence of images to compress, store, search and/or retrieve images of interest in the video data, the system comprising:

one or more processors in communication with a data storage device, the one or more processors configured to;

perform a segmenting process on the video data, wherein the segmenting process distinguishes different portions of the video data representing different objects and background images;

perform a wavelet transform process on each of the different portions to compress the video data, wherein the wavelet transform process encodes each of the different objects and background images in a respective wavelet pyramid representation;

identify one or more features associated with each of the different objects and background images; and

encode and index the one or more features in association with the respective wavelet pyramid representation of their respective different objects and background images to generate a feature indexed wavelet pyramid representation of the video data, wherein the feature indexed wavelet period representation comprises the respective wavelet pyramid representations and a separate encoded index of attributes associated with the objects encoded in the respective wavelet pyramid representation, wherein the attributes are in the group consisting of object detection ID, scheme, value, tracking ID and tracking information of the objects.

13. The system of claim 12 wherein the one or more processors are further configured to detect one or more shot boundaries in the feature indexed wavelet pyramid representation of the video data, wherein each of the shot boundaries separates different shots in the video data, and wherein each of the different shots represents a respective sequence of similar frames of the video data.

14. The system of claim 13 wherein the one or more processors are further configured to detect the one or more shot boundaries in the video data by identifying abrupt transitions of wavelet coefficients from frame to frame in the feature indexed wavelet pyramid representation of the video data, wherein the wavelet coefficients are associated with a key object in the respective sequence of similar frames.

15. The system of claim 13 wherein the one or more processors are further configured to detect the one or more shot boundaries in the video data by identifying abrupt transitions of the one or more features from frame to frame in the wavelet pyramid representation of the video data, wherein the one or more features are associated with a key object in the respective sequence of similar frames.

16. The system of claim 13 wherein the one or more processors are further configured to

identify a key frame in each of the different shots that includes one or more key objects and/or key features that are common to the respective sequence of similar frames within the respective shot;

identifying an association between each of the key frames and their corresponding shots.

17. The system of claim 16 wherein the one or more processors are further configured to store the video data based on the association.

18. The system of claim 17 wherein the one or more processors are further configured to

locate a portion of the video data associated with a selected one or more of the objects by searching the wavelet pyramid representation of only the key frames for video data associated with the selected one or more of the objects;

identify one or more of the key frames that are associated with the selected one or more of the objects; and

retrieve a respective shot represented by the one or more of the key frames that are associated with the one or more of the objects.

19. A non-transitory computer-readable medium having computer readable program codes embodied thereon for processing a sequence of images to compress, store, search and/or retrieve images of interest in the video data, the computer-readable codes including instructions that, when executed by a processor, cause the processor to:

perform a segmenting process on the video data, wherein the segmenting process distinguishes different portions of the video data representing different objects and background images;

perform a wavelet transform process on each of the different portions to compress the video data, wherein the wavelet transform process encodes each of the different objects and background images in a respective wavelet pyramid representation;

identify one or more features associated with each of the different objects and background images; and

encode the one or more features in association with the respective encoded wavelet pyramid representation of their respective different objects and background images to generate a feature indexed wavelet pyramid representation of the video data, wherein the feature indexed wavelet period representation comprises the respective wavelet pyramid representations and a separate encoded index of attributes associated with the objects encoded in the respective wavelet pyramid representation, wherein the attributes are in the group consisting of object detection ID, scheme, value, tracking ID and tracking information of the objects.

20. The non-transitory computer-readable medium of claim 19 further including computer-readable codes including instructions that, when executed by a processor, cause the processor to:

encode each object of each frame in the motion compensation method.

21. The non-transitory computer-readable medium of claim 20 further including computer-readable codes including instructions that, when executed by a processor, cause the processor to:

detect the one or more shot boundaries in the video data by identifying abrupt transitions of wavelet coefficients from frame to frame in the feature indexed wavelet pyramid representation of the video data, wherein the wavelet coefficients are associated with a key object in the respective sequence of similar frames; and

identify a plurality of partitions of the video data separated by the shot boundaries.

22. The non-transitory computer-readable medium of claim 21 further including computer-readable codes including instructions that, when executed by a processor, cause the processor to:

identify a key frame in each of the different shots that includes one or more key objects and/or key features that are common to the respective sequence of similar frames within the respective shot;

indexing an association between each of the key frames and their corresponding shots; and

storing the video data based on the indexing.

23. The non-transitory computer-readable medium of claim 22 further including computer-readable codes including instructions that, when executed by a processor, cause the processor to:

locate a portion of the video data associated with a selected one or more of the objects by searching the wavelet pyramid representation of only the key frames for video data associated with the selected one or more of the objects without decoding the feature indexed wavelet pyramid representation of the video data;

identify one or more of the key frames that are associated with the selected one or more of the objects; and

retrieve a respective shot represented by the one or more of the key frames that are associated with the one or more of the objects.

24. The non-transitory computer-readable medium of claim 22 further including computer-readable codes including instructions that, when executed by a processor, cause the processor to:

locate a portion of the video data associated with a selected one or more of the features by searching for the selected one or more of the feature in only the key frames for portions of the video data associated with the feature without decoding feature indexed wavelet pyramid representation of the video data;

identify one or more of the key frames that are associated with the selected one or more features; and

retrieve a respective shot represented by the one or more of the key frames that are associated with the one or more features.