Systems, methods, and apparatus for image retrieval

• US 10,089,330 B2
• Filed: 12/18/2014
• Issued: 10/02/2018
• Est. Priority Date: 12/20/2013
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Gao, Dashan, Zhong, Xin, Kandhadai, Ananthapadmanabhan Arasanipalai, Laver, Steven Douglas
• Primary Examiner(s): Leroux, Etienne
• Application Number:
  US14/576,006
• Publication Number:
  US 20150178320A1
• Time in Patent Office: 1,384 Days
• CPC Class Codes: G06F 16/35 Clustering; Classification
• CPC Class Codes: G06F 16/56 having vectorial format
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06F 16/7335 Graphical querying, e.g. query-by-region, query-by-sketch, query-by-trajectory, GUIs for designating a person/face/object as a query predicate end-user interface involving hot spots associated with the video H04N21/4725; end-user interface for selecting a Region of Interest H04N21/4728
• CPC Class Codes: G06F 16/78 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/7837 using objects detected or recognised in the video content
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 40/279 Recognition of textual entities
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44

Abstract

A method of image retrieval includes obtaining information identifying a plurality of selected objects and selecting one among a plurality of candidate geometrical arrangements. This method also includes, by at least one processor, and in response to the selecting, identifying at least one digital image, among a plurality of digital images, that depicts the plurality of selected objects arranged according to the selected candidate geometrical arrangement.

Claims

1. A method of image retrieval, the method comprising:

obtaining, at a processor, information selecting a plurality of objects represented in a digital image, the digital image associated with a location space and the plurality of selected objects distributed about the location space;

selecting, by the processor, a particular candidate geometrical arrangement of the plurality of selected objects among multiple candidate geometrical arrangements of the plurality of selected objects;

determining, at the processor, a reference position based on the particular candidate geometrical arrangement of the plurality of selected objects;

generating, by the processor, metadata descriptive of the particular candidate geometrical arrangement and the reference position; and

searching, by the processor, a metadata index corresponding to a plurality of digital images to identify at least one digital image from the plurality of digital images, the at least one digital image identified based on comparison of the metadata with an entry of the metadata index such that the at least one digital image depicts the plurality of selected objects, relative to the reference position, arranged according to the particular candidate geometrical arrangement.

2. The method of image retrieval according to claim 1, wherein the obtained information includes, for each of the plurality of selected objects, a label that is associated with the object, wherein the searching includes searching for the labels within the metadata index that is associated with the plurality of digital images.

3. The method of image retrieval according to claim 1, wherein the searching comprises searching, within metadata index that is associated with the plurality of digital images, for a label that is associated with the particular candidate geometrical arrangement of the plurality of selected objects, and wherein the reference position is one of a center of mass of the particular candidate geometrical arrangement of the plurality of selected objects, a top pixel or a bottom pixel among the plurality of selected objects, a vertical projection of the center of mass of the particular candidate geometrical arrangement of the plurality of selected objects to a top of the particular candidate geometrical arrangement of the plurality of selected objects or a bottom of the particular candidate geometrical arrangement of the plurality of selected objects, an average of positions of each of the plurality of selected objects of the particular candidate geometrical arrangement of the plurality of selected objects, or a position of a particular one of the plurality of selected objects.

4. The method of image retrieval according to claim 1, wherein the at least one digital image includes a plurality of frames from a first video file and a plurality of frames from a second video file.

5. The method of image retrieval according to claim 1, wherein the obtaining information selecting the plurality of selected objects is performed using at least one of speech recognition and a touchscreen, the speech recognition associated with at least one of:

a name of an object of the plurality of selected objects,

a particular arrangement of the plurality of selected objects using descriptors, and

the touchscreen configured to enable one of;

selection of the object via an associated icon,

highlighting the object within the digital image, and

surrounding at least a portion of the object within a bounding box, and

wherein the digital image is associated with one or more of a sporting events, a social events, an art performance, and security or surveillance monitoring.

6. The method of image retrieval according to claim 1, wherein a selection of the plurality of selected objects is based upon one of a bounding box, a bounding ellipse, and a lasso being placed around each object or a portion of each object of the plurality of selected objects, wherein the one of the bounding box, the bounding ellipse, and the lasso indicates a location and a size within the location space.

7. The method of image retrieval according to claim 1, wherein reference position corresponds to:

a center of mass of an arrangement of the plurality of selected objects,

a top pixel representing one of the plurality of selected objects,

a bottom pixel representing one of the plurality of selected objects, or

a vertical projection of the center of mass.

8. The method of image retrieval according to claim 1, wherein reference position is based on a position of a particular one of the plurality of selected objects or an average of positions of the plurality of selected objects in a ground plane, and wherein the metadata is based on an approximation of a count of the plurality of selected objects.

9. The method of image retrieval according to claim 1, further comprising receiving, at the processor, a set of multiple candidate geometrical arrangements of the plurality of selected objects, wherein the location space is divided into a plurality of regions, and wherein the particular candidate geometrical arrangement is selected based on a ratio between a first count of one or more particular objects in a first region of the location space and a second count of the one or more particular objects in a second region of the location space.

10. The method of image retrieval according to claim 1, further comprising:

clustering, by the processor, two or more frames having geometric arrangements of the plurality of selected objects that are similar to the particular candidate geometric arrangement; and

generating, by the processor, metadata descriptive of a cluster of the two or more frames.

11. The method of image retrieval according to claim 10, wherein the location space is divided into a plurality of regions, wherein a codebook corresponds to a division scheme associated with the location space, and wherein the metadata is associated with multiple codebooks.

12. The method of image retrieval according to claim 1, wherein the plurality of selected objects includes a first object having first coordinate data, a second object having second coordinate data, and a third object having third coordinate data, and wherein generating the metadata comprises:

mapping the first coordinate data to a first codeword associated with a codebook, the first codeword representing a first region of the location space;

mapping the second coordinate data to a second codeword associated with the codebook, the second codeword representing a second region of the location space; and

mapping the third coordinate data to a third codeword associated with the codebook, the third codeword representing a third region of the location space,

wherein the metadata includes the first codeword, the second codeword, and the third codeword.

13. The method of image retrieval according to claim 12:

wherein the metadata indicates a region of the location space which includes the reference position;

wherein the location space is one of a pixel coordinate space, a ground plane of a scene space, a two-dimensional space, and a three-dimensional space;

wherein the first region, the second region, and the third region divide the location space into regions of unequal size, the first region having a high density of the plurality of selected objects and the second region having a low density of the plurality of selected objects; and

wherein the first region is smaller than the smaller region, and more codewords are associated with the first region than the second region.

14. A non-transitory computer-readable medium storing instructions that when executed by a processor cause the processor to:

obtain information selecting a plurality of objects represented in a digital image, the digital image associated with a location space and the plurality of selected objects distributed about the location space;

select a particular candidate geometrical arrangement of the plurality of selected objects among multiple candidate geometrical arrangements of the plurality of selected objects;

determine a reference position based on the particular candidate geometrical arrangement of the plurality of selected objects;

generate metadata descriptive of the particular candidate geometrical arrangement and the reference position; and

search a metadata index corresponding to a plurality of digital images to identify at least one digital image, from the plurality of digital images, the at least one digital image identified based on comparison of the metadata with an entry of the metadata index such that the at least one digital image depicts the plurality of selected objects, relative to the reference position, arranged according to the particular candidate geometrical arrangement.

15. The non-transitory computer-readable medium according to claim 14, wherein the obtained information includes, for each object of the plurality of selected objects, a label that is associated with the object, and wherein the instructions, when executed by the processor, cause the processor to search for the labels within the metadata index that is associated with the plurality of digital images.

16. The non-transitory computer-readable medium according to claim 14, wherein the instructions, when executed by the processor, cause the processor to search, within the metadata index that is associated with the plurality of digital images, for a label that is associated with the particular candidate geometrical arrangement of the plurality of selected objects.

17. The non-transitory computer-readable medium according to claim 14, wherein the at least one digital image includes a plurality of frames from a first video file and a plurality of frames from a second video file.

18. The non-transitory computer-readable medium according to claim 14, wherein the information selecting the plurality of selected objects is obtained using at least one of speech recognition and a touchscreen.

19. An apparatus for image retrieval, the apparatus comprising:

means for obtaining information selecting a plurality of objects represented in a digital image, the digital image associated with a location space and the plurality of selected objects distributed about the location space;

means for selecting a particular candidate geometrical arrangement of the plurality of selected objects among multiple candidate geometrical arrangements of the plurality of selected objects;

means for determining a reference position based on the candidate geometrical arrangement of the plurality of selected objects;

means for generating metadata descriptive of the particular candidate geometrical arrangement and the reference position; and

means for searching a metadata index corresponding to a plurality of digital images to identify at least one digital image from the plurality of digital images, the at least one digital image identified based on comparison of the metadata with an entry of the metadata index such that the at least one digital image depicts the plurality of selected objects, relative to the reference position, arranged according to the particular candidate geometrical arrangement.

20. The apparatus according to claim 19, wherein the obtained information includes, for each object of the plurality of selected objects, a label that is associated with the object, wherein the means for searching for the labels includes searching for the labels within the metadata index that is associated with the plurality of digital images.

21. The apparatus according to claim 19, wherein the means for searching includes means for searching within the metadata index that is associated with the plurality of digital images for a label that is associated with the particular candidate geometrical arrangement of the plurality of selected objects.

22. The apparatus according to claim 19, wherein the at least one digital image includes a plurality of frames from a first video file and a plurality of frames from a second video file.

23. The apparatus according to claim 19, wherein the means for obtaining information selecting the plurality of selected objects is configured to obtain the information using at least one of speech recognition and a touchscreen.

24. An apparatus for image retrieval, the apparatus comprising:

a discriminator configured to obtain information selecting a plurality of objects represented in a digital image, the digital image associated with a location space and the plurality of selected objects distributed about the location space, and select a particular candidate geometrical arrangement of the plurality of selected objects among multiple candidate geometrical arrangements of the plurality of selected objects; and

a search engine configured to;

determine a reference position based on the candidate geometrical arrangement;

generate metadata descriptive of the particular candidate geometrical arrangement and the reference position; and

search a metadata index corresponding to a plurality of digital images to identify at least one digital image from the plurality of digital images, the at least one digital image identified based on comparison of the metadata with an entry of the metadata index such that the at least one digital image depicts the plurality of selected objects, relative to the reference position, arranged according to the particular candidate geometrical arrangement.

25. The apparatus according to claim 24, wherein the obtained information includes, for each of the plurality of selected objects, a label that is associated with the object, and wherein the search engine is configured to search for the labels within the metadata index that is associated with the plurality of digital images.

26. The apparatus according to claim 24, wherein the search engine is configured to search, within the metadata index that is associated with the plurality of digital images, for a label that is associated with the particular candidate geometrical arrangement of the plurality of selected objects.

27. The apparatus according to claim 24, wherein the at least one digital image includes a plurality of frames from a first video file and a plurality of frames from a second video file, and further comprising:

a camera configured to capture a video stream, the video stream including the plurality of frames of the first video file, the second video file, or both;

a memory, coupled to the camera, configured to store the first video file, the second video file, or both; and

a display screen, coupled to the camera, configured to display the video stream, an image of the first video file, an image of the second video file, or a combination thereof, wherein the display screen comprises a touchscreen configured to receive a touch input defining a target object, and wherein the display screen displays the target object, wherein the reference position is based on at least one point selected in the video stream or a position of an image of a selected object of the plurality of selected objects in the image of the first video file, the image of the second video file, or both.

28. The apparatus according to claim 24, wherein the discriminator is configured to obtain the information selecting the plurality of selected objects using at least one of speech recognition and a touchscreen, and wherein the reference position is a position of a speaker in a crowd.