Image retrieval

• US 8,462,224 B2
• Filed: 09/09/2010
• Issued: 06/11/2013
• Est. Priority Date: 06/01/2010
• Status: Active Grant

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Zhang, Tong, Wu, Peng
• Primary Examiner(s): HANNETT, JAMES M
• Application Number:
  US12/878,782
• Publication Number:
  US 20110292232A1
• Time in Patent Office: 1,006 Days
• Field of Search: 348/222.1
• US Class Current: 348/222.1
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: G06V 40/173 face re-identification, e.g. recognising unknown faces across different face tracks
• CPC Class Codes: G06V 40/179 metadata assisted face recognition
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Abstract

A method, comprising receiving query data including an image of a person, and detecting a face of the person in the image to create a detected face, generating face data and clothing data using the detected face including a measure of characteristics of the face and clothing of the person, and retrieving images and video for the person and a related group of people by comparing the face data and clothing data of the person with multiple measures for the characteristics of faces and clothing generated from a set of images and video in which other people appear wherein co-appearances of the person with at least some of the other people are used to retrieve content in which the person and people from the related group appear.

Claims

1. A method, comprising:

receiving query data including an image of a person;

detecting a face of the person in the image to create a detected face;

generating face data and clothing data using the detected face, wherein the face data includes a measure of characteristics of the face and the clothing data includes a measure of characteristics of the clothing of the person;

using the face data and the clothing data of the person along with face data and clothing data of other people contained in at least one of images and videos to determine a subset of people who co-appear with the person at least a predetermined number of times or for a predetermined period of time, wherein the subset of people comprise a related group of people to the person; and

retrieving content in which the person co-appears with at least one person of the related group of people.

2. A method as claimed in claim 1, wherein receiving the query data includes receiving a representative image frame generated from a portion of video in which the person is visible.

3. A method as claimed in claim 1, wherein the images and videos further include data representing a corresponding time and location of capture, the method further including sorting the images and videos on the basis of capture time and capture location, and wherein retrieving the content further includes using the capture time and capture location to reduce a search space of images and videos to a reduced set of images and videos captured within a time period and location of interest.

4. A method as claimed in claim 1, wherein retrieving further includes retrieving content in which the at least one person of the related group of people co-appears with the person at least a predetermined number of times in images from a set of images, or for at least a predetermined period of time in a video.

5. An image retrieval system comprising:

at least one image capture device to generate image data representing a set of video and still images of a predetermined area; and

a memory storing machine readable instructions to;

generate representative image frames from the video;

detect people appearing in the representative frames and still images in the set, and create a set of detected faces for the detected people; and

generate face data and clothing data for the detected people including characteristics of the detected faces and clothing of the people;

receive query data including at least one image of a user;

detect a user face in the at least one image of the user;

generate user face data and clothing data including characteristics of the face and clothing of the user therefrom;

use the face data and clothing data of the detected people to determine a subset of people who co-appear with the user at least a predetermined number of times or for a predetermined period of time;

retrieve content from the set in which the user co-appears with at least one person of the related group of people;

a processor to execute the machine readable instructions.

6. An image retrieval system as claimed in claim 5, further including:

a display to;

present the retrieved content to the user for confirmation; and

on the basis of the confirmation, to provide the user with information for a plurality of selections on the display representing a set of products associated with the retrieved content.

7. An image retrieval system as claimed in claim 5, wherein the machine readable instructions are further to:

provide remote access to content including images and/or video for the user over a web-based service accessible by the user.

8. An image retrieval system as claimed in claim 5, wherein the machine readable instructions are further to:

restrict the retrieval of content using control query data.

9. A non-transitory computer-readable medium storing computer-readable program instructions that when executed by a computer are to:

provide query data including an image of a person;

detect a face of the person in the image to create a detected face;

generate face data and clothing data from the image using the detected face, wherein the face data includes a measure of characteristics of the face and the clothing data includes a measure of characteristics of the clothing of the person;

use the face data and the clothing data of the person along with face data and clothing data of other people contained in at least one of images and videos to determine a subset of people who co-appear with the person at least a predetermined number of times or for a predetermined period of time, wherein the subset of people comprise an associated group of people to the person; and

retrieve content in which the person co-appears with at least one person of the associated group of people.

10. A non-transitory computer-readable medium as claimed in claim 9, wherein the instructions are further to:

retrieve at least a portion of video including a plurality of image frames from the raw image data by determining the presence of the person in the frames by matching at least one of the face and clothing data of the person to a person identified in the frames.

11. A non-transitory computer-readable medium as claimed in claim 9, wherein the instructions are further to:

to retrieve at least one of an image and portion of video in which the person consistently co-appears with a first person; and

to retrieve at least one of an image and portion of video in which the first person consistently co-appears with a second person, wherein co-appearances between the first person and the second person are determined by using images and/or video in which the person co-appears with the second person.

12. A non-transitory computer-readable medium as claimed in claim 9, wherein the instructions are further to:

to receive control query data for controlling the retrieval by restricting access to raw image data including images and/or video of a given person.

13. An image and video content retrieval apparatus, comprising:

a memory storing computer-readable instructions; and

a processor coupled to the memory, to execute the instructions, and based at least in part on the execution of the instructions, to;

use face data and clothing data of a user along with face data and clothing data of other people contained in at least one of images and videos to determine a subset of people who co-appear with the user at least a predetermined number of times or for a predetermined period of time, wherein the subset of people comprise a related group of people to the user; and

retrieve content captured using a network of image capture devices disposed to image a predetermined area by using a measure of the characteristics of the face data and the clothing data of the user to determine content in which the user appears; and

retrieve content in which the user consistently appears with at least one person in the related group of people.