Display device content selection through viewer identification and affinity prediction

• US 9,483,697 B2
• Filed: 09/30/2015
• Issued: 11/01/2016
• Est. Priority Date: 02/17/2009
• Status: Active Grant

• Current Assignee: Ikorongo Technology, LLC (Concert Technology Corporation)
• Original Assignee: Ikorongo Technology, LLC (Concert Technology Corporation)
• Inventors: Svendsen, Hugh Blake
• Primary Examiner(s): Haskins, Twyler
• Assistant Examiner(s): Quiett, Carramah J
• Application Number:
  US14/870,303
• Publication Number:
  US 20160019414A1
• Time in Patent Office: 398 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/50 of still image data
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 3/041 Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G06V 20/30 in albums, collections or shared content, e.g. social network photos or video
• CPC Class Codes: G06V 40/16 Human faces, e.g. facial parts, sketches or expressions
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: G06V 40/179 metadata assisted face recognition
• CPC Class Codes: G11B 27/19 by using information detectable on the record carrier
• CPC Class Codes: H04L 67/01 Protocols
• CPC Class Codes: H04L 67/306 User profiles
• CPC Class Codes: H04N 23/54 Mounting of pick-up tubes, electronic image sensors, deviation or focusing coils
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04N 23/61 based on recognised objects
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body
• CPC Class Codes: H04N 23/63 by using electronic viewfinders
• CPC Class Codes: H04N 23/66 Remote control of cameras or camera parts, e.g. by remote control devices
• CPC Class Codes: H04N 7/183 for receiving images from a single remote source
• CPC Class Codes: H04N 9/8715 : involving the mixing of the reproduced video signal with a non-recorded signal, e.g. a text signal

Abstract

Method and device are described for customizing the content selection to present on a display device based on identifying the viewer of the device. In one embodiment, the present disclosure relates to selecting from among a group of digital images, those images that are most likely to be of interest to the viewer. In general, many of the images available to the display device will be comprised of images containing subject faces. Using the relationship information to predict subject affinity, the display device computes an image affinity for each image by accumulating the individual subject affinity predictions between the viewer and each subject identified in each image. The image affinities are used to select images for presentation on the display device.

Claims

1. A server device comprising:

a communication interface operable to communicatively couple the server device to a display device over a network; and

a control system comprising at least a processor and a memory storing program codes operable to;

obtain social relationship information identifying a social relationship between a reference person and one of a plurality of other subjects that appear in a plurality of images, the plurality of images associated with the reference person;

determine a subject affinity defining a subject affinity score between the reference person and the one of the plurality of other subjects based on the social relationship information, the plurality of other subjects corresponding to subjects other than the reference person; and

determine an image affinity for each of the plurality of images based on the subject affinity for the one of the plurality of other subjects, each image affinity defining an image affinity score between the reference person and a corresponding one of the plurality of images based on the one of the plurality of other subjects appearing in the one of the plurality of images; and

select, by the server device, based on the image affinity for each of the plurality of images, one or more images from the plurality of images.

2. The server device of claim 1 wherein the control system is further operable to:

send image information identifying the one or more images to the display device over the network.

3. The server device of claim 2 wherein to send image information identifying the one or more images the control system is further operable to:

order, based on the image affinity for each of the plurality of images, the plurality of images.

6. The server device of claim 1 wherein one or more of the plurality of images contain a number of subjects chosen from the group consisting of no subjects and multiple subjects.

7. The server device of claim 1 wherein one or more of the plurality of images are stored at a source chosen from the group consisting of:

the display device,

the server device,

a social network server associated with the server device,

a photosharing server associated with the server device,

a secondary collection sharing device.

8. The server device of claim 2 wherein the control system is further operable to:

provide, to the display device, the one or more images identified by the image information.

9. The server device of claim 1 wherein the reference person is a viewer of the display device that has been matched to a subject appearing in the plurality of images.

10. The server device of claim 9 where in order to determine the subject affinity the control system is further operable to:

refine based on a proximity between a face of the reference person and faces of the other subjects in a same image in the plurality of images.

11. The server device of claim 10 where in order to determine the subject affinity the control system is further operable to:

determine the proximity between the face of the reference person and the faces of the other subjects in the same image based on dividing a distance between centers of circles enclosing the face of the reference person and the faces of the other subjects by a diagonal distance of the same image.

12. The server device of claim 9 where in order to determine the subject affinity the control system is further operable to:

identify face co-occurrence instances between the reference person and one of the other subjects appearing in a same image in the plurality of images; and

determine subject affinity from the social relationship inferred from subject face co-occurrence between the reference person and the other subjects in the plurality of images.

13. The server device of claim 9 where in order to determine the subject affinity the control system is further operable to:

determine viewer response to the other subjects in the image of the plurality of images; and

modify the viewer'"'"'s subject affinity for other subjects based on the viewer response.

14. The server device of claim 9 where in order to determine the image affinity the control system is further operable to:

resolve the subjects occurring in an image of the plurality of images; and

accumulate a contribution from the subjects according to the subject affinity between the viewer and the subjects in the image.

15. The server device of claim 1 further operable to:

communicate with a social network wherein the reference person and the one of the plurality of other subjects are members of the social network;

store information expressing friend relationships built through an invitation process;

identify the social relationship based on a social distance, the social distance defining a number of hops in a social graph based on the friend relationships between the reference person and the plurality of other subjects; and

determine the subject affinity based on the social relationship information.

16. The server device of claim 15 wherein other subjects with a lower social distance to the reference person are awarded a higher subject affinity.

17. The server device of claim 15 where in order to determine the subject affinity the control system is further operable to:

track location history of other devices associated with other subjects reporting time-stamped GPS locations of the other devices to the social network;

identify other subjects of closer average geographical proximity to the reference person as having a closer social relationship; and

award other subjects the closer social relationship to the reference person a higher subject affinity.

18. The server device of claim 15 where in order to determine the subject affinity the control system is further operable to:

refine the social relationship based on relationship qualifiers between the reference person and the one of the plurality of other subjects.

19. The server device of claim 15 where in order to determine the subject affinity the control system is further operable to:

refine the social relationship based on a duration of the friend relationship between the reference person and the one of the plurality of other subjects.

20. The server device of claim 15 where in order to determine the subject affinity the control system is further operable to:

refine the social relationship based on a frequency of communication occurring between the reference person and the one of the plurality of other subjects.

21. The server device of claim 15 where in order to determine the subject affinity the control system is further operable to:

refine the social relationship based on a type of communication occurring between the reference person and the one of the plurality of other subjects.

22. The server device of claim 15 wherein the social relationship is non-symmetric between the reference person and the one of the plurality of other subjects.

23. The server device of claim 15 where in order to determine the subject affinity the control system is further operable to:

refine the social relationship based on friend counts of the reference person and the one of the plurality of other subjects.

24. The server device of claim 1 where in order to determine the image affinity the control system is further operable to:

divide the image into zones; and

modify an image affinity score contribution of subjects according to the zones in which faces of the subjects appear.

25. The server device of claim 1 where in order to determine the image affinity the control system is further operable to:

determine a size of a subject face in an image of the plurality of images; and

modify an image affinity score contribution of an other subject corresponding to the subject face according to the size.

26. The server device of claim 25 where in order to determine the size of a subject face the control system is further operable to:

determine an area of a circle enclosing the subject face.

4. The server device of claim DR wherein the control system is further operable to:

obtain reference person information identifying the reference person; and

obtain image information identifying the plurality of images associated with the reference person.

5. The server device of claim 4 wherein to obtain image information the control system is further operable to:

obtain an index comprising;

for one or more images of the plurality of images, first one or more indicators to one or more subjects appearing in the one or more images; and

for one or more subjects of the plurality of subjects, second one or more indicators to one or more images in which the one or more subjects appear.

27. A system comprising:

a server device comprising;

a first communication interface operable to communicatively coupling the server device to a display device client; and

a first control system comprising a first processor and a first memory storing first program codes operable to;

obtain social relationship information identifying a social relationship between a reference person and one of a plurality of other subjects that appear in a plurality of images;

determine a subject affinity defining a subject affinity score between the reference person and the one of the plurality of other subjects based on the social relationship information, the plurality of other subjects corresponding to subjects other than the reference person; and

determine an image affinity for each of the plurality of images based on the subject affinity for the one of the plurality of other subjects, each image affinity defining an image affinity score between the reference person and a corresponding one of the plurality of images based on the one of the plurality of other subjects appearing in the one of the plurality of images;

select, by the server device, one or more images based on their image affinities; and

send, to the display device client, image presentation information, the image presentation information comprising one or more image indicators indicating the one or more images; and

the display device client operable to;

obtain information identifying the reference person;

send, to the server device, the information identifying the reference person; and

receive, from the server device, the image presentation information.

29. The system of claim 27 further comprising:

a display device comprising;

a second communication interface operable to communicatively couple the display device client to the server device; and

a second control system coupled to the second communication interface comprising a second processor and a second memory storing second program codes operable to;

execute the display device client.

28. A method of operating a server device comprising:

using at least a processor and memory for;

obtaining social relationship information identifying a social relationship between a reference person and one of a plurality of other subjects that appear in a plurality of images, the plurality of images associated with the reference person;

determining a subject affinity defining a subject affinity score between the reference person and the one of the plurality of other subjects based on the social relationship information between the reference person and the one of the plurality of other subjects, the plurality of other subjects corresponding to subjects other than the reference person; and

determining an image affinity for each of the plurality of images based on the subject affinity for the one of the plurality of other subjects, each image affinity defining an image affinity score between the reference person and a corresponding one of the plurality of images based on the one of the plurality of other subjects appearing in the one of the plurality of images; and

selecting, by the server device, based on the image affinity for each of the plurality of images, one or more images from the plurality of images.