RELATIONSHIP MAPPING EMPLOYING MULTI-DIMENSIONAL CONTEXT INCLUDING FACIAL RECOGNITION

• US 20110182485A1
• Filed: 03/19/2009
• Published: 07/28/2011
• Est. Priority Date: 03/20/2008
• Status: Active Grant

• Current Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Original Assignee: Face.com Incorporated (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Inventors: Shniberg, Moti, Hirsch, Gil, Shochat, Eden, Taigman, Yaniv
• Granted Patent: US 8,666,198 B2
• Time in Patent Office: Days
• US Class Current: 382/118
• CPC Class Codes: G06F 16/29 Geographical information databases
• CPC Class Codes: G06F 16/54 Browsing; Visualisation therefor
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06V 10/768 using context analysis, e.g. recognition aided by known co-occurring patterns
• CPC Class Codes: G06V 40/16 Human faces, e.g. facial parts, sketches or expressions
• CPC Class Codes: G06V 40/173 face re-identification, e.g. recognising unknown faces across different face tracks
• CPC Class Codes: H04L 65/403 Arrangements for multi-party communication, e.g. for conferences data switching systems for conference H04L12/18; arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities H04M3/56; television conferencing systems H04N7/15

Abstract

A system and method for mapping interpersonal relationships, the method including processing a multiplicity of images and contextual information relating thereto including creating and prioritizing a list of a plurality of candidate persons having at least a predetermined relationship with at least one person connected to at least one image, using multi-dimensional information including visually sensible information in the multiplicity of images and contextual information relating thereto and searching the list of a plurality of candidate persons based at least in part on the prioritizing to select at least one of the candidate persons as having at least a predetermined relationship with the at least one person.

Claims

1. A method for mapping interpersonal relationships comprising:

processing a multiplicity of images and contextual information relating thereto including creating and prioritizing a list of a plurality of candidate persons having at least a predetermined relationship with at least one person connected to at least one image, using multi-dimensional information including visually sensible information in said multiplicity of images and contextual information relating thereto; and

searching said list of a plurality of candidate persons based at least in part on said prioritizing to select at least one of said candidate persons as having at least a predetermined relationship with said at least one person.

2. A method for mapping interpersonal relationships according to claim 1 and wherein said searching also employs said multi-dimensional information.

3. A method for mapping interpersonal relationships according to claim 1 and wherein said multi-dimensional information includes:

at least one of

visual information appearing in an image;

geographical information appearing in said image;

visual background information appearing in said image;

images of other persons appearing in said image; and

person identifiers appearing in said image; and

at least one of

information relating to an image collection of which said image forms a part;

a time stamp associated with said image;

information not appearing in said image but associated therewith;

geographical information not appearing in said image;

visual background information appearing in another image; and

person identifiers not appearing in said image but otherwise associated therewith.

4. A method for mapping interpersonal relationships according to claim 1 and wherein said multi-dimensional information includes:

at least one of

visual information appearing in an image;

geographical information appearing in said image;

visual background information appearing in said image;

images of other persons appearing in said image; and

person identifiers appearing in said image; and

at least two of

information relating to an image collection of which said image forms a part;

a time stamp associated with said image;

information not appearing in said image but associated therewith;

geographical information not appearing in said image;

visual background information appearing in another image; and

person identifiers not appearing in said image but otherwise associated therewith.

5. A method for mapping interpersonal relationships according to claim 1 and wherein said multi-dimensional information includes:

at least one of

visual information appearing in an image;

geographical information appearing in said image;

visual background information appearing in said image;

images of other persons appearing in said image; and

person identifiers appearing in said image; and

at least three of

information relating to an image collection of which said image forms a part;

a time stamp associated with said image;

information not appearing in said image but associated therewith;

geographical information not appearing in said image;

visual background information appearing in another image; and

person identifiers not appearing in said image but otherwise associated therewith.

6. A method for mapping interpersonal relationships according to claim 1 and wherein said multi-dimensional information includes:

all of the following;

visual information appearing in an image;

geographical information appearing in said image;

visual background information appearing in said image;

images of other persons appearing in said image;

person identifiers appearing in said image;

information relating to an image collection of which said image forms a part;

a time stamp associated with said image;

information not appearing in said image but associated therewith;

geographical information not appearing in said image;

visual background information appearing in another image; and

person identifiers not appearing in said image but otherwise associated therewith.

7. A method for mapping interpersonal relationships according to claim 1 and wherein said at least one image is a composite image, which includes images of multiple persons, at least one of whom is unknown.

8. A method for mapping interpersonal relationships according to claim 1 and wherein said at least one image is a composite image, which includes images of multiple persons, none of whom are known.

9. A method for mapping interpersonal relationships according to claim 1 and wherein said at least one person connected with said at least one image appears in said image.

10. A method for mapping interpersonal relationships according to claim 1 and also comprising tagging said at least one person connected with said at least one image with a person identifier.

11. A method for mapping interpersonal relationships according to claim 1 and wherein said processing includes iterative generation of relationship maps based on at least visually sensible information and also on additional, non-visually sensible information related to persons who either appear in said at least one image or are otherwise associated therewith.

12. A method for mapping interpersonal relationships according to claim 1 and wherein said non-visually sensible information is meta-data associated with image data.

13. A method for mapping interpersonal relationships according to claim 12 and wherein said meta-data includes data derived from a social network.

14. A method for mapping interpersonal relationships according to claim 12 and wherein said meta-data includes data attached to the image data of said composite image.

15. A method for mapping interpersonal relationships according to claim 11 and wherein said iterative generation of relationship maps starts from a precursor relationship map, containing information on relationships of at least one known person in said at least one image.

16. A method for mapping interpersonal relationships according to claim 11 and wherein said relationship map is also based on inter-personal relationship data received from at least one of a social network and earlier instances of relationship mapping based on analysis of other images.

17. A method for mapping interpersonal relationships according to claim 11 and wherein said relationship map includes an indication of at least strength of the relationship between two persons.

18. A method for mapping interpersonal relationships according to claim 11 and wherein said relationship map includes a face representation which identifies each of the persons in the map.

19. A method for mapping interpersonal relationships according to claim 11 and wherein said relationship map includes an indication of whether each person in the map is a male or female.

20. A method for mapping interpersonal relationships according to claim 19 and wherein said indication of whether each person in the map is a male or female is provided by at least one of a social network and operation of image attribute recognition.

21. A method for mapping interpersonal relationships according to claim 1 and wherein said prioritizing employs an indication of whether a person is a male or female.

22. A method for mapping interpersonal relationships according to claim 1 and wherein said prioritizing employs an indication of whether a person appears in the same album in a social network as another person appears.

23. A method for mapping interpersonal relationships according to claim 1 and wherein said processing includes seeking candidate persons having at least a predetermined relationship with a known person in at least one image.

24. A method for mapping interpersonal relationships according to claim 23 and wherein said seeking candidate persons is carried out by starting with the generation of a list of candidate persons who have a temporal association with said known person based on visually-sensible information contained in said at least one image as well as non-visually sensible information.

25. A method for mapping interpersonal relationships according to claim 24 and wherein said non-visually sensible information includes at least one of the time and geographical location where said composite image was taken and an identification of an album on a social network with which it is associated.

26. A method for mapping interpersonal relationships according to claim 24 and wherein said non-visually sensible information is obtained at least by interfacing with social network APIs to find persons who appeared in other pictures in the same album, or persons that appeared in other albums taken in the same geographical location at the same time.

27. A method for mapping interpersonal relationships according to claim 24 and wherein said list of candidate persons is extended and further prioritized by analyzing relationships of the persons appearing in a relationship map.

28. A method for mapping interpersonal relationships according to claim 1 and wherein said prioritizing employs image attribute filtering.

29. A method for mapping interpersonal relationships according to claim 1 and wherein said processing includes facial representation generation performed on an unknown person in at least one image.

30. A method for mapping interpersonal relationships according to claim 29 and also comprising comparing said facial representation with previously generated facial representations of the candidate persons in accordance with and in an order established by said prioritizing.

31. A method for mapping interpersonal relationships according to claim 30 and wherein said comparing is terminated and a candidate person is selected when a combined priority/similarity threshold is reached for a given candidate person, said priority/similarity threshold taking into account the similarity of a facial representation of a candidate person to the facial representation of an unknown person, the priority of that candidate person established by the above-referenced prioritizing and the quality of the facial representation of the candidate person.

32. A method for mapping interpersonal relationships according to claim 31 and wherein user feedback confirming that the person whose image is believed to be a given person is or is not that person is employed in generation of a further iterative relationship map.

33. A method for mapping interpersonal relationships according to claim 31 and wherein user feedback confirming that the person whose image is believed to be a given person is or is not that person is employed in improving said face representation.

34. A method for mapping interpersonal relationships according to claim 1 and wherein said searching includes comparison of face representations of persons in said list of candidate persons with face representations of persons in said relationship map.

35. A method for mapping interpersonal relationships according to claim 11 and also comprising searching at least one of said relationship maps.

36. A method for mapping interpersonal relationships according to claim 35 and wherein said searching of said at least one of said relationship maps employs search terms including at least one of uniquely identified persons, an additional image of a person, relationships between various persons, gender and face representations.

37. A method for mapping interpersonal relationships according to claim 35 and wherein said searching of said at least one of said relationship maps employs a user interface.

38. A method for mapping interpersonal relationships according to claim 35 and wherein said searching of said at least one of said relationship maps is carried out via at least one social network having access to said at least one of said relationship maps.

39. A system for mapping interpersonal relationships comprising:

processing functionality operative to process a multiplicity of images and contextual information relating thereto including creating and prioritizing a list of a plurality of candidate persons having at least a predetermined relationship with at least one person connected to at least one image, using multi-dimensional information including visually sensible information in said multiplicity of images and contextual information relating thereto; and

searching functionality operative to search said list of a plurality of candidate persons based at least in part on said prioritizing to select at least one of said candidate persons as having at least a predetermined relationship with said at least one person.

40. A system for mapping interpersonal relationships according to claim 39 and wherein said searching functionality also employs said multi-dimensional information.

41. A system for mapping interpersonal relationships according to claim 39 and wherein said multi-dimensional information includes:

at least one of

visual information appearing in an image;

geographical information appearing in said image;

visual background information appearing in said image;

images of other persons appearing in said image; and

person identifiers appearing in said image; and

at least one of

information relating to an image collection of which said image forms a part;

a time stamp associated with said image;

information not appearing in said image but associated therewith;

geographical information not appearing in said image;

visual background information appearing in another image; and

person identifiers not appearing in said image but otherwise associated therewith.

42. A system for mapping interpersonal relationships according to claim 39 and wherein said multi-dimensional information includes:

at least one of

visual information appearing in an image;

geographical information appearing in said image;

visual background information appearing in said image;

images of other persons appearing in said image; and

person identifiers appearing in said image; and

at least two of

information relating to an image collection of which said image forms a part;

a time stamp associated with said image;

information not appearing in said image but associated therewith;

geographical information not appearing in said image;

visual background information appearing in another image; and

person identifiers not appearing in said image but otherwise associated therewith.

43. A system for mapping interpersonal relationships according to claim 39 and wherein said multi-dimensional information includes:

at least one of

visual information appearing in an image;

geographical information appearing in said image;

visual background information appearing in said image;

images of other persons appearing in said image; and

person identifiers appearing in said image; and

at least three of

information relating to an image collection of which said image forms a part;

a time stamp associated with said image;

information not appearing in said image but associated therewith;

geographical information not appearing in said image;

visual background information appearing in another image; and

person identifiers not appearing in said image but otherwise associated therewith.

44. A system for mapping interpersonal relationships according to claim 39 and wherein said multi-dimensional information includes:

all of the following;

visual information appearing in an image;

geographical information appearing in said image;

visual background information appearing in said image;

images of other persons appearing in said image;

person identifiers appearing in said image;

information relating to an image collection of which said image forms a part;

a time stamp associated with said image;

information not appearing in said image but associated therewith;

geographical information not appearing in said image;

visual background information appearing in another image; and

person identifiers not appearing in said image but otherwise associated therewith.

45. A system for mapping interpersonal relationships according to claim 39 and wherein said at least one image is a composite image, which includes images of multiple persons, at least one of whom is unknown.

46. A system for mapping interpersonal relationships according to claim 39 and wherein said at least one image is a composite image, which includes images of multiple persons, none of whom are known.

47. A system for mapping interpersonal relationships according to claim 39 and wherein said at least one person connected with said at least one image appears in said image.

48. A system for mapping interpersonal relationships according to claim 39 and also comprising tagging said at least one person connected with said at least one image with a person identifier.

49. A system for mapping interpersonal relationships according to claim 39 and wherein said processing includes iterative generation of relationship maps based on at least visually sensible information and also on additional, non-visually sensible information related to persons who either appear in said at least one image or are otherwise associated therewith.

50. A system for mapping interpersonal relationships according to claim 39 and wherein said non-visually sensible information is meta-data associated with image data.

51. A system for mapping interpersonal relationships according to claim 50 and wherein said meta-data includes data derived from a social network.

52. A system for mapping interpersonal relationships according to claim 50 and wherein said meta-data includes data attached to the image data of said composite image.

53. A system for mapping interpersonal relationships according to claim 49 and wherein said iterative generation of relationship maps starts from a precursor relationship map, containing information on relationships of at least one known person in said at least one image.

54. A system for mapping interpersonal relationships according to claim 49 and wherein said relationship map is also based on inter-personal relationship data received from at least one of a social network and earlier instances of relationship mapping based on analysis of other images.

55. A system for mapping interpersonal relationships according to claim 49 and wherein said relationship map includes an indication of at least strength of the relationship between two persons.

56. A system for mapping interpersonal relationships according to claim 49 and wherein said relationship map includes a face representation which identifies each of the persons in the map.

57. A system for mapping interpersonal relationships according to claim 49 and wherein said relationship map includes an indication of whether each person in the map is a male or female.

58. A system for mapping interpersonal relationships according to claim 57 and wherein said indication of whether each person in the map is a male or female is provided by at least one of a social network and operation of image attribute recognition.

59. A system for mapping interpersonal relationships according to claim 39 and wherein said prioritizing employs an indication of whether a person is a male or female.

60. A system for mapping interpersonal relationships according to claim 39 and wherein said prioritizing employs an indication of whether a person appears in the same album in a social network as another person appears.

61. A system for mapping interpersonal relationships according to claim 39 and wherein said processing functionality includes seeking candidate persons having at least a predetermined relationship with a known person in at least one image.

62. A system for mapping interpersonal relationships according to claim 61 and wherein said seeking candidate persons is carried out by starting with the generation of a list of candidate persons who have a temporal association with said known person based on visually-sensible information contained in said at least one image as well as non-visually sensible information.

63. A system for mapping interpersonal relationships according to claim 62 and wherein said non-visually sensible information includes at least one of the time and geographical location where said composite image was taken and an identification of an album on a social network with which it is associated.

64. A system for mapping interpersonal relationships according to claim 62 and wherein said non-visually sensible information is obtained at least by interfacing with social network APIs to find persons who appeared in other pictures in the same album, or persons that appeared in other albums taken in the same geographical location at the same time.

65. A system for mapping interpersonal relationships according to claim 62 and wherein said list of candidate persons is extended and further prioritized by analyzing relationships of the persons appearing in a relationship map.

66. A system for mapping interpersonal relationships according to claim 39 and wherein said prioritizing employs image attribute filtering.

67. A system for mapping interpersonal relationships according to claim 39 and wherein said processing includes facial representation generation performed on an unknown person in at least one image.

68. A system for mapping interpersonal relationships according to claim 67 and also comprising comparison functionality operative to compare said facial representation with previously generated facial representations of the candidate persons in accordance with and in an order established by said prioritizing.

69. A system for mapping interpersonal relationships according to claim 68 and wherein operation of said comparison functionality is terminated and a candidate person is selected when a combined priority/similarity threshold is reached for a given candidate person, said priority/similarity threshold taking into account the similarity of a facial representation of a candidate person to the facial representation of an unknown person, the priority of that candidate person established by the above-referenced prioritizing and the quality of the facial representation of the candidate person.

70. A system for mapping interpersonal relationships according to claim 69 and wherein user feedback confirming that the person whose image is believed to be a given person is or is not that person is employed in generation of a further iterative relationship map.

71. A system for mapping interpersonal relationships according to claim 69 and wherein user feedback confirming that the person whose image is believed to be a given person is or is not that person is employed in improving said face representation.

72. A system for mapping interpersonal relationships according to claim 39 and wherein said searching functionality includes comparison functionality operative to provide comparison of face representations of persons in said list of candidate persons with face representations of persons in said relationship map.

73. A system for mapping interpersonal relationships according to claim 49 and also comprising relationship map searching functionality.

74. A system for mapping interpersonal relationships according to claim 73 and wherein said relationship map searching functionality employs search terms including at least one of uniquely identified persons, an additional image of a person, relationships between various persons, gender and face representations.

75. A system for mapping interpersonal relationships according to claim 73 and wherein said relationship map searching functionality employs a user interface.

76. A system for mapping interpersonal relationships according to claim 73 and wherein said relationship map searching functionality is carried out via at least one social network having access to said at least one of said relationship maps.