Method and system of using image recognition and geolocation signal analysis in the construction of a social media user identity graph

• US 10,031,925 B2
• Filed: 10/15/2015
• Issued: 07/24/2018
• Est. Priority Date: 10/15/2014
• Status: Active Grant

• Current Assignee: ThinkCX Technologies, Inc.
• Original Assignee: ThinkCX Technologies, Inc.
• Inventors: Nielsen, Aaron David, Pang, Qiao, Mori, Greg, Yap, Jordan
• Primary Examiner(s): Cao, Phuong Thao
• Application Number:
  US14/884,751
• Publication Number:
  US 20160117347A1
• Time in Patent Office: 1,013 Days
• Field of Search: 707738
• CPC Class Codes: G06F 16/24575 using context
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/9024 Graphs; Linked lists G06F16/9027 takes precedence

Abstract

A social media identity/construct is created for a person by locating social media profiles of the person at least by one or both of i) recognizing image(s) in one or more social media postings related to the person and ii) correlating geolocation data embedded in one or more social media postings related to the person or the person'"'"'s social media connections, analyzing, monitor social media outlets and channels to create a social media profile for the person.

Claims

1. A non-transitory computer readable medium having stored thereon computer-executable instructions, the computer executable instructions causing a processor to execute a method of creating an integrated, collated social media profile for a person which comprises:

a) probing social media outlets and channels using one or more of social media application programming interfaces (APIs), data provided by third party data compilers, data aggregated directly from web crawlers, using selected person-specific search parameters and person specific context data to form a social media data stream;

b) from the social media data stream, identifying at least one photographic image from a first social media channel and at least one photographic image from a second social media channel;

c) comparing the at least one photographic image from the first social media channel and the at least one photographic image from the second social media channel via a plurality of image recognition steps, comprising for each image;

i) creating a 64 bit signature hash;

ii) converting the image down to 9×

8 pixels to remove unrequired details and to produce a low frequency 9×

8 image to ensure that different scaling and stretching will not affect a hash;

iii) converting the low frequency 9×

8 image into greyscale to simplify further computation;

iv) computing a greyscale colour difference between each pixel next to each other in each row of pixels, wherein each row will produce eight difference values for a total of 64 values;

v) creating a 64 bit hash wherein each bit is set to 0 or 1 depending on whether the respective left pixel is brighter than the right pixel and vi) comparing two images using Hamming distance to produce global image comparison output;

d) comparing the global image comparison output to a threshold to produce a set of resultant global images, wherein the set of resultant global images comprises at least one pair of photographic images from the first social media channel and the second social media channel of the at least one photographic image from the first social media channel and the at least one photographic image from the second social media channel having a global image comparison output less than the threshold;

e) comparing the set of resultant global images from the first social media channel and the set of resultant global images from the second social media channel via a plurality of image recognition steps, comprising for each image, i) identifying one or more features/keypoints;

ii) assigning descriptors to the one or more features/keypoints so that they can be recognized and compared to other images;

iii) weighing the one or more features/keypoints;

iv) matching the one or more features/keypoints;

f) from the social media data stream, i) identifying a plurality of geolocation data inputs related to the person and/or social media connections of the person;

ii) performing a centroid-based analysis on the plurality of geolocation data inputs to form clusters;

iii) classifying and filtering the clusters; and

iv) selecting one or more probable locations of the person based upon the clusters; and

g) determining and corroborating with confidence the photographic images and the geolocation data inputs which are assignable to the person and creating the integrated, collated social media profile, displayable on a dashboard, for the person, wherein the integrated, collated social media profile for the person comprises at least one matched photographic image of the set of resultant global images.

2. The non-transitory computer readable medium of claim 1 wherein comparisons between all descriptors in a pair of images are conducted, distances are sorted according to Hamming distance (d(n)) and an overall score(s) for the image is determined by first sorting all the distances in ascending order as follows:

3. The non-transitory computer readable medium of claim 1 wherein the selected person-specific search parameters are deterministic and are selected from the group consisting of email address, usernames, identification codes and other recognition entry points to a social network related to the person.

4. The non-transitory computer readable medium of claim 1 wherein the selected person-specific search parameters are semi-deterministic and are selected from the group consisting of names, locations (address, geo coordinates, city), companies, phone numbers, related people (spouse, parent, child, roommate, workmate, classmate, friend), personal names and photographs.

5. The non-transitory computer readable medium of claim 1 wherein the selected person-specific search parameters comprises context data selected from the group consisting of authorization tokens for the social networks associated with a search, and a location associated with the search.

6. The non-transitory computer readable medium of claim 1 wherein geolocation data inputs related to the person comprise at least one of:

EXIF geotag data from one or more publicly posted photos;

data from websites that append geo-coordinates in meta data;

third party social media sites that append location data to user posts;

social media that include user phone numbers in profiles, bios, or posts; and

social media that include location data in profiles, bios, or posts.

7. The non-transitory computer readable medium of claim 1 wherein the plurality of geolocation data inputs at step f) related to the person are cross-referenced with the at least one time cue and/or spatial cue selected from the group consisting of time of day that a given post was created, data relating to post clustering, data relating to outlier extractions, outputs from an analysis of natural language for location clues and outputs from an analysis of the person'"'"'s social networks.

8. The non-transitory computer readable medium of claim 1 wherein the plurality of geolocation data inputs at step f) related to the person are cross-referenced with at least one other time cue and/or spatial cue.

9. The non-transitory computer readable medium of claim 1 wherein geolocation data inputs comprise artifacts selected from the group comprising one or more online profiles of the person, online social media posts of the person, images (photographic and video) posted online of or by the person, one or more online profiles of a social media connection of the person, online social media posts of a social media connection of the person, images (photographic and video) posted online of or by a social media connection of the person.

10. The non-transitory computer readable medium of claim 7, wherein the computer executable instructions further cause the processor to perform the following operations, comprising:

producing at least one possible location for the person based on the plurality of cross-referenced geolocation data inputs; and

triangulating the at least one possible location against at least one other data source.

11. The non-transitory computer readable medium of claim 10, wherein the at least one other data source further comprises at least one of:

known address information for the person, national change of address database, address registry, marriage registry, general web search, or a third party vendor.

12. The non-transitory computer readable medium of claim 1 wherein identifying the one or more features/keypoints comprises determining local image features that are identifiable through one more image transformations.

13. The non-transitory computer readable medium of claim 1 wherein identifying the one or more features/keypoints comprises determining local image features using Maximally Stable Extremal Regions (MSER).

14. The non-transitory computer readable medium of claim 1 wherein the one or more features/keypoints comprise a salient area encompassing a feature or keypoint.

15. The non-transitory computer readable medium of claim 1 wherein the descriptors comprise binary descriptors for efficient comparison and matching.

16. The non-transitory computer readable medium of claim 1 wherein the descriptors comprise Binary Robust Invariant Scalable Keypoint (“

BRISK”

) descriptors.

17. A non-transitory computer readable medium having stored thereon computer-executable instructions, the computer executable instructions causing a processor to execute a method of creating an integrated, collated social media profile for a person which comprises:

a) probing social media outlets and channels using one or more of social media application programming interfaces (APIs), data provided by third party data compilers, data aggregated directly from web crawlers, using selected person-specific search parameters and person specific context data to form a social media data stream;

b) from the social media data stream, identifying at least one photographic image from a first social media channel and at least one photographic image from a second social media channel;

c) comparing the at least one photographic image from the first social media channel and the at least one photographic image from the second social media channel via a plurality of image recognition steps, comprising for each image;

i) converting the image down to 8×

8 pixels to remove unrequired details and to produce a low frequency image with 64 total pixels;

ii) converting the 8×

8 pixels into grey scale values to simplify further computation;

iii) computing a mean value for all of the 8×

8 pixels;

iv) creating a 64 bit hash where each bit is set to 0 or 1 depending on whether the grey scale value is above or below the mean value;

v) comparing two images using Hamming distance to produce a global image comparison output;

d) comparing the global image comparison output to a threshold to produce a set of resultant global images, wherein the set of resultant global images comprises at least one pair of photographic images from the first social media channel and the second social media channel of the at least one photographic image from the first social media channel and the at least one photographic image from the second social media channel having a global image comparison output less than the threshold;

e) comparing the set of resultant global images from the first social media channel and the set of resultant global images from the second social media channel via a plurality of image recognition steps, comprising for each image, i) identifying one or more features/keypoints;

ii) assigning descriptors to the one or more features/keypoints so that they can be recognized and compared to other images;

iii) weighing the one or more features/keypoints;

iv) matching the one or more features/keypoints;

f) from the social media data stream, i) identifying a plurality of geolocation data inputs related to the person and/or social media connections of the person;

ii) performing a centroid-based analysis on the plurality of geolocation data inputs to form clusters;

iii) classifying and filtering the clusters; and

iv) selecting one or more probable locations of the person based upon the clusters; and

g) determining and corroborating with confidence the photographic images and the geolocation data inputs which are assignable to the person and creating the integrated, collated social media profile, displayable on a dashboard, for the person, wherein the integrated, collated social media profile for the person comprises at least one matched photographic image of the set of resultant global images.

18. The non-transitory computer readable medium of claim 17, wherein comparisons between all descriptors in a pair of images are conducted, distances are sorted according to Hamming distance (d(n)) and an overall score (s) for the image is determined by first sorting all the distances in ascending order as follows:

19. A non-transitory computer readable medium having stored thereon computer-executable instructions, the computer executable instructions causing a processor to execute a method of creating an integrated, collated social media profile for a person which comprises:

a) probing social media outlets and channels using one or more of social media application programming interfaces (APIs), data provided by third party data compilers, data aggregated directly from web crawlers, using selected person-specific search parameters and person specific context data to form a social media data stream;

b) from the social media data stream, identifying at least one photographic image from a first social media channel and at least one photographic image from a second social media channel;

c) comparing the at least one photographic image from the first social media channel and the at least one photographic image from the second social media channel via a plurality of image recognition steps, comprising for each image;

i) converting the image down to 32×

32 pixels in order to simplify Discrete Cosine Transform (DCT) computation;

ii) computing the DCT of the 32×

32 image to produce a 32×

32 collection of frequency scalars;

iii) selecting the top left 8×

8 scalars and disregarding remaining scalars;

iv) computing a mean average of the selected DCT scalars;

v) creating a 64 bit hash where each bit is set to 0 or 1 depending on whether each of the 64 DCT scalars is above or below the mean average;

vi) comparing two images using Hamming distance to produce a global image comparison output;

d) comparing the global image comparison output to a threshold to produce a set of resultant global images, wherein the set of resultant global images comprises at least one pair of photographic images from the first social media channel and the second social media channel of the at least one photographic image from the first social media channel and the at least one photographic image from the second social media channel having a global image comparison output less than the threshold;

e) comparing the set of resultant global images from the first social media channel and the set of resultant global images from the second social media channel via a plurality of image recognition steps, comprising for each image, i) identifying one or more features/keypoints;

ii) assigning descriptors to the one or more features/keypoints so that they can be recognized and compared to other images;

iii) weighing the one or more features/keypoints;

iv) matching the one or more features/keypoints;

f) from the social media data stream, i) identifying a plurality of geolocation data inputs related to the person and/or social media connections of the person;

ii) performing a centroid-based analysis on the plurality of geolocation data inputs to form clusters;

iii) classifying and filtering the clusters; and

iv) selecting one or more probable locations of the person based upon the clusters; and

g) determining and corroborating with confidence the photographic images and the geolocation data inputs which are assignable to the person and creating the integrated, collated social media profile, displayable on a dashboard, for the person, wherein the integrated, collated social media profile for the person comprises at least one matched photographic image of the set of resultant global images.

20. The non-transitory computer readable medium of claim 19, wherein comparisons between all descriptors in a pair of images are conducted, distances are sorted according to Hamming distance (d(n)) and an overall score (s) for the image is determined by first sorting all the distances in ascending order as follows: