Identifying image comments from similar images

• US 10,732,783 B2
• Filed: 12/28/2015
• Issued: 08/04/2020
• Est. Priority Date: 12/28/2015
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Yang, Kuiyuan, Ma, Wei-Ying, Liu, Chunyan, Li, Di, Xu, Pengfei, Rui, Yong, Wang, Baoxun
• Primary Examiner(s): Lee, Ting Z
• Application Number:
  US14/981,816
• Publication Number:
  US 20170185236A1
• Time in Patent Office: 1,681 Days
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: H04L 51/02 using automatic reactions or user delegation, e.g. automatic replies or chatbot-generated messages
• CPC Class Codes: H04L 51/04 Real-time or near real-time messaging, e.g. instant messaging [IM]
• CPC Class Codes: H04L 51/10 Multimedia information
• CPC Class Codes: H04L 51/52 for supporting social networking services

Abstract

An image chat application generates comments to images based on features of the images. In one example, the image chat application searches through a repository of stored image-comment pairs to identify a stored image that is similar to the image, and generates a comment to the image based on an identified stored image-comment pair. In another example, the image chat application may identify and tag particular objects that dominate an image, and may generate a comment to the image based on characteristics of those particular objects. In this second example, the image chat application further generates a comment to the image based on comments previously associated with the identified tag.

Claims

1. A system comprising:

one or more processors;

a computer-readable medium coupled to the one or more processors, including one or more modules that are executable by the one or more processors to;

receive data indicating an image;

extract, by the one or more processors, a first set of feature representations from a first object of the image and a second set of feature representations from a second object of the image;

determine a similarity between the image and individual stored images of a dataset of stored images, based at least in part on the first set of feature representations of the image and the second set of feature representations of the image;

identify a particular stored image from the dataset of stored images, based at least in part on the similarity;

identify a first comment, based at least in part on the particular stored image and the first set of feature representations of the image;

identify a second comment, based at least in part on the particular stored image and the second set of feature representations of the image;

generate, autonomously in response to identifying the first comment and the second comment, a comment by combining a portion of the first comment and a portion of the second comment; and

cause a display of the comment on a user interface of a client device, the comment being displayed as a response to a display of the image.

2. The system of claim 1, wherein the first set of feature representations from the image corresponds to the first object being a first dominant object of the image and the second set of feature representations from the image corresponds to the second object being a second dominant object of the image.

3. The system of claim 1, wherein the client device is associated with a user or a user group, and wherein the individual stored images of the dataset of stored images include at least one tag that identifies a dominant object of the individual stored images, the at least one tag being based at least in part on particular tags that are commonly associated with the user or the user group.

4. The system of claim 1, wherein the one or more modules are further executable by the one or more processors to:

determine that the image is displayed on a web-based social media platform, based at least in part on the data indicating the image; and

retrieve the image from the web-based social media platform.

5. A system comprising:

one or more processors;

a computer-readable medium coupled to the one or more processors, including one or more modules that are executable by the one or more processors to;

receive data indicating an image;

extract, by the one or more processors, a dominant object from the image;

compare the dominant object with a dataset of tagged images to select a first tagged image from the dataset of tagged images similar to the image;

identify a first tag that is associated with the first tagged image;

compare the dominant object with additional datasets of tagged images, the additional datasets associated with the identified first tag;

identify, from the comparison of the dominant object with the additional datasets, a second tagged image that is similar to the dominant object from the image;

identify a second tag that is associated with the second tagged image;

generate a comment using the second tag to describe the dominant object; and

cause a display of the comment on a user interface of a client device, the comment being displayed as a response to a display of the image.

6. The system of claim 5, wherein the one or more modules are executable to use a dataset index to identify at least one association of the additional datasets with the first tag that is associated with the first tagged image.

7. The system of claim 6, wherein the one or more modules are further executable to use the dataset index to identify refinements of the first tag that is associated with the first tagged image.

8. One or more computer-readable storage media including one or more modules that are executable by one or more processors to cause a computing system to:

receive data indicating an image;

determine a similarity between the image and individual stored images of a dataset of stored images, based at least in part on a first set of feature representations of the image including a first object and a second set of feature representations of the image including a second object;

identify a particular stored image from the dataset of stored images, based at least in part on the similarity;

identify a first particular comment from a plurality of comments associated with the particular stored image and the first set of feature representations of the image;

identify a second particular comment from the plurality of comments, based at least in part on the particular stored image and the second set of feature representations of the image;

generate, autonomously in response to the identifications of the first particular comment and the second particular comment, a particular comment by combining portions of the first particular comment and portions of the second particular comment; and

cause a display of the particular comment on a user interface of a client device, the particular comment being displayed as a response to a display of the image.

9. The one or more computer-readable storage media of claim 8, wherein the plurality of comments is based at least in part on one or more of:

comments previously posted by users on a social media platform;

or

stock comments.

10. The one or more computer-readable storage media of claim 8, wherein the one or more modules are further executable by the one or more processors to select the first particular comment randomly from the plurality of comments.

11. The one or more computer-readable storage media of claim 8, wherein the one or more modules are further executable by the one or more processors to identify a tag associated with the particular stored image, and the first particular comment is identified from the plurality of comments based at least in part on the tag.

12. A computer-implemented method, comprising:

receiving, via a client device, data indicating an image;

extracting, from the image, a first set of feature representations having a first feature vector associated with a first object of the image and a second set of feature representations having a second feature vector associated with a second object of the image;

determining a similarity between the image and individual stored images of a dataset of stored images, based at least in part on the first feature vector and the second feature vector;

identifying a particular stored image from the dataset of stored images, based at least in part on the similarity;

identifying a plurality of comments based at least in part on the particular stored image;

identifying a first comment from the plurality of comments, based at least in part on the particular stored image and the first feature vector;

identifying a second comment from the plurality of comments, based at least in part on the particular stored image and the second feature vector;

generating, autonomously in response to identifying the first comment and the second comment, a comment by combining a portion of the first comment and a portion of the second comment; and

causing a display of the comment on a user interface of the client device as a response to a display of the image.

13. The computer-implemented method of claim 12, wherein the plurality of comments is associated with the individual stored images or is associated with one or more tags, the one or more tags being associated with the individual stored images.

14. The computer-implemented method of claim 12, further comprising selecting the second comment randomly from the plurality of comments.

15. The computer-implemented method of claim 12, wherein the individual stored images of the dataset of stored images include one or more tags that identify a dominant object of the individual stored images, the method further comprising identifying a tag from the one or more tags that is associated with the particular stored image, wherein identifying the plurality of comments is further based at least in part on the tag.

16. The computer-implemented method of claim 12, further comprising reducing a first length of the first feature vector and a second length of the second feature vector to a same predetermined feature vector length.

17. The computer-implemented method of claim 12, further comprising reducing a first length of the first feature vector and a second length of the second feature vector progressively through a plurality of non-linear mapping layers associated with the dataset of stored images and the image.

18. The computer-implemented method of claim 17, wherein reducing a first length of the first feature vector and a second length of the second feature vector is performed using at least one of a Top-K most frequent feature dimensionality reduction technique, a principle component analysis technique, or a local sensitive hashing technique.

19. A computer-implemented method, comprising:

receiving, via a client device, data indicating an image;

extracting, from the image, a first set of feature representations having a first feature vector;

determining a similarity between the image and individual stored images of a dataset of stored images, based at least in part on the first feature vector and a second feature vector extracted from the individual stored images;

identifying a particular stored image from the dataset of stored images, based at least in part on the similarity;

identifying a plurality of comments based at least in part on the particular stored image; and

causing, a display of at least a particular comment of the plurality of comments on a user interface of the client device as a response to a display of the image, wherein the individual stored images of the dataset of stored images include one or more tags that identify a dominant object of the individual stored images, the method further comprising identifying a tag from the one or more tags that is associated with the particular stored image, wherein identifying the plurality of comments is further based at least in part on the tag, wherein the tag is a first tag, the particular stored image is a first particular stored image, and the dataset of stored images is a first dataset of stored images, and further comprising;

determining that a number of the plurality of comments is greater than a predetermined threshold number of comments;

identifying a second tag that is associated with the first tag, based at least in part on a dataset index that identifies an association between the first tag and the second tag;

identifying a second dataset of stored images that is associated with the second tag;

determining a similarity between the image and stored images of the second dataset of stored images;

identifying a second particular stored image from the second dataset of stored images, based at least in part on the similarity; and

wherein identifying the plurality of comments is further based at least in part on additional comments associated with the second tag.

20. The computer-implemented method of claim 19, wherein the first tag identifies the dominant object of the image as a tangible entity, and wherein the second tag further describes features commonly associated with the tangible entity.