Grouping digital images based on detected objects

• US 9,875,426 B2
• Filed: 11/08/2016
• Issued: 01/23/2018
• Est. Priority Date: 02/06/2014
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Tohidi, Maryam
• Primary Examiner(s): Dunphy, David F
• Application Number:
  US15/346,700
• Publication Number:
  US 20170053188A1
• Time in Patent Office: 441 Days
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 18/23 Clustering techniques
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06V 10/762 using clustering, e.g. of similar faces in social networks
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: H04N 23/64 Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image

Abstract

A user captures images on a user computing device. The user signs in to an application, which transmits the user'"'"'s images to an account management system, which recognizes objects within the images and assigns one or more object categories to the images and recognizes multiple images comprising objects in a common object category. After receiving user consent, the application groups the images on the user computing device according to object category. The user computing device captures an image of another object. The application transmits the image to the account management system, which detects objects within the image, identifies the object category, and saves the image to the corresponding object category group on the user computing device. After receiving user consent, the account management system finds information for each image in the object category group of images and transmits the information to the user computing device.

Claims

1. A computer-implemented method to organize digital images based on objects detected in images, comprising:

receiving, by one or more computing devices, a set of images, wherein the images are captured by a computing device operated by a user, the set of images comprising location data corresponding to each image of the set of images, wherein the location data is identified by the computing device operated by the user;

identifying, by the one or more computing devices, one or more objects within each image of the set of images;

assigning, by the one or more computing devices, one or more object category labels to each image of the set of images based on the one or more objects identified within each image and based on the location data corresponding to each image;

for each assigned object category label;

detecting, by the one or more computing devices, two or more images within the set of images having the particular assigned object category label;

grouping, by the one or more computing devices and in response to detecting the two or more images within the set of images having the particular assigned object category label, the two or more detected images; and

transmitting, by the one or more computing devices and to the computing device operated by the user, an organized set of images, wherein the organized set of images comprises the set of images arranged in groups of two or more images, each group comprising images assigned a respective particular object category label.

2. The method of claim 1, further comprising:

receiving, by the one or more computing devices and from the computing device operated by the user, one or more subsequent images, wherein the one or more subsequent images are captured by the computing device operated by the user;

identifying, by the one or more computing devices, one or more objects within each image of one or more subsequent images;

assigning, by the one or more computing devices, one or more object category labels to each image of the one or more subsequent images based on the one or more objects identified within each subsequent image;

detecting, by the one or more computing devices, that one or more of the subsequent images has one or more of the same object category labels as a group of two or more images with a particular assigned category label within the organized set of images;

adding, by the one or more computing devices, the one or more of the subsequent images to the group comprising the two or more images assigned the particular assigned object category label within the organized set of images; and

transmitting, by the one or more computing devices and to the computing device operated by the user, a request for consent to find information related to each image of the group of images corresponding to the particular assigned object category label of one or more of the subsequent images.

3. The method of claim 2, further comprising receiving, by the one or more computing devices, location data corresponding to each image of the one or more subsequent images, wherein the location data is captured by the computing device operated by the user.

4. The method of claim 3, wherein assigning the one or more object category labels is further based on the location data corresponding to each image.

5. The method of claim 1, further comprising, receiving, by the one or more computing devices and from the computing device, time stamp data corresponding to each image of the set of images, wherein the time stamp data is identified by the computing device operated by the user, and wherein grouping the one or more detected images within the set of images having a common object category label is based on the time stamp data.

6. The method of claim 1, further comprising:

receiving, using the one or more computing devices and from the computing device operated by the user, a user consent to find information;

identifying, using the one or more computing devices, information relating to each image of the group of images corresponding to the object category label of one or more of the subsequent images;

organizing, using the one or more computing devices, the information to correspond to each image of the group of images corresponding to the object category label of one or more of the subsequent images; and

transmitting, using the one or more computing devices and to the computing device operated by the user, the organized information.

7. The method of claim 6, wherein the information relating to each image of the group of one or more images having a common object category label comprises a price and availability for the object within the image at a merchant associated with the location data.

8. The method of claim 6, wherein the information comprises a set of options for purchasing one or more products associated with the one or more objects in each image in the set of images.

9. The method of claim 6, wherein the information comprises a set of hyperlinks to one or more merchants offering one or more products associated with the one or more objects in each image in the set of images for purchase.

10. A computer program product, comprising:

a non-transitory computer-readable medium having computer-executable program instructions embodied thereon that when executed by a computer cause the computer to organize digital images based on objects detected in images, the computer-executable program instructions comprising;

computer-executable program instructions to receive a set of images captured by a computing device operated by a user, the set of images comprising location data corresponding to each image of the set of images, wherein the location data is identified by the computing device operated by the user;

computer-executable program instructions to identify one or more objects within each image of the set of images;

computer-executable program instructions to assign one or more object category labels to each image of the set of images based on the one or more objects identified within each image and based on the location data corresponding to each image;

for each assigned object category label;

computer-executable program instructions to detect two or more images within the set of images having the particular assigned object category label;

computer-executable program instructions to group the two or more detected images in response to detecting the two or more images within the set of images having the particular assigned object category label; and

computer-executable program instructions to transmit, to the computing device operated by the user, an organized set of images, wherein the organized set of images comprises the set of images arranged in groups of two or more images, each group comprising images assigned a respective particular object category label.

11. The computer program product of claim 10, further comprising:

computer-executable program instructions to receive, from the computing device operated by the user, one or more subsequent images, wherein the one or more subsequent images are captured by the computing device operated by the user;

computer-executable program instructions to identify one or more objects within each image of one or more subsequent images;

computer-executable program instructions to assign one or more object category labels to each image of the one or more subsequent images based on the one or more objects identified within each subsequent image;

computer-executable program instructions to detect that one or more of the subsequent images has one or more of the same object category labels as a group of one or more images with a particular assigned category label within the organized set of images;

computer-executable program instructions to add the one or more of the subsequent images to the group comprising the two or more images assigned the particular assigned object category label within the organized set of images to update the organized set of images;

computer-executable program instructions to identify information relating to each image of the group of images corresponding to the object category label of one or more of the subsequent images;

computer-executable program instructions to organize the information to correspond to each image of the group of images corresponding to the object category label of one or more of the subsequent images; and

computer-executable program instructions to transmit, to the computing device operated by the user, the updated organized set of images comprising the organized information.

12. The computer program product of claim 10, further comprising computer-executable program instructions to receive location data corresponding to each image of the one or more subsequent images, wherein the location data is identified by the computing device operated by the user.

13. The computer program product of claim 10, further comprising:

computer-executable program instructions to receive, from the computing device operated by the user, a request to find information for a group of images having a common object category label;

computer-executable program instructions to identify information relating to each image of the group of two or more images having a common object category label;

computer-executable program instructions to organize the information to correspond to each image of the group of two or more images having a common object category label; and

computer-executable program instructions to transmit, to the computing device operated by the user, the organized information.

14. A system to organize digital images based on objects detected in images, comprising:

a storage device; and

a processor communicatively coupled to the storage device, wherein the processor executes application code instructions that are stored in the storage device to cause the system to;

receive a set of images, wherein the images are captured by a computing device operated by a user, the set of images comprising location data corresponding to each image of the set of images, wherein the location data is identified by the computing device operated by the user;

identify one or more objects within each image of the set of images;

assign one or more object category labels to each image of the set of images based on the one or more objects identified within each image and based on the location data corresponding to each image;

for each assigned object category label;

detect two or more images within the set of images having the particular assigned object category label;

group two or more detected images in response to detecting the two or more images within the set of images having the particular assigned object category label; and

transmit, to the computing device operated by the user, an organized set of images, wherein the organized set of images comprises the set of images arranged in groups of two or more images, each group comprising images assigned a respective particular object category label.

15. The system of claim 14, wherein the processor is further configured to execute computer-executable instructions stored in the storage medium to cause the system to:

receive, from the computing device operated by the user, one or more subsequent images, wherein the one or more subsequent images are captured by the computing device operated by the user;

identify one or more objects within each image of one or more subsequent images;

assign one or more object category labels to each image of the one or more subsequent images based on the one or more objects identified within each subsequent image;

detect that one or more of the subsequent images has one or more of the same object category labels as a group of one or more images with a particular assigned category label within the organized set of images;

add the one or more of the subsequent images to the group comprising the two or more images assigned the particular assigned object category label within the organized set of images to update the organized set of images; and

transmit the updated organized set of images to the computing device operated by the user.

16. The system of claim 15, wherein the processor is further configured to execute computer-executable instructions stored in the storage medium to cause the system to:

transmit, to the computing device operated by the user, a request for consent to find information related to each image of the group of images corresponding to the object category label of one or more of the subsequent images, wherein the user consents to find information;

receive, from the computing device operated by the user, a user consent to find information;

identify information relating to each image of the group of images corresponding to the object category label of one or more of the subsequent images;

organize the information to correspond to each image of the group of images corresponding to the object category label of one or more of the subsequent images; and

transmit, to the computing device operated by the user, the organized information.

17. The system of claim 14, wherein the processor is further configured to execute application code instructions stored on the storage device to cause the system to:

identify information relating to each image of a group of images having a common object category label;

organize the information to correspond to each image of the group of two or more images having a common object category label; and

transmit, to the computing device operated by the user, the organized information.