Image-based CAPTCHA exploiting context in object recognition

• US 9,317,676 B2
• Filed: 07/08/2013
• Issued: 04/19/2016
• Est. Priority Date: 02/19/2010
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Liu, Jia, Zhu, Bin Benjamin, Li, Qiujie, Li, Shipeng, Xu, Ning
• Primary Examiner(s): Bayat, Ali
• Application Number:
  US13/937,083
• Publication Number:
  US 20130298195A1
• Time in Patent Office: 1,016 Days
• Field of Search: 382/100, 382/103, 382/164, 382/173, 382/282, 382/283, 382/298, 382/305, 707/709, 707/710
• US Class Current: 1/1
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 21/36 by graphic or iconic representation
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 2210/22 Cropping
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation

Abstract

Techniques for an image-based CAPTCHA for object recognition are described. The disclosure describes adding images to a database by collecting images by querying descriptive keywords to an image search engine or crawling images from the Internet.

The disclosure describes generating the image-based CAPTCHA. The image is retrieved from the database, along with objects having significant values. An object is cropped from its image. The portion on the image where the object has been cropped is filled with image inpainting. The process obtains other objects from the database. The object is mixed among the other objects to from a set of candidate objects. A user is asked to select “the object” from the set of candidate objects that fits or matches the image. The image-based CAPTCHA evaluates whether a response, the selection, is from a human or a bot.

Claims

1. A computer storage device storing computer-executable instructions that, when executed, perform acts comprising:

displaying, on a device interface, an image and a set of candidate objects for object recognition, the set of candidate objects including a specific candidate object and one or more other candidate objects, the specific candidate object corresponding to a missing portion of the image, the one or more other candidate objects including at least a first candidate object that does not correspond to the image;

detecting a selection of a candidate object from the set of candidate objects; and

determining whether the selection was made by a human based on whether the candidate object corresponds to the specific candidate object.

2. The computer storage device of claim 1, wherein the determining includes:

determining that the selection of the candidate object is made by a bot in response to identifying the candidate object selected as different from the specific candidate object;

or

determining that the selection of the candidate object is made by a human in response to identifying the candidate object as the specific candidate object.

3. The computer storage device of claim 1, further comprising:

segmenting the image into objects;

assigning a value of significance to each of the objects;

determining that one or more of the objects are significant by determining that each of the one or more objects has an assigned value that is greater than a threshold;

removing the one or more of the objects that are significant in response to determining that a corresponding object includes low level features that are similar to additional low level features in the image; and

adding a remaining significant object into a database as one of the set of candidate objects.

4. The computer storage device of claim 1, further comprising calculating an additional value for an object, the additional value based at least in part on an estimation of a probability that the object is recognizable by humans.

5. The computer storage device of claim 1, further comprising:

retrieving the image and objects segmented from the image from a database; and

cropping an object from the image, wherein the image surrounds the object and the image visually associates the object as a part of the image.

6. The computer storage device of claim 5, further comprising:

filling a cropped portion of the image; and

generating the one or more other candidate objects such that each candidate object has a first shape and a first size similar to a second shape and a second size of the object cropped from the image.

7. The computer storage device of claim 1, further comprising:

evaluating the candidate object based on an aligned position of the candidate object with respect to a position of a cropped object in the image;

in response to determining that the aligned position of the candidate object does not match the position of the cropped object in the image, denying accessing to an application; and

in response to determining that the aligned position of the candidate object matches the position of the cropped object in the image, granting access to the application.

8. The computer storage device of claim 1, wherein the image is obtained by one of crawling for the image from an Internet, querying for the image using keywords for the image through a search engine, or obtaining the image from a private collection.

9. A system, comprising:

one or more processors;

one or more modules stored in the memory and executable by the one or more processors to;

display an image in a first display area, the image having a missing object and an area of the image with the missing object filled in with image inpainting;

display a set of candidate objects in a second display area, the set of candidate objects including the missing object distributed among other candidate objects,

display the image with a candidate object selected from the set of candidate objects in the first display area in response to a drag and drop of the candidate object into the first display area,

evaluate the candidate object with respect to the image based on visual recognition;

determine that the candidate object is selected by a bot in response to identifying the candidate object as not belonging to the image; and

determine that the candidate object is selected by a human in response to identifying the candidate object as the missing object that belongs in the image.

10. The system of claim 9, wherein display of the set of candidate objects includes displaying the set of candidate objects arranged in a carousel formation.

11. The system of claim 9, wherein display of the set of candidate object includes displaying the set of candidate objects arranged in columns and rows.

12. The system of claim 9, further comprising:

detect an aligned position of the candidate object with respect to a position of the missing object in the image;

in response to determining that the aligned position of the candidate object does not match a position of the missing object in the image, deny access to an application; and

in response to determining that the aligned position of the candidate object matches the position of the missing object in the image, grant access to the application.

13. The system of claim 9, wherein the image inpainting provides unnatural details that hint to a human that the candidate object belongs in the area filled with the image inpainting.

14. The system of claim 9, wherein the set of candidate objects share one or more low level features, the low level features including smoothness, texture, and color difference of neighboring segments.

15. The system of claim 9, wherein the candidate object includes semantic information that is different from semantic information of other candidate objects.

16. A computer-implemented method, comprising:

displaying, on a device interface, a set of candidate objects for object recognition and an image having a missing object, the set of candidate images including the missing object and a first candidate object, wherein the first candidate object does not correspond to the image;

detecting a selection of a candidate object from the set of candidate objects; and

determining whether the selection was made by a human based on whether the candidate object corresponds to the missing object.

17. The computer-implemented method of claim 16, wherein the determining includes:

determining that the selection of the candidate object is made by a bot in response to identifying the candidate object as different from the missing object;

or

determining that the selection of the candidate object is made by a human in response to identifying the candidate object as the missing object.

18. The computer-implemented method of claim 16, further comprising:

detecting an aligned position of the candidate object with respect to a position of the missing object in the image; and

in response to determining that the aligned position of the candidate object does not match a position of the missing object in the image, denying access to an application; and

in response to determining that the aligned position of the candidate object matches the position of the missing object in the image, granting access to the application.

19. The computer-implemented method of claim 16, wherein an area of the image that surrounds the missing object includes image inpainting that provides unnatural details, the unnatural details hinting to a human that the candidate object belongs in the area filled with the image inpainting.

20. The computer-implemented method of claim 16, wherein the set of candidate objects share one or more low level features, the low level features including smoothness, texture, and color difference of neighboring segments, and wherein the candidate object includes semantic information that is different from the semantic information of other candidate objects in the set of candidate objects.