Object identification in visual media

• US 9,619,705 B1
• Filed: 05/08/2015
• Issued: 04/11/2017
• Est. Priority Date: 03/14/2013
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Nordstrom, Paul G.
• Primary Examiner(s): Bayat, Ali
• Application Number:
  US14/707,495
• Time in Patent Office: 704 Days
• Field of Search: 382218
• CPC Class Codes: G06F 2218/12 Classification; Matching
• CPC Class Codes: G06V 10/255 Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 20/20 in augmented reality scenes

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for determining the identity of an object in an image where the object in the image is in a disassembled state. In one aspect, a method includes accessing previous interactive sessions, each of the interactive sessions including images of a reference object in one or more disassembled states and each of the interactive sessions specifying an identity of the reference object in an assembled state; processing an image of a first object to identify characteristics of the first object, the first object being in a disassembled state in the image; comparing the image of the first object in the disassembled state to images of reference objects in disassembled states; and determining an identity of the first object based on the comparison and the identities of the reference objects in assembled states specified in the interactive sessions.

Claims

1. A method comprising:

accessing a plurality of images depicting a reference object in disassembled states, wherein the reference object in a disassembled state requires the assembly of one or more additional parts to be in the assembled state;

determining that the reference object is depicted in particular images of the plurality of images in particular states of disassembly based on changes in the depiction of the reference object between two or more images;

processing an image of a first object to identify characteristics of the first object, the first object being in a disassembled state in the image;

comparing the image of the first object in the disassembled state to images of the reference object in disassemble states; and

identifying one or more additional parts required for the first object in the disassembled state to be in an assembled state based on the comparison of the image of the first object in the disassembled state to images of the reference object in disassemble states.

2. The method of claim 1, further comprising:

identifying one or more assembled parts of the first object in the disassembled state based on the comparison, each of the one or more assembled parts being an existing part of the first object in the disassembled state.

3. The method of claim 1, further comprising:

identifying one or more broken parts of the first object in the disassembled state based on the comparison, each of the one or more broken parts being an existing part of the first object in the disassembled state that is broken.

4. The method of claim 1, further comprising determining the first object by identifying the first object as being most similar to the reference object.

5. The method of claim 1, comprising:

comparing the image of the reference object in an assembled state to images of other reference objects in assembled states; and

identifying one or more broken parts of the reference object in the assembled state based on the comparison of the image of the reference object in the assembled state to the images of the other reference objects in assembled states, each of the one or more broken parts being an existing part of the reference object in the assembled state that is broken.

6. The system of claim 1, the operation further comprising:

identifying one or more assembled parts of the first object in the disassembled state based on the comparison, each of the one or more assembled parts being an existing part of the first object in the disassembled state.

7. The system of claim 1, the operations further comprising:

identifying one or more broken parts of the first object in the disassembled state based on the comparison, each of the one or more broken parts being an existing part of the first object in the disassembled state that is broken.

8. The system of claim 1, the operations further comprising determining the first object by identifying the first object as being most similar to the reference object.

9. The system of claim 1, the operations further comprising:

comparing the image of the reference object in an assembled state to images of other reference objects in assembled states; and

identifying one or more broken parts of the reference object in the assembled state based on the comparison of the image of the reference object in the assembled state to the images of the other reference objects in assembled states, each of the one or more broken parts being an existing part of the reference object in the assembled state that is broken.

10. A system comprising:

one or more data processors; and

instructions stored on computer-readable storage apparatus that when executed by the one or more data processors cause the one or more data processors to perform operations comprising;

accessing a plurality of images depicting a reference object in disassembled states, wherein the reference object in a disassembled state requires the assembly of one or more additional parts to be in the assembled state;

determining that the reference object is depicted in particular images of the plurality of images in particular states of disassembly based on changes in the depiction of the reference object between two or more images;

processing an image of a first object to identify characteristics of the first object, the first object being in a disassembled state in the image;

comparing the image of the first object in the disassembled state to images of the reference object in disassemble states; and

identifying one or more additional parts required for the first object in the disassembled state to be in an assembled state based on the comparison of the image of the first object in the disassembled state to images of the reference object in disassemble states.

11. A non-transitory computer-readable storage medium having instructions stored thereon, which, when executed by one or more data processors, cause the one or more processor to perform operations comprising:

accessing a plurality of images depicting a reference object in disassembled states, wherein the reference object in a disassembled state requires the assembly of one or more additional parts to be in the assembled state;

determining that the reference object is depicted in particular images of the plurality of images in particular states of disassembly based on changes in the depiction of the reference object between two or more images;

processing an image of a first object to identify characteristics of the first object, the first object being in a disassembled state in the image;

comparing the image of the first object in the disassembled state to images of the reference object in disassemble states; and

identifying one or more additional parts required for the first object in the disassembled state to be in an assembled state based on the comparison of the image of the first object in the disassembled state to images of the reference object in disassemble states.

12. The non-transitory computer-readable storage medium of claim 11, the operation further comprising:

identifying one or more assembled parts of the first object in the disassembled state based on the comparison, each of the one or more assembled parts being an existing part of the first object in the disassembled state.

13. The non-transitory computer-readable storage medium of claim 11, the operations further comprising:

identifying one or more broken parts of the first object in the disassembled state based on the comparison, each of the one or more broken parts being an existing part of the first object in the disassembled state that is broken.

14. The non-transitory computer-readable storage medium of claim 11, the operations further comprising determining the first object by identifying the first object as being most similar to the reference object.

15. The non-transitory computer-readable storage medium of claim 11, the operations further comprising:

comparing the image of the reference object in an assembled state to images of other reference objects in assembled states; and

identifying one or more broken parts of the reference object in the assembled state based on the comparison of the image of the reference object in the assembled state to the images of the other reference objects in assembled states, each of the one or more broken parts being an existing part of the reference object in the assembled state that is broken.