Using photogrammetry to aid identification and assembly of product parts

• US 10,168,152 B2
• Filed: 10/02/2015
• Issued: 01/01/2019
• Est. Priority Date: 10/02/2015
• Status: Active Grant

• Current Assignee: Wayfair LLC (Wayfair Inc.)
• Original Assignee: International Business Machines Corporation
• Inventors: Bender, Michael, Childress, Rhonda L., Kumhyr, David B., Spisak, Michael J.
• Primary Examiner(s): Duley, Janese
• Application Number:
  US14/873,260
• Publication Number:
  US 20170097236A1
• Time in Patent Office: 1,187 Days
• Field of Search: None
• CPC Class Codes: G01C 11/06 by comparison of two or more pictures of the same area
• CPC Class Codes: G06F 30/00 Computer-aided design [CAD]
• CPC Class Codes: G06Q 30/016 After-sales
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06V 20/653 by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces
• CPC Class Codes: G06V 2201/12 Acquisition of 3D measurements of objects

Abstract

A user may be aided in modifying a product that is an assemblage of parts. This aid may involve a processor obtaining images of a target part captured by the user on a mobile device camera. The processor may compare, based on the captured images and a plurality of images of identified parts, the target part to the identified parts. Based on the comparison, the processor may determine an identity of the target part. This aid may also involve a processor obtaining images of a first configuration of a partial assembly of the product captured by a mobile device camera. The processor may compare, based on the captured images, the first configuration to a correct configuration of the partial assembly. Based on the comparison, the processor may determine that the first configuration does not match the correct configuration and may notify the user accordingly.

Claims

1. A computer-implemented method for aiding a user in modifying a product, wherein the product is an assemblage of a plurality of parts, the method comprising:

obtaining, by a processor, a plurality of images of a target part of the product captured by the user on a mobile device camera;

generating, by the processor and based on the plurality of images of the target part, a three-dimensional model of the target part;

comparing, by the processor, the target part to a plurality of identified parts by;

overlaying a three-dimensional model of each identified part onto the three-dimensional model of the target part; and

determining, for each overlaid three-dimensional model of an identified part, if corresponding dimensions between the overlaid three-dimensional model of the identified part and the three-dimensional model of the target part are the same lengths; and

determining, by the processor and based on the comparing, an identity of the target part.

2. The method of claim 1, further comprising:

notifying the user of the identity of the target part via the mobile device.

3. The method of claim 1, further comprising:

obtaining an identifier of the product, wherein the product identifier is selected by the user on the mobile device; and

selecting, prior to the comparing and by the processor, the three-dimensional models of the identified parts from a database based on the obtained product identifier.

4. The method of claim 3, wherein the database includes images of parts sorted by product identifiers of products in which the parts are included.

5. The method of claim 1, further comprising:

providing the user with the identity of the target part and a video showing how to install the target part within the assemblage.

6. A computer program product for aiding a user in obtaining a suitable replacement for a broken part included an assembly of a product, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to perform a method comprising:

obtaining, by the processor, a plurality of images of the broken part captured by a mobile device camera;

generating, by the processor and based on the plurality of images of the broken part, a three-dimensional model of the broken part;

obtaining, by the processor, a plurality of images of a potential replacement part captured by the mobile device camera;

generating, by the processor and based on the plurality of images of the potential replacement part, a three-dimensional model of the potential replacement part;

comparing the three-dimensional model of the broken part to the three-dimensional model of the potential replacement part by overlaying the models on top of each other;

identifying, by the processor and based on the comparing, that functional portions of the broken part that are designed to be points of connection with other parts of the product assembly are the same as corresponding functional portions of the potential replacement part and that at least one ornamental feature of the broken part is different from a corresponding at least one ornamental feature of the potential replacement part;

determining, by the processor, based on the identification that the corresponding functional portions are the same, and despite the identification that the corresponding at least one ornamental feature is different, that the potential replacement part is a suitable replacement for the broken part;

notifying, by the processor, the user that the potential replacement part is a suitable replacement for the broken part.

7. A system for aiding a user in identifying an unidentified part of a plurality of parts to be included in an assembly of a product, the system comprising:

a memory;

a processor in communication with the memory, the processor configured to perform a method comprising;

obtaining a plurality of images of the unidentified part of the product captured by the user on a mobile device camera;

determining, based on the plurality of images of the unidentified part, lengths of dimensions of the unidentified part;

generating, based on the plurality of images of the unidentified part, a three-dimensional model of the unidentified part;

obtaining an identifier of the product, wherein the product identifier is selected by the user on the mobile device;

selecting, based on the obtained product identifier and from a product database containing information about different parts included in a plurality of different products, the plurality of parts from among the different parts;

obtaining, based on the selection and from the product database, lengths of dimensions of the plurality of parts;

comparing the lengths of dimensions of the unidentified part to the lengths of the corresponding dimensions of the plurality of parts;

identifying, based on the compared lengths, a subset of the plurality of parts that have dimension lengths within a threshold of the corresponding dimension lengths of the unidentified part;

obtaining, based on the identification of the subset and from the product database, a three-dimensional model of each part within the subset of the plurality of parts;

comparing, for each part within the subset, the three-dimensional model of the unidentified part to the three-dimensional model of that part within the subset by overlaying the three-dimensional model of that part within the subset onto the three-dimensional model of the unidentified part;

determining, based on the compared models, an identify of the unidentified part; and

notifying the user of the identity of the unidentified part.