3D model generation from 2D images

• US 10,318,102 B2
• Filed: 01/25/2016
• Issued: 06/11/2019
• Est. Priority Date: 01/25/2016
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Inc.
• Inventors: Ito, Daichi, Mech, Radomir, Carr, Nathan A., Fukusato, Tsukasa
• Primary Examiner(s): Broome, Said
• Application Number:
  US15/005,927
• Publication Number:
  US 20170212661A1
• Time in Patent Office: 1,233 Days
• Field of Search: None
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2200/04 involving 3D image data

Abstract

Techniques and systems are described to generate a three-dimensional model from two-dimensional images. A plurality of inputs is received, formed through user interaction with a user interface. Each of the plurality of inputs define a respective user-specified point on the object in a respective one of the plurality of images. A plurality of estimated points on the object are generated automatically and without user intervention. Each of the plurality of estimated points corresponds to a respective user-specified point for other ones of the plurality of images. The plurality of estimated points is displayed for the other ones of the plurality of images in the user interface by a computing device. A mesh of the three-dimensional model of the object is generated by the computing device by mapping respective ones of the user-specified points to respective ones of the estimated points in the plurality of images.

Claims

1. In a digital medium environment to generate a three-dimensional model of an object from a plurality of images having different two-dimensional views of the object, a method implemented by a computing device, the method comprising:

receiving a plurality of inputs by the computing device, the plurality of inputs formed based on user interaction with a user interface, each of the plurality of inputs defining a respective user-specified point on the object in a respective image of the plurality of images;

generating, by the computing device, a plurality of estimated points on the object, each of the plurality of estimated points corresponding to a respective said user-specified point and defined in real time for multiple different images of the plurality of images than the respective image defining the respective user-specified point;

displaying, by the computing device, the plurality of estimated points in the user interface;

mapping, by the computing device, the user-specified points to the estimated points, respectively, in the plurality of images; and

generating, by the computing device, a mesh of the three-dimensional model of the object based on the mapping.

2. The method as described in claim 1, wherein the generating of the plurality of points and the displaying are performed in real time as the plurality of inputs are received by the computing device.

3. The method as described in claim 1, wherein at least one of the plurality of estimated points is moveable in relation to the object through user interaction with the user interface as correcting an error in an estimated location of the at least one of the plurality of estimated points.

4. The method as described in claim 1, wherein a size of the object in at least one of the plurality of images is different than a size of the object in another one of the plurality of images.

5. The method as described in claim 1, wherein the generating of the mesh includes normalizing locations of the user-specified points and locations of the estimated points.

6. The method as described in claim 1, wherein the generating of the mesh includes mirroring at least one of the user-specified points or at least one of the estimated points to model a portion of the object that is not viewable in the plurality of images.

7. The method as described in claim 1, wherein the generating of the plurality of estimated points includes selecting a template automatically and without user intervention.

8. The method as described in claim 1, further comprising receiving at least one input based on user interaction with the user interface that selects a template from a plurality of templates that is to be used to generate a respective said estimated point from a respective said user-specified point.

9. The method as described in claim 1, further comprising displaying the plurality of images simultaneously in the user interface by the computing device, each of the plurality of images including a different respective view of the object.

10. The method as described in claim 1, further comprising receiving, by the computing device, one or more image identification inputs describing a respective view of the object that is captured by respective ones of the plurality of images and wherein the generating of the mesh is based at least in part a relationship of the respective views of the object.

11. The method as described in claim 1, further comprising receiving one or more inputs that specify a location of the object in respective ones of the plurality of images that is used as part of the generating of the plurality of estimated points.

12. In a digital medium environment to generate a three-dimensional model of an object from a plurality of images having different two-dimensional views of the object, a system comprising:

a point manager module implemented at least partially in hardware of a processing system and a computer-readable storage medium to;

receive a plurality of inputs specifying a plurality of user-specified points based on user interaction with a user interface, each of the plurality of inputs defining a respective said user-specified point on the object in a respective image of the plurality of images;

generate a plurality of estimated points on the object, each of the plurality of estimated points corresponding to a respective said user-specified point and defined in real time for different multiple images of the plurality of images that the respective image defining the respective user-specified point;

display the plurality of points in the user interface; and

a mesh generation module implemented at least partially in hardware of the processing system and the computer-readable storage medium to map the plurality of user-specified points to the plurality of estimated points, respectively, and generate a mesh of the three-dimensional model of the object based on the mapping.

13. The system as described in claim 12, wherein at least one of the plurality of estimated points is configured to be moveable in relation to the object based on user interaction with the user interface as correcting an error in an estimated location of the at least one of the plurality of estimated points.

14. The system as described in claim 12, wherein the point manager module is configured to display the plurality of estimated points in real time as the plurality of inputs are received.

15. The system as described in claim 12, wherein a size of the object in at least one of the plurality of images is different than a size of the object in another one of the plurality of images.

16. The system as described in claim 12, wherein the mesh generation module is configured to normalize locations of the user-specified points with respect to locations of the estimated points.

17. The system as described in claim 12, wherein the mesh generation module is configured to mirror at least one of the user-specified points or the estimated points to model a portion of the object that is not viewable in the plurality of images.

18. In a digital medium environment to generate a three-dimensional model of an object from a plurality of images having different two-dimensional views of the object, a method implemented by a computing device, the method comprising:

receiving an input, formed based on user interaction with a user interface of the computing device, defining a user-specified point on the object in a first image of the plurality of images;

displaying an estimated point in real time on the object by the computing device, the estimated point corresponding the user-specified point for a second image of the plurality of images;

receiving at least one input, by the computing device, based on a user interaction with the user interface, the at least one input defining movement of the estimated point on the object in the second image as correcting an error in an estimated location of the at least one of the plurality of estimated points; and

generating a mesh, by the computing device, of the three-dimensional model of the object, the generating including mapping the moved estimated point to the user-specified point.

19. The method as described in claim 18, wherein the displaying is performed in real time as the input defining the user-specified point on the object is received.

20. The method as described in claim 18, wherein a size of the object in at least one of the plurality of images is different than a size of the object in another one of the plurality of images.