AUGMENTED REALITY BASED ITEM MODELING

• US 20190325653A1
• Filed: 04/23/2018
• Published: 10/24/2019
• Est. Priority Date: 04/23/2018
• Status: Active Grant

• Current Assignee: Houzz Inc.
• Original Assignee: Houzz Inc.
• Inventors: Yip, Aaron
• Granted Patent: US 10,467,817 B1
• Time in Patent Office: Days
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06T 15/005 General purpose rendering architectures
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2210/04 Architectural design, interior design

Abstract

Disclosed are various embodiments for simulating one or more virtual objects (e.g., renders) in specified spatial areas of a real-world environment. Options of item models for modeling in a given spatial area can be filtered based on specified dimensions and identified features of an image of a given spatial area. A selected item model can be rendered and continuously updated on a display device as the client device is physical moved.

Claims

1. A method comprising:

displaying, on a display device of a client device, an image sequence depicting a real-world environment, the image sequence generated using an image sensor of the client device;

generating, on the client device, a virtual frame in a three-dimensional (3D) model of a room, the virtual frame created by receiving placements, via inputs to the display device, of multiple virtual points in the image sequence as the image sequence is displayed on the display device, the multiple virtual points being endpoints of one or more virtual dimensions of the virtual frame, the virtual frame being a two-dimensional parallelogram having a defined first dimension, a defined second dimension, and an undefined third dimension;

transmitting, to a networked server, a request specifying 3D models of items that have first dimension values that fit the defined first dimension and that have second dimension values that fit the defined second dimension of the virtual frame generated on the client device;

displaying, on a user interface of the display device, a plurality of items matching the defined first dimension and the defined second dimension and have various third dimension values due to the undefined third dimension of the virtual frame; and

in response to a selection of an item of the plurality of items, displaying, on the display device, a modified image sequence that shows the image sequence and one or more renders of the selected item, the one or more renders being renders of a 3D model of the selected item as arranged at the virtual frame in the 3D model of the room.

2. The method of claim 1, further comprising:

generating, on the client device, an environment category for the real-world environment by applying one or more convolutional neural networks to the image sequence, the one or more convolutional neural networks trained on image features in real-world images.

3. The method of claim 2, wherein the request to the networked server further specifies 3D models of items that match the environment category.

4. The method of claim 2, wherein the environment category is from a plurality of environment categories that describe different types of environments, and wherein the real-world images are images of the different types of environments.

5. The method of claim 4, wherein the different types of environments include one or more of:

bedroom, bathroom, kitchen, living room, or outside.

6. The method of claim 4, wherein the one or more convolutional neural networks is an object recognition neural network, and the image features are images of real-world items.

7. The method of claim 6, wherein generating the environment category comprises:

identifying, using the one or more convolutional neural networks, a real-world item depicted in the image sequence; and

determining the environment category based on the identified real-world item being pre-associated with the environment category.

8. The method of claim 1, wherein the defined first dimension is a fixed height data value generated from placing some of the multiple virtual points of the virtual frame, and the defined second dimension is a fixed width data value generated from placing some of the multiple virtual points of the virtual frame, and the undefined third dimension is a height dimension.

12. The method of claim 3, wherein the one or more convolutional neural networks generates a plurality of environment categories including the environment category;

and wherein the environment category is selected based on meeting a pre-specified threshold.

13. The method of claim 12, further comprising:

identifying additional environment categories that meet the pre-specified threshold, wherein the request further specifies 3D models of items that match the additional environment categories.

14. The method of claim 1, wherein the client device is a mobile client device and the display device substantially fills one side of the mobile client device.

9. (canceled)

10. (canceled)

11. (canceled)

15. A system comprising:

one or more processors of a machine;

an image sensor;

a display device; and

a memory storing instructions that, when executed by the one or more processors, cause the machine to perform operations comprising;

displaying, on the display device, an image sequence depicting a real-world environment, the image sequence generated using the image sensor;

generating a virtual frame in a three-dimensional (3D) model of a room, the virtual frame created by receiving placements, via inputs to the display device, of multiple virtual points in the image sequence as the image sequence is displayed on the display device, the multiple virtual points being endpoints of one or more virtual dimensions of the virtual frame, the virtual frame being a two-dimensional parallelogram having a defined first dimension, a defined second dimension, and an undefined third dimension;

transmitting, to a networked server, a request specifying 3D models of items that have first dimension values that fit the defined first dimension and that have second dimension values that fit the defined second dimension of the virtual frame;

displaying, on a user interface in the display device, a plurality of items matching the defined first dimension and the defined second dimension and have various third dimension values due to the undefined third dimension of the virtual frame;

in response to a selection of an item of the plurality of items, displaying, on the display device, a modified image sequence that shows the image sequence and one or more renders of the selected item, the one or more renders being renders of a 3D model of the selected item as arranged at the virtual frame in the 3D model of the room.

16. The system of claim 15, the operations further comprising:

generating an environment category for the real-world environment by applying one or more convolutional neural networks to the image sequence, the one or more convolutional neural networks trained on image features in real-world images.

17. The system of claim 16, wherein the request to the networked server further specifies 3D models of items that match the environment category.

18. The system of claim 16, wherein the environment category is from a plurality of environment categories that describe different types of environments, and wherein the real-world images are images of the different types of environments.

19. A non-transitory machine-readable storage medium embodying instructions that, when executed by a machine, cause the machine to perform operations comprising:

displaying on a display device, an image sequence depicting a real-world environment, the image sequence generated using an image sensor;

generating a virtual frame in a three-dimensional (3D) model of a room, the virtual frame created by receiving placements, via inputs to the display device, of multiple virtual points in the image sequence as the image sequence is displayed on the display device, the multiple virtual points being endpoints of one or more virtual dimensions of the virtual frame, the virtual frame being a two-dimensional parallelogram having a defined first dimension, a defined second dimension, and an undefined third dimension;

transmitting, to a networked server, a request specifying 3D models of items that have first dimension values that fit the defined first dimension and that have second dimension values that fit the defined second dimension of the virtual frame;

displaying, on a user interface in the display device, a plurality of items matching the defined first dimension and the defined second dimension and have various third dimension values due to the undefined third dimension of the virtual frame;

in response to a selection of an item of the plurality of items, displaying, on the display device, a modified image sequence that shows the image sequence and one or more renders of the selected item, the one or more renders being renders of a 3D model of the selected item as arranged at the virtual frame in the 3D model of the room.

20. The non-transitory machine-readable storage medium of claim 19, the operations further comprising:

generating, an environment category for the real-world environment by applying one or more convolutional neural networks to the image sequence, the one or more convolutional a neural networks trained on image features in real-world images.