SYSTEM FOR MULTIMEDIA SPATIAL ANNOTATION, VISUALIZATION, AND RECOMMENDATION

• US 20180374276A1
• Filed: 05/24/2018
• Published: 12/27/2018
• Est. Priority Date: 04/04/2016
• Status: Active Grant

• Current Assignee: Occipital, Inc.
• Original Assignee: Occipital, Inc.
• Inventors: Powers, Jeffrey Roger, Reddy, Vikas, Schiff, Alex
• Granted Patent: US 10,395,435 B2
• Time in Patent Office: Days
• CPC Class Codes: G06Q 30/0631 Item recommendations
• CPC Class Codes: G06Q 30/0643 Graphical representation of items or shoppers
• CPC Class Codes: G06T 15/005 General purpose rendering architectures
• CPC Class Codes: G06T 15/10 Geometric effects
• CPC Class Codes: G06T 15/50 Lighting effects
• CPC Class Codes: G06T 17/205 Re-meshing
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2210/04 Architectural design, interior design
• CPC Class Codes: G06T 2215/16 Using real world measurements to influence rendering
• CPC Class Codes: G06T 2219/004 Annotating, labelling
• CPC Class Codes: G06T 2219/2004 Aligning objects, relative positioning of parts
• CPC Class Codes: G06T 2219/2021 Shape modification
• CPC Class Codes: G06T 7/62 of area, perimeter, diameter or volume

Abstract

A system configured to provide a three-dimensional representation of a physical environment. The three-dimensional representation including annotation data associated with particular objects and/or viewpoints of the three-dimensional representation. In some cases, the viewpoints may be rendered using image data associated with a photograph captured from a corresponding viewpoint within the physical environment.

Claims

1. A system comprising:

one or more processors;

non-transitory computer-readable media storing computer-executable instructions, which when executed by the one or more processors cause the one or more processors to perform operations comprising;

determining physical characteristics of a physical environment from a three dimensional (3D) virtual model of the physical environment;

identifying one or more objects within the physical environment based at least in part on the 3D virtual model and one or more stored object models;

determining features of individual ones of the one or more objects;

determining user preferences based at least in part on the 3D virtual model;

identifying a recommended object for the physical environment based at least in part on the characteristics of the physical environment, the features of the individual one or more objects, and the user preferences;

selecting a location for the recommended object within the 3D virtual model;

rendering the recommended object within the 3D virtual model at the location;

annotating the 3D virtual model at the location, the annotation including information related to the recommended object and a purchase option;

receiving a user selection of the purchase option; and

placing an order for the recommended object on behalf of the user.

2. The system as recited in claim 1, wherein the physical characteristics of the physical environment include geometrics of the physical environment, materials associated with the physical environment, lighting conditions, proximity to objects, stylistic similarity to other objects, or stylistic similarity to the physical environment.

3. The system as recited in claim 1, wherein the operations further comprise:

obtaining a plurality of first images of a physical environment; and

generating the 3D virtual model based at least partly on the plurality of first images, prior to determining the physical characteristics of the physical environment.

4. The system as recited in claim 1, wherein the operations further comprises displaying the 3D virtual model to the user as a 3D virtual representation of the physical environment.

5. The system as recited in claim 1, wherein identifying a recommended object includes determining the recommended object fits within the available geometries of the physical environment.

6. The system as recited in claim 1, wherein identifying the recommended object and selecting the location is based at least in part on at least one of:

one or more stylistic similarities between the recommended object and the one or more objects;

climate or environmental related data;

geolocation of the physical environment;

age of the one or more objects;

age of the physical environment;

physical properties of the recommended object;

physical properties of the one or more objects;

proximity between surfaces in the physical environment, the one or more objects, and the location;

or

physical properties of the physical environment.

7. The system as recited in claim 1, wherein selecting a location includes identifying a first object of the one or more objects to replace with the recommended object and setting the location as the location of the first object.

8. The system as recited in claim 1, wherein:

identifying a recommended object includes receiving an object input from a service provider within the 3D model; and

selecting a location includes receiving a location input from the service provider.

9. A method comprising:

receiving image data associated with a physical environment;

generating a 3D virtual model of the physical environment;

communicatively coupling a user of a 3D virtual model with a service provider, the communicatively coupling allowing both the user and the service provider to interact with the 3D virtual model and each other within the 3D virtual model;

receiving an input to alter at least one of an object or a characteristic of a surface within the 3D model from the service provider; and

re-rendering the 3D virtual model based at least in part on the input.

10. The method as recited in claim 9, further comprising:

receiving a user request for a service via the 3D virtual model;

generating a list of service providers based at least in part on the service and data known about the user;

presenting the list of service providers to the user via the 3D virtual model;

receiving a selection of a particular service provider included on the list of service provider from the user via the 3D virtual model; and

wherein communicatively coupling the user of a 3D virtual model with the service provider includes communicatively coupling the user of a 3D virtual model with particular service provider.

11. The method as recited in claim 9, further comprising placing an order behalf of the user with a third-party system, the order related to at least one of the object or the surface.

12. The method as recited in claim 9, further comprising providing at least one statistic or metric related to the physical environment associated with the 3D virtual model to the service provider, prior to receiving the input.

13. The method as recited in claim 9, at least one of

estimating changes to a usage metric about the physical environment, caused by the input to 3D virtual model over time;

or

re-rendering effects of the changes within the 3D virtual model to illustrate the effects on the physical environment

14. The method as recited in claim 9, further comprising receiving a second input to alter a object, set of objects, or theme of the physical environment;

and re-rendering the 3D virtual model to include those changes while the 3D virtual model is being viewed by the user and the service provider.

15. The method as recited in claim 9, further comprising highlighting the object or the characteristic of the surface to bring the object or the characteristic of the surface to the attention of the service provider, the highlighting indicating damage to the object or the surface.

16. A method comprising:

communicatively coupling a user of the 3D virtual model with a service provider, the communicatively coupling allowing both the user and the service provider to interact with the 3D virtual model and each other within the 3D virtual model;

providing user data and physical environment data to the service provider via the 3D virtual model;

receiving a recommendation related to an object within the physical environment from the service provider;

re-rendering the object within the 3D virtual model based at least in part on the recommendation; and

placing an order based on the recommendation related to the object on behalf of the user with a third-party system.

17. The method as recited in claim 16, further comprising:

determining physical characteristics of a physical environment from a three dimensional (3D) virtual model of the physical environment;

identifying one or more objects within the physical environment based at least in part on the 3D virtual model and one or more stored object models;

determining features of individual ones of the one or more objects; and

wherein the recommendation is related to the features.

18. The method as recited in claim 16, further comprising:

selecting the service provider to communicatively couple to the user via the 3D virtual model based at least in part on user preferences associated with the user.

19. The method as recited in claim 18, further comprising:

receiving selecting of a location for the recommended object within the 3D virtual model; and

wherein re-rendering the object within the 3D virtual model based at least in part on the recommendation includes rendering the recommended object at the location.

20. The method as recited in claim 19, further comprising:

receiving an input from the service provider related to the object; and

annotating the 3D virtual model at the location, the annotation including information related to the input from the service provider.