Virtualizing objects using object models and object position data

• US 10,192,115 B1
• Filed: 12/13/2017
• Issued: 01/29/2019
• Est. Priority Date: 12/13/2017
• Status: Active Grant

• Current Assignee: Lowe's Companies, Inc.
• Original Assignee: Lowe's Companies, Inc.
• Inventors: Sheffield, Mason E., Shabtai, Josh
• Primary Examiner(s): Rudolph, Vincent
• Assistant Examiner(s): Patel, Pinalben
• Application Number:
  US15/840,567
• Time in Patent Office: 412 Days
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06F 16/9537 Spatial or temporal dependent retrieval, e.g. spatiotemporal queries
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: G06V 20/20 in augmented reality scenes

Abstract

Described herein are a system and methods for generating a record of objects, as well as respective positions for those objects, with respect to a user. In some embodiments, a user may use a user device to scan an area that includes one or more objects. The one or more objects may be identified from image information obtained from the user device. Positional information for each of the one or more objects may be determined from depth information obtained from a depth sensor installed upon the user device. In some embodiments, the one or more objects may be mapped to object models stored in an object model database. The image information displayed on the user device may be augmented so that it depicts the object models associated with the one or more objects instead of the actual objects.

Claims

1. A method of generating a catalog of positioned objects, comprising:

receiving sensor output obtained using one or more input sensors of a user device;

identifying, within the sensor output, one or more objects;

determining, based on the sensor output and based on a location of the user device, a position of the one or more objects;

mapping the one or more objects to one or more corresponding object models within an object model database;

identifying that the one or more objects are to be associated with the determined position;

storing an indication of the one or more object models and the determined position of the one or more objects;

generating an extrapolated image of an area where the one or more objects are located that depicts the area without the one or more objects; and

causing the extrapolated image to be displayed on a display screen such that the one or more objects appear to be absent from the sensor output.

2. The method of claim 1, wherein the sensor output includes at least image information and depth information associated with the one or more objects.

3. The method of claim 1, further comprising:

receiving, from a second user device, a second location associated with the second user device;

determining, based on the second location and the determined position of the one or more objects, that the second user device is within a vicinity of the one or more objects; and

providing, to the second user device, the one or more object models.

4. The method of claim 3, further comprising causing the second user device to augment an image displayed on a display screen of the second user device such that the one or more object model appears to be positioned in the determined position of the one or more objects.

5. The method of claim 1, further comprising:

determining, based on the identification of the one or more objects, whether any one or more other objects not present in the image might be of interest to a person using the method; and

recommending at least one of the one or more other objects to the person.

6. The method of claim 1, further comprising:

identifying that the one or more objects are to be associated with a user based on the user being associated with the user device; and

storing an indication of the user with respect to the one or more object models and the determined position of the one or more objects.

7. The method of claim 1, further comprising:

detecting an event which impacts a geographic area;

determining that the one or more objects are located within the geographic area based on the determined position of the one or more objects; and

assessing an impact of the event based on one or more attributes of the one or more objects.

29. The method of claim 1, further comprising:

providing the one or more object models to the user device; and

causing the one or more object models to be depicted within the extrapolated image.

8. A system comprising:

one or more camera devices;

a processor; and

a memory including instructions that, when executed with the processor, cause the system to, at least;

receive sensor data from the one or more camera devices;

identify, within the sensor data, at least one object, the object being distinct from an area in which the object is located;

determine, based on the sensor data and a location of the one or more camera devices, a position of the at least one object;

match the at least one object with at least one object model stored in an object model database;

store an indication of the at least one object model and the determined position of the at least one object;

generate an extrapolated image of an area where the at least one object is located that depicts the area without the at least one object; and

cause the extrapolated image to be displayed on a display screen such that the at least one object appears to be absent from the sensor output.

9. The system of claim 8, wherein the determined position includes a location and orientation of the at least one object.

10. The system of claim 8, wherein the location of the one or more camera devices comprises GPS coordinates of a user device on which the one or more camera devices was installed.

11. The system of claim 8, wherein one of the one or more camera devices comprises a depth sensor.

12. The system of claim 8, wherein the instructions, when executed with the processor, cause the system to match the at least one object to the at least one object model using one or more machine learning techniques.

13. The system of claim 8, wherein the instructions, when executed with the processor, cause the system to identify the at least one object using one or more object recognition techniques.

14. The system of claim 8, wherein the memory further includes instructions that, when executed with the processor, cause the system to, at least:

determine, based on the identity of the one or more objects, whether any one or more other objects might be of interest to a person using the method; and

recommend at least one of the one or more other objects to the person.

15. An apparatus comprising:

a camera device configured to capture image information;

a depth sensor device configured to capture depth information;

a display screen; and

a mobile application stored in a computer-readable medium that, when executed, causes the apparatus to, at least;

identify, within image information captured by the camera device, one or more objects;

determine a position of the one or more objects from depth information captured by the depth sensor;

provide, to a server in communication with the apparatus, an indication of the one or more objects and the position of the one or more objects to cause the server to generate an extrapolated image of an area where the one or more objects are located that depicts the area without the one or more objects; and

display, on the display screen, the extrapolated image such that the one or more objects appear to be absent from the sensor output.

16. The apparatus of claim 15, wherein the mobile application, when executed, further causes the apparatus to receive, from the server, one or more object models associated with the one or more objects.

17. The apparatus of claim 16, wherein the mobile application, when executed, further causes the one or more object models to be displayed upon the display screen in the area where the one or more objects are located.

18. The apparatus of claim 17, wherein the mobile application, when executed, causes the at least one object to be removed from the image information displayed on the display and replaced by the one or more object models.

19. The apparatus of claim 16, wherein the mobile application, when executed, causes the one or more object models to be received upon determining that the apparatus is within a vicinity of the determined position.

20. The apparatus of claim 15, wherein the mobile application, when executed, causes the apparatus to identify a category associated with the one or more objects.

21. The apparatus of claim 15, wherein the mobile application, when executed, causes the apparatus to identify a type or model of the one or more objects.

22. The apparatus of claim 15, wherein the mobile application, when executed, further causes the apparatus to, at least:

determine, based on the identity of the one or more objects, whether any one or more other objects might be of interest to a person using the apparatus; and

recommend at least one of the one or more other objects to the person.

23. A method of generating a catalog of positioned objects, comprising:

receiving sensor output obtained using one or more input sensors of a user device;

identifying, within the sensor output, one or more objects;

determining, based on the sensor output and based on a location of the user device, a position of the one or more objects;

mapping the one or more objects to one or more corresponding object models within an object model database;

identifying that the one or more objects are to be associated with the determined position;

storing an indication of the one or more object models and the determined position of the one or more objects;

receiving, from a second user device, a second location associated with the second user device;

determining, based on the second location and the determined position of the one or more objects, that the second user device is within a vicinity of the one or more objects; and

providing, to the second user device, the one or more object models.

24. The method of claim 23, further comprising:

causing the second user device to augment an image displayed on a display screen of the second user device such that the one or more object model appears to be positioned in the determined position of the one or more objects.

25. A system comprising:

one or more camera devices;

a processor; and

a memory including instructions that, when executed with the processor, cause the system to, at least;

receive sensor data from the one or more camera devices;

identify, within the sensor data, at least one object, the object being distinct from an area in which the object is located;

determine, based on the sensor data and a location of the one or more camera devices, a position of the at least one object;

match the at least one object with at least one object model stored in an object model database;

store an indication of the at least one object model and the determined position of the at least one object

receive, from a second system, a location associated with the second system;

determine, based on the location and the determined position of the at least one object, that the second system is within a vicinity of the at least one object; and

provide, to the second system, the at least one object model.

26. The system of claim 25, further comprising a display screen, wherein the memory further includes instructions that, when executed with the processor, cause the display screen to display the received at least one second object model so that it appears to be in a location of the at least one second object.

27. An apparatus comprising:

a camera device configured to capture image information;

a depth sensor device configured to capture depth information;

a mobile application stored in a computer-readable medium that, when executed, causes the apparatus to, at least;

identify, within image information captured by the camera device, one or more objects;

determine a position of the one or more objects from depth information captured by the depth sensor;

provide, to a server in communication with the apparatus, an indication of the one or more objects and the position of the one or more objects to cause the server to store the indication of the one or more objects in relation to the position of the one or more objects; and

receive, from the server in response to the server determining that a location of the apparatus is within a vicinity of at least one second object, at least one second object model associated with the at least one second object.

28. The apparatus of claim 27, wherein the mobile application further causes the apparatus to display, on a display screen of the apparatus, the received at least one second object model so that it appears to be in a location of the at least one second object.