Systems and methods for three dimensional environmental modeling

• US 10,339,644 B2
• Filed: 05/07/2018
• Issued: 07/02/2019
• Est. Priority Date: 12/03/2015
• Status: Active Grant

• Current Assignee: Echosense LLC
• Original Assignee: Quasar Blu, LLC
• Inventors: Thomas, Mark, Vanbeneden, Michael J.
• Primary Examiner(s): Ahmed, Samir A
• Application Number:
  US15/973,286
• Publication Number:
  US 20180293719A1
• Time in Patent Office: 421 Days
• CPC Class Codes: G06T 15/503 Blending, e.g. for anti-aliasing
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80

Abstract

A system, according to various embodiments, receives images such as videos and photographs from one or more portable computing devices associated with one or more individuals while the portable computing devices are in a particular position within a particular location at a particular time. The system determines a virtual position within a 3-D representation of the particular location that generally corresponds to the particular position and combines the images with the 3-D representation to generate an enhanced 3-D representation of the particular location. The system may create one or more avatars to help users better understand how people will look and move within the space. Users may then login to an enhanced 3-D representation application to view a recreation of the activity at a particular time at a particular location.

Claims

1. A computer-implemented method for generating an enhanced representation of a particular location, the method comprising:

i. receiving, by at least one computer processor, a first image from a first portable computing device associated with a first individual taken while the first portable computing device is in a first position within a particular location at a first time;

ii. receiving, by at least one computer processor, a second image from a second portable computing device associated with a second individual taken while the second portable computing device is in a second position within the particular location at a second time;

iii. determining, by at least one computer processor, a first virtual position within a 3-D representation of the particular location that generally corresponds to the first position;

iv. determining, by at least one computer processor, a second virtual position within the 3-D representation of the particular location that generally corresponds to the second position;

v. after receiving the first and second image, combining, by at least one computer processor, the first image, the second image, and the 3-D representation so that the first position of the first image generally corresponds to the first virtual position within the 3-D representation and the second position of the second image generally corresponds to the second virtual position within the 3-D representation; and

vi. at least partially in response to combining the first image, the second image, and the 3-D representation, generating, by at least one computer processor, an enhanced representation of the particular location that includes the first image, the second image, and the 3-D representation of the particular location.

2. The computer-implemented method of claim 1, wherein the particular location is a dining establishment.

3. The computer-implemented method of claim 2, wherein the first and second individuals are patrons of the dining establishment.

4. The computer-implemented method of claim 3, wherein the first and second individuals each respectively use a same particular software application on the first and second portable computing devices to capture, respectively, the first and second images.

5. The computer-implemented method of claim 2, wherein the first and second individuals each respectively use a same particular software application on the first and second portable computing devices to capture, respectively, the first and second images.

6. The computer-implemented method of claim 5, wherein the first image is a 2-D video.

7. The computer-implemented method of claim 6, wherein the second image is a 2-D video.

8. The computer-implemented method of claim 7, wherein at least a portion of the first and second 2-D videos were taken at the same time.

9. The computer-implemented method of claim 8, wherein the enhanced representation of the particular location is an enhanced 3-D representation of the particular location.

10. The computer-implemented method of claim 9, wherein the method comprises playing back the first and second 2-D videos to recreate a scene that occurred within the dining establishment and to synchronize the playback of the first and second videos so that at least one sound, captured in both the first video and the second video, is heard at substantially the same time during the synchronized playback of the first and second videos.

11. The computer-implemented method of claim 10, wherein the method comprises synchronizing playback of the first and second videos by identifying a particular sound within the videos and determining the timing of the playback of both the first video and the second video so that the particular sound, captured in both the first video and the second video, is heard at substantially the same time during the synchronized playback of the first and second videos.

12. The computer-implemented method of claim 10, wherein the method comprises synchronizing playback of the first and second videos based on respective time indicators associated with the respective first and second videos.

13. The computer-implemented method of claim 5, wherein the first image is a 2-D still picture.

14. The computer-implemented method of claim 1, wherein the method further comprises:

a. determining a first orientation of the first portable computing device using a first gyroscope associated with the first portable computing device;

b. determining a second orientation of the second portable computing device using a second gyroscope associated with the second portable computing device;

c. determining, at least partially based on the determined first orientation and second orientation, a position within the particular location for video playback.

15. The computer-implemented method of claim 1, wherein the step of combining the first image, the second image, and the 3-D representation further comprises:

a. combining a portion of the first image with a portion of the second image; and

b. generating the enhanced representation of the particular location that includes the portion of the first image, the portion of the second image, and the 3-D representation of the particular location, wherein;

i. the portion of the first image comprises less than all of the first image, and the portion of the second image comprises less than all of the second image.

16. The computer-implemented method of claim 1, wherein the method comprises:

a. identifying a first particular object within the first image;

b. identifying a second particular object within the second image;

c. identifying the first particular object within the 3-D representation of the particular location;

d. identifying the second particular object within the 3-D representation of the particular location;

e. superimposing the first image over the 3-D representation of the space so that the first object is in substantially the same position in the first image and the 3-D representation; and

f. superimposing the second image over the 3-D representation of the space so that the second object is in substantially the same position in the second image and the 3-D representation.

17. The computer-implemented method of claim 16, wherein the first particular object is an object selected from a group consisting of:

a. a clock;

b. a piece of furniture;

c. a piece of art;

d. a door; and

e. a window.

18. The computer-implemented method of claim 1, wherein the method comprises generating one or more avatars in the virtual 3-D representation of the particular location, wherein the one or more avatars comprise a depiction of one or more individuals in the particular location.

19. A system for generating an enhanced 3-D representation of a space, the system comprising:

a. at least one processor; and

b. memory, wherein the computer system is adapted for;

i. receiving a 2-D digital image taken in the space from a portable computing device associated with an individual while the portable computing device is in a particular position within the space;

ii. determining a virtual position within a 3-D representation of the space that generally corresponds to the particular position in the 2-D digital image;

iii. combining the 2-D digital image and the 3-D representation of the space so that the particular position of the 2-D digital image generally corresponds to the virtual position within the 3-D representation of the space;

iv. generating an enhanced 3-D representation of the space that includes the combination of the 2-D digital image and the 3-D representation of the space; and

v. generating one or more avatars in the enhanced 3-D representation of the space, wherein the one or more avatars comprise a depiction of one or more individuals in the space.

20. The system of claim 19, wherein the step of generating one or more avatars in the enhanced 3-D representation of the space further comprises:

a. receiving one or more movement patterns for the one or more individuals in the space during a particular time period, wherein the one or more individuals are using a particular application on one or more portable computing devices associated with the one or more individuals;

b. determining one or more virtual movement patterns within the 3-D representation of the space that generally correspond to the one or more movement patterns of the one or more individuals during the particular time period; and

c. generating an enhanced 3-D representation of the space that includes the generated one or more avatars moving in the one or more virtual movement patterns in the 3-D representation of the space.

21. The system of claim 19, wherein the system is further adapted for:

a. receiving a second 2-D digital image taken in the space from a second portable computing device associated with a second individual while the second portable computing device is in a second particular position within the space;

b. determining a second virtual position within the 3-D representation of the space that generally corresponds to the second particular position in the second 2-D digital image;

c. combining the second 2-D digital image and the 3-D representation of the space so that the second particular position of the second 2-D digital image generally corresponds to the second virtual position within the 3-D representation of the space; and

d. generating an enhanced 3-D representation of the space that includes the combination of the second 2-D digital image and the 3-D representation of the space.

22. The system of claim 21, wherein the first and second 2-D digital images are videos.

23. The system of claim 22, wherein at least a portion of the 2-D digital image and the second 2-D digital image were taken at the same time.

24. The system of claim 23, wherein the system is further adapted for:

a. combining the first 2-D digital image, the second 2-D digital image, and the 3-D representation of the space so that the particular position of the 2-D digital image generally corresponds to the virtual position within the 3-D representation of the space and the second particular position of the second 2-D digital image generally corresponds to the second virtual position within the 3-D representation; and

b. generating an enhanced 3-D representation of the space that includes the combination of the 2-D digital image, the second 2-D digital image, and the 3-D representation of the space.

25. The system of claim 24, wherein the step of combining the 2-D digital image, the second 2-D digital image, and the 3-D representation comprises superimposing the 2-D and the second 2-D digital images over the 3-D representation of the space.

26. A system for generating a virtual reality representation of a particular location, the system comprising:

a. at least one processor; and

b. memory, wherein the computer system is adapted for;

i. receiving a first image from a first portable computing device associated with a first individual taken while the first portable computing device is in a first position within a particular location at a first time;

ii. determining a first virtual position within a 3-D representation of the particular location that generally corresponds to the first position;

iii. after receiving the first image, combining the first image and the 3-D representation so that the first position of the first image generally corresponds to the first virtual position within the 3-D representation; and

iv. at least partially in response to combining the first image and the 3-D representation, generating a virtual reality representation of the particular location that includes the first image and the 3-D representation of the particular location.

27. The system of claim 26, wherein the system is further adapted for:

i. receiving a second image from a second portable computing device associated with a second individual taken while the second portable computing device is in a second position within the particular location at a second time;

ii. determining a second virtual position within the 3-D representation of the particular location that generally corresponds to the second position;

iii. combining the second image and the 3-D representation so that the second position of the second image generally corresponds to the second virtual position within the 3-D representation; and

iv. at least partially in response to combining the second image and the 3-D representation, generating a virtual reality representation of the particular location that includes the first image, the second image, and the 3-D representation of the particular location.

28. The system of claim 27, wherein the first and second images are 2-D videos.

29. The system of claim 27, wherein at least a portion of the first and second images were taken at least substantially at the same time.

30. The system of claim 27, wherein combining the first image, the second image, and the 3-D representation further comprises:

a. converting the first image into a first 3-D video image;

b. converting the second image into a second 3-D video image; and

c. generating a virtual reality representation of the particular location by superimposing the first 3-D video image and the second 3-D video image over the 3-D representation of the particular location.

31. The system of claim 30, wherein the virtual reality representation of the particular location comprises a 3-D video.