System and method for capturing and sharing a location based experience

• US 10,600,235 B2
• Filed: 12/19/2017
• Issued: 03/24/2020
• Est. Priority Date: 02/23/2012
• Status: Active Grant

• Current Assignee: Charles D. Huston, Chris Coleman
• Original Assignee: Charles D. Huston, Chris Coleman
• Inventors: Huston, Charles D., Coleman, Chris
• Primary Examiner(s): Robinson, Terrell M
• Application Number:
  US15/847,148
• Publication Number:
  US 20180108172A1
• Time in Patent Office: 826 Days
• CPC Class Codes: G06F 16/00 Information retrieval; Database structures therefor; File system structures therefor
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06F 16/954 Navigation, e.g. using categorised browsing
• CPC Class Codes: G06Q 30/0255 based on user history
• CPC Class Codes: G06Q 30/0277 Online advertisement
• CPC Class Codes: G06Q 50/01 Social networking
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 19/003 Navigation within 3D models or images
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2219/024 Multi-user, collaborative environment
• CPC Class Codes: G06T 2219/028 Multiple view windows (top-side-front-sagittal-orthogonal)
• CPC Class Codes: G06T 7/521 from laser ranging, e.g. using interferometry; from the projection of structured light
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: H04N 9/8205 involving the multiplexing of an additional signal and the colour video signal

Abstract

A system and method for capturing a location based experience at an event including a plurality of mobile devices having a camera employed near a point of interest to capture random, crowdsourced images and associated metadata near said point of interest. In a preferred form, the images include depth camera information from prepositioned devices around the point of interest during the event. A network communicates images, depth information, and metadata to build a 3D model of the region, preferably with the location of contributors known. Users connect to this experience platform to view the 3D model from a user selected location and orientation and to participate in experiences with, for example, a social network.

Claims

1. A method to build or update a 3D model, comprising:

wirelessly collecting crowdsourced images and associated metadata captured near a point of interest with a plurality of mobile devices accompanying a number of crowdsource contributors, one or more mobile devices having a depth camera, wherein the associated metadata includes approximate location of a mobile device, an orientation of the camera, and range between the depth camera and a target at or near said point of interest;

receiving at least some of said images and associated metadata including range metadata from said mobile devices at an image processing server; and

processing at least some of the received images and associated metadata including range metadata on said image processing server to determine the location of one or more targets proximate said point of interest to build or update a 3D model near the point of interest,

selecting a first image and associated metadata including range metadata from a first mobile device based at least in part on the proximity of the approximate mobile device camera location and orientation to said point of interest,

selecting a second image and associated metadata including range metadata from a mobile device based at least in part on the proximity of the approximate mobile device camera location and orientation to said point of interest,

determining a common target in the first and second images,

determining a first location of said common target in the 3D model using at least in part said associated metadata including range metadata from said first and second image, said range metadata including range between a mobile device depth camera and said common target.

2. The method of claim 1, at least some of said mobile devices comprising a depth camera having two or more conventional cameras in a stereo triangulation.

3. The method of claim 1, the depth camera comprising a time of flight camera, a structured light sensor, a stereo triangulation or a light field camera.

4. The method of claim 1, wherein said processing step includes using an existing 3D model proximate the common target and updating said existing 3D model using said first location of said common target.

5. The method of claim 1, including inserting a message into said 3D model at a geo-referenced position relative to the common target first location.

6. The method of claim 1, selecting a third image having associated metadata including range metadata from a mobile device to said common target, identifying said common target in the third image, and determining a second location of said common target in the 3D model using range metadata from said mobile device to said common target.

7. A system for developing a 3D model for use in artificial reality comprising:

a network for wirelessly receiving images and metadata from a plurality of mobile devices at least some having a camera to capture images and associated metadata near a point of interest, wherein the metadata for each image includes approximate location of the mobile device and the approximate orientation of the camera,

at least some of said mobile devices including a depth camera capturing range metadata between the depth camera and targets at or near said point of interest,

an image processing server connected to the network for receiving said images and metadata, wherein the server processes the images and metadata to develop a 3D model of one or more targets proximate the point of interest based at least in part on images and range metadata from said depth camera,

the image processing server being operable to select a first image and associated metadata including range proximate to the point of interest using the approximate location of a first mobile device and the orientation of a first camera associated with said first image,

the image processing server being operable to select a second image and associated metadata including range proximate to the point of interest using the approximate location of a second mobile device and the orientation of a camera associated with said second image, and

said 3D model comprising the geo-referenced location of a number of said targets near said point of interest where target location is based at least in part on mobile device approximate location and range metadata from said depth camera.

8. The system of claim 7 wherein the images are crowdsourced at random from users of the mobile devices.

9. The system of claim 7, including an experience platform having said 3D model near the point of interest and a user connected to the experience platform can view targets associated with a user selected location and orientation.

10. The system of claim 7, wherein the image processing server is operable to stitch a number of images together to form a panorama.

11. The system of claim 7, the mobile device approximate location metadata comprising GPS, Wi-Fi, Cell ID, repeater, or RSS information.

12. The system of claim 7, wherein said 3D model comprises a mesh having a number of feature points and some of said geo-referenced targets serving as feature points in said mesh.

13. The system of claim 7, including devices having a depth sensor in fixed locations proximate said point of interest.

14. The system of claim 7, wherein some of the mobile devices including a depth camera incorporated into glasses worn by a mobile user.

15. The system of claim 7, the metadata including time of image acquisition and said first and second images are selected based at least in part on time of image acquisition.

16. The system of claim 7, said time of image acquisition being about the time of an event near the point of interest.

17. The system of claim 7, at least some of the mobile devices including a light field camera for capturing at least a portion of a light field proximate the mobile device location and orientation.

18. The system of claim 7, at least some of the mobile devices being operable to insert a geo-referenced message into the 3D model at or near said point of interest.

19. A method to build or update a 3D model for use in an artificial reality application comprising:

crowdsourcing a plurality of random images and associated metadata wirelessly transmitted from a plurality of mobile devices each having a camera where the metadata includes an approximate camera location and orientation and at least some of said crowdsourced images include depth metadata;

selecting a first image and associated depth metadata including approximate range between a first mobile device and a point of interest, said selecting based at least in part on a proximity of the approximate first mobile device camera location and orientation to a point of interest;

selecting a second image and associated depth metadata including approximate range between a mobile device and a point of interest, said selecting based at least in part on a proximity of the approximate camera location and orientation to said point of interest;

processing the first and second images to—

identify one or more targets in the first and second images at or near the point of interest,

detect a target common in each image using a feature point detection and matching process;

determining a first location of said common target in a 3D model using the camera location, orientation, and depth metadata a associated with said first and second images.

20. The method of claim 19, selecting a third image from a mobile device based at least in part on a proximity of an approximate camera location and orientation to the point of interest, identifying said common target in the third image, and determining a second location of said common target in the 3D model using the camera location and orientation metadata from said first, second, and third images.

21. The method of claim 20, wherein one or more additional images are selected based on proximity of camera location and orientation and subsequent locations of said common target in the 3D model is determined.

22. The method of claim 21, applying a converging algorithm to additional images to determine more precise locations of the common target.

23. The method of claim 19, wherein the second image is acquired from the first mobile device where the second image is acquired at a first mobile device camera location different than the approximate location of the first image.

24. The method of claim 19, where in the second image is acquired from a second mobile device where the second image is acquired at a camera location different than the approximate location of the first image.

25. The method of claim 19, wherein the first location of the common target is determined using photogrammetry based on camera location, orientation, and depth metadata from said first and second images.

26. The method of claim 19, wherein a first mobile device includes a depth camera and the first location of the common target is determined using at least in part a range between the depth camera of the first mobile device and the common target.

27. The method of claim 26, the depth camera comprises a time of flight camera, a structured light sensor, a stereo triangulation or a light field camera.

28. The method of claim 19, wherein the feature point detection and matching process to detect a target common in each image comprises a SIFT or SIFT variation approach.

29. The method of claim 19, including inserting a message into said 3D model at a geo-referenced position relative to the common target location.

30. The method of claim 29, including viewing the message in the 3D model comprising an advertisement based on context.

31. The method of claim 19, including updating a preexisting 3D model using said common target location.

32. The method of claim 19, wherein some of the images are randomly captured by crowdsource from users equipped with mobile devices, with the images captured about the same time to build or update a 3D model representative of an approximate time.

33. The method of claim 19, one of the mobile devise includes a light field camera for capturing a light field near the common target.