System and method for capturing and sharing a location based experience

• US 9,965,471 B2
• Filed: 06/17/2015
• Issued: 05/08/2018
• Est. Priority Date: 02/23/2012
• Status: Active Grant

• Current Assignee: Charles D. Huston, Chris Coleman
• Original Assignee: Charles D. Huston, Chris Coleman
• Inventors: Huston, Charles D., Coleman, Chris
• Primary Examiner(s): Drennan, Barry
• Assistant Examiner(s): ROBINSON, TERRELL M
• Application Number:
  US14/741,626
• Publication Number:
  US 20150287241A1
• Time in Patent Office: 1,056 Days
• CPC Class Codes: G06F 16/00 Information retrieval; Database structures therefor; File system structures therefor
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06F 16/954 Navigation, e.g. using categorised browsing
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2219/024 Multi-user, collaborative environment
• CPC Class Codes: G06T 2219/028 Multiple view windows (top-side-front-sagittal-orthogonal)
• CPC Class Codes: H04N 9/8205 involving the multiplexing of an additional signal and the colour video signal

Abstract

A system and method for capturing a location based experience at an event including a plurality of mobile devices having a camera employed near a point of interest to capture random, crowdsourced images and associated metadata near said point of interest. In a preferred form, the images include depth camera information from prepositioned devices around the point of interest during the event. A network communicates images, depth information, and metadata to build a 3D model of the region, preferably with the location of contributors known. Users connect to this experience platform to view the 3D model from a user selected location and orientation and to participate in experiences with, for example, a social network.

Claims

1. A method for experiencing an event at or near a point of interest, comprising:

collecting depth information, and associated metadata at or near a point of interest from one or more prepositioned devices positioned at one or more known locations proximate said point of interest, each device having a depth camera, wherein the captured depth information includes range between a depth camera and one or more points near said point of interest and said metadata include an orientation of the depth camera;

collecting random images and associated metadata randomly captured at or near said point of interest during the event from a plurality of mobile devices accompanying a number of contributors, each mobile device having a depth camera, wherein the associated metadata for each image includes location of the mobile device, an orientation of the camera, and depth information;

receiving at least some of said images, depth information and metadata from some of said devices from a network;

receiving said images, depth information and metadata at an experience platform connected to the network;

processing the images, depth information and metadata to build a 3D model at or near the point of interest; and

viewing the 3D model of the event including at least some of the random images from different perspectives at or near the point of interest.

2. The method of claim 1, at least some of said depth cameras of said mobile devices comprising a stereo triangulation camera.

3. The method of claim 1, wherein the depth camera comprises a time of flight camera, a structured light sensor, or a plenoptic camera.

4. The method of claim 1, wherein said processing step includes using an existing 3D model at or near said point of interest and enhancing said existing 3D model using said random images, depth information, and metadata.

5. The method of claim 1, wherein said random images are crowdsourced from a plurality of contributors without coordination among said contributors.

6. The method of claim 1, wherein said viewing step includes viewing some of said contributors during the event including viewing an avatar of a contributor during said event.

7. The method of claim 1, wherein a remote user views said 3D model during the event.

8. The method of claim 1, wherein said prepositioned devices capture depth information and associated metadata prior to said event to contribute to at least a portion of said 3D model.

9. The method of claim 1, wherein said mobile devices include a depth camera capturing depth information during said event and comprising a time of flight camera, a structured light sensor, stereo triangulation camera, or a plenoptic camera.

10. A method of viewing an event near a point of interest, comprising:

collecting a plurality of random event images and associated metadata randomly acquired near said point of interest during said event and using one or more depth cameras to capture depth information including range between a depth camera and a target;

processing the captured event images, depth information and associated metadata to build a 3D target model of the event including event images near said point of interest;

storing the 3D target model in an experience platform connected to a network;

accessing the experience platform using the network to access the 3D target model;

selecting a viewing location near said point of interest; and

viewing the 3D target model using the selected location during the time of the event.

11. The method of claim 10, including viewing within the 3D target model an advertisement based on context.

12. The method of claim 10, said targets comprising persons, including removing unknown persons from the 3D target model during viewing and including members identified using a social network during viewing.

13. The method of claim 10, wherein at least some of said random images are randomly captured by crowdsource from users equipped with mobile devices having depth cameras without coordination of targets or time of acquisition among users.

14. The method of claim 13, wherein each mobile device includes at least one depth camera for acquiring range between said depth camera and one or more targets.

15. The method of claim 14, wherein the depth camera is a time of flight camera, a structured light sensor, stereo triangulation camera or a plenoptic camera.

16. The method of claim 10, wherein said viewing step includes a user remote from said selected location.

17. The method of claim 10, including analyzing said random images and metadata to determine users'"'"' interests based on images acquired and/or time of acquisition.

18. The method of claim 10, including inserting an augmented reality message onto the 3D target model for viewing.

19. A system of creating an event experience near a point of interest, comprising:

a communication network including a connection for collecting data from one or more prepositioned data collection devices positioned at one or more known locations proximate said point of interest, and including a depth camera for collecting depth information comprising range between a depth camera and a target near said point of interest and metadata;

a communications network for collecting random images and associated metadata randomly acquired from a plurality of mobile devices near the point of interest during the event;

an experience platform connected to said one or more communication networks for receiving depth information, images, and metadata, and having a processor for building a 3D model near said point of interest using said depth information, images, and metadata; and

a communication network for distributing said 3D model for viewing by a plurality of users, wherein the users can view the 3D model of the event during the event.

20. The system of claim 19, some of the users being remote from said point of interest, said processor being operable to build said 3D model for viewing from a point of view selected by said remote user.