REAL-TIME SHARED AUGMENTED REALITY EXPERIENCE

• US 20160133230A1
• Filed: 11/11/2014
• Published: 05/12/2016
• Est. Priority Date: 11/11/2014
• Status: Abandoned Application

• Current Assignee: Bent Image Lab, LLC
• Original Assignee: Bent Image Lab, LLC
• Inventors: Daniels, Oliver Clayton, Daniels, David Morris, Di Carlo, Raymond Victor
• Application Number:
  US14/538,641
• Publication Number:
  US 20160133230A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/147 using display panels
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G09G 2370/022 Centralised management of display operation, e.g. in a server instead of locally
• CPC Class Codes: G09G 2370/04 Exchange of auxiliary data, i.e. other than image data, between monitor and graphics controller

Abstract

A system is provided for enabling a shared augmented reality experience. The system comprises zero, one or more on-site devices for generating augmented reality representations of a real-world location, and one or more off-site devices for generating virtual augmented reality representations of the real-world location. The augmented reality representations include data and or content incorporated into live views of a real-world location. The virtual augmented reality representations of the AR scene incorporate images and data from a real world location and include additional content used in an AR presentation. The on-site devices synchronize the content used to create the augmented reality experience with the off-site devices in real time such that the augmented reality representations and the virtual augmented reality representations are consistent with each other.

Claims

23. A system for providing a shared augmented reality experience, the system comprising:

one or more on-site devices for generating augmented reality representations of a real-world location; and

one or more off-site devices for generating virtual augmented reality representations of the real-world location;

wherein the augmented reality representations include content visualized and incorporated with live views of the real-world location;

wherein the virtual augmented reality representations include the content visualized and incorporated with live views in a virtual augmented reality world representing the real-world location; and

wherein the on-site devices synchronize the data of the augmented reality representations with the off-site devices such that the augmented reality representations and the virtual augmented reality representations are consistent with each other.

24. The method of claim 23, wherein there are zero off-site devices, and the on-site devices communicate through either a peer-to-peer network, a mesh network, or an ad hoc network.

25. The system of claim 23, wherein an on-site device is configured to identify a user instruction to change data or content of the on-site device'"'"'s internal representation of AR;

and

wherein the on-site device is further configured to send the user instruction to other on-site devices and off-site devices of the system so that the augmented reality representations and the virtual augmented reality representations within the system reflect the change to the data or content consistently in real time.

26. The system of claim 23, wherein an off-site device is configured to identify a user instruction to change the data or content in the virtual augmented reality representation of the off-site device;

and

wherein the off-site device is further configured to send the user instruction to other on-site devices and off-site devices of the system so that the augmented reality representations and the virtual augmented reality representations within the system reflect the change to the data or content consistently in real time.

27. The system of claim 23, further comprising:

a server for relaying and/or storing communications between the on-site devices and the off-site devices, as well as the communications between on-site devices, and the communications between off-site devices.

28. The system of claim 23, wherein the users of the on-site and off-site devices participate a shared augmented reality event.

29. The system of claim 23, wherein the users of the on-site and off-site devices are represented by avatars of the users visualized in the augmented reality representations and virtual augmented reality representations;

and wherein augmented reality representations and virtual augmented reality representations visualize that the avatars participate in a shared augmented reality event in a virtual location or scene as well as a corresponding real-world location.

30. A computer device for sharing augmented reality experiences, the computer device comprising of:

a network interface configured to receive environmental, position, and geometry data of a real-world location from an on-site device in proximity to the real-world location;

the network interface further configured to receive augmented reality data or content from the on-site device;

an off-site virtual augmented reality engine configured to create a virtual representation of the real-world location based on the environmental data including position and geometry data received from the on-site device; and

an engine configured to reproduce the augmented reality content in the virtual representation of reality such that the virtual representation of reality is consistent with the augmented reality representation of the real-world location (AR scene) created by the on-site device.

1. A computer-implemented method for providing a shared augmented reality experience, the method comprising:

receiving, at an on-site device in proximity to a real-world location, a location coordinate of the on-site device;

sending, from the on-site device to a server, a request for available AR content and for position and geometry data of objects of the real-world location based on the location coordinate;

receiving, at the on-site device, AR content as well as environmental data including position and geometry data of objects of the real-world location;

visualizing, at the on-site device, an augmented reality representation of the real-world location by presenting augmented reality content incorporated into a live view of the real-world location;

forwarding, from the on-site device to an off-site device remote to the real-world location, the AR content as well as the position and geometry data of objects in the real-world location to enable the off-site device to visualize a virtual representation of the real-world by creating virtual copies of the objects of the real-world location, wherein the off-site device incorporates the AR content in the virtual representation; and

synchronizing a change to the augmented reality representation on the on-site device with the virtual augmented reality representation on the off-site device.

2. The method of claim 1, further comprising:

synchronizing a change to the virtual augmented reality representation on the off-site device with the augmented reality representation on the on-site device.

3. The method of claim 1, wherein the change to the augmented reality representation on the on-site device is sent to the off-site device asynchronously.

4. The method of claim 1, wherein the synchronizing comprises:

receiving, from an input component of the on-site device, a user instruction to create, alter, move or remove augmented reality content in the augmented reality representation;

updating, at the on-site device, the augmented reality representation based on the user instruction; and

forwarding, from the on-site device to the off-site device, the user instruction such that the off-site device can update its virtual representation of the augmented reality scene according to the user instruction.

5. The method of claim 1, further comprising:

receiving, at the on-site device from the off-site device, a user instruction for the off-site device to create, alter, move or remove augmented reality content in its virtual augmented reality representation; and

updating, at the on-site device, the augmented reality representation based on the user instruction such that the status of the augmented reality content is synchronized between the augmented reality representation and the virtual augmented reality representation.

6. The method of claim 1, further comprising:

capturing environmental data including but not limited to live video of the real-world location, live geometry and exiting texture information, by the on-site device.

7. The method of claim 1, further comprising:

sending, from the on-site device to the off-site device, the textural image data of the objects of the real-world location.

8. The method of claim 1, wherein the synchronizing comprises:

synchronizing a change to the augmented reality representation on the on-site device with multiple virtual augmented reality representations on multiple off-site devices and multiple augmented reality representations on other on-site devices.

9. The method of claim 1, wherein the augmented reality content comprises a video, an image, a piece of artwork, an animation, text, a game, a program, a sound, a scan or a 3D object.

10. The method of claim 9, wherein the augmented reality content contains a hierarchy of objects including but not limited to shaders, particles, lights, voxels, avatars, scripts, programs, procedural objects, images, or visual effects, or wherein the augmented reality content is a subset of an object.

11. The method of claim 1, further comprising:

establishing, by the on-site device, a hot-editing augmented reality event by automatically or manually sending invitations or allowing public access to multiple on-site or off-site devices.

12. The method of claim 1, wherein the on-site device maintains its point of view of the augmented reality at the location of the on-site device at the scene.

13. The method of claim 12, wherein the virtual augmented reality representation of the off-site device follows the point of view of the on-site device.

14. The method of claim 1, wherein the off-site device maintains its point of view of the virtual augmented reality representation as a first person view from the avatar of the user of the off-site device in the virtual augmented reality representation, or as a third person view of the avatar of the user of the off-site device in the virtual augmented reality representation.

15. The method of claim 1, further comprising:

capturing, at the on-site or off-site device, a facial expression or a body gesture of a user of said device;

updating, at said device, a facial expression or a body positioning of the avatar of the user of the device in the augmented reality representation; and

sending, from the device to all other devices, information of the facial expression or the body gesture of the user to enable the other devices to update the facial expression or the body positioning of the avatar of the user of said device in the virtual augmented reality representation.

16. The method of claim 1, wherein communications between the on-site device and the off-site device are transferred through a central server, a cloud server, a mesh network of device nodes, or a peer-to-peer network of device nodes.

17. The method of claim 1, further comprising:

forwarding, by the on-site device to another on-site device, the AR content as well as the environmental data including the position and the geometry data of the objects of the real-world location, to enable the other on-site device to visualize the AR content in another location similar to the real-world location proximate to the on-site device; and

synchronizing a change to the augmented reality representation on the on-site device with another augmented reality representation on the other on-site device.

18. The method of claim 1, wherein the change to the augmented reality representation on the on-site device is stored on an external device and persists from session to session.

19. The method of claim 18, wherein the change to the augmented reality representation on the on-site device persists for a predetermined amount of time before being erased from the external device.

20. The method of claim 19, wherein communications between the on-site device and the other on-site device are transferred though an ad hoc network.

21. The method of claim 20, wherein the change to the augmented reality representation does not persist from session to session, or from event to event.

22. The method of claim 1, further comprising:

extracting data needed to track real-world object(s) or feature(s), including but not limited to geometry data, point cloud data, and textural image data, from public or private sources of real world textural, depth, or geometry information, e.g. Google Street View, Google Earth, and Nokia Here;

using techniques such as photogrammetry and SLAM.

31. The system of claim 30, wherein the computer device is remote to the real-world location.

32. The system of claim 30, wherein the network interface is further configured to receive a message indicating that the on-site device has altered the augmented reality overlay object in the augmented reality representation or scene;

and

wherein the data and content engine is further configured to alter the augmented reality content in the virtual augmented reality representation based on the message.

33. The system of claim 30, further comprising:

An input interface configured to receive a user instruction to alter the augmented reality content in the virtual augmented reality representation or scene;

wherein the overlay engine is further configured to alter the augmented reality content in the virtual augmented reality representation based on the user instruction; and

wherein the network interface is further configured to send an instruction from a first device to a second device to alter an augmented reality overlay object in an augmented reality representation of the second device.

34. The system of claim 30, wherein

the instruction was sent from the first device which is an on-site device to the second device which is an off-site device;

or

the instruction was sent from the first device which is an off-site device to the second device which is an on-site device;

or

the instruction was sent from the first device which is an on-site device to the second device which is an on-site device;

or

the instruction was sent from the first device which is an off-site device to the second device which is an off-site device.

35. The system of claim 30, wherein the position and geometry data of the real-world location include data collected using any or all of the following:

fiducial marker technology, simultaneous localization and mapping (SLAM) technology, global positioning system (GPS) technology, dead reckoning technology, beacon triangulation, predictive geometry tracking, image recognition and or stabilization technologies, photogrammetry and mapping technologies, and any conceivable locating or specific positioning technology.

45. The method of claim 36, wherein the data of the AR Vector is manipulated in a variety of ways in order to achieve a variety of effects, including, but not limited to:

changing the speed, color, shape, and scaling.

36. A method for sharing augmented reality positional data and the relative time values of that positional data, the method comprising:

receiving, from at least one on-site device, positional data and the relative time values of that positional data, collected from the motion of the on-site device;

creating an augmented reality (AR) three-dimensional Vector based on the positional data and the relative time values of that positional data;

placing the augmented reality Vector at a location where the positional data was collected; and

visualizing a representation of the augmented reality Vector with a device.

37. The method of claim 36, wherein the representation of the augmented reality Vector includes additional information through the use of color shading and other data visualization techniques.

38. The method of claim 36, wherein the AR Vector defines the edge or surface of a piece of AR content, or otherwise acts as a parameter for that piece of AR content.

39. The method of claim 36, wherein the included information about the relative time that each point of positional data was captured at, on the on-site device allows for the calculation of velocity, acceleration and jerk data.

40. The method of claim 39, further comprising:

Creating from the positional data and the relative time values of that positional data, objects and values including but not limited to an AR animation, an AR ballistics visualization, or a path of movement for an AR object.

41. The method of claim 36, wherein the device'"'"'s motion data that is collected to create the AR Vector is generated from sources including, but not limited to, the internal motion units of the on-site device.

42. The method of claim 36, wherein the AR Vector is created from input data not related to the device'"'"'s motion, generated from sources including but not limited to RF trackers, pointers, or laser scanners.

43. The method of claim 36, wherein the AR Vector is accessible by multiple digital and mobile devices, wherein the digital and mobile device can be on-site or off-site, wherein the AR Vector is viewed in real time or asynchronously.

44. The method of claim 36, wherein one or more on-site digital devices or one or more off-site digital devices can create and edit the AR Vector. Creations and edits to the AR Vector can be seen by multiple on-site and off-site users live, or at a later time. Creation and editing, as well as viewing creation and editing, can either be done by multiple users simultaneously, or over a period of time.

47. The method of claim 36, wherein the AR Vector positional data can be altered so that the relationship between the altered and unaltered data is linear.

48. The method of claim 36, wherein the AR Vector positional data can be altered so that the relationship between the altered and unaltered data is nonlinear.

49. The method of claim 36, further comprising:

A piece of AR content which uses multiple augmented reality Vectors as parameters.

50. The method of claim 36, wherein the AR Vector can be a distinct element of content, independent of a specific location or piece of AR content. They can be copied, edited and/or moved to different positional coordinates.

51. The method of claim 36, further comprising:

using the AR Vector to create content for different kinds of AR applications, including but not limited to;

surveying, animation, light painting, architecture, ballistics, training, gaming, and national defense.

45-1. The method of claim 35, wherein various types of input can be used to create or change the AR Vector'"'"'s positional data vector, including, but not limited to:

midi boards, styli, electric guitar output, motion capture, and pedestrian dead reckoning enabled devices.