PROTECTING PRIVACY IN WEB-BASED IMMERSIVE AUGMENTED REALITY

• US 20150143459A1
• Filed: 11/15/2013
• Published: 05/21/2015
• Est. Priority Date: 11/15/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Molnar, David, Ofek, Eyal, Moshchuk, Alexander, Gal, Ran, Shapira, Lior, Burger, Douglas Christopher, MacIntyre, Blair, Livshits, Benjamin, Vilk, John, Wang, Jiahe
• Granted Patent: US 9,679,144 B2
• Time in Patent Office: Days
• US Class Current: 726/2
• CPC Class Codes: G06F 21/60 Protecting data
• CPC Class Codes: G06F 21/6245 Protecting personal data, e.g. for financial or medical purposes
• CPC Class Codes: G06F 21/629 to features or functions of an application
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 67/12 specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
• CPC Class Codes: H04L 67/131 Protocols for games, networked simulations or virtual reality
• CPC Class Codes: H04L 67/535 Tracking the activity of the user network monitoring arrangements H04L43/00; recording of computer activity G06F11/34
• CPC Class Codes: H04W 12/02 Protecting privacy or anonymity, e.g. protecting personally identifiable information [PII]
• CPC Class Codes: H04W 12/65 Environment-dependent, e.g. using captured environmental data

Abstract

An “AR Privacy API” provides an API that allows applications and web browsers to use various content rendering abstractions to protect user privacy in a wide range of web-based immersive augmented reality (AR) scenarios. The AR Privacy API extends the traditional concept of “web pages” to immersive “web rooms” wherein any desired combination of existing or new 2D and 3D content is rendered within a user'"'"'s room or other space. Advantageously, the AR Privacy API and associated rendering abstractions are useable by a wide variety of applications and web content for enhancing the user'"'"'s room or other space with web-based immersive AR content. Further, the AR Privacy API is implemented using any existing or new web page coding platform, including, but not limited to HTML, XML, CSS, JavaScript, etc., thereby enabling existing web content and coding techniques to be smoothly integrated into a wide range of web room AR scenarios.

Claims

1. A computer-implemented process for using a web browser to render content in a 3D space, comprising:

using a computer to perform process actions for;

evaluating data from a plurality of sensors to infer environmental information regarding a space and regarding persons and contents within that space;

evaluating the environmental information to identify a plurality of surfaces, referred to as “

screens,”

in the space upon which content can be rendered by one or more output devices; and

generating one or more semantic abstractions for one or more of the “

screens”

that allow content to be rendered on the corresponding “

screens”

without disclosing environmental information regarding the “

screens”

to a source of the rendered content.

2. The computer-implemented process of claim 1 further comprising steps for associating a permission state with one or more elements of the environmental information.

3. The computer-implemented process of claim 2 wherein one or more of the permission states are individually granted to particular sources of the rendered content.

4. The computer-implemented process of claim 2 wherein one or more of the permission states are granted to multiple sources of the rendered content.

5. The computer-implemented process of claim 1 wherein the semantic abstractions are interpreted by a trusted renderer that receives content from an untrusted source and determines which “

screen”

that content is to be rendered on without allowing the untrusted source to obtain information regarding that “

screen.”

6. The computer-implemented process of claim 5 wherein a JavaScript-based API provides an interface between the trusted renderer and each untrusted source.

7. The computer-implemented process of claim 1 wherein a constraint solver is used to evaluate the environmental data to identify the “

screens”

in the space upon which content can be rendered.

8. The computer-implemented process of claim 7 wherein the constraint solver evaluates the environmental data relative to any semantic abstractions associated with any corresponding screen to identify which of the “

screens”

are to be used to render particular content.

9. The computer-implemented process of claim 1 wherein the semantic abstractions define relative relationships between two or more of the “

screens.”

10. A system for rendering content to construct an augmented reality environment, comprising:

a device for evaluating a plurality of sensors for inferring environmental information regarding a space and regarding persons and contents within that space;

a device for evaluating the environmental information to identify a plurality of surfaces in the space upon which content can be rendered;

a device for associating one or more semantic abstractions with one or more of the surfaces;

a device for interpreting the semantic abstractions to associate specific content with one or more corresponding surfaces; and

a device for rendering the content to the corresponding surfaces without disclosing corresponding environmental information regarding the surfaces to a source of the rendered content.

11. The system of claim 10 further comprising steps for associating a permission state with one or more elements of the environmental information.

12. The system of claim 11 wherein one or more of the permission states are individually granted to particular sources of the rendered content.

13. The system of claim 11 wherein one or more of the permission states are granted to multiple sources of the rendered content.

15. The system of claim 10 wherein a constraint solver is used to evaluate the environmental data to identify the surfaces relative to any semantic abstractions associated with any corresponding surface to associate specific content with one or more corresponding surfaces.

16. The system of claim 10 wherein the semantic abstractions define relative relationships between two or more of the surfaces.

14. The system 10 wherein a JavaScript-based API provides a trusted interface between the device for rendering content and an untrusted source of that content.

17. A computer-readable medium having computer executable instructions stored therein for rendering content in a 3D space, said instructions causing a computing device to execute a method comprising:

evaluating data from a plurality of sensors to infer environmental information regarding surfaces, objects and persons within a 3D space;

evaluating the environmental information to identify a plurality of surfaces in the 3D space upon which content can be rendered;

associating one or more semantic abstractions with one or more of the surfaces;

using a constraint solver to evaluate the environmental data relative to any semantic abstractions associated with any corresponding surface to associate specific content with one or more corresponding surfaces; and

rendering the content to the corresponding surfaces without disclosing corresponding environmental information to a source of the rendered content.

18. The computer-readable medium of claim 17 further comprising associating a permission state with one or more elements of the environmental information.

19. The computer-readable medium of claim 17 further comprising a JavaScript-based API that provides a trusted interface for rendering content from an untrusted source of that content.

20. The computer-readable medium of claim 17 wherein the semantic abstractions define relative relationships between two or more of the surfaces.