Protecting privacy in web-based immersive augmented reality

• US 9,679,144 B2
• Filed: 11/15/2013
• Issued: 06/13/2017
• Est. Priority Date: 11/15/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Ofek, Eyal, Vilk, John, Burger, Douglas Christopher, Wang, Jiahe, Gal, Ran, Shapira, Lior, Livshits, Benjamin, Molnar, David, Moshchuk, Alexander, MacIntyre, Blair
• Primary Examiner(s): SHAIFER HARRIMAN, DANT B
• Application Number:
  US14/082,051
• Publication Number:
  US 20150143459A1
• Time in Patent Office: 1,306 Days
• Field of Search: 726 2
• CPC Class Codes: G06F 21/60 Protecting data
• CPC Class Codes: G06F 21/6245 Protecting personal data, e.g. for financial or medical purposes
• CPC Class Codes: G06F 21/629 to features or functions of an application
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 67/12 specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
• CPC Class Codes: H04L 67/131 Protocols for games, networked simulations or virtual reality
• CPC Class Codes: H04L 67/535 Tracking the activity of the user network monitoring arrangements H04L43/00; recording of computer activity G06F11/34
• CPC Class Codes: H04W 12/02 Protecting privacy or anonymity, e.g. protecting personally identifiable information [PII]
• CPC Class Codes: H04W 12/65 Environment-dependent, e.g. using captured environmental data

Abstract

An “AR Privacy API” provides an API that allows applications and web browsers to use various content rendering abstractions to protect user privacy in a wide range of web-based immersive augmented reality (AR) scenarios. The AR Privacy API extends the traditional concept of “web pages” to immersive “web rooms” wherein any desired combination of existing or new 2D and 3D content is rendered within a user'"'"'s room or other space. Advantageously, the AR Privacy API and associated rendering abstractions are useable by a wide variety of applications and web content for enhancing the user'"'"'s room or other space with web-based immersive AR content. Further, the AR Privacy API is implemented using any existing or new web page coding platform, including, but not limited to HTML, XML, CSS, JavaScript, etc., thereby enabling existing web content and coding techniques to be smoothly integrated into a wide range of web room AR scenarios.

Claims

1. A computer-implemented process for using a web browser to render content in a 3D space, comprising:

using a computer to perform process actions for;

evaluating data from a plurality of sensors to infer environmental information regarding a space and regarding persons and contents within that space, the environmental information comprising any combination of geometric characteristics of the space and geometric characteristics and positions of contents and persons within the space;

evaluating the environmental information to identify a plurality of surfaces in the space upon which content can be rendered by one or more output devices;

labeling one or more of the surfaces with semantic annotations based in part on the environmental information, each semantic annotation comprising one or more physical characteristics and a physical location of a corresponding one of the surfaces;

receiving content via a networking device from any combination of websites, web pages and web rooms, the content further comprising one or more abstractions that declaratively specify how that content is to be adapted relative to any of a plurality of semantic annotations;

evaluating the abstractions in combination with the semantic annotations associated with the surfaces to determine one or more corresponding surfaces on which the content is to be rendered; and

applying the abstractions and corresponding semantic annotations to adapt and render the content onto the corresponding surfaces, via one or more output devices, without disclosing the environmental information to any of the websites, web pages and web rooms.

2. The computer-implemented process of claim 1 further comprising steps for associating a permission state with one or more elements of the environmental information.

3. The computer-implemented process of claim 2 wherein one or more of the permission states are individually granted to particular websites, web pages and web rooms.

4. The computer-implemented process of claim 2 wherein one or more of the permission states are granted to multiple websites, web pages and web rooms.

5. The computer-implemented process of claim 1 wherein the abstractions are interpreted by a trusted renderer that receives content from one or more of the websites, web pages and web rooms and determines which surface that content is to be rendered onto without allowing the websites, web pages and web rooms to obtain information regarding that surface.

6. The computer-implemented process of claim 5 wherein a JavaScript-based API provides an interface between the trusted renderer and each websites, web pages and web rooms.

7. The computer-implemented process of claim 1 wherein a constraint solver is used to evaluate the environmental data to determine the surfaces in the space upon which content is to be rendered.

8. The computer-implemented process of claim 7 wherein the constraint solver evaluates the environmental data and the abstractions of the content relative to the semantic annotations associated with corresponding surfaces to identify which of the surfaces are to be used to render particular content.

9. The computer-implemented process of claim 1 wherein the abstractions further define relative positional relationships between two or more particular segments of the content.

10. A system for rendering content to construct an augmented reality environment, comprising:

a device for evaluating a plurality of sensors for inferring environmental information regarding a space and regarding persons and contents within that space, the environmental information comprising any combination of geometric characteristics of the space and geometric characteristics and positions of contents and persons within the space;

a device for evaluating the environmental information to identify a plurality of surfaces in the space upon which content can be rendered;

a device for associating one or more semantic annotations with one or more of the surfaces, each semantic annotation comprising one or more physical characteristics and a physical location of a corresponding one of the surfaces;

a networking device for receiving content from one or more websites, web pages and web rooms, the content further comprising one or more abstractions that declaratively specify how the content is to be adapted relative to any of a plurality of semantic annotations;

a device for interpreting the abstractions of the content to associate that content with one or more corresponding surfaces based on the semantic annotations associated with those surfaces; and

an output device, comprising any combination of monitors, projectors, touchscreens, tablets, phones, and media players, for rendering the content onto the corresponding surfaces based on the corresponding abstractions and semantic annotations without disclosing corresponding environmental information regarding the surfaces to the websites, web pages or web rooms that were the source of the content.

11. The system of claim 10 further comprising steps for associating a permission state with one or more elements of the environmental information.

12. The system of claim 11 wherein one or more of the permission states are individually granted to particular sources of the content.

13. The system of claim 11 wherein one or more of the permission states are granted to multiple sources of the content.

15. The system of claim 10 wherein a constraint solver is used to evaluate the environmental data to identify the surfaces relative to any semantic annotations associated with any corresponding surface to associate specific content with one or more corresponding surfaces based on one or more of the abstractions of that content.

16. The system of claim 10 wherein the abstractions further define relative positional relationships between two or more segments of the content.

14. The system 10 wherein a JavaScript-based API provides a trusted interface between the device for rendering content and the source of that content.

17. A computer-readable device having computer executable instructions stored therein for rendering content in a 3D space, said instructions causing a computing device to execute a method comprising:

evaluating data from a plurality of sensors to infer environmental information regarding surfaces, objects and persons within a 3D space, the environmental information comprising any combination of geometric characteristics of the space and geometric characteristics and positions of objects and persons within the space;

evaluating the environmental information to identify a plurality of surfaces in the 3D space upon which content can be rendered;

associating one or more semantic annotations with one or more of the surfaces, each semantic annotation comprising one or more physical characteristics and a physical location of a corresponding one of the surfaces;

receiving content from one or more websites, web pages and web rooms via a communications device, the content further comprising one or more abstractions that declaratively specify how the content is to be adapted relative to one or more of a plurality of semantic annotations;

applying a constraint solver to evaluate the environmental data and the semantic annotations associated with any corresponding surface to associate specific content with one or more corresponding surfaces based on the abstractions of that content; and

applying one or more output devices for rendering the content onto the corresponding surfaces based on the corresponding abstractions and semantic annotations without disclosing corresponding environmental information to the websites, web pages and web rooms that were the source of the rendered content.

18. The computer-readable device of claim 17 further comprising associating a permission state with one or more elements of the environmental information.

19. The computer-readable device of claim 17 further comprising a JavaScript-based API that provides a trusted interface for rendering content from the source of that content.

20. The computer-readable device of claim 17 wherein the semantic annotations further define relative relationships between two or more of the surfaces.