Methods and systems for dynamically customizing a scene for presentation to a user

• US 10,388,072 B2
• Filed: 10/31/2016
• Issued: 08/20/2019
• Est. Priority Date: 10/31/2016
• Status: Active Grant

• Current Assignee: Verizon Patent and Licensing Incorporated (Verizon Communications Inc.)
• Original Assignee: Verizon Patent and Licensing Incorporated (Verizon Communications Inc.)
• Inventors: Egeler, Christian, Hansen, Xavier, Khalid, Mohammad Raheel
• Primary Examiner(s): Chow, Jeffrey J
• Application Number:
  US15/339,656
• Publication Number:
  US 20180122142A1
• Time in Patent Office: 1,023 Days
• Field of Search: None
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 17/10 Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2210/61 Scene description

Abstract

An exemplary scene customization system detects a selection, by a user, of a customization scheme to be applied to an original immersive scene throughout a scene customization session to be experienced by the user by way of an artificial reality device. The system dynamically generates a custom immersive scene based on an application of the selected customization scheme to the original immersive scene by receiving data representative of the original immersive scene, developing a dynamic volumetric model of the original immersive scene based on the received data representative of the original immersive scene, identifying one or more objects included in the original immersive scene, and replacing the one or more objects with one or more custom objects in accordance with the applied customization scheme. The system also presents the custom immersive scene to the user by way of the artificial reality device throughout the scene customization session.

Claims

1. A method comprising:

detecting, by a scene customization system, a selection, by a user, of a customization scheme to be applied to an original immersive scene throughout a scene customization session to be experienced by the user by way of an artificial reality device;

dynamically generating, by the scene customization system and based on an application of the selected customization scheme to the original immersive scene, a custom immersive scene by

receiving data representative of the original immersive scene,

developing, based on the received data representative of the original immersive scene, a dynamic volumetric model of the original immersive scene,

identifying, within the dynamic volumetric model of the original immersive scene, a first object of a first object type and a second object of a second object type, the first and second objects each included in the original immersive scene,

replacing, within the dynamic volumetric model and in accordance with the applied customization scheme, the first identified object with a custom object that is predetermined within the applied customization scheme to correspond to the first object type,

determining, based on a non-augmentable object list associated with the selected customization scheme, that the second identified object of the second object type is a non-augmentable object corresponding with a predefined entry on the non-augmentable object list, and

abstaining from replacing, in response to the determining that the second identified object is the non-augmentable object, the second identified object within the dynamic volumetric model; and

presenting, by the scene customization system throughout the scene customization session, the custom immersive scene to the user by way of the artificial reality device.

2. The method of claim 1, wherein:

the identifying of the first object included in the original immersive scene includes;

identifying, based on the received data representative of the original immersive scene, one or more features of the first object,

matching the one or more features of the first object with one or more corresponding features of a known object documented in an object library associated with the selected customization scheme, and

determining, based on the matching of the one or more features, that the first object is an instance of the known object documented in the object library; and

the custom object with which the first object is replaced is a custom object that is predetermined, within the selected customization scheme, to correspond to the known object.

3. The method of claim 1, wherein:

the identifying of the first object included in the original immersive scene includes;

identifying, based on the received data representative of the original immersive scene, one or more features of the first object,

determining, based on the one or more features of the first object, that the first object type is a known object type documented in an object library associated with the selected customization scheme; and

the custom object with which the first object is replaced is a custom object that is predetermined, within the selected customization scheme, to correspond to the known object type.

4. The method of claim 1, wherein:

the original immersive scene includes a first real-world scene that is distinct from a second real-world scene surrounding the user throughout the scene customization session;

the receiving of the data representative of the original immersive scene includes receiving data transmitted by a scene modeling system, the scene modeling system having elements that are disposed in a vicinity of the first real-world scene to capture data representative of the first real-world scene; and

the developing of the dynamic volumetric model of the original immersive scene includes receiving, as part of the received data transmitted by the scene modeling system,

data representative of a dynamic volumetric model of the first real-world scene, the data representative of the dynamic volumetric model of the first real-world scene created by the scene modeling system based on the captured data representative of the first real-world scene, and

continually updating the received dynamic volumetric model of the first real-world scene based on the received data transmitted by the scene modeling system as the data transmitted by the scene modeling system continues to be transmitted and received.

5. The method of claim 1, wherein:

the original immersive scene includes a real-world scene surrounding the user throughout the scene customization session;

the receiving of the data representative of the original immersive scene includes directly capturing, throughout the scene customization session, data representative of the real-world scene surrounding the user; and

the developing of the dynamic volumetric model of the original immersive scene includes

creating a dynamic volumetric model of the real-world scene surrounding the user based on the directly captured data representative of the real-world scene surrounding the user, and

continually updating the created dynamic volumetric model of the real-world scene surrounding the user based on the directly captured data representative of the real-world scene surrounding the user as the data representative of the real-world scene surrounding the user continues to be directly captured throughout the scene customization session.

6. The method of claim 5, wherein the data representative of the real-world scene surrounding the user, and upon which the creating and the continually updating of the dynamic volumetric model is based, is captured by way of one or more sensors of the artificial reality device used by the user to experience the scene customization session.

7. The method of claim 1, further comprising determining, by the scene customization system, that the original immersive scene includes a known real-world scene documented in a scene library associated with the selected customization scheme;

wherein the dynamically generating of the custom immersive scene based on the application of the selected customization scheme to the original immersive scene is customized according to at least one preconfigured customization documented within the scene library in association with the known real-world scene.

8. The method of claim 1, further comprising:

determining, by the scene customization system while the presenting of the custom immersive scene to the user is ongoing, that another user using another artificial reality device is experiencing the original immersive scene; and

providing, by the scene customization system in response to the determining that the other user is experiencing the original immersive scene, data representative of the custom immersive scene being presented to the user by way of the artificial reality device, the data representative of the custom immersive scene configured to facilitate another dynamic generation of the custom immersive scene and another presentation of the custom immersive scene to the other user by way of the other artificial reality device.

9. The method of claim 1, wherein the selected customization scheme is associated with a theme related to a time period.

10. The method of claim 1, wherein the selected customization scheme is associated with a theme related to at least one of:

a work of fiction;

a particular aesthetic; and

a real-world place.

11. A method comprising:

detecting, by a scene customization system, a selection, by a user, of a customization scheme from a library of customization schemes provided by an augmented reality provider associated with the scene customization system, the customization scheme selected to be applied to a real-world scene surrounding the user throughout a scene customization session to be experienced by the user by way of an immersive headset device;

dynamically generating, by the scene customization system and based on an application of the selected customization scheme to the real-world scene surrounding the user, a custom immersive scene by

directly capturing, throughout the scene customization session, data representative of the real-world scene surrounding the user,

creating, based on the directly captured data representative of the real-world scene surrounding the user, a dynamic volumetric model of the real-world scene surrounding the user,

continually updating, based on the directly captured data representative of the real-world scene surrounding the user as the data representative of the real-world scene surrounding the user continues to be directly captured throughout the scene customization session, the created dynamic volumetric model of the real-world scene surrounding the user,

identifying, within the dynamic volumetric model of the real-world scene surrounding the user, a first object of a first object type and a second object of a second object type, the first and second objects each in the real-world scene surrounding the user,

replacing, within the dynamic volumetric model and in accordance with the applied customization scheme, the first identified object with a custom object that is predetermined within the applied customization scheme to correspond to the first object type,

determining, based on a non-augmentable object list associated with the selected customization scheme, that the second identified object of the second object type is a non-augmentable object corresponding with a predefined entry on the non-augmentable object list, and

abstaining from replacing, in response to the determining that the second identified object is the non-augmentable object, the second identified object within the dynamic volumetric model; and

presenting, by the scene customization system throughout the scene customization session, the custom immersive scene to the user by way of the immersive headset device.

12. A system comprising:

a processor; and

a memory communicatively coupled with the processor and storing instructions that, when executed by the processor, direct the processor to;

detect a selection, by a user, of a customization scheme to be applied to an original immersive scene throughout a scene customization session to be experienced by the user by way of an artificial reality device;

dynamically generate, based on an application of the selected customization scheme to the original immersive scene, a custom immersive scene by

receiving data representative of the original immersive scene,

developing, based on the received data representative of the original immersive scene, a dynamic volumetric model of the original immersive scene,

identifying, within the dynamic volumetric model of the original immersive scene, a first object of a first object type and a second object of a second object type, the first and second objects each included in the original immersive scene,

replacing, within the dynamic volumetric model and in accordance with the applied customization scheme, the first identified object with a custom object that is predetermined within the applied customization scheme to correspond to the first object type,

determining, based on a non-augmentable object list associated with the selected customization scheme, that the second identified object of the second object type is a non-augmentable object corresponding with a predefined entry on the non-augmentable object list, and

abstaining from replacing, in response to the determining that the second identified object is the non-augmentable object, the second identified object within the dynamic volumetric model; and

present, throughout the scene customization session, the custom immersive scene to the user by way of the artificial reality device.

13. The system of claim 12, wherein:

the instructions direct the processor to perform the identifying of the first object by;

identifying, based on the received data representative of the original immersive scene, one or more features of the first object,

matching the one or more features of the first object with one or more corresponding features of a known object documented in an object library associated with the selected customization scheme, and

determining, based on the matching of the one or more features, that the first object is an instance of the known object documented in the object library; and

the custom object with which the first object is replaced is a custom object that is predetermined, within the selected customization scheme, to correspond to the known object.

14. The system of claim 12, wherein:

the instructions direct the processor to perform the identifying of the first object by;

identifying, based on the received data representative of the original immersive scene, one or more features of the first object,

determining, based on the one or more features of the first object, that the first object type is a known object type documented in an object library associated with the selected customization scheme; and

the custom object with which the first object is replaced is a custom object that is predetermined, within the selected customization scheme, to correspond to the known object type.

15. The system of claim 12, wherein:

the original immersive scene includes a first real-world scene that is distinct from a second real-world scene surrounding the user throughout the scene customization session;

the instructions direct the processor to perform the receiving of the data representative of the original immersive scene by receiving data transmitted by a scene modeling system, the scene modeling system having elements that are disposed in a vicinity of the first real-world scene to capture data representative of the first real-world scene; and

the instructions direct the processor to perform the developing of the dynamic volumetric model of the original immersive scene by;

receiving, as part of the received data transmitted by the scene modeling system, data representative of a dynamic volumetric model of the first real-world scene, the data representative of the dynamic volumetric model of the first real-world scene created by the scene modeling system based on the captured data representative of the first real-world scene, and

continually updating the received dynamic volumetric model of the first real-world scene based on the received data transmitted by the scene modeling system as the data transmitted by the scene modeling system continues to be transmitted and received.

16. The system of claim 12, wherein:

the original immersive scene includes a real-world scene surrounding the user throughout the scene customization session;

the instructions direct the processor to perform the receiving of the data representative of the original immersive scene by directly capturing, throughout the scene customization session and by way of one or more sensors of the artificial reality device, data representative of the real-world scene surrounding the user; and

the instructions direct the processor to perform the developing of the dynamic volumetric model of the original immersive scene by

creating a dynamic volumetric model of the real-world scene surrounding the user based on the directly captured data representative of the real-world scene surrounding the user, and

continually updating the created dynamic volumetric model of the real-world scene surrounding the user based on the directly captured data representative of the real-world scene surrounding the user as the data representative of the real-world scene surrounding the user continues to be directly captured throughout the scene customization session.

17. The system of claim 12, wherein:

the instructions, when executed by the processor, further direct the processor to determine that the original immersive scene includes a known real-world scene documented in a scene library associated with the selected customization scheme; and

the dynamic generation of the custom immersive scene based on the application of the selected customization scheme to the original immersive scene is customized according to at least one preconfigured customization documented within the scene library in association with the known real-world scene.

18. The system of claim 12, wherein the instructions, when executed by the processor, further direct the processor to:

determine, while the presenting of the custom immersive scene to the user is ongoing, that another user using another artificial reality device is experiencing the original immersive scene; and

provide, in response to the determination that the other user is experiencing the original immersive scene, data representative of the custom immersive scene being presented to the user by way of the artificial reality device, the data representative of the custom immersive scene configured to facilitate another dynamic generation of the custom immersive scene and another presentation of the custom immersive scene to the other user by way of the other artificial reality device.

19. The system of claim 12, wherein the selected customization scheme is associated with a theme related to a time period.

20. The system of claim 12, wherein the selected customization scheme is associated with a theme related to at least one of:

a work of fiction;

a particular aesthetic; and

a real-world place.