Methods and systems for virtualizing a target object within a mixed reality presentation

• US 10,380,803 B1
• Filed: 03/26/2018
• Issued: 08/13/2019
• Est. Priority Date: 03/26/2018
• Status: Active Grant

• Current Assignee: Verizon Patent and Licensing Incorporated (Verizon Communications Inc.)
• Original Assignee: Verizon Patent and Licensing Incorporated (Verizon Communications Inc.)
• Inventors: Jaafar, Ali, Randhawa, Jaskirat Singh, Mahmood, Abdul
• Primary Examiner(s): Johnson, Motilewa Good
• Application Number:
  US15/936,339
• Time in Patent Office: 505 Days
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2219/2004 Aligning objects, relative positioning of parts
• CPC Class Codes: G06T 2219/2021 Shape modification

Abstract

An exemplary mixed reality presentation system accesses video data representative of a real-world environment proximate to a mixed reality player device that is configured to present, to a user of the mixed reality player device, a mixed reality presentation. In the mixed reality presentation, one or more virtual objects included within a virtual domain associated with the real-world environment are presented together with real objects included within the real-world environment. The mixed reality presentation system identifies, based on the accessed video data, a target object among the real objects included within the real-world environment. The mixed reality presentation system extracts the target object from the mixed reality presentation and replaces, within the mixed reality presentation, the target object with a virtual object included within the virtual domain. The virtual object implements a representation of the target object. Corresponding methods are also disclosed.

Claims

1. A method comprising:

accessing, by a mixed reality presentation system, video data representative of a real-world environment proximate to a mixed reality player device that is configured to present, to a user of the mixed reality player device, a mixed reality presentation in which one or more virtual objects included within a virtual domain associated with the real-world environment are presented together with real objects included within the real-world environment;

identifying, by the mixed reality presentation system based on the accessed video data, a target object among the real objects included within the real-world environment, the target object casting a shadow upon one or more of the real objects included within the real-world environment;

extracting, by the mixed reality presentation system, the target object from the mixed reality presentation by

overlaying a first virtual object included within the virtual domain onto the mixed reality presentation in alignment with the target object, the first virtual object implementing an extraction object for the target object, and

overlaying a shadow extraction object onto the one or more real objects upon which the shadow is cast in order to remove the shadow within the mixed reality presentation; and

replacing, by the mixed reality presentation system, the target object within the mixed reality presentation by overlaying a second virtual object included within the virtual domain onto the mixed reality presentation out of alignment with the target object, the second virtual object implementing a representation of the target object.

2. The method of claim 1, wherein the second virtual object that replaces the target object within the mixed reality presentation implements an animated representation of the target object that appears to move, within the mixed reality presentation and over a period of time, from a first location coincident with a location of the target object within the real-world environment to a second location distinct from the location of the target object within the real-world environment.

3. The method of claim 1, wherein:

the extracting of the target object from the mixed reality presentation is performed by extracting a first part and a second part of the target object from the mixed reality presentation; and

the second virtual object that replaces the extracted first and second parts of the target object within the mixed reality presentation implements a representation of only the first part of the target object and not the second part of the target object.

4. The method of claim 1, wherein the overlaying of the first virtual object onto the mixed reality presentation in alignment with the target object includes:

identifying an occluded area of the real-world environment that is not represented in the accessed video data due to being occluded from view, from a vantage point of a device capturing the accessed video data, by the target object;

determining that the occluded area is not occluded from view, within the mixed reality presentation from a vantage point of the user, by the second virtual object that replaces the target object;

generating the first virtual object to include an extrapolation of video data representative of the occluded area based on a representation within the accessed video data of one or more real objects associated with the occluded area; and

overlaying the first virtual object onto the occluded area within the mixed reality presentation.

5. The method of claim 4, wherein:

the overlaying of the first virtual object onto the mixed reality presentation in alignment with the target object further includes accessing data representative of one or more non-occluded images of the one or more real objects associated with the occluded area; and

the generating of the first virtual object to include the extrapolation of the video data representative of the occluded area is performed based on the one or more non-occluded images of the one or more real objects associated with the occluded area.

6. The method of claim 1, further comprising:

identifying, by the mixed reality presentation system based on the accessed video data, a light source that illuminates the target object within the real-world environment;

determining, by the mixed reality presentation system based on the identified light source, a characteristic of the light source; and

simulating, by the mixed reality presentation system and based on the determined characteristic of the light source, the identified light source within the virtual domain of the mixed reality presentation,

wherein the simulating of the identified light source within the virtual domain includes the overlaying of the shadow extraction object onto the one or more real objects upon which the shadow is cast.

7. The method of claim 6, wherein the simulating of the identified light source within the virtual domain further includes simulating an effect of the light source on the second virtual object that replaces the target object within the mixed reality presentation.

8. The method of claim 6, wherein the simulating of the identified light source within the virtual domain further includes simulating an effect of the light source on an extrapolated area of the real-world environment that is presented, within the mixed reality presentation, in place of an area of the real-world environment that is occluded from view by the target object and not by the second virtual object that replaces the target object.

9. The method of claim 6, wherein the overlaying of the shadow extraction object onto the one or more real objects upon which the shadow is cast is performed by simulating an effect of the light source on an area of the real-world environment that is affected, with respect to lighting, by the extracting of the target object.

10. The method of claim 1, wherein the overlaying of the second virtual object onto the mixed reality presentation out of alignment with the target object includes:

generating a three-dimensional (“

3D”

) model of the target object to be the second virtual object implementing the representation of the target object;

integrating the 3D model of the target object into the virtual domain of the mixed reality presentation; and

manipulating the 3D model of the target object to behave differently, within the virtual domain of the mixed reality presentation, than the target object behaves within the real-world environment.

11. The method of claim 10, wherein the manipulating of the 3D model of the target object to behave differently than the target object is performed based on at least one of:

a user command from the user to direct the 3D model to behave in a particular manner that is different from how the target object behaves within the real-world environment; and

artificial intelligence associated with the 3D model and configured to cause the 3D model to automatically behave in the particular manner without the user command.

12. The method of claim 1, wherein:

the second virtual object that replaces the target object within the mixed reality presentation is an animated character; and

the method further comprises

identifying, by the mixed reality presentation system, an additional target object among the real objects included within the real-world environment while the animated character is presented within the mixed reality presentation and while the additional target object is included within a field of view of the animated character,

extracting, by the mixed reality presentation system, the additional target object from the mixed reality presentation by overlaying a third virtual object included within the virtual domain onto the mixed reality presentation in alignment with the additional target object, the third virtual object implementing an extraction object for the additional target object,

replacing, by the mixed reality presentation system, the additional target object within the mixed reality presentation by overlaying a fourth virtual object included within the virtual domain onto the mixed reality presentation out of alignment with the additional target object, and

presenting, by the mixed reality presentation system within the mixed reality presentation, an interaction between the animated character and the fourth virtual object.

13. The method of claim 1, embodied as computer-executable instructions on at least one non-transitory computer-readable medium.

14. A method comprising:

accessing, by a mixed reality presentation system, video data representative of a real-world environment proximate to a mixed reality player device that is configured to present, to a user of the mixed reality player device, a mixed reality presentation in which one or more virtual objects included within a virtual domain associated with the real-world environment are presented together with real objects included within the real-world environment;

identifying, by the mixed reality presentation system based on the accessed video data, a target object among the real objects included within the real-world environment, the target object emulating a character and casting a shadow upon one or more of the real object included within the real-world environment;

extracting, by the mixed reality presentation system, the target object from the mixed reality presentation by

overlaying a first virtual object included within the virtual domain onto the mixed reality presentation in alignment with the target object, the first virtual object implementing an extraction object for the target object, and

overlaying a shadow extraction object onto the one or more real objects upon which the shadow is cast in order to remove the shadow within the mixed reality presentation;

generating, by the mixed reality presentation system, a second virtual object implemented as an animated three-dimensional (“

3D”

) model of the character emulated by the target object;

integrating, by the mixed reality presentation system, the animated 3D model of the character into the virtual domain to be presented within the mixed reality presentation;

identifying, by the mixed reality presentation system, an additional target object among the real objects included within the real-world environment while the animated 3D model is presented within the mixed reality presentation and while the additional target object is included within a field of view of the animated 3D model of the character;

extracting, by the mixed reality presentation system, the additional target object from the mixed reality presentation by overlaying a third virtual object included within the virtual domain onto the mixed reality presentation in alignment with the additional target object, the third virtual object implementing an extraction object for the additional target object;

replacing, by the mixed reality presentation system, the additional target object within the mixed reality presentation by overlaying a fourth virtual object included within the virtual domain onto the mixed reality presentation out of alignment with the additional target object; and

presenting, by the mixed reality presentation system within the mixed reality presentation, an interaction between the animated 3D model of the character and the fourth virtual object.

15. The method of claim 14, embodied as computer-executable instructions on at least one non-transitory computer-readable medium.

16. A system comprising:

at least one physical computing device that

accesses video data representative of a real-world environment proximate to a mixed reality player device that is configured to present, to a user of the mixed reality player device, a mixed reality presentation in which one or more virtual objects included within a virtual domain associated with the real-world environment are presented together with real objects included within the real-world environment;

identifies, based on the accessed video data, a target object among the real objects included within the real-world environment, the target object casting a shadow upon one or more of the real objects included within the real-world environment;

extracts the target object from the mixed reality presentation by

overlaying a first virtual object included within the virtual domain onto the mixed reality presentation in alignment with the target object, the first virtual object implementing an extraction object for the target object, and

overlaying a shadow extraction object onto the one or more real objects upon which the shadow is cast in order to remove the shadow within the mixed reality presentation; and

replaces the target object within the mixed reality presentation by overlaying a second virtual object included within the virtual domain onto the mixed reality presentation out of alignment with the target object, the second virtual object implementing a representation of the extracted target object.

17. The system of claim 16, wherein the second virtual object that replaces the target object within the mixed reality presentation implements an animated representation of the target object that appears to move, within the mixed reality presentation and over a period of time, from a first location coincident with a location of the target object within the real-world environment to a second location distinct from the location of the target object within the real-world environment.

18. The system of claim 16, wherein the at least one physical computing device overlays the first virtual object onto the mixed reality presentation in alignment with the target object by:

identifying an occluded area of the real-world environment that is not represented in the accessed video data due to being occluded from view, from a vantage point of a device capturing the accessed video data, by the target object;

determining that the occluded area is not occluded from view, within the mixed reality presentation from a vantage point of the user, by the second virtual object that replaces the target object;

generating the first virtual object to include an extrapolation of video data representative of the occluded area based on a representation within the accessed video data of one or more real objects associated with the occluded area; and

overlaying the first virtual object onto the occluded area within the mixed reality presentation.

19. The system of claim 16, wherein:

the at least one physical computing device further;

identifies, based on the accessed video data, a light source that illuminates the target object within the real-world environment;

determines, based on the identified light source, a characteristic of the light source; and

simulates, based on the determined characteristic of the light source, the identified light source within the virtual domain of the mixed reality presentation; and

the simulation of the identified light source within the virtual domain includes the overlaying of the shadow extraction object onto the one or more real objects upon which the shadow is cast.

20. The system of claim 16, wherein:

the second virtual object that replaces the target object within the mixed reality presentation is an animated character; and

the at least one physical computing device further

identifies an additional target object among the real objects included within the real-world environment while the animated character is presented within the mixed reality presentation and while the additional target object is included within a field of view of the animated character,

extracts the additional target object from the mixed reality presentation by overlaying a third virtual object included within the virtual domain onto the mixed reality presentation in alignment with the additional target object, the third virtual object implementing an extraction object for the additional target object,

replaces the additional target object within the mixed reality presentation by overlaying a fourth virtual object included within the virtual domain onto the mixed reality presentation out of alignment with the additional target object, and

presents, within the mixed reality presentation, an interaction between the animated character and the fourth virtual object.