Systems and methods for utilizing a living entity as a marker for augmented reality content

• US 10,636,188 B2
• Filed: 02/09/2018
• Issued: 04/28/2020
• Est. Priority Date: 02/09/2018
• Status: Active Grant

• Current Assignee: Nicholas T. Hariton
• Original Assignee: Nicholas T. Hariton
• Inventors: Hariton, Nicholas T.
• Primary Examiner(s): Salvucci, Matthew
• Application Number:
  US15/893,498
• Publication Number:
  US 20190251720A1
• Time in Patent Office: 809 Days
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 13/80 2D [Two Dimensional] animation, e.g. using sprites
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 10/143 Sensing or illuminating at different wavelengths
• CPC Class Codes: G06V 10/225 based on a marking or identifier characterising the area
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: G06V 40/18 Eye characteristics, e.g. of the iris
• CPC Class Codes: G10L 17/00 Speaker identification or verification techniques

Abstract

Systems and methods for utilizing living entities as markers for virtual content in an augmented reality environment are discussed herein. The virtual content may comprise objects, surfaces, textures, effects, and/or other content visibly manifested in views of the augmented reality environment. In some implementations, the virtual content may comprise an avatar and/or other full- or partial-body virtual content object depicted based on the living entity. A living entity and multiple linkage points for the living entity may be detected within the field of view of a user. Based on the arrangement of the linkage points, virtual content may be rendered and appear superimposed over or in conjunction with a view of the living entity in the augmented reality environment. In some implementations, the rendering of virtual content in the augmented reality environment may be triggered by the arrangement of the multiple linkage points for a given living entity.

Claims

1. A system configured to utilize living entities as markers for virtual content viewed in an augmented reality environment, the system comprising:

a display device configured to superimpose images of visual virtual content over a real-world view of a user to create a visual effect of the augmented reality environment being present in a real world, wherein the real-world view of the user is an outlook of the real world from the point of view of the user;

one or more physical computer processors configured by computer readable instructions to;

detect a living entity within a field of view of the real-world view of the user, the living entity being a separate and discrete entity from the user, wherein the field of view is the area that comprises the real-world view of the user, wherein detecting the living entity visible within the field of view includes;

obtaining a first image of the field of view visible via the display device;

obtaining a second image of the field of view visible via the display device; and

detecting the living entity visible within the field of view based on a comparison of the first image and the second image;

detect multiple linkage points for the living entity;

determine an arrangement of the multiple linkage points, wherein the arrangement defines the relative position of each of the linkage points to each of the other linkage points of the multiple linkage points;

dynamically determine virtual content information based on the arrangement, wherein the determined virtual content information defines (i) a virtual content item to be displayed when the arrangement is present and (ii) a correlation between the arrangement and a reference frame of the virtual content item;

obtain the determined virtual content information;

generate an image of the virtual content item to be displayed in the augmented reality environment based on the living entity, wherein the image of the virtual content item is generated based at least on the virtual content information and the field of view; and

cause the image of the virtual content item to be displayed in the augmented reality environment so that the image of the virtual content item is superimposed over the living entity in the real-world view of the user such that the user views the image of the virtual content item in the augmented reality environment via the display device.

2. The system of claim 1, wherein the arrangement is determined at a first point in time, and wherein the one or more processors are further configured to:

determine a second arrangement of the multiple linkage points at a second point in time, wherein the second arrangement of the multiple linkage points is different from the arrangement of the multiple linkage points at the first point in time;

identify an action of the living entity based on the first arrangement and the second arrangement;

obtain second virtual content information defining a second virtual content item to be displayed when the action is identified;

generate an image of the second virtual content item to be displayed in the augmented reality environment based on the living entity; and

cause the image of the second virtual content item to be displayed in the augmented reality environment so that the image of the second virtual content item is superimposed over the living entity in the real-world view of the user such that the user views the image of the second virtual content item in the augmented reality environment via the display device.

3. The system of claim 2, wherein the second virtual content item comprises an animation associated with the action.

4. The system of claim 1, wherein the one or more processors are further configured to:

determine an identification of the living entity that distinguishes the living entity from other living entities, wherein the virtual content information is obtained based on the identification of the living entity.

5. The system of claim 4, wherein the identification of the living entity comprises an indication of a type of living entity and/or a group of living entities.

6. The system of claim 1, wherein a position of each of the multiple linkage points in the real world defines the reference frame of the virtual content item with respect to the real world and the living entity.

7. The system of claim 1, wherein the image of the virtual content item is generated based further on a position of the display device in the real world and the position of each of the multiple linkage points in the real world.

8. The system of claim 1, wherein the image of the virtual content item is generated based further on the size of the arrangement of the multiple linkage points within the field of view of the user.

9. The system of claim 1, wherein the field of view is defined based on location information and orientation information, the location information indicating at least a current location associated with the display device, and the orientation information indicating at least a pitch angle, a roll angle, and a yaw angle associated with the display device.

10. The system of claim 1, wherein to detect the living entity and the multiple linkage points defined with respect to the living entity, the one or more processors are further configured to:

receive a signal from a transponder of the living entity, wherein the signal from the transponder indicates a location of the transponder and includes triggering information that identifies virtual content to be displayed in the augmented reality environment based on the living entity;

obtain virtual content information based on the triggering information;

detect at least a first linkage point for the living entity based on the transponder, wherein the signal from the transponder defines the position of the first linkage point with respect to the living entity; and

detect one or more additional linkage points based on the position of the first linkage point with respect to the living entity, wherein the multiple linkage points include at least the first linkage point and the one or more additional linkage points.

11. A method of utilizing living entities as markers for virtual content viewed in an augmented reality environment, the method comprising:

detecting a living entity within a field of view of a real-world view of a user, the living entity being a separate and discrete entity from the user, wherein the field of view is the area that comprises the real-world view of the user, wherein the real-world view of the user is an outlook of a real world from the point of view of the user, wherein detecting the living entity visible within the field of view comprises;

obtaining a first image of the field of view visible via the display device;

obtaining a second image of the field of view visible via the display device; and

detecting the living entity visible within the field of view based on a comparison of the first image and the second image;

detecting multiple linkage points for the living entity;

determining an arrangement of the multiple linkage points, wherein the arrangement defines the relative position of each of the linkage points to each of the other linkage points of the multiple linkage points;

dynamically determining virtual content information based on the arrangement, wherein the determined virtual content information defines (i) a virtual content item to be displayed when the arrangement is present and (ii) a correlation between the arrangement and a reference frame of the virtual content item;

obtaining the determined virtual content information;

generating an image of the visual virtual content item to be displayed in the augmented reality environment based on the living entity, wherein the image of the virtual content item is generated based at least on the virtual content information and the field of view; and

causing the image of the virtual content item to be displayed in the augmented reality environment so that the image of the virtual content item is superimposed over the living entity in the real-world view of the user such that the user views the image of the virtual content item in the augmented reality environment via a display device, wherein the display device is configured to superimpose images of visual virtual content over the real-world view of the user to create a visual effect of the augmented reality environment being present in the real world.

12. The method of claim 11, wherein the arrangement is determined at a first point in time, the method further comprising:

determining a second arrangement of the multiple linkage points at a second point in time, wherein the second arrangement of the multiple linkage points is different from the arrangement of the multiple linkage points at the first point in time;

identifying an action of the living entity based on the first arrangement and the second arrangement;

obtaining second virtual content information defining a second virtual content item to be displayed when the action is identified;

generating an image of the second virtual content item to be displayed in the augmented reality environment based on the living entity; and

causing the image of the second virtual content item to be displayed in the augmented reality environment so that the image of the second virtual content item is superimposed over the living entity in the real-world view of the user such that the user views the image of the second virtual content item in the augmented reality environment via the display device.

13. The method of claim 12, wherein the second virtual content item comprises an animation associated with the action.

14. The method of claim 11, the method further comprising:

determining an identification of the living entity that distinguishes the living entity from other living entities, wherein the virtual content information is obtained based on the identification of the living entity.

15. The method of claim 14, wherein the identification of the living entity comprises an indication of a type of living entity and/or a group of living entities.

16. The method of claim 11, wherein a position of each of the multiple linkage points in the real world defines the reference frame of the virtual content item with respect to the real world and the living entity.

17. The method of claim 11, wherein the image of the virtual content item is generated based further on a position of the display device in the real world and the position of each of the multiple linkage points in the real world.

18. The method of claim 11, wherein the image of the virtual content item is generated based further on the size of the arrangement of the multiple linkage points within the field of view of the user.

19. The method of claim 11, wherein the field of view is defined based on location information and orientation information, the location information indicating at least a current location associated with the display device, and the orientation information indicating at least a pitch angle, a roll angle, and a yaw angle associated with the display device.

20. The method of claim 11, wherein detecting the living entity and the multiple linkage points defined with respect to the living entity comprises:

receiving a signal from a transponder of the living entity, wherein the signal from the transponder indicates a location of the transponder and includes triggering information that identifies virtual content to be displayed in the augmented reality environment based on the living entity;

obtaining virtual content information based on the triggering information;

detecting at least a first linkage point for the living entity based on the transponder, wherein the signal from the transponder defines the position of the first linkage point with respect to the living entity; and

detecting one or more additional linkage points based on the position of the first linkage point with respect to the living entity, wherein the multiple linkage points include at least the first linkage point and the one or more additional linkage points.