Augmented reality user interface control

• US 10,430,016 B2
• Filed: 12/22/2017
• Issued: 10/01/2019
• Est. Priority Date: 12/22/2017
• Status: Active Grant

• Current Assignee: Snap, Inc.
• Original Assignee: Snap, Inc.
• Inventors: Monastyrshyn, Yurii, Pyschchenko, Oleksandr
• Primary Examiner(s): Tran, Tam T
• Application Number:
  US15/853,264
• Publication Number:
  US 20190196663A1
• Time in Patent Office: 648 Days
• Field of Search: None
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06V 10/25 Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation

Abstract

Systems, devices, media, and methods are presented for controlling user interface with an object depicted within the user interface. The systems and methods initiates an augmented reality mode configured to present augmented reality elements within a graphical user interface. A face is detected within a field of view of an image capture component and presented within the graphical user interface. In response to detecting the face, the systems and methods sequentially present a set of augmented reality elements within the graphical user interface. A subset of the augmented reality elements and the face may be depicted contemporaneously. A movement is detected of at least a portion of the face relative to a first augmented reality element and presentation of the first augmented reality element is modified and at least one second augmented reality element is presented.

Claims

1. A method, comprising:

initiating, by one or more processors of a device, an augmented reality mode configured to present augmented reality elements within a graphical user interface presented on a display component of the device;

detecting a face within a field of view of an image capture component coupled to the device;

in response to detecting the face, sequentially presenting a set of augmented reality elements within the graphical user interface, such that a subset of augmented reality elements and the face are depicted within the graphical user interface contemporaneously;

detecting a movement of at least a portion of the face relative to a first augmented reality element of the subset of augmented reality elements; and

in response to detecting the movement, modifying presentation of the first augmented reality element and causing presentation of at least one second augmented reality element of the subset of augmented reality elements.

2. The method of claim 1, wherein the movement of the portion of the face is a movement of a reference feature and further comprises:

identifying a first state of the reference feature, the first state corresponding to first positions of a set of landmark points corresponding to the reference feature; and

determining a change of the reference feature from the first state to a second state corresponding to second positions of at least one landmark point of the set of landmark points.

3. The method of claim 2, further comprising:

determining a facial location of the reference feature;

determining a spatial location of the first augmented reality element within the graphical user interface, the spatial location determined at a time corresponding to the reference feature changing from the first state to the second state;

determining a proximity of the spatial location to the facial location as being within a proximity threshold; and

based on the proximity and in response to the reference feature changing to the second state, presenting the at least one second augmented reality element within the graphical user interface along with the face.

4. The method of claim 3, wherein the reference feature is a mouth, the first state is an open state of the mouth, and the second state is a closed state of the mouth, and modifying presentation of the first augmented reality element further comprises:

determining the first augmented reality element, at the spatial location, obscures at least a portion of the mouth in the first state, the portion of the mouth obscured by the first augmented reality element being an interior surface of the mouth;

in response to the first augmented reality element obscuring the portion of the mouth, modifying presentation of the first augmented reality element for a period of time prior to terminating presentation of the first augmented reality element; and

causing presentation of the at least one second augmented reality element spaced a distance apart from the face and the reference feature, the at least one second augmented reality element moving along a trajectory toward the face.

5. The method of claim 1, wherein detecting the movement of the face further comprises:

identifying a chance of position of the face, within the graphical user interface, from a first position to a second position;

in response to identifying the second position, determining a head location within the graphical user interface at a first time; and

determining a spatial location of the first augmented reality element at a second time.

6. The method of claim 5, further comprising:

determining a proximity of the spatial location, at the second time, relative to the head location exceeds a miss threshold;

modifying presentation of the first augmented reality element by modifying one or more of a size, a color, and a position of the first augmented reality element over a period of time; and

terminating presentation of the first augmented reality element after modifying presentation of the first augmented reality element and prior to causing presentation of the at least one second augmented reality element.

7. The method of claim 5, further comprising:

determining a proximity of the spatial location, at the second time, relative to the head location is below a miss threshold;

determining a termination position of the first augmented reality element at a third time, the termination position being within an boundary of the face; and

modifying presentation of the first augmented reality element by modifying the first augmented reality element from a first state to a second state, the second state of the first augmented reality element remaining at the termination position on the face for a period of time.

8. The method of claim 1, wherein modifying presentation of the first augmented reality element further comprises:

causing presentation of the first augmented reality element traveling in a first direction relative to the face;

in response to the movement of the portion of the face, determining a contact the first augmented reality element and a region of the face; and

in response to the contact, causing presentation of the first augmented reality element traveling in a second direction relative to the face.

9. The method of claim 8, wherein causing presentation of the at least one second augmented reality element further comprises:

causing presentation of an augmented reality element representing the contact between the first augmented reality element and the region of the face; and

causing presentation of an augmented reality element representing an interaction history of the face relative to the first augmented reality element.

10. The method of claim 1, further comprising:

responsive to the movement of the portion of the face and modifying presentation of the first augmented reality element, modifying a depiction of the face within the graphical user interface to generate a modified face; and

causing presentation of the modified face contemporaneous with presentation of one or more of the first augmented reality element and the at least one second augmented reality element.

11. A system, comprising:

one or more processors; and

a non-transitory processor-readable storage medium coupled to the one or more processors, the non-transitory processor-readable storage medium storing processor executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising;

initiating, by the one or more processors, an augmented reality mode configured to present augmented reality elements within a graphical user interface presented on a display component of the device;

detecting, by the one or more processors, a face within a field of view of an image capture component coupled to the device;

in response to detecting the face, sequentially presenting, by the one or more processors, a set of augmented reality elements within the graphical user interface, such that a subset of augmented reality elements and the face are depicted within the graphical user interface contemporaneously;

detecting, by the one or more processors, a movement of at least a portion of the face relative to a first augmented reality element of the subset of augmented reality elements; and

in response to detecting the movement, modifying, by the one or more processors, presentation of the first augmented reality element and causing presentation of at least one second augmented reality element of the subset of augmented reality elements.

12. The system of claim 11, wherein the movement of the portion of the face is a movement of a reference feature and further comprises:

identifying a first state of the reference feature, the first state corresponding to first positions of a set of landmark points corresponding to the reference feature; and

determining a change of the reference feature from the first state to a second state corresponding to second positions of at least one landmark point of the set of landmark points.

13. The system of claim 12, wherein the operations further comprise:

determining a facial location of the reference feature;

determining a spatial location of the first augmented reality element within the graphical user interface, the spatial location determined at a time corresponding to the reference feature changing from the first state to the second state;

determining a proximity of the spatial location to the facial location as being within a proximity threshold; and

based on the proximity and in response to the reference feature changing to the second state, presenting the at least one second augmented reality element within the graphical user interface along with the face.

14. The system of claim 11, wherein detecting the movement of the face further comprises:

identifying a chance of position of the face, within the graphical user interface, from a first position to a second position;

in response to identifying the second position determining a head location within the graphical user interface at a first time; and

determining a spatial location of the first augmented reality element at a second time.

15. The system of claim 11, wherein modifying presentation of the first augmented reality element further comprises:

causing presentation of the first augmented reality element traveling in a first direction relative to the face;

in response to the movement of the portion of the face, determining a contact the first augmented reality element and a region of the face; and

in response to the contact, causing presentation of the first augmented reality element traveling in a second direction relative to the face.

16. A non-transitory processor-readable storage medium storing processor executable instructions that, when executed by a processor of a machine, cause the machine to perform operations comprising:

initiating, by one or more processors of a device, an augmented reality mode configured to present augmented reality elements within a graphical user interface presented on a display component of the device;

detecting, by the one or more processors, a face within a field of view of an image capture component coupled to the device;

in response to detecting the face, sequentially presenting, by the one or more processors, a set of augmented reality elements within the graphical user interface, such that a subset of augmented reality elements and the face are depicted within the graphical user interface contemporaneously;

detecting, by the one or more processors, a movement of at least a portion of the face relative to a first augmented reality element of the subset of augmented reality elements; and

in response to detecting the movement, modifying, by the one or more processors, presentation of the first augmented reality element and causing presentation of at least one second augmented reality element of the subset of augmented reality elements.

17. The non-transitory processor-readable storage medium of claim 16, wherein the movement of the portion of the face is a movement of a reference feature and further comprises:

identifying a first state of the reference feature, the first state corresponding to first positions of a set of landmark points corresponding to the reference feature; and

determining a change of the reference feature from the first state to a second state corresponding to second positions of at least one landmark point of the set of landmark points.

18. The non-transitory processor-readable storage medium of claim 17, wherein the operations further comprise:

determining a facial location of the reference feature;

determining a spatial location of the first augmented reality element within the graphical user interface, the spatial location determined at a time corresponding to the reference feature changing from the first state to the second state;

determining a proximity of the spatial location to the facial location as being within a proximity threshold; and

based on the proximity and in response to the reference feature changing to the second state, presenting the at least one second augmented reality element within the graphical user interface along with the face.

19. The non-transitory processor-readable storage medium of claim 16, wherein detecting the movement of the face further comprises:

identifying a chance of position of the face, within the graphical user interface, from a first position to a second position;

in response to identifying the second position determining a head location within the graphical user interface at a first time; and

determining a spatial location of the first augmented reality element at a second time.

20. The non-transitory processor-readable storage medium of claim 16, wherein modifying presentation of the first augmented reality element further comprises:

causing presentation of the first augmented reality element traveling in a first direction relative to the face;

in response to the movement of the portion of the face, determining a contact the first augmented reality element and a region of the face; and

in response to the contact, causing presentation of the first augmented reality element traveling in a second direction relative to the face.