Audio responsive augmented reality

• US 10,445,936 B1
• Filed: 08/01/2017
• Issued: 10/15/2019
• Est. Priority Date: 08/01/2016
• Status: Active Grant

• Current Assignee: Snap, Inc.
• Original Assignee: Snap, Inc.
• Inventors: Charlton, Ebony James, Dargis, Jokubas, Evans, Michael John, Hare, Samuel Edward, McPhee, Andrew James, Murphy, Robert Cornelius, Pilipski, Eitan
• Primary Examiner(s): Ge, Jin
• Application Number:
  US15/666,349
• Time in Patent Office: 805 Days
• Field of Search: 345589
• CPC Class Codes: G06F 3/16 Sound input; Sound output speech processing G10L
• CPC Class Codes: G06F 3/165 Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2200/24 involving graphical user interfaces [GUIs]
• CPC Class Codes: G06T 2219/2012 Colour editing, changing, or manipulating; Use of colour codes
• CPC Class Codes: G06T 2219/2016 Rotation, translation, scaling
• CPC Class Codes: G06T 2219/2021 Shape modification
• CPC Class Codes: H04S 7/40 Visual indication of stereophonic sound image

Abstract

Systems, devices, media, and methods are presented for generating responsive augmented reality elements in a graphical user interface. The systems and methods receive a video stream having one or more frames containing image data and audio data. The systems and methods identify a set of coordinates within a portion of the one or more frames and identify one or more audio characteristics within the audio data of the video stream. In response to the one or more audio characteristics, the systems and methods generate one or more graphical interface elements and detect a change in the audio data within the video stream. The systems and methods modify the one or more graphical interface elements in a second portion of the video stream in response to the change in the audio data.

Claims

1. A method, comprising:

receiving, by one or more processors, a video stream comprising one or more frames containing image data and audio data;

detecting one or more audio characteristics within the audio data of the video stream;

identifying, based on the one or more audio characteristics, one or more graphical interface elements that correspond to the one or more audio characteristics detected within the audio data;

displaying the identified one or more graphical interface elements while the one or more audio characteristics continue to be detected within the audio data associated with multiple frames of the video stream, the one or more graphical interface elements displayed in a first portion of the video stream, wherein a first portion of the one or more graphical interface elements is persistently displayed while the one or more audio characteristics continue to be detected within the audio data associated with the multiple frames of the video stream, and wherein a second portion of the one or more graphical interface elements is temporarily displayed for a predetermined period of time simultaneously with the first portion of the one or more graphical interface elements;

detecting a change in the audio data within the video stream; and

in response to detecting the change in the audio data, modifying the one or more graphical interface elements in a second portion of the video stream.

2. The method of claim 1 further comprising:

detecting, at a gyroscope component, a change in orientation of a client device;

determining the change in orientation exceeds a predetermined orientation threshold;

generating a coordinate mesh within at least a portion of the one or more frames of the video stream, wherein the graphical interface elements are generated for display using the coordinate mesh; and

resuming display of the one or more graphical interface elements when an object present in a first frame of the one or more frames in the image data re-enters a second frame of the one or more frames in the image data.

3. The method of claim 1, wherein the one or more audio characteristics detected within the audio data include volume or frequency, wherein the one or more graphical interface elements are generated for display in a first color, wherein modifying the one or more graphical interface elements comprises gradually changing a color of the one or more graphical interface elements from the first color to a second color in response to detecting a change in the volume or frequency.

4. The method of claim 1 further comprising:

generating a first graphical interface element associated with the one or more audio characteristics, the first graphical interface element generated with a set of first behaviors associated with at least one audio characteristic of the one or more audio characteristics; and

generating a set of second graphical interface elements associated with the one or more audio characteristics, the set of second graphical interface elements generated with a set of second behaviors associated with at least one audio characteristic of the one or more audio characteristics.

5. The method of claim 4, wherein modifying the one or more graphical interface elements in the second portion of the video stream further comprises:

modifying one or more visual characteristic of the first graphical interface element according to the set of first behaviors based on the change in the audio data;

modifying one or more visual characteristics of one or more second graphical interface elements of the set of second graphical interface elements according to the set of second behaviors based on the change in the audio data; and

causing presentation of the modified first graphical interface element and the modified one or more second graphical interface elements within the second portion of the video stream.

6. The method of claim 5, wherein the one or more visual characteristic of the set of second graphical interface elements is selected from a group consisting of a color value, an opacity value, a brightness value, a size, a position of a second graphical interface element, and a relative position of two or more second graphical interface elements.

7. The method of claim 1 further comprising:

determining one or more positions for the one or more graphical interface elements with respect to a set of coordinates in a first frame of the video stream;

identifying a change in angular position of the client device between the first frame and a subsequent frame; and

generating one or more second positions for the one or more graphical interface elements with respect to the set of coordinates based on the change in angular position of the client device.

8. The method of claim 1 further comprising:

tracking one or more positions of the one or more graphical interface elements across the one or more frames of the video stream.

9. A device comprising:

one or more processors; and

computer readable storage medium comprising instructions that, when executed by the one or more processors of a device, cause the one or more processors to perform operations comprising;

receiving, by one or more processors, a video stream comprising one or more frames containing image data and audio data;

detecting one or more audio characteristics within the audio data of the video stream;

identifying, based on the one or more audio characteristics, one or more graphical interface elements that correspond to the one or more audio characteristics detected within the audio data;

displaying the identified one or more graphical interface elements while the one or more audio characteristics continue to be detected within the audio data associated with multiple frames of the video stream, the one or more graphical interface elements displayed in a first portion of the video stream, wherein a first portion of the one or more graphical interface elements is persistently displayed while the one or more audio characteristics continue to be detected within the audio data associated with the multiple frames of the video stream, and wherein a second portion of the one or more graphical interface elements is temporarily displayed for a predetermined period of time simultaneously with the first portion of the one or more graphical interface elements;

detecting a change in the audio data within the video stream; and

in response to detecting the change in the audio data, modifying the one or more graphical interface elements in a second portion of the video stream.

10. The device of claim 9, wherein the operations further comprise:

detecting, at a gyroscope component, a change in orientation of a client device;

determining the change in orientation exceeds a predetermined orientation threshold;

generating a coordinate mesh within at least a portion of the one or more frames of the video stream, wherein the graphical interface elements are generated for display using the coordinate mesh; and

resuming display of the one or more graphical interface elements when an object present in a first frame of the one or more frames in the image data re-enters a second frame of the one or more frames in the image data.

11. The device of claim 9, wherein the operations further comprise:

generating a first graphical interface element associated with the one or more audio characteristics, the first graphical interface element generated with a set of first behaviors associated with at least one audio characteristic of the one or more audio characteristics; and

generating a set of second graphical interface elements associated with the one or more audio characteristics, the set of second graphical interface elements generated with a set of second behaviors associated with at least one audio characteristic of the one or more audio characteristics.

12. The device of claim 11, wherein modifying the one or more graphical interface elements in the second portion of the video stream further comprises:

modifying one or more visual characteristic of the first graphical interface element according to the set of first behaviors based on the change in the audio data;

modifying one or more visual characteristics of one or more second graphical interface elements of the set of second graphical interface elements according to the set of second behaviors based on the change in the audio data; and

causing presentation of the modified first graphical interface element and the modified one or more second graphical interface elements within the second portion of the video stream.

13. The device of claim 9, wherein the operations further comprise:

determining one or more positions for the one or more graphical interface elements with respect to a set of coordinates in a first frame of the video stream;

identifying a change in angular position of the client device between the first frame and a subsequent frame; and

generating one or more second positions for the one or more graphical interface elements with respect to the set of coordinates based on the change in angular position of the client device.

14. The device of claim 9, wherein the operations further comprise:

tracking one or more positions of the one or more graphical interface elements across the one or more frames of the video stream.

15. A non-transitory computer readable storage medium comprising non-transitory instructions that, when executed by one or more processors of a device, cause the device to perform operations comprising:

receiving, by one or more processors, a video stream comprising one or more frames containing image data and audio data;

detecting one or more audio characteristics within the audio data of the video stream;

identifying, based on the one or more audio characteristics, one or more graphical interface elements that correspond to the one or more audio characteristics detected within the audio data;

displaying the identified one or more graphical interface elements while the one or more audio characteristics continue to be detected within the audio data associated with multiple frames of the video stream, the one or more graphical interface elements displayed in a first portion of the video stream, wherein a first portion of the one or more graphical interface elements is persistently displayed while the one or more audio characteristics continue to be detected within the audio data associated with the multiple frames of the video stream, and wherein a second portion of the one or more graphical interface elements is temporarily displayed for a predetermined period of time simultaneously with the first portion of the one or more graphical interface elements;

detecting a change in the audio data within the video stream; and

in response to detecting the change in the audio data, modifying the one or more graphical interface elements in a second portion of the video stream.

16. The non-transitory computer readable storage medium of claim 15, wherein the operations further comprise:

detecting, at a gyroscope component, a change in orientation of a client device;

determining the change in orientation exceeds a predetermined orientation threshold;

generating a coordinate mesh within at least a portion of the one or more frames of the video stream, wherein the graphical interface elements are generated for display using the coordinate mesh; and

resuming display of the one or more graphical interface elements when an object present in a first frame of the one or more frames in the image data re-enters a second frame of the one or more frames in the image data.

17. The non-transitory computer readable storage medium of claim 15, wherein the operations further comprise:

generating a first graphical interface element associated with the one or more audio characteristics, the first graphical interface element generated with a set of first behaviors associated with at least one audio characteristic of the one or more audio characteristics; and

generating a set of second graphical interface elements associated with the one or more audio characteristics, the set of second graphical interface elements generated with a set of second behaviors associated with at least one audio characteristic of the one or more audio characteristics.

18. The non-transitory computer readable storage medium of claim 17, wherein modifying the one or more graphical interface elements in the second portion of the video stream further comprises:

modifying one or more visual characteristic of the first graphical interface element according to the set of first behaviors based on the change in the audio data;

modifying one or more visual characteristics of one or more second graphical interface elements of the set of second graphical interface elements according to the set of second behaviors based on the change in the audio data; and

causing presentation of the modified first graphical interface element and the modified one or more second graphical interface elements within the second portion of the video stream.

19. The non-transitory computer readable storage medium of claim 15, wherein the operations further comprise:

determining one or more positions for the one or more graphical interface elements with respect to a set of coordinates in a first frame of the video stream;

identifying a change in angular position of the client device between the first frame and a subsequent frame; and

generating one or more second positions for the one or more graphical interface elements with respect to the set of coordinates based on the change in angular position of the client device.