Audio output for panoramic images

• US 10,261,749 B1
• Filed: 11/30/2016
• Issued: 04/16/2019
• Est. Priority Date: 11/30/2016
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Wilcox, Eric, Tran, Thanh
• Primary Examiner(s): Brahmachari, Mandrita
• Application Number:
  US15/365,601
• Time in Patent Office: 867 Days
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/0485 Scrolling or panning
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/165 Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: G06V 20/20 in augmented reality scenes

Abstract

Implementations relate to audio output for panoramic images. In some implementations, a computer-implemented method includes causing a first portion of a panoramic image to be displayed by a display device of a user device, and receiving user input at the user device. Based on the user input, the method determines at least a part of a second portion of the panoramic image for display, the second portion associated with a first audio segment and a second audio segment, selects one of the first audio segment and the second audio segment, and causes a change in display of the panoramic image, where the change includes transitioning the display from the first portion to the second portion. The method causes output of the selected audio segment by an audio output device in communication with the user device.

Claims

1. A computer-implemented method comprising:

causing a first portion of a panoramic image to be displayed by a display device of a user device;

receiving user input at the user device;

based on the user input;

determining at least a part of a second portion of the panoramic image for display, the second portion associated with a first audio segment and a second audio segment, wherein the first audio segment is associated with a first range of scrolling rate and the second audio segment is associated with a second range of scrolling rate;

causing a change in display of the panoramic image, wherein the change includes transitioning the display from the first portion to the second portion, wherein the transitioning includes scrolling the panoramic image within a display view of the user device;

selecting one of the first audio segment and the second audio segment based on the user input and based on at least one of;

a rate of the transitioning or a direction of the transitioning, wherein selecting one of the first audio segment and the second audio segment includes;

selecting the first audio segment in response to the scrolling of the panoramic image being at a rate within the first range of scrolling rate, and

selecting the second audio segment in response to the scrolling of the panoramic image being at a rate within the second range of scrolling rate; and

causing output of the selected audio segment by an audio output device in communication with the user device.

2. The computer-implemented method of claim 1 wherein the user input includes a gesture provided by a user on a touchscreen of the display device, and wherein selecting one of the first audio segment and the second audio segment is based on a type of the gesture.

3. The computer-implemented method of claim 1 wherein the first audio segment is associated with a first range of zooming rate and the second audio segment is associated with a second range of zooming rate, wherein the transitioning includes zooming the panoramic image within a display view of the user device, and wherein selecting one of the first audio segment and the second audio segment includes:

selecting the first audio segment in response to the zooming of the panoramic image being at a rate within the first range of zooming rate, and

selecting the second audio segment in response to the zooming of the panoramic image being at a rate within the second range of zooming rate.

4. The computer-implemented method of claim 1 wherein selecting one of the first audio segment and the second audio segment is further based on one or more stored user preferences.

5. The computer-implemented method of claim 1 further comprising determining at least one image characteristic associated with the second portion of the panoramic image, wherein selecting one of the first audio segment and the second audio segment is further based on the at least one image characteristic associated with the second portion.

6. The computer-implemented method of claim 1 wherein selecting one of the first audio segment and the second audio segment is further based on one or more device characteristics of the user device, wherein the device characteristics include at least one of:

an orientation of the user device in space, and a sensed motion of the user device in space.

7. The computer-implemented method of claim 1 wherein selecting one of the first audio segment and the second audio segment is further based on one or more current environmental characteristics of the user device, wherein the current environmental characteristics include at least one of:

a current time, a current date, and a geographic location of the user device.

8. The computer-implemented method of claim 1 wherein selecting one of the first audio segment and the second audio segment is further based on an ambient sound level surrounding the user device.

9. The computer-implemented method of claim 1 wherein selecting one of the first audio segment and the second audio segment is further based on a respective priority level associated with each of the first audio segment and the second audio segment.

10. The computer-implemented method of claim 1 wherein the first range of scrolling rate does not overlap the second range of scrolling rate.

11. The computer-implemented method of claim 1 wherein causing the first portion of the panoramic image to be displayed includes causing a spherical display of the first portion, and wherein the transitioning the display from the first portion to the second portion includes causing a spherical display of the second portion.

12. A device comprising:

a memory;

a display device;

an audio output device; and

at least one processor coupled to the display device and to the audio output device and configured to access the memory, the at least one processor configured to perform operations comprising;

causing a first portion of a panoramic image to be displayed by the display device;

receiving user input at the device;

based on the user input,

determining at least part of a second portion of the panoramic image for display, the second portion associated with a first audio segment and a second audio segment, wherein the first audio segment is associated with a first range of zooming rate and the second audio segment is associated with a second range of zooming rate;

causing a change in display of the panoramic image, wherein the change includes transitioning the display from the first portion to the second portion, wherein the transitioning includes zooming the panoramic image within a display view on the display device;

selecting one of the first audio segment and the second audio segment based on the user input and based on at least one of;

a rate of the transitioning or a direction of the transitioning, wherein selecting one of the first audio segment and the second audio segment includes;

selecting the first audio segment in response to the zooming of the panoramic image being at a rate within the first range of zooming rate, and

selecting the second audio segment in response to the zooming of the panoramic image being at a rate within the second range of zooming rate; and

causing output by the audio output device of the selected one of the first audio segment and the second audio segment.

13. The device of claim 12 wherein the at least one processor is configured to perform operations further comprising detecting at least one image content feature depicted in the at least part of the second portion of the panoramic image,

wherein the at least one processor is configured to perform the selecting one of the first audio segment and the second audio segment further based on the at least one image content feature and to include:

determining one or more image content categories of the at least one image content feature; and

determining which of the first audio segment and the second audio segment is triggered by the one or more image content categories,

wherein the selected audio segment corresponds to the triggered one of the first audio segment and the second audio segment.

14. The device of claim 13 wherein the at least one processor is configured to perform the determining which of the first audio segment and the second audio segment is triggered by the one or more image content categories to include:

determining one or more audio categories for the first audio segment and the second audio segment; and

determining which of the one or more audio categories is associated with the one or more image content categories based on stored associations, wherein the triggered one of the first audio segment and the second audio segment has the associated one or more audio categories.

15. The device of claim 12 wherein the at least one processor is configured to perform the selecting one of the first audio segment and the second audio segment further based on the change in the display of the panoramic image caused by the user input.

16. A non-transitory computer readable medium having stored thereon software instructions that, when executed by a processor, cause the processor to perform operations including:

causing a first portion of a panoramic image to be displayed by a display device of a user device;

receiving user input at the user device;

based on the user input;

determining at least a part of a second portion of the panoramic image for display, the second portion associated with a first audio segment and a second audio segment, wherein the first audio segment is associated with a first range of scrolling rate and the second audio segment is associated with a second range of scrolling rate, wherein the first range of scrolling rate does not overlap the second range of scrolling rate;

causing a change in display of the panoramic image, wherein the change includes transitioning the display from the first portion to the second portion, wherein the transitioning includes scrolling the panoramic image within a display view of the user device;

selecting one of the first audio segment and the second audio segment based on the user input and based on at least one of;

a rate of the transitioning or a direction of the transitioning, wherein selecting one of the first audio segment and the second audio segment includes;

selecting the first audio segment in response to the scrolling of the panoramic image being at a rate within the first range of scrolling rate, and

selecting the second audio segment in response to the scrolling of the panoramic image being at a rate within the second range of scrolling rate; and

causing output of the selected audio segment by an audio output device in communication with the user device.