Relative position-inclusive device interfaces

• US 9,274,744 B2
• Filed: 10/12/2010
• Issued: 03/01/2016
• Est. Priority Date: 09/10/2010
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Hart, Gregory M., Bezos, Jeffrey P., Kwee, Frances MHH, Brown, James Samuel
• Primary Examiner(s): Rao, Andy
• Assistant Examiner(s): SOSANYA, OBAFEMI OLUDAYO
• Application Number:
  US12/902,986
• Publication Number:
  US 20120062729A1
• Time in Patent Office: 1,967 Days
• Field of Search: 348/135
• US Class Current: 1/1
• CPC Class Codes: G06F 1/1626 with a single-body enclosure integrating a flat display, e.g. Personal Digital Assistants [PDAs]
• CPC Class Codes: G06F 1/1686 the I/O peripheral being an integrated camera
• CPC Class Codes: G06F 2200/1637 Sensing arrangement for detection of housing movement or orientation, e.g. for controlling scrolling or cursor movement on the display of an handheld computer
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/165 Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: H04M 2250/52 including functional features of a camera constructional features concerning the integration of cameras in portable phones H04M1/0264
• CPC Class Codes: H04N 7/142 Constructional details of the terminal equipment, e.g. arrangements of the camera and the display

Abstract

A computing device can analyze image or video information to determine a relative position of an active user. The computing device can optimize audio or video data capture based at least in part upon the relative location. The device can capture audio using one or more microphones pointing toward the relative location of the active user, and can use other microphones to determine audio from other sources to be removed from the captured audio. If video data is being captured, a video capture element can be adjusted to focus primarily on the active user. The position of the user can be monitored so the audio and video data capture can be adjusted accordingly.

Claims

1. A method of modifying user input to a computing device, comprising:

capturing first image data of a user using a first image capture element of the computing device;

capturing second image data of the user using a second image capture element, the first image capture element and the second image capture element being positioned on a same side of the computing device;

analyzing the first image data and the second image data to detect at least one action performed by the user;

determining, using a triangulation determination based at least in part on the first image data and the second image data, a relative position of the user with respect to the computing device;

selecting, based at least in part on the relative position of the user, a first audio capture element from a plurality of audio capture elements, the first audio capture element being closer to the user than a remainder of the plurality of audio capture elements;

capturing audio data using the plurality of audio capture elements of the computing device;

determining background audio data included in the audio data based at least in part on the relative position of the user, the background audio data corresponding to sound captured by the remainder of the plurality of audio capture elements;

generating modified audio data by removing data corresponding to at least a portion of the background audio data included in the audio data; and

providing the modified audio data as input to the computing device.

2. The method of claim 1, further comprising:

updating the relative position of the user based on the first image data and the second image data.

3. The method of claim 1, further comprising:

capturing video data of the user using at least one of the first image capture element or the second image capture element, wherein the at least one of the first image capture element or the second image capture element is oriented to capture the video data from a viewable area at least partially including the relative position of the user.

4. The method of claim 1, wherein generating the modified audio data includes removing substantially all of the background audio data from the audio data.

29. The method of claim 1, wherein the triangulation determination of the first image data and the second image data is based at least in part on angles associated with respective positions of the first image capture element and the second image capture element relative to the user.

5. A method of modifying input to a computing device, comprising:

capturing first image data and second image data using at least two image capture elements of the computing device, the at least two image capture elements being positioned on a same side of the computing device, the first image data and the second image data including respective image representations of an active audio source;

determining a relative position of the active audio source with respect to the computing device, based, at least in part, on a triangulation determination based at least in part on the first image data and the second image data;

selecting, based at least in part on the relative position of the active audio source, a first audio capture element from a plurality of audio capture elements, the first audio capture element being closer to the active audio source than a remainder of the plurality of audio capture elements;

capturing active audio data using the plurality of audio capture elements;

updating the relative position of the active audio source based at least in part upon third image data of the active audio source captured using the at least two image capture elements to obtain an updated relative position of the active audio source;

adjusting the first audio capture element to capture the active audio data from a direction at least partially towards the updated relative position of the active audio source;

determining background audio data within the active audio data based at least in part on the updated relative position of the active audio source, the background audio data corresponding to sound emitted from locations other than the updated relative position of the active audio source; and

modifying the active audio data to remove data corresponding to at least a portion of the background audio data.

6. The method of claim 5, further comprising:

analyzing the first image data to detect at least one action corresponding to the active audio source.

7. The method of claim 6, wherein analyzing the first image data comprises executing at least one image analysis algorithm for determining a motion of a user, a gesture of the user, speech formation by the user, or a state of a sound-producing device.

8. The method of claim 6, wherein determining the relative position of the active audio source with respect to the computing device further comprises:

validating an identity of the active audio source.

9. The method of claim 5, further comprising:

determining a second relative position of a secondary audio source with respect to the computing device;

capturing secondary audio data using at least one audio capture element of the computing device, the at least one audio capture element being configured to capture the secondary audio data from a direction at least partially towards the second relative position of the secondary audio source; and

filtering the secondary audio data from the active audio data.

10. The method of claim 5, further comprising:

analyzing the active audio data and second audio data captured from any other audio source; and

filtering of the active audio data based on the second audio data.

11. The method of claim 5, wherein capturing the active audio data is based at least in part upon an identity of the active audio source.

12. The method of claim 5, further comprising:

capturing video data of the active audio source using at least one image capture element of the computing device currently oriented to capture the video data from a viewable area at least partially including the relative position of the active audio source.

13. The method of claim 5, wherein the at least two image capture elements include at least one of an image capture element, an ultrasonic transceiver, an infrared transceiver, a laser, or a radio transceiver.

14. The method of claim 5, wherein the relative position of the active audio source is updated based upon at least one of movement of the active audio source or movement of the computing device.

15. The method of claim 5, wherein determining the relative position of the active audio source comprises identifying the active audio source within a defined area with respect to the computing device.

16. The method of claim 5, further comprising transmitting a response signal from the computing device in the direction at least partially towards the updated relative position of the active audio source.

17. The method of claim 5, wherein the active audio source is a primary user of the computing device or another person in closest proximity to the computing device.

18. The method of claim 5, further comprising:

determining a person speaking out of a plurality of people within a proximity of the computing device to be the active audio source.

19. A computing device, comprising:

a processor;

a memory device including instructions operable to be executed by the processor to perform a set of actions, enabling the processor to;

capture first image data and second image data using at least two image capture elements of the computing device, the at least two image capture elements being positioned on a same side of the computing device, the first image data and second image data including respective image representations of an active audio source;

determine a relative position of the active audio source with respect to the computing device, based, at least in part, on a triangulation determination based at least in part on the first image data and the second image data;

select, based at least in part on the relative position of the active audio source, a first audio capture element from a plurality of audio capture elements, the first audio capture element being closer to the active audio source than a remainder of the plurality of audio capture elements;

capture active audio data using the plurality of audio capture elements;

update the relative position of the active audio source based at least in part upon third image data of the active audio source captured using the at least two image capture elements to obtain an updated relative position; and

adjust the first audio capture element to capture the active audio data from a direction at least partially towards the updated relative position of the active audio source;

determine background audio data within the active audio data based at least in part on the updated relative position of the active audio source, the background audio data corresponding to sound emitted from locations other than the updated relative position of the active audio source; and

modify the active audio data to remove data corresponding to at least a portion of the background audio data.

20. The computing device of claim 19, wherein the instructions when executed further cause the computing device to:

analyze the first image data to detect at least one action corresponding to the active audio source.

21. The computing device of claim 19, wherein the instructions when executed further cause the computing device to:

determine a second relative position of a secondary audio source with respect to the computing device;

capture secondary audio data using at least one audio capture element of the computing device, the at least one audio capture element being configured to capture the secondary audio data from a direction at least partially towards the second relative position of the secondary audio source; and

filter the secondary audio data from the active audio data.

22. The computing device of claim 19, wherein capturing the active audio data is based at least in part upon an identity of the active audio source.

23. The computing device of claim 19, wherein the instructions when executed further cause the computing device to:

capture video data of the active audio source using at least one image capture element of the computing device oriented to capture the video data from a viewable area at least partially including the updated relative position of the active audio source.

24. A non-transitory computer-readable storage medium storing processor-executable instructions for controlling a computing device, comprising:

program code for capturing first image data and second image data using at least two image capture elements of the computing device, the at least two image capture elements being positioned on a same side of the computing device, the first image data and the second image data including respective image representations of an active audio source;

program code for determining a relative position of the active audio source with respect to the computing device based, at least in part, on a triangulation determination based at least in part on the first image data and the second image data;

program code for selecting, based at least in part on the relative position of the active audio source, a first audio capture element from a plurality of audio capture elements, the first audio capture element being closer to the active audio source than a remainder of the plurality of audio capture elements;

program code for capturing active audio data using the first audio capture element and the remainder of the plurality of audio capture elements of the computing device;

program code for updating the relative position of the active audio source based at least in part upon third image data of the active audio source captured using the at least two image capture elements to obtain an updated relative position; and

program code for adjusting the first audio capture element to capture the active audio data from a direction at least partially towards the updated relative position of the active audio source with respect to the computing device;

program code for determining background audio data within the active audio data based at least in part on the updated relative position of the active audio source, the background audio data corresponding to sound emitted from locations other than the updated relative position of the active audio source; and

program code for modifying the active audio data to remove data corresponding to at least a portion of the background audio data.

25. The non-transitory computer-readable storage medium of claim 24, further comprising:

program code for analyzing the first image data to detect at least one action corresponding to the active audio source.

26. The non-transitory computer-readable storage medium of claim 24, further comprising:

program code for determining a second relative position of a secondary audio source with respect to the computing device;

program code for capturing secondary audio data using at least one audio capture element of the computing device, the at least one audio capture element being configured to capture the secondary audio data from a direction at least partially towards the second relative position of the secondary audio source; and

program code for filtering the secondary audio data from the active audio data.

27. The non-transitory computer-readable storage medium of claim 24, wherein capturing the active audio data is based at least in part upon an identity of the active audio source.

28. The non-transitory computer-readable storage medium of claim 24, further comprising:

program code for capturing video data of the active audio source using at least one image capture element of the computing device oriented to capture the video data from a viewable area at least partially including the relative position of the active audio source.