RECOGNITION INTERFACES FOR COMPUTING DEVICES

• US 20140337800A1
• Filed: 12/20/2013
• Published: 11/13/2014
• Est. Priority Date: 05/09/2013
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Rooney, Sean Anthony, Agnetta, Bryan Todd, Park, Jae Pum, Mains, Richard Leigh, Marashi, Suzan, Gray, Timothy Thomas, Salazar, Gray Anthony, Sommer, Steven Steven, Cummins, Charles Eugene
• Granted Patent: US 9,927,949 B2
• Time in Patent Office: Days
• US Class Current: 715/835
• CPC Class Codes: G06F 2203/04801 Cursor retrieval aid, i.e. visual aspect modification, blinking, colour changes, enlargement or other visual cues, for helping user do find the cursor in graphical user interfaces
• CPC Class Codes: G06F 3/002 Specific input/output arrangements not covered by G06F3/01 - G06F3/16 other optical apparatus G02B27/00
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/0485 Scrolling or panning
• CPC Class Codes: G06V 10/235 based on user input or interaction
• CPC Class Codes: H04M 1/2755 by optical scanning
• CPC Class Codes: H04M 2250/22 including a touch pad, a touch sensor or a touch detector digitizers for touch screens G06F3/041
• CPC Class Codes: H04M 2250/52 including functional features of a camera constructional features concerning the integration of cameras in portable phones H04M1/0264

Abstract

A computing device can utilize a recognition mode wherein an interface utilizes graphical elements, such as virtual fireflies or other such elements, to indicate objects that are recognized or identified. As objects are recognized, fireflies perform one or more specified actions to indicate recognition. A ribbon or other user-selectable icon is displayed indicates a specific action that the device can perform with respect to the respective object. As additional objects are recognized, additional ribbons are created and older ribbons can be moved off screen and stored for subsequent retrieval or search. The fireflies disperse when the objects are no longer represented in captured sensor data, and can be animated to move towards representations of new objects as features of those objects are identified as potential object features, in order to communicate a level of recognition for a current scene or environment.

Claims

1. A computing device, comprising:

at least one processor;

a display screen;

at least one camera; and

memory including instructions that, when executed by the at least one processor, cause the computing device to;

generate video data using the at least one camera;

display, on the display screen as the video data is generated, video content based on at least a portion of the video data;

recognize at least a first object and a second object represented in the video content;

cause a first plurality of graphical elements to indicate the first object has been recognized;

cause a second plurality of the graphical elements to indicate the second object has been recognized;

display a first user-selectable icon on the display screen for the first object, the first user-selectable icon conveying a first action that the computing device is configured to perform in response a touch event occurring on the display screen proximate the first user-selectable icon, wherein the first action is based at least in part upon a first object type;

display a second user-selectable icon on the display screen for the second object, the second user-selectable icon conveying a second action that the computing device is configured to perform in response a touch event occurring on the display screen proximate the second user-selectable icon, wherein the second action is based at least in part upon a second object type; and

in response to more than a threshold number of user-selectable icons being generated for display, where the threshold number is greater than two, remove at least one excess user-selectable icon from the display screen, the excess user-selectable icon being subsequently accessible to the user using at least one navigational approach.

2. The computing device of claim 1, wherein the instructions when executed further cause the computing device to:

capture audio data using a microphone of the computing device; and

display a third user-selectable icon on the display screen for the audio data, the third user-selectable icon conveying a third action that the computing device is configured to perform in response a touch event occurring on the display screen proximate the third user-selectable icon, wherein the third action is based at least in part upon a type of recognized audio in the audio data.

3. The computing device of claim 1, wherein the instructions when executed further cause the computing device to:

cause the first user-selectable icon to be displayed to appear to be stacked on top of the second user-selectable icon on the display screen.

4. The computing device of claim 1, wherein the instructions when executed further cause the computing device to:

cause information for a previously-displayed user-selectable icon to be stored by the computing device; and

cause the previously-displayed user-selectable icon to be displayed in response to a user performing a swiping motion on the display screen, wherein the swiping motion is able to be used to locate any previously-displayed user-selectable icon.

5. The computing device of claim 1, wherein the instructions when executed further cause the computing device to:

provide a search option enabling a search to be executed against information stored for any currently displayed user-selectable icon and any previously-displayed user-selectable icon.

16. The computer-implemented method of claim 5, further comprising:

determining a priority for each user-selectable icon corresponding to an object currently represented in the acquired sensor data; and

positioning for display on the display screen at least a subset of the user-selectable icons based at least in part upon the priority.

17. The computer-implemented method of claim 16, wherein determining the priority is based at least in part upon at least one of a location of a respective actionable object in the acquired sensor data, a type of the respective actionable object, historical user data, a date of last access, a distance to location for creation, or a time at which each of the user-selectable icons was generated.

6. A computer-implemented method, comprising:

acquiring sensor data obtained using at least one sensor of a computing device;

recognizing, using at least one processor of the computing device, a first object and a second object represented in the acquired sensor data;

causing a first plurality of graphical elements to indicate, on a display screen of the computing device, that the first object has been recognized;

causing a second plurality of the graphical elements to indicate, on the display screen, that the second object has been recognized;

displaying a first user-selectable icon on the display screen for the first object, the first user-selectable icon conveying a first action that the computing device is configured to perform with respect to the first object;

displaying a second user-selectable icon on the display screen for the second object, the second user-selectable icon conveying a second action that the computing device is configured to perform with respect to the second object;

displaying up to a threshold of user-selectable icons, wherein the threshold number is at least two; and

removing at least one excess user-selectable icon from the display screen, the excess user-selectable icon being subsequently accessible to the user using at least one navigational approach.

7. The computer-implemented method of claim 6, further comprising:

storing information of the first user-selectable icon on the computing device; and

displaying the first user-selectable icon on the display screen for at least a determined period of time after the object is no longer represented in the acquired sensor data.

8. The computer-implemented method of claim 6, further comprising:

animating the first plurality of graphical elements to indicate that the first object has been recognized by forming an animated bounding box proximate a representation of the first object displayed on the display screen, the animated bounding box conveying the first action that the computing device is configured to perform with respect to the first object.

9. The computer-implemented method of claim 6, further comprising:

displaying a first label and a second label on the first user-selectable icon, the first label conveying a first action that the computing device is configured to perform with respect to the first object and the second label conveying a different action that the computing device is configured to perform with respect to the first object.

10. The computer-implemented method of claim 6, further comprising:

causing at least one user-selectable icon to be removed from display in response to an additional user-selectable icon causing the number of user-selectable icons to be displayed to exceed an allowable threshold.

11. The computer-implemented method of claim 10, further comprising:

automatically storing information for any user-selectable icon no longer displayed on the computing device, wherein the information once stored is accessible using at least one of a scroll action, a touch-screen-based pull action, or a search request.

12. The computer-implemented method of claim 10, further comprising:

identifying a previously-recognized object represented in the acquired sensor data; and

causing a user-selectable icon to be displayed using the information stored for the previously-recognized object.

13. The computer-implemented method of claim 6, wherein a plurality of user-selectable icons is capable of being displayed at least one of stacked or positioned in a sequence.

14. The computer-implemented method of claim 13, wherein at least a subset of the plurality of user-selectable icons enable more than one action to be performed with respect to the respective object.

15. The computer-implemented method of claim 14, further comprising:

selecting a primary action of the more than one action based at least in part upon historical data associated with the user, information for the primary action being displayed on the respective user-selectable icon.

18. A non-transitory computer-readable storage medium storing instructions that, when executed by at least one processor of a computing device, cause the computing device to:

capture sensor data using at least one sensor of the computing device;

recognize a first object and a second object represented in the acquired sensor data;

cause a first plurality of graphical elements to indicate, on a display screen of the computing device, that the first object has been recognized;

cause a second plurality of the graphical elements to indicate, on the display screen, that the second object has been recognized;

display a first user-selectable icon on the display screen for the first object, the first user-selectable icon conveying a first action that the computing device is configured to perform with respect to the first object;

display a second user-selectable icon on the display screen for the second object, the second user-selectable icon conveying a second action that the computing device is configured to perform with respect to the second object; and

in response to more than a threshold number of user-selectable icons being generated for display, where the threshold number is greater than two, remove at least one excess user-selectable icon from the display screen, the excess user-selectable icon being subsequently accessible to the user using at least one navigational approach.

19. The non-transitory computer-readable storage medium of claim 18, wherein the instructions when executed further cause the computing device to:

store information of the first user-selectable icon on the computing device; and

display the first user-selectable icon on the display screen for at least a determined period of time after the object is no longer represented in the acquired sensor data.

20. The non-transitory computer-readable storage medium of claim 18, wherein the instructions when executed further cause the computing device to:

animate the first plurality of graphical elements to indicate that the first object has been recognized by forming an animated bounding box proximate a representation of the first object displayed on the display screen, the animated bounding box conveying the first action that the computing device is configured to perform with respect to the first object.