Method, medium, and system for online ordering using sign language

• US 9,230,160 B1
• Filed: 08/27/2012
• Issued: 01/05/2016
• Est. Priority Date: 08/27/2012
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Kanter, Max L.
• Primary Examiner(s): Zimmerman, Matthew
• Application Number:
  US13/595,384
• Time in Patent Office: 1,226 Days
• Field of Search: 705/26.1, 705/27.1
• US Class Current: 1/1
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/033 Pointing devices displaced or positioned by the user , e.g. mice, trackballs, pens or joysticks; Accessories therefor digitisers characterised by the transducing means G06F3/041
• CPC Class Codes: G06Q 30/0601 Electronic shopping [e-shopping]
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: G09B 21/009 Teaching or communicating with deaf persons G09B21/04 takes precedence; deaf and dumb persons G09B21/00

Abstract

With a sign language computer interface, a user may perform one or more gestures in accordance with a recognized sign language, and have the gestures translated into information or instructions for use by one or more computers. Such interfaces may capture video imagery of a user performing gestures using one or more cameras, sense the gestures expressed in the video imagery, search a library for phrases corresponding to the gestures, and display one or more of the phrases to the user, who may indicate whether the phrases are consistent with his or her intended communication. The interfaces permit a user to communicate with computer-based systems the exclusive use of gestures, and without a single keystroke or mouse click.

Claims

1. A non-transitory computer-readable medium having a computer-executable component which causes a computer system to perform a method enabling an online marketplace to process an order using sign language, the method comprising:

causing a display of information pertaining to at least one item on a first user interface using the at least one processor;

generating a query for a response related to the at least one item;

receiving video data from a customer in response to the query, the video data comprising a sequence of activity;

analyzing the video data to identify at least one gesture by the customer included in the sequence of activity;

identifying at least one phrase as related to the at least one gesture by the customer;

causing a display of the at least one phrase on a second user interface;

requesting a confirmation from the customer that the at least one phrase corresponds to the at least one gesture by the customer included in the sequence of activity;

determining whether the confirmation is received from the customer within a predetermined time period;

if the confirmation is received from the customer within the predetermined time period, processing the order for the at least one item; and

if the confirmation is not received from the customer within the predetermined time period, requesting a phrase corresponding to the at least one gesture from the customer.

2. The non-transitory computer-readable medium of claim 1, further comprising:

searching a dictionary for a plurality of potentially matching phrases corresponding to the at least one gesture by the customer included in the sequence of activity, wherein the dictionary comprises a plurality of gestures corresponding to a plurality of phrases;

calculating a confidence interval for each of the plurality of potentially matching phrases; and

selecting the one of the potentially matching phrases having the highest confidence interval as the at least one gesture by the customer.

3. The non-transitory computer-readable medium of claim 2, wherein the at least one phrase is identified based at least in part on a context of the query, and

wherein the context relates to at least one of an identity of the customer, a location of the customer, a purchasing history of the customer or the at least one item.

4. A computer-implemented method enabling an online marketplace to process an order using sign language from a customer, comprising:

causing a display of information relating to at least one product available at the online marketplace using at least one computer processor, wherein the information comprises at least one of a name of the at least one product, a price of the at least one product, a size of the at least one product or an image of the at least one product;

receiving a communication relating to the at least one product over a network from a customer in response to the display of the information, the communication including video imagery;

analyzing the video imagery to identify at least one gesture included in the video imagery;

determining at least one phrase as potentially corresponding to the at least one gesture according to a recognized sign language by the at least one computer processor;

requesting a confirmation from the customer that the at least one phrase corresponds to the at least one gesture by the customer included in the video imagery;

determining whether the confirmation is received from the customer within a predetermined time period;

if the confirmation is received from the customer within the predetermined time period, processing the order for the at least one product; and

if the confirmation is not received from the customer within the predetermined time period, requesting a phrase corresponding to the at least one gesture from the customer.

5. The computer-implemented method according to claim 4, wherein identifying the at least one phrase as potentially corresponding to the at least one gesture according to the recognized sign language further comprises:

searching a library for information corresponding to the at least one gesture,

wherein the library includes a plurality of phrases corresponding to a plurality of gestures according to the recognized sign language.

6. The computer-implemented method according to claim 4, wherein identifying at least one phrase as potentially corresponding to the at least one gesture according to a recognized sign language further comprises:

identifying a plurality of phrases as potentially corresponding to the at least one gesture, and

wherein the method further comprises;

calculating a confidence interval for each of the plurality of phrases with respect to the at least one gesture;

ranking the plurality of phrases according to the confidence intervals; and

causing a display of at least some of the plurality of phrases ranked according to the confidence intervals using the at least one computer processor,

wherein the confidence interval is a measure of a probability that one of the plurality of phrases corresponds to the at least one gesture according to the recognized sign language.

7. The computer-implemented method according to claim 4, wherein identifying the at least one phrase as potentially corresponding to the at least one gesture according to a recognized sign language further comprises:

identifying a plurality of phrases as potentially corresponding to the at least one gesture, and

wherein the method further comprises;

calculating a confidence interval for each of the plurality of phrases with respect to the at least one gesture;

identifying one of the plurality of phrases having a highest confidence interval; and

causing a display of the one of the plurality of phrases having the highest confidence interval.

8. The computer-implemented method according to claim 4, wherein identifying the at least one phrase as potentially corresponding to the at least one gesture according to a recognized sign language further comprises:

determining a context of the communication; and

identifying the at least one phrase as potentially corresponding to the at least one gesture according to the recognized sign language based at least in part on the context of the communication.

9. The computer-implemented method according to claim 8, wherein the context relates to at least one of an identity of the customer, a location of the customer, a dialect associated with the customer, an idiolect associated with the customer, or the at least one product.

10. The computer-implemented method according to claim 8, wherein the context relates to at least one of a product that was previously purchased by the customer, a product that was previously considered for purchase by the customer, a product that was previously viewed by the customer or a category of products including a product that was previously purchased by the customer.

11. The computer-implemented method according to claim 4, further comprising:

if the confirmation is received from the customer prior to the predetermined time,

updating the library to indicate that the at least one phrase corresponds to the at least one gesture.

12. The computer-implemented method according to claim 4, wherein the video imagery is captured using at least one camera associated with the at least one computer processor.

13. The computer-implemented method according to claim 12, wherein the video imagery is captured using two cameras,

wherein each of the two cameras captures the video imagery from a different perspective,

wherein the method further comprises:

obtaining a stereoscopic impression of the video imagery using the at least one computer processor,

wherein the at least one phrase is identified as potentially corresponding to the at least one gesture based at least in part on the stereoscopic impression.

14. The computer-implemented method according to claim 4, wherein the at least one gesture comprises at least one facial expression or at least one movement of at least one hand.

15. The computer-implemented method according to claim 4, wherein the recognized sign language is the American Sign Language.

23. The computer-implemented method of claim 4, further comprising:

if the confirmation is not received from the customer within the predetermined time period,

updating the library to indicate that the at least one phrase does not correspond to the at least one gesture.

16. A computer-implemented method enabling an online marketplace to process an order using sign language from a customer, comprising:

receiving a request for service from a customer over a network, the request including a first set of video imagery;

analyzing the request for service to identify a first gesture expressed in the first set of video imagery;

translating, by a computer processor, the first gesture into at least a first phrase according to a sign language;

preparing a response to the request for service based at least in part on the first phrase;

transmitting the response to the customer, the response including a second set of video imagery;

causing a display of the second set of video imagery to the customer, wherein the second set of video imagery comprises a request for a confirmation that at least the first phrase corresponds to at least the first gesture by the customer included in the first set of video imagery;

determining whether the confirmation is received from the customer within a predetermined time period;

if the confirmation is received from the customer within the predetermined time period, processing the order for the at least one product; and

if the confirmation is not received from the customer within the predetermined time period, requesting a phrase corresponding to the first gesture from the customer.

17. The computer-implemented method according to claim 16, wherein translating the first gesture into at least the first phrase according to the sign language further comprises:

identifying a context of the request for service; and

translating the first gesture into at least a second phrase based at least in part on the context of the request for service, and

wherein the context of the request for service relates to at least one of a location of the customer, an item purchased by the customer, a dialect associated with the customer, an idiolect associated with the customer, at least one item previously purchased by the customer or at least one item previously considered for purchase by the customer.

18. The computer-implemented method according to claim 16,

wherein at least a second gesture is expressed in the second set of video imagery, and

wherein at least the second gesture corresponds to at least a second phrase according to the sign language.

19. The computer-implemented method according to claim 16, wherein the second set of video imagery comprises a text-based message.

20. A mobile computer device configured to interact with an online marketplace for processing an order using sign language from a customer, comprising:

at least one computer processor;

at least one data store; and

at least one camera,

wherein the mobile computer device is connected to a network, and

wherein the mobile computer device is configured with a computer program which when executed causes the mobile device to perform a method comprising;

capturing a video data file depicting a user performing at least one gesture using the at least one camera;

analyzing the video data file to recognize the at least one gesture;

searching a library for at least one word potentially corresponding to the at least one gesture according to a recognized sign language;

requesting a confirmation from the user that the at least one word corresponds to the at least one gesture;

determining whether the confirmation is received from the user within a predetermined time period;

if the confirmation is received from the user within the predetermined time period, processing the order for the at least one product; and

if the confirmation is not received from the user within the predetermined time period, requesting a phrase corresponding to the at least one gesture from the user.

21. The mobile computer device according to claim 20, wherein searching the library for the at least one word potentially corresponding to the at least one gesture further comprises:

determining a context associated with the video data file; and

selecting the library based at least in part on the context of the video data file.

22. The mobile computer device according to claim 21,

wherein the context relates to at least one of a location of the user, an item purchased by the user, a purchasing history of the user, a dialect associated with the user or an idiolect associated with the user.