MULTI DISPARATE GESTURE ACTIONS AND TRANSACTIONS APPARATUSES, METHODS AND SYSTEMS

• US 20150012426A1
• Filed: 01/06/2014
• Published: 01/08/2015
• Est. Priority Date: 01/04/2013
• Status: Abandoned Application

• Current Assignee: Visa International Service Association (Visa, Inc.)
• Original Assignee: Visa International Service Association (Visa, Inc.)
• Inventors: Purves, Thomas, Hua, Julian, Rutherford, Robert
• Application Number:
  US14/148,576
• Publication Number:
  US 20150012426A1
• Time in Patent Office: Days
• US Class Current: 705/41
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06Q 20/321 using wearable devices
• CPC Class Codes: G06Q 20/326 Payment applications installed on the mobile devices
• CPC Class Codes: G06Q 20/384 using social networks
• CPC Class Codes: G06Q 20/386 using messaging services or messaging apps using mobile network messaging services for payment, e.g. SMS G06Q20/3255
• CPC Class Codes: G06Q 30/0623 Item investigation
• CPC Class Codes: G06Q 30/0631 Item recommendations
• CPC Class Codes: G06Q 30/0639 Item locations
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command

Abstract

The Multi Disparate Gesture Actions And Transactions Apparatuses, Methods And Systems (“MDGAAT”) transform gesture, video, and audio inputs via MDGAAT components into action, augmented reality, and transaction outputs. receiving from a wallet user multiple gesture actions within a specified temporal quantum; determining composite constituent gestures, gesture manipulated objects, and user account information from the received multiple gesture actions; determining via a processor a composite gesture action associated with the determined composite constituent gestures and gesture manipulated objects; and executing via a processor the composite gesture action to perform a transaction with a user account specified by the user account information.

Claims

11. A processor-implemented method comprising:

providing check-in information to a merchant store, the check-in information i) being associated with a user, and ii) being stored on the user'"'"'s mobile device, wherein the user has an account with the merchant store;

accessing, based on the provided check-in information, an identifier for the user, wherein the identifier is associated with the account;

detecting, via a sensor, a first gesture that is performed by the user, the first gesture being directed to an item that is included in the merchant store, wherein the first gesture is detected after the providing of the check-in information to the merchant store;

providing the detected first gesture to the merchant store;

determining an action associated with the detected first gesture;

performing the action associated with the detected first gesture, wherein the performing of the action associated with the detected first gesture modifies the account with information related to the item;

detecting, via the sensor, a second gesture that is performed by the user, wherein the second gesture is detected after the performing of the action associated with the detected first gesture;

providing the detected second gesture to the merchant store;

determining an action associated with the detected second gesture, wherein the action associated with the detected second gesture initiates a payment transaction between the user and the merchant store; and

performing the action associated with the detected second gesture.

12. The method of claim 11, wherein the check-in information is generated by the user snapping a quick response (QR) code that is provided by the merchant store.

13. The method of claim 11, wherein the check-in information is sent to a remote server.

14. The method of claim 11, wherein the check-in information includes geo-location information of the user.

15. The method of claim 11, wherein the merchant store assigns a sales clerk to the user upon the providing of the check-in information to the merchant store.

16. The method of claim 15, wherein the sales clerk comprises a local representative of the merchant store or a remote representative of the merchant store.

17. The method of claim 11, wherein the account includes loyalty information for the user or a past purchasing history with the merchant store for the user.

18. The method of claim 11, wherein the performing of the action:

generates a user inquiry about a location of the item;

generates a user request to communicate with a sales clerk for shopping assistance;

modifies the account with the information related to the item by placing the item in a virtual shopping cart of the user;

or

modifies the account with the information related to the item by placing the item on a wish list of the user.

19. The method of claim 18, wherein the communication with the sales clerk is conducted via:

in-person communication between the user and the sales clerk;

video chat;

audio chat;

instant messages;

or

text messages.

20. The method of claim 11, further comprising:

providing, based on the detected first gesture, a product purchase recommendation to the user, wherein the product purchase recommendation includes product items related to the item.

21. A processor-implemented method comprising:

obtaining a visual capture of a reality scene via a visual device, the visual capture of the reality scene including an object that identifies a subset of data included in a user account;

performing image analysis on the visual capture via an image analysis tool of the visual device, wherein the object is identified based on the image analysis, and wherein the visual device accesses the subset of data based on the identified object;

generating, based on the subset of data, an augmented reality display that is viewed by a user, the user i) being associated with the subset of data, and ii) using the visual device to obtain the visual capture;

detecting a gesture performed by a user, wherein the gesture is directed to a user interactive area included in the augmented reality display;

providing the detected gesture to the visual device, the visual device being configured to determine an action associated with the detected gesture, wherein the determined action is based on one or more aspects of the augmented reality display; and

performing the action associated with the detected gesture, wherein the performing of the action modifies the subset of data based on information relating to the user interactive area.

22. The method of claim 21, wherein the gesture includes a fingertip motion within the augmented reality display.

23. The method of claim 21, wherein the visual device comprises a pair of smart glasses that is worn by the user.

24. The method of claim 21, wherein the visual device comprises an optical head-mounted display (OHMD) that is worn by the user, and wherein the OHMD includes:

a front camera to capture a line of sight of the user; and

a rear camera to capture eye movement of the user.

25. The method of claim 21, wherein the image analysis includes:

barcode reading;

quick response (QR) code decoding;

or

optical character recognition.

26. The method of claim 21, wherein the augmented reality display includes virtual information labels overlaid atop the reality scene.

27. The method of claim 26, wherein the gesture includes a fingertip motion that causes movement of one or more of the virtual information labels.

28. The method of claim 21, wherein the subset of data includes information relating to:

a collection of bills;

products available for purchase by the user;

a news feed;

a social media feed;

a calendar;

or

a desktop display.

29. The method of claim 21, further comprising:

detecting, at a sensor, a voice command that is vocalized by the user;

generating the augmented reality display based on the subset of data and the voice command.

30. The method of claim 21, wherein the augmented reality display is not visible to a second user that is not operating the visual device.

31. A processor-implemented method comprising:

obtaining a visual capture of a reality scene via a visual device, the visual capture including an image of a customer, wherein the visual device is operated by personnel of a merchant store;

performing image analysis on the visual capture via an image analysis tool of the visual device;

identifying, based on the image analysis, an identifier for the customer that is depicted in the image, the identifier being associated with a user account of the customer; and

generating, via the visual device, an augmented reality display that includes i) the image of the customer, and ii) additional image data that surrounds the image of the customer, the augmented reality display being viewed by the personnel of the merchant store,

wherein the additional image data is based on the user account of the customer and is indicative of prior behavior by the customer.

32. The method of claim 31, wherein the additional image data that surrounds the image of the customer comprises a glow, a semi-transparent bound box that covers the image of the customer, or text related to the customer.

33. The method of claim 31, wherein the prior behavior by the customer indicates whether the customer has paid for an item, whether the customer has provided check-in information to the merchant store, an item previously purchased at the merchant store by the customer, or an amount of money spent by the customer in the merchant store.

34. The method of claim 31, wherein the visual device comprises a pair of smart glasses that is worn by the personnel of the merchant store.

35. The method of claim 31, wherein the visual device comprises an optical head-mounted display (OHMD) that is worn by the personnel of the merchant store, and wherein the OHMD includes:

a front camera to capture a line of sight of the personnel; and

a rear camera to capture eye movement of the personnel.

36. The method of claim 31, wherein the image analysis includes:

barcode reading;

quick response (QR) code decoding;

optical character recognition;

or

facial recognition.

37. The method of claim 31, further comprising:

determining an amount of shopping assistance desired by the customer based on the additional image data;

or

communicating with the customer based on the additional image data.

38. The method of claim 37, wherein the communication with the customer is conducted via:

in-person communication between the customer and the personnel;

video chat;

audio chat;

instant messages;

or

text messages.

39. The method of claim 31, further comprising:

providing, based on the additional image data, a product purchase recommendation or a coupon to the customer.

40. The method of claim 31, wherein the augmented reality display is not visible to a user that is not operating the visual device.

41. A processor-implemented method comprising:

obtaining one or more visual captures of a reality scene via a visual device, the one or more visual captures including i) a first image of a bill to be paid, and ii) a second image of a person or object that is indicative of a financial account;

performing image analysis on the one or more visual captures via an image analysis tool of the visual device, wherein the person or object that is indicative of the financial account is identified based on the image analysis, and wherein an itemized expense included on the bill to be paid is identified based on the image analysis;

generating, via the visual device, an augmented reality display that includes a user interactive area, the user interactive area being associated with the itemized expense;

detecting, via a sensor, a gesture performed by a user of the visual device, the gesture being directed to the user interactive area;

providing the detected gesture to the visual device, wherein the visual device is configured to determine an action associated with the detected gesture; and

performing the action associated with the detected gesture, the performing of the action being configured to associate the itemized expense with the financial account.

42. The method of claim 41, wherein the financial account is associated with a payment card or a bank account.

43. The method of claim 41, wherein the gesture includes a fingertip motion within the augmented reality display.

44. The method of claim 43, wherein the fingertip motion moves an image associated with the itemized expense to the second image in the augmented reality display.

45. The method of claim 41, further comprising:

obtaining information pertaining to the financial account based on the identification of the person or object that is indicative of the financial account.

46. The method of claim 41, wherein the image analysis includes:

barcode reading;

quick response (QR) code decoding;

optical character recognition;

or

facial recognition.

47. The method of claim 41, wherein the visual device comprises a pair of smart glasses that is worn by the user.

48. The method of claim 41, wherein the visual device comprises an optical head-mounted display (OHMD) that is worn by the user, and wherein the OHMD includes:

a front camera to capture a line of sight of the user; and

a rear camera to capture eye movement of the user.

49. The method of claim 41, wherein the sensor includes a camera-enabled device that is configured to record a video or obtain one or more photographs.

50. The method of claim 49, wherein the camera-enabled device is included on the visual device or the camera-enabled device is separate from the visual device.

51. A processor-implemented method comprising:

obtaining a visual capture of a reality scene via a visual device, the visual capture including i) an image of a store display of a merchant store, and ii) an object that is associated with a first item and a second item, wherein the merchant store sells the first item and the second item, and wherein the store display includes the first item and the second item;

performing image analysis on the visual capture via an image analysis tool of the visual device, wherein the object is identified in the visual capture based on the image analysis;

storing an image of a user at the visual device, wherein the visual device is operated by the user or worn by the user;

generating, at the visual device, an interactive display that includes the image of the user and one or more user interactive areas, the one or more user interactive areas being associated with an image of the first item or an image of the second item;

detecting, via a sensor, a gesture performed by the user, wherein the detected gesture is directed to the one or more user interactive areas, and wherein the detected gesture is provided to the visual device; and

determining an action associated with the gesture and performing the action at the visual device, wherein the performing of the action updates the interactive display based on the image of the first item or the image of the second item, and wherein the updating of the interactive display causes the image of the user to be modified based on the image of the first item or the image of the second item.

52. The method of claim 51, wherein the first item comprises a first article of clothing, wherein the second item comprises a second article of clothing.

53. The method of claim 52, wherein the image of the user is modified to depict the user wearing the first article of clothing or the second article of clothing.

54. The method of claim 52, wherein the action causes scrolling through the first article of clothing and the second article of clothing in the interactive display.

55. The method of claim 51, wherein the image analysis includes:

barcode reading;

quick response (QR) code decoding;

optical character recognition;

facial recognition;

or

recognition of a pattern, shape, or predetermined image within the visual capture.

56. The method of claim 51, wherein the visual device comprises a pair of smart glasses that is worn by the user.

57. The method of claim 51, wherein the visual device comprises an optical head-mounted display (OHMD) that is worn by the user, and wherein the OHMD includes:

a front camera to capture a line of sight of the user; and

a rear camera to capture eye movement of the user.

58. The method of claim 51, wherein the interactive display includes additional information related to the first item or the second item.

59. The method of claim 58, wherein the additional information includes:

a price of the first item or the second item;

an availability of the first item or the second item;

a product purchase recommendation that includes information on items related to the first item or the second item;

or

a social media feed related to the first item or the second item.

60. The method of claim 51, wherein the interactive display includes an interactive area associated with a purchase action, and wherein manipulating the interactive area associated with the purchase action initiates a purchase transaction between the user and the merchant store for the first item or the second item.

61. A processor-implemented method comprising:

obtaining a visual capture of a reality scene via a visual device, wherein the visual capture includes an image of an item sold by a merchant store;

performing image analysis on the visual capture via an image analysis tool of the visual device, wherein the item sold by the merchant store is identified based on the image analysis; and

generating an augmented reality display at the visual device, wherein the augmented reality display includes i) the image of the item sold by the merchant store, and ii) additional image data that surrounds the image of the item,

wherein the additional image data that surrounds the image of the item is based on a list of one or more store items that is associated with a user, wherein the list of the one or more store items includes the item sold by the merchant store, and wherein the visual device is operated by the user or worn by the user.

62. The method of claim 61, wherein the list of the one or more store items comprises a shopping list entered by the user.

63. The method of claim 62, wherein the shopping list is generated by extracting product item information from a previous sales receipt of the user.

64. The method of claim 61, wherein the list of the one or more store items comprises a listing of items for which payment has been made to the merchant store by the user.

65. The method of claim 61, wherein the additional image data that surrounds the image of the item comprises a glow, a semi-transparent bound box that covers the image of the item sold by the merchant store, or text related to the item.

66. The method of claim 61, wherein the additional image data includes:

a price of the item;

an availability of the item.

67. The method of claim 61, wherein the image analysis includes:

barcode reading;

quick response (QR) code decoding;

or

optical character recognition.

68. The method of claim 61, wherein the visual device comprises a pair of smart glasses that is worn by the user.

69. The method of claim 61, wherein the visual device comprises an optical head-mounted display (OHMD) that is worn by the user, and wherein the OHMD includes:

a front camera to capture a line of sight of the user; and

a rear camera to capture eye movement of the user.

70. The method of claim 61, wherein the additional image data includes:

a product purchase recommendation that includes information on items related to the item;

or

a social media feed related to the item.

71. A processor-implemented method comprising:

displaying, at a television, a virtual store display that includes an image of an item, wherein a merchant store sells the item, and wherein the merchant store provides data to the television to generate the virtual store display;

obtaining a visual capture of the television via a visual device, wherein the visual capture includes at least a portion of the virtual store display;

performing image analysis on the visual capture via an image analysis tool of the visual device;

identifying the image of the item in the visual capture based on the image analysis;

generating an interactive display at the visual device, the interactive display including a user interactive area and a second image of the item;

detecting, via a sensor, a gesture performed by a user, the gesture being directed to the user interactive area of the interactive display;

providing the detected gesture to the visual device;

determining, at the visual device, an action associated with the detected gesture; and

performing the action associated with the detected gesture, wherein the performing of the action updates the interactive display.

72. The method of claim 71, wherein the visual device comprises a pair of smart glasses that is worn by the user.

73. The method of claim 71, wherein the visual device comprises an optical head-mounted display (OHMD) that is worn by the user, and wherein the OHMD includes:

a front camera to capture a line of sight of the user; and

a rear camera to capture eye movement of the user.

74. The method of claim 71, wherein the image analysis includes:

barcode reading;

quick response (QR) code decoding;

facial recognition;

or

recognition of a pattern, shape, or predetermined image within the visual capture.

75. The method of claim 71, wherein the sensor includes a camera-enabled device that is configured to record a video or obtain one or more photographs.

76. The method of claim 75, wherein the camera-enabled device is included on the visual device or the camera-enabled device is separate from the visual device.

77. The method of claim 76, wherein the camera-enabled device is included on the television or the camera-enabled device is separate from the television.

78. The method of claim 71, wherein the gesture includes multiple gesture actions within a predetermined period of time, the method further comprising:

determining composite constituent gestures, gesture manipulated objects in the interactive display, and an account of the user from the multiple gesture actions; and

determining the action associated with the determined composite constituent gestures and gesture manipulated objects.

79. The method of claim 71, wherein the action is a payment transaction between the user and the merchant store.

80. The method of claim 79, further comprising:

obtaining authorization credentials for the payment transaction from the user to the merchant store.

81. A processor-implemented method comprising:

detecting, at a sensor, a voice command that is vocalized by a first entity, wherein the voice command initiates a payment transaction to a second entity;

providing the detected voice command to a visual device that is operated by the first entity;

obtaining, at the visual device, a visual capture of a reality scene, wherein the visual capture of the reality scene includes an image of the second entity;

performing, at an image analysis tool of the visual device, image analysis on the obtained visual capture, wherein the image analysis tool identifies the image of the second entity in the visual capture;

reporting to the visual device that the second entity is in proximity to the first entity based on the identifying of the image of the second entity by the image analysis tool; and

completing the payment transaction from the first entity to the second entity based on the reporting.

82. The method of claim 81, wherein the second entity is:

a financial payment card having an account resolvable identifier;

a sales bill including a QR code;

or

a metro card.

83. The method of claim 81, wherein the payment transaction comprises:

a fund transfer from a first financial payment card of the first entity to a second financial payment card of the second entity.

85. The method of claim 81, further comprising:

generating a security alert request when the second entity comprises a financial payment card with a cardholder; and

sending the security alert to the cardholder of the second entity.

86. The method of claim 81, wherein the visual device comprises a pair of smart glasses that is worn by the first entity.

87. The method of claim 81, wherein the visual device comprises an optical head-mounted display (OHMD) that is worn by the first entity, and wherein the OHMD includes:

a front camera to capture a line of sight of the first entity; and

a rear camera to capture eye movement of the first entity.

88. The method of claim 81, wherein the image analysis includes:

barcode reading;

quick response (QR) code decoding;

optical character recognition;

facial recognition;

or

recognition of a pattern, shape, or predetermined image within the visual capture.

89. The method of claim 81, wherein the payment transaction includes:

a bill payment from the first financial payment card to a merchant store for a product purchase;

or

a fund refill from the first financial payment card to a metro card.

90. The method of claim 81, further comprising:

authenticating an identity of the second entity, wherein the authenticating includes;

requesting a user to input a passcode;

or

requesting the user to speak a predetermined sentence or phrase.

84-1. The method of claim 84, wherein obtaining the authorization credentials for the payment transaction include:

requesting a user to input a passcode for user identity confirmation;

or

requesting the user to speak a predetermined sentence or phrase.

91. A processor-implemented method comprising:

receiving from a wallet user multiple gesture actions within a specified temporal quantum;

determining composite constituent gestures, gesture manipulated objects, and user account information from the received multiple gesture actions;

determining via a processor a composite gesture action associated with the determined composite constituent gestures and gesture manipulated objects; and

executing via a processor the composite gesture action to perform a transaction with a user account specified by the user account information.

92. The method of claim 91, wherein the multiple gesture actions contain a video file.

93. The method of claim 91, wherein the multiple gesture actions contain at least one image file.

94. The method of claim 91, wherein the wherein the multiple gesture actions contain an audio file.

95. The method of claim 91, wherein the multiple gesture actions contain both at least one image file and an audio file.

96. The method of claim 91, wherein the transaction is a payment transaction between the user and a second entity.

97. The method of claim 91, wherein the transaction is a payment transaction between the user'"'"'s payment device and second payment device also owned by the user.