Wearable augmented reality eyeglass communication device including mobile phone and mobile computing via virtual touch screen gesture control and neuron command

• US 9,153,074 B2
• Filed: 08/22/2013
• Issued: 10/06/2015
• Est. Priority Date: 07/18/2011
• Status: Expired due to Fees

• Current Assignee: Andrew H.B. Zhou, Dylan T.X. Zhou, Tiger T.G. Zhou
• Original Assignee: Andrew H.B. Zhou, Dylan T.X. Zhou, Tiger T.G. Zhou
• Inventors: Zhou, Dylan T X, Zhou, Tiger T G, Zhou, Andrew H B
• Primary Examiner(s): WALSH, DANIEL I
• Application Number:
  US13/973,146
• Publication Number:
  US 20130346168A1
• Time in Patent Office: 775 Days
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0426 tracking fingers with respect to a virtual keyboard projected or printed on the surface virtual keyboards on touch screens G06F3/04886
• CPC Class Codes: G06Q 20/306 using TV related infrastructures external card used for payment with client device H04N21/4185
• CPC Class Codes: G06Q 20/308 using the Internet of Things
• CPC Class Codes: G06Q 20/321 using wearable devices
• CPC Class Codes: G06Q 20/3276 using a pictured code, e.g. barcode or QR-code, being read by the M-device
• CPC Class Codes: G06Q 30/0641 Shopping interfaces
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G08B 21/06 indicating a condition of sleep, e.g. anti-dozing alarms

Abstract

Provided are an augmented reality eyeglass communication device and a method for facilitating shopping using an augmented reality eyeglass communication device. The augmented reality eyeglass communication device may comprise a frame, and a right earpiece and a left earpiece connected to the frame. Furthermore, the eyeglass communication device may comprise a processor configured to receive one or more commands of a user, perform operations associated with the commands of the user, receive product information, and process the product information. The eyeglass communication device may comprise a display connected to the frame and configured to display data received from the processor. In addition to that, the eyeglass communication device may comprise a transceiver electrically connected to the processor and configured to receive and transmit data over a wireless network. The eyeglass communication device may comprise a Subscriber Identification Module card slot, a camera, an earphone, a microphone, and a charging unit.

Claims

1. An augmented reality eyeglass communication device comprising:

an eyeglass frame having a first end and a second end;

a right earpiece and a left earpiece, wherein the right earpiece is connected to the first end of the frame and the left earpiece is connected to the second end of the frame;

a camera disposed on the frame, the right earpiece or the left earpiece, the camera being configured to;

track a hand gesture command of a user,

capture a sequence of images containing a finger of the user and virtual objects of a virtual keypad displayed by the eyeglass communication device and operable to provide input to the eyeglass communication device by the user, finger motions in relation to virtual objects being detected based on the sequence, wherein one or more gestures are recognized based on the finger motions, wherein the one or more gestures define user commands input to the eyeglass communication device,

capture a skeletal representation of a body of the user, a virtual skeleton being computed based on the skeletal representation, and body parts being mapped to segments of the virtual skeleton, wherein the capturing is performed in real time,

a processor disposed in the frame, the right earpiece or the left earpiece and configured to;

receive one or more hand gesture commands of the user, wherein the one or more hand gesture commands comprise displaying product information comprising product description and product pricing of one or more products fetched from a networked database in response to user input of identifiers of the one or more products into the processor and displaying location information associated with the one or more products determined by the eyeglass communication device, including displaying a route on a map of a store to guide the user to the location within the store to obtain the product, and changing the frequency of a WiFi signal of the eyeglass communication device,

perform the one or more hand gesture commands of the user,

process the one or more hand gesture commands tracked by the camera, the hand gesture command being inferred from a collection of vertices and lines in a three dimensional mesh associated with a hand of the user,

derive parameters from the hand gesture command using a template database, the template database storing captured storing deformable two dimensional templates of a human hand, a deformable two dimensional template of the human hand being associated with a set of points on outline of the human hand;

receive product information, and

process the product information;

at least one display connected to the frame and configured to display data received from the processor corresponding to each of the one or more hand gesture commands, the display comprising;

an optical prism element embedded in the display, and

a projector embedded in the display, the projector being configured to project the data received from the processor to the optical prism element and to project the data received from the processor to a surface in environment of the user, the data including a virtual touch screen environment;

a transceiver electrically coupled to the processor and configured to receive and transmit data over a wireless network;

a Subscriber Identification Module (SIM) card slot disposed in the frame, the right earpiece or the left earpiece and configured to receive a SIM card;

at least one earphone disposed on the right earpiece or the left earpiece;

a microphone configured to sense a voice command of the user, wherein the voice command is operable to perform commands of the one or more hand gestures commands; and

a charging unit connected to the frame, the right earpiece or the left earpiece;

at least one electroencephalograph sensor configured to sense brain activity of the user and provide an alert when undesired brain activity is sensed;

a gesture recognition unit including at least three dimensional gesture recognition sensors, a range finder, a depth camera, and a rear projection system, the gesture recognition unit being configured to track the hand gesture command of the user, the hand gesture command being processed by the processor, wherein the hand gesture command is associated with the vertices and lines of the hand of the user, the vertices and lines being in a specific relation;

a band configured to secure the augmented reality eyeglass communication device on a head of the user;

wherein the augmented reality eyeglass communication device is configured to perform phone communication functions, and wherein the eyeglass communication device is operable to calculate a total price for the one or more products, encode the total price into a code that is scannable by a merchant scanning device, and wherein the eyeglass communication device is operable to communicate with the merchant scanning device and perform a payment transaction for the one or more products.

2. The device of claim 1, further comprising a sensor, wherein the sensor includes a motion sensing unit configured to sense head movement of the user, and an eye-tracking unit configured to track eye movement of the user.

3. The device of claim 1, wherein the voice command includes a voice memo, and a voice message.

4. The device of claim 1, wherein the microphone is configured to sense voice data and to transmit the voice data to the processor.

5. The device of claim 1, wherein the charging unit includes one or more solar cells configured to charge the device, a wireless charger accessory, and a vibration charger configured to charge the devices using natural movement vibrations.

6. The device of claim 1, wherein the user interacts with the data projected to the surface in environment, the interaction being performed through the hand gesture command.

7. The device of claim 1, wherein the gesture recognition unit is configured to identify multiple hand gesture commands of the user or gestures of a human, hand gesture commands of the user or gestures of a human including depth data, finger data, and hand data.

8. The device of claim 1, wherein the processing of the hand gesture command includes correlating of the hand gesture command with a template from a template database.

9. The device of claim 1, wherein the rear projector system is configured to project the virtual touch screen environment in front of the user, the hand gesture command being captured combined with the virtual touch screen environment.

10. The device of claim 1, wherein the display is configured as a prescription lens, a non-prescription lens, a safety lens, a lens without diopters, or a bionic contact lens, the bionic contact lens including integrated circuitry for wireless communication.

11. The device of claim 1, wherein the display includes a see-through material to display simultaneously a picture of real world and data requested by the user.

12. The device of claim 1, wherein the camera is configured to capture front-, rear-, top-, left- or right-side area around the device.

13. The device of claim 1, wherein the camera is configured to simultaneously perform video recording and image capturing.

14. The device of claim 1, the device further comprising one or more of the following:

a memory slot, a memory unit inserted into the memory slot, a vibration unit, a Universal Serial Bus (USB) slot, a light indicator, an on/off button, a reset button, an accelerometer determining an activity of the user, and a GPS unit disposed on the frame, the right earpiece or the left earpiece.

15. The device of claim 1, wherein the processor operates on an operational system, wherein the operating system includes iOS, Android, Windows Mobile, Asha, Linux, Nemo Mobile, Blackberry, Symbian, and other operating systems.

16. The device of claim 1, wherein the processor is configured to establish connection with a network to view text, photo or video data, maps, listen to audio data, watch multimedia data, receive and send e-mails, and perform payments.

17. The device of claim 1, wherein the camera is configured to scan a barcode, the scanned barcode being processed by the processor to retrieve payment information and product information encoded in the barcode and enable self-checkout.

18. The device of claim 1, wherein the camera is configured to capture an image of a product, the captured image being processed by the processor to retrieve the product information.

19. The device of claim 1, further comprising a RFID reader to read a RFID tag of a product, the read RFID tag being processed by the processor to retrieve the product information.

20. The device of claim 1, wherein the surface in environment of the user includes a vertical surface in environment of the user, a horizontal surface in environment of the user, an inclined surface in environment of the user, a surface of a physical object in environment of the user, and a part of a body of the user.

21. The device of claim 1, further comprising a Wi-Fi module and a Wi-Fi signal detecting sensor, wherein the Wi-Fi signal detecting sensor is configured to detect change of a Wi-Fi signal caused by the hand gesture command of the user and communicate data associated with the detected change to the processor.

22. The device of claim 21, wherein the processor is further configured to:

process the data associated with the detected change of the Wi-Fi signal; and

perform the detected hand gesture command in accordance with the processed data.

23. The device of claim 1, wherein the device synchronizes with one or more external devices in real time, tracks a geographical location of the one or more external devices in real time, and provides communication capabilities using an embedded emergency button configured to give a medical alert signal, a request for help signal, or another informational signal.

24. The device of claim 1, wherein the device is used as a hands-free mobile computing device.

25. The device of claim 1, wherein the gesture recognition unit is configured to track non-verbal communication of a human, the non-verbal communication including one or more of the following:

a gesture, a sign, a directional indication, and a facial gesture.

26. The device of claim 1, wherein the at least one electroencephalograph (EEG) sensor senses one or more electrical impulses associated with the brain activity, the one or more electrical impulses being translated in one or more commands.

27. The device of claim 26, wherein the one or more electrical impulses are used to optimize brain fitness and performance of the user, measure and monitor cognitive health and well being of the user.

28. The device of claim 1, wherein the camera and EEG sensors are configured to track one or more facial expressions of the user, the one or more facial expressions including a blink, a wink, a surprise expression, a frown, a clench, and a smile.

29. The device of claim 1, wherein the camera is configured to track a blinking of the user, the blinking being associated with video recording and image capturing.

30. The device of claim 1, further comprising an embedded transmitter configured to produce one or more signals, the one or more signals being associated with a remote radio control, a two way radio alert, a medical care alert, a radar, a door opener, an operation transporting vehicle, a navigational beacon, and a toy.

31. The device of claim 1, further comprising a communication circuit, the communication circuit including one or more of the following:

a Bluetooth module, a WiFi module, a communication port including a universal serial bus (USB) port, a parallel port, an infrared transceiver port, a radiofrequency transceiver port, wherein the device communicates with external devices using the communication circuit.

32. The device of claim 1, further comprising one or more control elements to control operation or functions of the device.

33. The device of claim 1, further comprising one or more biometric sensors to sense biometric parameters of the user, the biometric parameters being stored to memory and processed by the processor to receive historical biometric data.

34. The device of claim 33, wherein the one or more biometric sensors include sensors for measuring one or more of the following:

a blood pressure, a pulse, a heart rate, a glucose level, a body temperature, an environment temperature, and arterial properties, the measuring shown on the display.

35. The device of claim 34, wherein one or more automatic alerts are provided based on the measuring, the one or more automatic alerts including visual alerts, audio alerts, and voice alerts.

36. The device of claim 1, further comprising one or more accelerometers to track activity of the user, the activity of the user including calories burned, sleep quality, breaths per minute, snoring breaks, steps walked, and distance walked.

37. The device of claim 36, wherein the device controls snoring by sensing a position of the user using the one or more accelerometers.

38. The device of claim 1, wherein the device is compatible with one or more of the following network standards:

GSM, CDMA, LTE, IMS, Universal Mobile Telecommunication System (UMTS), RFID, 4G, 5G, 6G and upper.

39. The device of claim 1, wherein the processor is further configured to download applications, receive and send text, video, multimedia data.

40. The device of claim 1, wherein device is used as a standalone system operating via a WiFi module or a Subscriber Identity Module (SIM) card.

41. The device of claim 1, wherein the three dimensional gesture recognition sensors capture three dimensional data in real time, the three dimensional data being millimeter exact.

42. The device of claim 1, wherein the virtual touch screen environment is see-through.

43. The device of claim 1, wherein virtual objects in the virtual touch screen environment are moveable and deformable, wherein moves and deformations may be performed based on motions of the user hand.

44. The device of claim 1, wherein the data projected by the projector to the optical prism element is perceived by a human eye as located at a distance of 3 to 8 meters.

45. The device of claim 1, wherein the device provides gesture tracking, surface tracking, and code example tracking.

46. The device of claim 1, wherein the user interacts with virtual objects visualized in the virtual touch screen environment.

47. The device of claim 1, further comprising two cameras, one for each eye of the user, each of the two cameras having a 23 degree field of view.

48. The device of claim 1, wherein the camera is configured to capture a sequence of images, the images containing a hand of the user, the images being processed by the processor to recognize the hand gesture command.

49. The device of claim 1, further comprising a receiver configured to sense a change in frequency of a WiFi signal, wherein the change is caused by a move of a user hand, the change being processed by the processor and associated with a command.

50. The device of claim 49, wherein the command includes controlling temperature settings, adjusting a volume on a stereo, flipping a channel on a television set, or shutting off a lights, and causing a fireplace to blaze to life.

51. The device of claim 49, wherein the change in frequency is sensed in a line of sight of the user, outside the line of sight of the user, and through a wall.

52. The device of claim 49, wherein the sensing is activated by a combination of gestures.

53. The device of claim 49, wherein the sensing is performed by the microphone.

54. The device of claim 1, further comprising a camera lens configured to track eye movements, the eye movements being interpreted as a command, the command including a scrolling command, a swiping command.

55. The device of claim 1, further comprising a gesture sensor configured to measure electrical activity associated with a muscle movement, wherein the muscle movement is interpreted as a command, the command being associated with wireless control of video games, phones, drones, vehicles, and other digital products.

56. The device of claim 1, wherein access to the device is controlled by one or more of the following:

a password, a Personal Identification Number (PIN) code, and biometric authorization, the biometric authorization including fingerprint scanning, palm scanning, face scanning, and retina scanning, wherein the scanning is performed using one or more biometric sensors.

57. The device of claim 56, further comprising a fingerprint reader configured to scan a fingerprint, the scanned fingerprint being matched to one or more approved fingerprints, wherein the access to the device is granted based on the matching.

58. The device of claim 1, further comprising:

a GPS module configured to track geographical location of the device;

an alert unit configured to alert the user about one or more events by one or more of the following;

vibration and sound;

one or more subscriber identification module (SIM) cards;

one or more additional memory units;

a physical interface configured to receive memory devices external to the device, wherein the physical interface includes a microSecureDigital (microSD) slot;

a two-way radio transceiver for communication purposes; and

an emergency button configured to send an alarm signal.

59. The device of claim 58, wherein the vibration and sound of the alert unit may be used by a guide tool and an exercise learning service.

60. The device of claim 1, wherein the device is configured to analyze one or more music records stored in a memory unit, communicate, over a network, with one or more music providers, and display data about music records suggested by the music providers for sale, wherein the data displayed include music records being similar to the music records stored in the memory unit of the device.

61. The device of claim 1, wherein the processor is configured to communicate with a gambling cloud service or a gaming cloud service, exchange gambling or gaming data with the gambling cloud service or the gaming cloud service, and, based on a user request, transfer payments related to gambling or gaming using payment data of the user associated with an account of the user in the cloud service, using payment data of the user stored in a memory unit or using a swipe card reader to read payment card data.

62. The device of claim 1, wherein the device is configured to perform functions of a phone and a computer.

63. A method for facilitating shopping using an augmented reality eyeglass communication device, the augmented reality eyeglass communication device comprising the augmented reality eyeglass communication device of claim 1, and the method comprising:

receiving, by a processor of the augmented reality eyeglass communication device, product information associated with products comprised in a list of products of a user;

receiving, by the processor, location information associated with location of the user;

searching, based on the product information, by the processor, a database associated with a store for availability, location and pricing information associated with the products;

receiving, by the processor, the availability, location and pricing information associated with the product; and

displaying, by a display of the augmented reality eyeglass communication device, the availability, location and pricing information associated with the product.

64. The method of claim 63, further comprising:

plotting a route for the user on a map of the store based on the availability, location and pricing information associated with the product and the location information associated with the location of the user; and

displaying, by the display, the route for the user.

65. The method of claim 63, further comprising

receiving, by the processor, a command of the user to provide description of a product present in the store;

receiving, by the processor, information associated with the product present in the store, wherein receiving information associated with the product present in the store includes taking a picture of the product, scanning a barcode of the product and reading a RFID tag of the product;

processing, by the processor, the received information associated with the product present in the store;

searching, based on the received information associated with the product, by the processor, the description of the product in a database available in a network;

receiving, by the processor, the description of the product; and

displaying, by the display, the description of the product present in the store.

66. The method of claim 65, further comprising:

tracking, by a camera, a hand gesture command of the user;

processing, by the processor, the hand gesture command of the user; and

projecting, by a projector, the description of the product to a surface in environment of the user according to the hand gesture command, wherein the surface in environment of the user includes a vertical surface in environment of the user, a horizontal surface in environment of the user, an inclined surface in environment of the user, a surface of a physical object in environment of the user, and a part of a body of the user.

67. The method of claim 63, further comprising:

receiving, by the processor, information about the products put by the user into a shopping cart to enable self-checkout; and

removing the products put by the user into the shopping cart from the list of products of the user.

68. The method of claim 67, further comprising:

generating a payment request based on the information about the products put by the user into the shopping cart; and

sending the payment request to a financial organization to perform a payment.

69. The method of claim 67, further comprising:

notifying the user if a product comprised in the list of products of the user is not available in the store; and

searching, by the processor, availability information associated with the not available product in a database of a store located proximate to the location of the used based on location information associated with the location of the user.

70. The method of claim 69, further comprising

searching, the database associated with the store for information about a product having the same characteristics as the not available product;

receiving, by the processor, the information about the product having the same characteristics as the not available product; and

displaying, by the display, the information about the product having the same characteristics as the not available product.

71. The method of claim 63, further comprising:

receiving payment data associated with another user, the payment data including payment account information of another user; and

transferring an amount, based on the payment data associated with the another user, from the payment account of another user to a payment account of the user, information on the payment account of the user being stored in a memory unit of the device or a server.

72. The method of claim 63, further comprising scanning a barcode, the barcode including a one-dimensional barcode, a two-dimensional barcode, a three-dimensional barcode, a quick response code, a snap tag code, and other machine readable codes.

73. The method of claim 72, wherein the barcode encodes one or more of the following:

payment data, personal data, credit card data, debit card data, gift card data, prepaid card data, bank checking account data, and digital cash data.

74. The method of claim 72, wherein the barcode includes electronic key data, the barcode being scannable by a web-camera of an access control system and processed by the access control system, wherein access to an item related to the access control system is granted based on the processing.

75. The method of claim 72, wherein the barcode includes a link to a web-resource, a payment request, advertising information, and other information.

76. The method of claim 63, further comprising hands free check-in scanning and hands free check-out scanning.

77. The method of claim 63, further comprising, using a voice or neuron command:

performing a hands free video call;

sending a message;

taking a picture;

recording a video; and

getting directions to a location.

78. The method of claim 63, wherein the augmented reality eyeglass communication device includes one or more of the following:

a Software Development Kit and an Application Programming Interface.

79. The method of claim 63, wherein the augmented reality eyeglass communication device is configured to make and receive calls over a radio link while moving around a wide geographic area via a cellular network, access a public phone network, send and receive text, photo, and video messages, access internet, capture videos and photos, and play games.

80. A computer-implemented method for purchasing a product item in a retail environment using an augmented reality eyeglass communication device, the augmented reality eyeglass communication device comprising the augmented reality eyeglass communication device of claim 1, the method comprising:

using the augmented reality eyeglass communication device, sending a request to read one or more product codes;

reading the one or more product codes corresponding to one or more products;

based on the reading, receiving product information;

transmitting data on the reading to a payment processing system;

receiving a request, from a customer, to calculate a total price of the reading;

based on the request, calculating the total price of the reading,

authorizing payment and transmitting authorization to the payment processing system; and

selectively receiving a compensation based on predetermined criteria.

81. The computer-implemented method of claim 80, further comprising receiving a request, from a user, to pay for the one or more products with a restored payment card.

82. The computer-implemented method of claim 80, wherein the total price is encoded in a code, the total price being scannable by a merchant scanning device to transfer payment for the one or more products.

83. The computer-implemented method of claim 80, further comprising:

receiving a request from a user to determine an aisle location of the one or more products; and

determining the aisle location of the one or more products.

84. The computer-implemented method of claim 80, wherein a user creates one or more product lists.

85. The computer-implemented method of claim 80, further comprising connecting to a wireless network of a merchant.

86. The computer-implemented method of claim 80, further comprising receiving digital coupons and offers to make a purchase.

87. The computer-implemented method of claim 80, further comprising receiving promotional offers and advertising.

88. The computer-implemented method of claim 87, wherein the promotional offers and advertising are received from one or more of the following:

a merchant, a mobile payment service provider, a third party.

89. The computer-implemented method of claim 80, wherein the compensation includes one or more of the following:

cashback, a discount, and a gift card.

90. The computer-implemented method of claim 80, wherein a user receives a digital receipt by email, the digital receipt containing detailed information on cashback and discount.

91. The computer-implemented method of claim 80, further comprising receiving a request, from a user, to check a total amount and price of the reading.

92. The computer-implemented method of claim 91, further comprising selectively removing one or more items of the reading.

93. The computer-implemented method of claim 80, further comprising placing a remote order for home delivery of one or more unavailable products.

94. A method for accessing game and multimedia data and performing communications using an augmented reality eyeglass communication device, the augmented reality eyeglass communication device comprising the augmented reality eyeglass communication device of claim 1, and the method comprising:

receiving a user request to display the game and multimedia data or perform communication;

communicating, over a network, with a game and multimedia server to transfer game and multimedia data or a communication server to transfer communication data; and

displaying the game and multimedia data or the communication data on a display of the augmented reality eyeglass communication device.

95. The computer-implemented method of claim 94, further comprising:

receiving, by a processor of the augmented reality eyeglass communication device, a user command;

transferring the user command to the game and multimedia server;

processing the user command; and

transferring game and multimedia data related to the processing to the augmented reality eyeglass communication device.

96. The computer-implemented method of claim 94, further comprising:

receiving, by a processor of the augmented reality eyeglass communication device, incoming communication data;

notifying the user about the incoming communication data.

97. The computer-implemented method of claim 96, further comprising:

receiving, by a processor of the augmented reality eyeglass communication device, a user command in response to the incoming communication data;

displaying the incoming communication data.

98. The computer-implemented method of claim 97, further comprising:

generating an audible sound, wherein the sound corresponds to the incoming communication data.

99. The computer-implemented method of claim 97, further comprising:

transferring the game and multimedia data or the incoming communication data to a television set, a set-top box, a computer, a laptop, a smartphone, or a wearable personal digital device.