SYSTEM AND METHODS FOR ON-BODY GESTURAL INTERFACES AND PROJECTION DISPLAYS

• US 20170123487A1
• Filed: 10/28/2016
• Published: 05/04/2017
• Est. Priority Date: 10/30/2015
• Status: Active Grant

• Current Assignee: Ostendo Technologies Inc.
• Original Assignee: Ostendo Technologies Inc.
• Inventors: El-Ghoroury, Hussein S., Hazra, Siddharth S.
• Granted Patent: US 11,106,273 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/015 Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour

Abstract

A wearable system with a gestural interface for wearing on, for instance, the wrist of a user. The system comprises an ultrasonic transceiver array structure and may comprise a pico projector display element for displaying an image on a surface. User anatomical feature inputs are received in the form of ultrasound signals representative of a spatio-temporal cross-section of the wrist of the user by articulating wrist, finger and hand postures, which articulations are translated into gestures. Inputs from inertial and other sensors are used by the system as part of the anatomical feature posture identification method and device. Gestures are recognized using a mathematically-modeled, simulation-based set of biological metrics of tissue objects, which gestures are converted to executable computer instructions. Embodiments of the system disclosed herein may also be used to monitor biometric and health data over computer networks or using onboard systems.

Claims

1. A method for generating a gesture command using a wearable gesture recognition device in a human-computer interface system comprising the steps of:

providing a mathematical model based on a rigid body motion of the wearable gesture recognition device and on a set of biological metrics of a plurality of predetermined internal or external objects of an anatomical feature;

obtaining a set of ultrasound signal data from the anatomical feature of a user of the plurality of objects by mapping a spatio-temporal cross-section of the anatomical feature of the user;

identifying a predetermined property of the objects by correlating the ultrasound signal data with the mathematical model;

estimating a posture of the user'"'"'s anatomical feature based on the identified property of the objects; and

generating a gesture command based on the estimated posture.

2. The method of claim 1 wherein the mapped objects comprise one or more of muscle, tendon, ligament, vasculature, bone, dermis, epidermis, epithelial, connective muscular and nervous tissue of the user.

3. The method of claim 1 wherein the property comprises at least one of a radius, position, relative position to another object, angle, relative angle to another object, pulsatility, deformation, echogenicity, velocity, displacement, extension, compression, appearance, disappearance, and differential changes of each property, of the object.

4. The method of claim 1 wherein the posture estimate is further based on a received sensor output from a sensor element configured to measure a predetermined event, physical change or state of the user.

5. The method of claim 4 wherein the sensor element is comprised of an accelerometer;

a gyroscope;

a pressure sensor;

a GPS sensor;

a microphone;

an electromagnetic imaging sensor;

an ambient light sensor;

a color sensor;

an angular sensor;

an electrophysiology sensor;

an electromyographic sensor;

a bioacoustic sensor;

a photoplethysmographic sensor, an RFID antenna;

an ultrasonic ranging sensor;

a high frequency transducer array;

an electromagnetic field sensor or a proximity sensor.

6. The method of claim 4 wherein estimated kinematics of the estimated posture is used to generate a soft tissue deformation map.

7. The method of claim 6 wherein the soft tissue deformation map is used to compensate and calibrate a projected display.

8. The method of claim 4 further comprising the step of generating a predetermined executable computer instruction based on the gesture command.

9. The method of claim 8 wherein the computer instruction set modifies a display information projected on a non-linear surface based on an input received from the color sensor or imaging sensor configured to image the non-linear surface.

10. The method of claim 9 wherein the non-linear surface is a body surface of the user.

11. The method of claim 8 wherein the computer instruction set deletes a user-selected digital content in a computer memory.

12. The method of claim 8 wherein the computer instruction set advances or returns to a user-selected digital content in a computer memory.

13. The method of claim 8 wherein the computer instruction set triggers an image capture in a digital camera system.

14. The method of claim 8 wherein the computer instruction set triggers a panoramic image capture in a digital camera system.

15. The method of claim 8 wherein the computer instruction set modifies a three dimensional light field display.

16. The method of claim 8 wherein the computer instruction set controls a two dimensional cursor on a computer display.

17. The method of claim 8 wherein the computer instruction set controls a three dimensional cursor in a three dimensional light field display.

18. The method of claim 8 wherein the computer instruction set interacts with a computer game.

19. The method of claim 8 wherein the computer instruction set interacts with a computer game that is displayed on two hands of a user.

20. The method of claim 8 wherein the computer instruction set monitors or controls a piece of industrial equipment.

21. The method of claim 8 wherein the computer instruction is used to remotely pilot a land, air or marine vehicle.

22. The method of claim 8 wherein the computer instruction set interacts with a home entertainment system.

23. The method of claim 8 wherein the computer instruction set controls a power tool or robotic system.

24. The method of claim 8 wherein the computer instruction set controls a robotic surgical system.

25. The method of claim 8 wherein the computer instruction set maps and renders a spatial dimension or geometry of an object.

26. The method of claim 8 wherein the computer instruction set measures a force, torque and moment applied by a limb of a user.

27. The method of claim 8 wherein the computer instruction set interacts with three dimensional content for digital content generation.

28. The method of claim 8 wherein the computer instruction set is used for tissue elastography-aided monitoring.

29. The method of claim 8 wherein the computer instruction set is used to monitor a joint for impactive stress, posture or motion for physiotherapy.

30. The method of claim 8 wherein the computer instruction set is used in a user interaction with printed content.

31. The method of claim 8 wherein the computer instruction set is used in a user interaction with displayed augmented reality content.

32. The method of claim 8 wherein the computer instruction is used in a user interaction with displayed virtual reality content.

33. The method of claim 8 wherein the computer instruction set is used in an inventory discrimination application.

34. The method of claim 8 wherein the computer instruction set is used to create a virtual immersive display.

35. The method of claim 8 wherein the computer instruction set is used in an immersive multi-player game.

36. The method of claim 8 wherein the computer instruction set modifies a display information on a projection surface.

37. The method of claim 36 wherein the display information comprises a computer menu screen comprising a graphical user interface icon.

38. The method of claim 1 wherein the mathematical model comprises a lumped, finite or mixed elements model.

39. The method of claim 1 wherein the mathematical model comprises a dynamic and deformable mesh.

40. The method of claim 1 wherein the mathematical model comprises at least one numerical optimization routine that operates subjected to kinematic constraints.

41. The method of claim 1 wherein the mathematical model comprises at least one unsupervised statistical learning model.

42. The method of claim 1 wherein the mathematical model comprises at least one supervised statistical learning model.

43. The method of claim 1 wherein the mathematical model comprises at least one probabilistic graphical model.

44. A method for remotely monitoring a property of an anatomical feature comprising the steps of:

providing a mathematical model based on a rigid body motion of a wearable gesture recognition device and on a set of biological metrics of a plurality of predetermined internal or external objects of an anatomical feature;

obtaining a set of ultrasound signal data from the anatomical feature of a user of the plurality of objects by mapping a spatio-temporal cross-section of the anatomical feature of the user;

identifying a predetermined property of the objects by correlating the ultrasound signal data with the mathematical model; and

transmitting the identified property of the object in the form of computer-readable information over a network to a remote location.

45. A wearable gesture recognition system for a human-computer interface comprising:

a band element configured to be secured on an anatomical feature of a user;

a plurality of spaced-apart ultrasonic transceivers disposed at predetermined locations on the band element;

the transceivers configured to transmit an ultrasound signal and acquire a reflected signal representative of a spatio-temporal map of a cross-section of the anatomical feature of a user when the band is worn by the user; and

processing circuitry configured to receive, process and output the acquired spatio-temporal map to a signal processor.

46. The system of claim 45 wherein the transceiver is a two dimensional array of individual ultrasound transceiver subelements each having a top and a bottom addressable matrix electrode;

an acoustic coupling layer;

an acoustic lensing layer;

an acoustic matching layer; and

a transmission active/passive impedance layer.

47. The system of claim 45 wherein the band element is configured to be secured around the perimeter of the limb of a user.

48. The system of claim 47, further comprising:

a wearable near eye display system in communication with the band element,

wherein the processing circuitry is further configured to receive information from the wearable near eye display system for processing, to modify the spatiotemporal map of the cross-section of the anatomical feature according to the information received from the wearable near eye display system, and to output the modified spatiotemporal map to the signal processor.

49. The system of claim 47, further comprising:

a mobile communications device in communication with the band element,

wherein the processing circuitry is further configured to receive information from the mobile communications device for processing, to modify the spatiotemporal map of the cross-section of the anatomical feature according to the information received from the mobile communications device, and to output the modified spatiotemporal map to the signal processor.

50. The system of claim 47 wherein the band element comprises a variable tension element configured to provide a predetermined tension on the band element when worn by a user.

51. The system of claim 45 further comprising a sensor element including at least one of an accelerometer and a gyroscope configured to sense a position of the anatomical feature in a three-dimensional volume;

and a position-dependent gesture interface in a three dimensional light field display system comprising an estimated posture of the user'"'"'s anatomical feature and the sensed position.

52. The system of claim 51 wherein the position-dependent gesture interface is configured for the user to interact with or manipulate a digital content that is viewed by the user in the light field display system.

53. The system of claim 52 wherein the display system is a wearable near eye display system comprising an augmented reality display system or a virtual reality display system.

54. The system of claim 52 wherein the digital content is displayed on a direct view light field display system.

55. The system of claim 51 further comprising a wireless interface for wirelessly transferring the sensed position and estimated posture to an external computer or display system.

56. The system of claim 55 wherein the light field display system is a wearable near eye display system comprising an augmented reality display system or virtual reality display system.

57. The system of claim 55 wherein the display system is a direct view light field display system.

58. The system of claim 45 further comprising a pico projector worn on the anatomical feature configured to project a digital content, objects or icons for display on a body surface of the user or an external surface, wherein the user interacts with, manipulates or selects the digital content, objects or icons via a gesture command.

59. The system of claim 58 further comprising a gyroscope or accelerometer configured to select a portion of the digital content projected on the body of the user or external surface using a hand position of the user.