Systems and methods for computer assisted operation

• US 10,149,958 B1
• Filed: 05/06/2016
• Issued: 12/11/2018
• Est. Priority Date: 07/17/2015
• Status: Active Grant

• Current Assignee: Bao Tran, Ha Tran
• Original Assignee: Bao Tran, Ha Tran
• Inventors: Tran, Bao, Tran, Ha
• Primary Examiner(s): Beasley, Deirdre L
• Application Number:
  US15/148,739
• Time in Patent Office: 949 Days
• Field of Search: 348 53
• CPC Class Codes: A61B 2034/101 Computer-aided simulation of surgical operations
• CPC Class Codes: A61B 2034/107 Visualisation of planned trajectories or target regions
• CPC Class Codes: A61B 34/10 Computer-aided planning, simulation or modelling of surgical operations
• CPC Class Codes: A61M 2021/005 images, e.g. video
• CPC Class Codes: A61M 21/00 Other devices or methods to cause a change in the state of consciousness; Devices for producing or ending sleep by mechanical, optical, or acoustical means, e.g. for hypnosis
• CPC Class Codes: A61M 2205/332 Force measuring means
• CPC Class Codes: A61M 2205/3375 Acoustical, e.g. ultrasonic, measuring means
• CPC Class Codes: A61M 2205/507 Head Mounted Displays [HMD]
• CPC Class Codes: A61M 2230/04 Heartbeat characteristics, e.g. ECG, blood pressure modulation
• CPC Class Codes: A61M 2230/202 partial carbon oxide pressure, e.g. partial dioxide pressure (P-CO2)
• CPC Class Codes: A61M 2230/205 partial oxygen pressure (P-O2)
• CPC Class Codes: A61M 2230/30 Blood pressure A61M2230/04 takes precedence
• CPC Class Codes: A61M 2230/50 Temperature
• CPC Class Codes: A61M 2230/60 Muscle strain, i.e. measured on the user
• CPC Class Codes: A61M 2230/63 Motion, e.g. physical activity
• CPC Class Codes: A61M 2230/65 Impedance, e.g. conductivity, capacity
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/015 Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 2201/034 : of medical instruments
• CPC Class Codes: G06V 40/19 : Sensors therefor
• CPC Class Codes: G16H 20/30 : relating to physical therapies or activities, e.g. physiotherapy, acupressure or exercising
• CPC Class Codes: G16H 20/60 : relating to nutrition control, e.g. diets
• CPC Class Codes: G16H 30/40 : for processing medical images, e.g. editing
• CPC Class Codes: G16H 40/63 : for local operation
• CPC Class Codes: G16H 50/20 : for computer-aided diagnosis, e.g. based on medical expert systems
• CPC Class Codes: G16H 50/50 : for simulation or modelling of medical disorders

Abstract

A reality system includes a display aimed at a retina, the display providing 3D images with different depth view points; a variable reflectance mirror to selectably turn on or off view of an outside environment in front of the person'"'"'s eye; a processor coupled to the camera and to the variable reflectance mirror to selectably switch between augmented reality and virtual reality; and a wireless transceiver coupled to the transceiver to communicate with a remote processor.

Claims

1. A system, comprising:

a display in a wearable housing with camera and sensors;

a mirror coupled to the display to selectably turn on or off view of an outside environment in front of the person'"'"'s eye to provide augmented reality, virtual reality, or a combination thereof;

a processor in the wearable housing coupled to the camera and to the variable reflectance mirror to selectably switch between augmented reality and virtual reality, the processor including code to perform motion tracking and understanding an environment with points or planes using accelerometer sensor and estimating light or color in the environment using one video camera without a depth sensor in a mobile phone;

code to capture images from a plurality of angles of the environment;

code to acquire sensor data from sensors and optimize features extracted from each image and sensor data, where a feature conveys data unique to the image at a specific pixel location;

a learning machine in the wearable housing including graphical processing unit (GPU wherein the GPU discriminates between learned user gestures in real-time and wherein the learning machine determines content to be presented based on previous activities or selections while generating content including virtual images to be integrated with surrounding images; and

a wireless transceiver coupled to the transceiver to communicate with a remote processor.

2. The system of claim 1, comprising a sensor or a gyroscope coupled to the display.

3. The system of claim 1, comprising a sound transducer coupled to the wireless transceiver to communicate audio.

4. The system of claim 1, comprising one or more bins to receive preselected objects.

5. The system of claim 1, comprising a plurality of cameras to detect eye movement, mouth movement, and hand gesture.

6. The system of claim 5, wherein the cameras are mounted on a head worn device.

7. The system of claim 5, wherein the cameras are positioned in front of the person.

8. The system of claim 1, comprising a variable reflectance mirror, wherein a selective reflectance of light is based at least in part on a desired visualization mode, wherein a desired visualization mode is one of an augmented reality mode, a virtual reality mode, a blended reality mode, and a combination of augmented and virtual reality modes.

9. The system of claim 1, comprising code to allow a reflectance of light from the outside environment when the head-mounted user display device is turned off, such that the person only views the entirely physical objects.

10. The system of claim 1, comprising code for displaying either entirely virtual objects, entirely physical objects or a combination of virtual objects and physical objects.

11. The system of claim 1, wherein the processor performs emotion sensing when blood vessels dilate.

12. The system of claim 1, comprising one or more side view variable reflectance mirrors to selectably turn on or off view of an outside environment behind the person'"'"'s eye.

13. The system of claim 12, comprising a graphical code for real-time spectral processing.

14. The system of claim 1, comprising virtual pain killer code to distract the person and generate an experience of extreme, disturbing, or unexpected fear, stress, or pain, and that involves or threatens serious injury, perceived serious injury, or death to the person or someone else.

15. The system of claim 1, comprising a conveyor carrying a plurality of objects for spectrum analysis one object at a time.

16. The system of claim 1, comprising multimedia code to stimulate release of adrenaline and to noradrenaline from the medulla of the adrenal glands based on a spectral analysis of the object.

17. The system of claim 1, comprising multimedia code to stimulate release of catecholamines at neuroreceptor sites based on a spectral analysis of an object.

18. The system of claim 1, comprising code to stimulate neuron firings in a locus ceruleus based on a spectral analysis of the object.

19. The system of claim 1, comprising multimedia code to stimulate release of chemicals from one or more organs or glands based on a spectral analysis of an object.

20. The system of claim 1, comprising code executed by the processor to receive spectral sensor data to determine attributes of food represented by a multi-dimensional food attribute vector including food color;

food texture;

food shape;

food size or scale;

geographic location of selection, purchase, or consumption;

timing of day, week, or special event;

common food combinations or pairings;

image brightness, resolution, or lighting direction;

infrared light reflection;

spectroscopic analysis; and

person-specific historical eating patterns.