SECURED MOBILE MAINTENANCE AND OPERATOR SYSTEM INCLUDING WEARABLE AUGMENTED REALITY INTERFACE, VOICE COMMAND INTERFACE, AND VISUAL RECOGNITION SYSTEMS AND RELATED METHODS

• US 20160124501A1
• Filed: 09/29/2015
• Published: 05/05/2016
• Est. Priority Date: 10/31/2014
• Status: Active Grant

• Current Assignee: the united states of america as represented by the secretary of the navy
• Original Assignee: the united states of america as represented by the secretary of the navy
• Inventors: Lam, Jack, Navarro, Philippe
• Granted Patent: US 9,746,913 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 21/00 Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
• CPC Class Codes: G06F 21/32 using biometric data, e.g. fingerprints, iris scans or voiceprints
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback

Abstract

Secured remote maintenance, configuration management, and systems engineering apparatuses and methods including wearable augmented reality (AR) interface systems are provided. Embodiments can support secure and remote configuration setting changes (CSC) to a system or subsystem of interest (SSoI) using a head mounted device (HMD), a camera, a visual interface section, and a processing section including a plurality of processing instructions operable to operate a command input (CI) interface using one or more user command input CIs (e.g. voice or motion/gesture input recognition) using a secure user authentication systems. HMD machine vision and pattern recognition systems visually identify a SSoI, displaying a 3D model(s) of the SSoI on the HMD'"'"'s visual interface using the AR interface system, obtaining a plurality of SSoI data, and displaying one or more of the SSoI data in relation to the 3D model on the HMD visual interface to support various tasks including CI directed CSCs.

Claims

1. A secured mobile maintenance and operator system comprising:

a head mounting device (HMD) comprising a plurality of components wherein said components comprise;

a microphone;

an image capture system oriented towards a user field of view (FOV);

a HMD visual interface comprising a transparent section positioned over said user FOV and a visual interface system configured to display a plurality of visual elements including a graphical user interface onto said transparent section;

a processing section including a plurality of machine readable processing instructions operable to operate said plurality of components and a command input interface using one or more user command inputs (CI) comprising voice inputs or motion/gesture input recognition;

a first transceiver and input/output interface configured to communicate with a second transceiver system;

a secure user authentication system (SUAS) configured to receive user access requests as one of said voice or gesture inputs;

a machine vision and pattern recognition systems configured to visually identify a system or subsystem of interest (SSoI) in said FOV; and

an augmented reality (AR) control section operable to generate a configuration interface display configured for displaying at least one 3D model of the SSoI on the HMD visual interface, orienting the 3D model to align and scale over the SSoI in said FOV on said transparent section based on orientation, position, distance, and perspective of said transparent section in relation to said SSoI, obtaining a plurality of SSoI data through said transceiver and input/output section from a remote server comprising SSoI configuration data, displaying one or more of the SSoI data in relation to the 3D model on the HMD visual interface, orienting the SSoI data to align with over top of or next to components of the SSoI displayed on the transparent section of the HMD visual interface;

wherein one or more of said processing instructions comprise machine readable instructions to execute a change of a configuration setting of a selected SSoI based on one or more said voice inputs or motion/gesture inputs associated with selection and changing one or more configuration settings of a selected said SSoI displayed in said HMD visual interface;

wherein one or more of said processing instructions comprise machine readable instructions to execute a change of the HMD visual interface to display one or more said SSoI data based on user'"'"'s selection.

2. A system as in claim 1, further comprising said second transceiver, a network system coupled to said second transceiver, and said remote server communicatively coupled with said network system.

3. A secured mobile maintenance and operator system comprising:

a head mounted device (HMD) including a plurality of components comprising;

a visual interface system comprising a visual overlay display system comprising a transparent element configured to generate visual graphical user interfaces (GUI) positioned within a field of view (FOV) of a wearer of said HMD on said transparent element;

at least one video camera configured to record visual inputs oriented to capture images in at least said FOV;

a microphone configured to receive aural inputs in proximity to the HMD comprising user voice command inputs (CI);

a position and orientation section configured to determine position and orientation of the HMD with respect to a reference framework comprising a layout or map of an internal area of a structure and predetermined structures within in said internal area;

a machine vision and pattern recognition system;

an augmented reality (AR) section configured to generate one or more three dimensional (3D) models of a system or subsystem of interest (SSoI) on said visual interface;

a CI control section configured to control one or more CI interfaces and said microphone;

a wireless communication system configured to communicate with said one or more remote data servers and with said one or more SSoI;

a user authentication system (UAS) interface configured to authenticate a user'"'"'s access credentials and allow access with said one or more remote systems comprising maintenance, configuration management, or systems engineering data stores using the wireless communication system and generate user authentication interfaces and input interfaces for user input of authentication data and enabling access to said one or more remote data servers; and

a control section comprising a memory and machine processing control element configured to operate said HMD including said plurality of components and control operation of said machine vision and recognition system, perform data storage and data recall functions with said plurality of components including said memory and said one or more remote data servers, said AR section configured to selectively superimpose at least one said GUI comprising a plurality of computer-generated images on said transparent element comprising a visual overlay display on or within said FOV thus providing a composite view, and said CI control section, said wireless communication system, and said UAS interface;

wherein said control section is configured to execute a local user login process to request the user to input a first authentication data to be granted access to the HMD, wherein the control section is configured to operate said one or more video camera with said machine vision and pattern recognition system that is configured to selectively pattern match identify said SSoI based on said CI or an automatic pattern match identification mode for SSoI'"'"'s matched with remote pattern matching data stored in said one or more remote data servers or within a local pattern matching data stored in said memory, displaying one or more elements of said 3D model of the selected SSoI on the one or more visual interface of the HMD over the SSoI using said AR section, communicating with SSoI or a SSoI control or monitoring system (SSoICoMs) to obtain a plurality of SSoI data with the SUAS, and displaying one or more of the SSoI data in relation to the 3D model on the HMD visual interface;

wherein the control section is configured to operate the HMD to obtain and display a comparison of said plurality of SSoI data from one or more said SSoIs or said SSoICoMs for SSoIs in said FOV including SSoI operational status data including current operational state and actual SSoI configurable parameters with one or a plurality of predetermined SSoI control settings used to control said SSoI including default configurable parameter settings associated with one of a plurality of SSoI operational modes including startup, operations including power levels, and maintenance operational states;

wherein the control section is configured to operate the HMD and said UAS to obtain and display said comparison of said plurality of SSoI data from one or more said SSoIs or said SSoICoMs for SSoIs in said FOV upon pattern recognition of one or more said SSoIs within said FOV using said machine vision and pattern recognition system or upon receiving a CI request for a change of at least one configurable parameter of the SSoI;

wherein said HMD is configured to execute an authentication process for the CI request with user access data using one or more authentication modes comprising said user access credentials comprising a user name and password, a stored PKI certificate data with a user code or pin.

4. The system as in claim 3, wherein said one or more CI interfaces comprises one or more interfaces comprising voice, keyboard, motion, or gesture recognition CI interface control sections.

5. The system as in claim 3, wherein said machine vision and pattern recognition system that is configured to selectively pattern match identify said SSoI identifies said SSoI by identifying a bar code attached to the SSoI.

6. The system as in claim 3, further comprising said one or more remote data servers comprising a local or remote data storage and at least a portion of said UAS.

7. The system as in claim 3, wherein said HMD further comprises a retina image capture camera positioned to capture an image of said user'"'"'s retina, wherein said HMD is further configured to execute said authentication process further comprising a retina biometric user authentication mode that includes capturing a retina image from said user and controlling said UAS to compare said retina image to a stored retina image associated with said user.

8. The system as in claim 3, wherein said position and orientation section comprises an inertial navigation unit comprising a three axis gyro and three-axis accelerometer configured to determine orientation of said HMD with respect to said SSoI and a reference plane defined by a structure said SSoI is mounted within.

9. The system as in claim 3, wherein said wireless communication section and said control section communicate with a SSoI wireless communication section to configure said at least one said at least one configurable parameter of the SSoI based on said CI request.

10. A secured mobile maintenance and operator system comprising:

a first section including a first head-mounted device comprising a camera, a microphone, a transceiver, a power unit, a control section, a transparent element positioned in user'"'"'s field of view (FOV), an image projector configured to project a plurality of images on said transparent element, a pattern recognition system configured to receive images from said camera and match elements of said image with one or more of a plurality of locally or remotely stored patterns including patterns associated with a system or subsystem of interest (SSoI), a first user interface generation section configured for generating a plurality of graphical user interfaces that are projected onto said transparent (GUI), a signal and data input/output interface, a transceiver, and a power unit, and a storage section operable to store a plurality of data and a plurality of non-transitory machine readable instructions configured to control one or more elements of said first section including said control section;

wherein said plurality of data comprises a first, second, third, fourth, fifth, sixth, seventh, eighth, and ninth data, said first data comprising configuration data including a list of said SSoIs, said second data comprising said three-dimensional models of said one or more SSoI in said first data, said third data comprising configurable parameters associated with SSoIs, said fourth data comprising a list of environment conditions experienced by said SSoI, said fifth data comprising predetermined parameter values and sensor values for a plurality of operation environment and a plurality of configurable parameter type associated with one or more of SSoIs in said first data, said data comprising a list of current parameter and sensor value data from one or more of said SSoI listed in said first data, said seventh data comprising a list of user information used for authentication access, said eighth data comprising a list of voice commands, and said ninth data comprises a plurality of gesture command data including gesture pattern recognition data;

wherein said plurality of machine readable computer processing instructions comprise;

a first plurality of machine readable instructions operable to identify one or more said SSoIs using said video camera to capture an image of one of said SSoIs and use said an pattern recognition section to match at least one said stored patterns with said image of one or more said SSoIs;

a second plurality of machine readable instructions operable to display a plurality of 3D models from second data on said transparent element using said image projector;

a third plurality of machine readable instructions operable to communicate with a remote data server by operating said transceiver and signal and data input/output interface;

a fourth plurality of machine readable instructions operable to encrypt a plurality of network data messages transmitted by said transceiver and signal and data input/output interface;

a fifth plurality of machine readable instructions operable to query and collect a plurality of SSoI current configuration setting data from said SSoI through said transceiver and signal and data interface and store in said third data comprising a plurality of configurable parameter type, said fifth plurality of machine readable instructions further operates said first section to display one or more of said third data on said transparent element showing current sensor values and current parameters of said SSoI within said FOV;

a sixth plurality of machine readable instructions operable to collect a plurality of predetermined setting data associated with said SSoI and a plurality of operation environment data associated with SSoI identified and store in said eighth data and displaying the information on said transparent section using said image projector to show said default settings required for said SSoI in a current operational mode in an operation environment;

a seventh plurality of machine readable instructions operable to operate said microphone to obtain a one or a plurality of voice inputs from said microphone and compare the voice inputs to plurality of stored voice patterns associated with one or more voice command list data respectively associated with one or more of a plurality of voice command execution machine readable instructions, determining a match between said voice input and said voice command list data, and executing a matched said voice command data execution machine readable instructions;

an eighth plurality of machine readable instructions operable to receive a user gesture input image from said camera and compare the user gesture input image in said ninth data and execute a gesture command data execution machine readable instruction if matched;

a ninth plurality of machine readable instructions operable to change a plurality of configurable parameters associated with the SSoI based on said matched voice input and said voice command data list; and

a tenth plurality of machine readable instructions operable to request user access information to allow the user to make changes to the configurable parameters in said third data.

11. The system as in claim 10, wherein said pattern recognition system configured to receive images from said camera and match elements of said image with one or more of said plurality of locally or remotely stored patterns including patterns associated with a system or subsystem of interest (SSoI) is configured to read bar codes attached to the SSoI to identify said SSoI based on stored bar code identification data.