Interactive system control apparatus and method

• US 10,296,359 B2
• Filed: 02/15/2016
• Issued: 05/21/2019
• Est. Priority Date: 02/25/2015
• Status: Active Grant

• Current Assignee: BAE Systems Plc
• Original Assignee: BAE Systems Plc
• Inventors: Whiteford, Christopher James, Colosimo, Nicholas Giacomo Robert, Wright, Julian David
• Primary Examiner(s): Rayan, Mihir K
• Application Number:
  US15/551,085
• Publication Number:
  US 20180032139A1
• Time in Patent Office: 1,191 Days
• CPC Class Codes: G05B 19/048 Monitoring; Safety
• CPC Class Codes: G05B 2219/40131 Virtual reality control, programming of manipulator
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: G09G 3/003 to produce spatial visual effects
• CPC Class Codes: H04N 23/54 Mounting of pick-up tubes, electronic image sensors, deviation or focusing coils
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Abstract

A mixed reality control apparatus for a system having at least one remote data source requiring at least one physical control input, the apparatus comprising a headset for placing over a user'"'"'s eyes, in use, the headset including a screen, the apparatus further including a processor configured to receive data from the at least one remote data source and display the data on the screen within a three-dimensional virtual environment, and image capture means for capturing images of the real world environment in the vicinity of the user, the processor being further configured to: blend at least portions or objects of the images of the real world environment into the three-dimensional virtual environment to create a mixed reality environment, including the data, to be displayed on the screen; and generate a virtual representation of the at least one physical control input and blend the virtual representation into the mixed reality environment at a selected location, and generate a marker representative of the selected location; wherein the apparatus is configured to receive a user input in association with the marker, identify the user input as an action in respect of the control input, or the virtual representation thereof, and generate a control signal to effect the action accordingly.

Claims

1. A mixed reality control apparatus for a system comprising at least one remote data source requiring at least one physical control input, the apparatus comprising a headset for placing over a user'"'"'s eyes, in use, said headset including a screen, the apparatus further comprising a processor configured to receive data from said at least one remote data source and display said data on said screen within a three-dimensional virtual environment, and at least one image capture device for capturing images of the real world environment in the vicinity of the user, the processor being further configured to:

blend at least portions or objects of said images of said real world environment into said three-dimensional virtual environment to create a mixed reality environment, including said data, to be displayed on said screen; and

generate a virtual representation of said at least one physical control input and blend said virtual representation into said mixed reality environment at a selected location, and generate a marker representative of said selected location;

wherein the apparatus is configured to receive a user input in association with said marker, identify said user input as an action in respect of said control input, or said virtual representation thereof, and generate a control signal to effect said action accordingly, and

wherein said user input comprises a predefined bodily movement by said user in association with said marker, and said processor is configured to identify a bodily movement, determine from the relative location of said bodily movement in relation to said marker the control input to which said user input relates, determined from said bodily movement the control action required to be performed and generate a control signal to effect said action accordingly.

2. The apparatus according to claim 1, comprising a pair of spatially separated image capture devices for capturing respective images of the real world environment in the vicinity of the user, said processor being configured to define a depth map using respective image frame pairs to produce three-dimensional image data.

3. The apparatus according to claim 2, wherein said image capture devices are mounted on said headset so as to be substantially aligned with a user'"'"'s eyes, in use.

4. The apparatus according to claim 2, wherein said predefined bodily movement is a predefined user hand gesture.

5. The apparatus according to claim 2, further comprising a database in which is stored data representative of a predefined set of bodily movements and control actions to which they relate, and said processor is configured to extract an image of a user'"'"'s bodily movement from said captured images, compare said movement with said predefined set of bodily movements to determine a match and, thereby, to identify a required control action, and generate a control single operative to cause a required control action to be performed in relation to data displayed within said mixed reality environment.

6. The apparatus according claim 1, wherein said marker is in a fixed relative location within said three-dimensional virtual environment displayed on said screen.

7. The apparatus according claim 1, wherein said marker is associated with a user'"'"'s body part, and the processor is configured to track movement of said user'"'"'s body part and adjust the relative location of said virtual representation of said control input accordingly.

8. The apparatus according to claim 1, wherein said virtual representation of said control input comprises an image of a corresponding physical control device, and said user input comprises a physical action representative of physical manipulation of said physical control device.

9. The apparatus according to claim 1, configured to selectively move said data and/or said virtual representation of said control input and associated marker in response to a user input.

10. A mixed reality control apparatus for a system comprising at least one remote data source requiring at least one physical control input, the apparatus comprising a headset for placing over a user'"'"'s eyes, in use, said headset including a screen, the apparatus further comprising a processor configured to receive data from said at least one remote data source and display said data on said screen within a three-dimensional virtual environment, and at least one image capture device for capturing images of the real world environment in the vicinity of the user, the processor being further configured to:

blend at least portions or objects of said images of said real world environment into said three-dimensional virtual environment to create a mixed reality environment, including said data, to be displayed on said screen; and

generate a virtual representation of said at least one physical control input and blend said virtual representation into said mixed reality environment at a selected location, and generate a marker representative of said selected location;

wherein the apparatus is configured to receive a user input in association with said marker, identify said user input as an action in respect of said control input, or said virtual representation thereof, and generate a control signal to effect said action accordingly, and

wherein said processor is configured to determine, from said images of said real world captured by said at least one image capture device, the direction of a user'"'"'s gaze, and affix an associated said marker thereto, such that said virtual representation of said control input remains constantly within said user'"'"'s field of view.