System and method for virtual engineering

• US 9,170,648 B2
• Filed: 04/03/2012
• Issued: 10/27/2015
• Est. Priority Date: 04/03/2012
• Status: Active Grant

• Current Assignee: The Boeing Co.
• Original Assignee: The Boeing Co.
• Inventors: Roth, Andrew S.
• Primary Examiner(s): Nguyen, Kimbinh T
• Application Number:
  US13/438,692
• Publication Number:
  US 20130257904A1
• Time in Patent Office: 1,302 Days
• Field of Search: 345/163, 345/157, 345/629, 345/633
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 2203/0382 Plural input, i.e. interface arrangements in which a plurality of input device of the same type are in communication with a PC
• CPC Class Codes: G06F 2203/0384 Wireless input, i.e. hardware and software details of wireless interface arrangements for pointing devices
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/014 Hand-worn input/output arrangements, e.g. data gloves
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry

Abstract

A method and system for generating a virtual image of a virtual environment is provided. A virtual reality manager receives hand position data for at least one hand of a user from a hand system. The virtual reality manager receives head position data for a head of the user from a head-mounted system. The virtual reality manager identifies image-based position data and a current frame of reference for a current time using a target image corresponding to the current time. The virtual reality manager generates virtual image control data for the current time using the hand position data, the head position data, the image-based position data, and the current frame of reference. The virtual image control data is configured for use by a virtual image application.

Claims

1. A method for generating a virtual image of a virtual environment, the method comprising:

receiving, at a virtual reality manager, hand position data for at least one hand of a user from a hand system;

receiving, at the virtual reality manager, head position data for a head of the user from a head-mounted system;

identifying, by the virtual reality manager, image-based position data and a current frame of reference for a current time using a target image corresponding to the current time;

generating, by the virtual reality manager, virtual image control data for the current time using the hand position data, the head position data, the image-based position data, and the current frame of reference, wherein the virtual image control data is configured for use by a virtual image application; and

generating, via a feedback controller, the virtual image control data with a desired level of accuracy using a finger position error, a relative hand position error, and a head position error;

wherein finger position error is a difference between positions of fingers identified in the hand position data and positions of the fingers simulated in a current virtual image;

wherein relative hand position error is a difference between positions of hands identified in the hand position data and positions of virtual hands in the current virtual image; and

wherein head position error is a difference between position of the head identified in the head position data and position of the head in the current virtual image.

2. The method of claim 1 further comprising:

sending, by the virtual reality manager, the virtual image control data to the virtual image application; and

generating, by the virtual image application, the current virtual image of the virtual environment for the current time using the virtual image control data for the current time.

3. The method of claim 2, wherein the step of generating the current virtual image of the virtual environment for the current time using the virtual image control data for the current time comprises:

positioning a virtual left hand and a virtual right hand in the current virtual image based on the virtual image control data, wherein the virtual left hand is an image representing a left hand of the user and the virtual right hand is an image representing a right hand of the user.

4. The method of claim 2 further comprising:

displaying, by the virtual reality manager, the current virtual image of the virtual environment to the user on a display device associated with the head-mounted system, wherein the current virtual image replaces a previous virtual image displayed on the display device.

5. The method of claim 1, wherein the step of generating the virtual image control data for the current time using the hand position data, the head position data, the image-based position data, and the current frame of reference comprises:

synchronizing, by the virtual reality manager, the hand position data and the head position data with respect to time.

6. The method of claim 1 further comprising:

generating left hand position data using a left glove configured to substantially conform to a left hand of the user; and

generating right hand position data using a right glove configured to substantially conform to a right hand of the user, wherein the left hand position data and the right hand position data form the hand position data.

7. The method of claim 1, wherein the step of generating the virtual image control data for the current time comprises:

identifying modified hand position data using left hand position data in the hand position data, right hand position data in the right hand position data, and the image-based position data;

identifying modified head position data using the head position data and the current frame of reference; and

generating the virtual image control data using the modified hand position data, the modified head position data, and a set of constraints.

8. The method of claim 7, wherein the step of generating the virtual image control data for the current time further comprises:

identifying the set of constraints using user data, wherein the user data is based on a set of user images.

9. The method of claim 7, wherein the step of generating the virtual image control data using the modified hand position data, the modified head position data, and the set of constraints comprises:

identifying the finger position error, the relative hand position error, and the head position error using the feedback controller, the modified hand position data, the modified head position data, and the set of constraints.

10. The method of claim 1, wherein the virtual environment is a simulation of an engineering environment selected from one of a design environment, a manufacturing environment, a computer environment, a testing environment, a data management environment, an inspection environment, and an operations environment.

11. The method of claim 1 further comprising:

filtering out the head position data to use only the hand position data when in a static mode;

using the head position data and the hand position data when in a dynamic mode; and

displaying no virtual images when in a transparent mode.

12. A virtual reality system comprising:

a head-mounted system configured to be worn relative to a head of a user; and

a virtual reality manager associated with the head-mounted system, wherein the virtual reality manager is configured to;

receive hand position data for at least one hand of the user from a hand system;

receive head position data for the head of the user from a sensor system in the head-mounted system;

identify image-based position data and a current frame of reference for a current time using a target image corresponding to the current time; and

generate virtual image control data for the current time using the hand position data, the head position data, the image-based position data, and the current frame of reference, wherein the virtual image control data is configured for use by a virtual image application; and

a control data generator of a data coordinator of the virtual reality manager, wherein the control data generator is configured to use a finger position error, a relative hand position error, and a head position error to generate the virtual image control data;

wherein finger position error is a difference between positions of fingers identified in the hand position data and positions of the fingers simulated in a current virtual image;

wherein relative hand position error is a difference between positions of hands identified in the hand position data and positions of virtual hands in the current virtual image; and

wherein head position error is a difference between position of the head identified in the head position data and position of the head in the current virtual image.

13. The virtual reality system of claim 12, wherein the virtual reality manager is configured to send the virtual image control data to the virtual image application and receive the current virtual image of a virtual environment for the current time from the virtual image application.

14. The virtual reality system of claim 12 further comprising:

a display device associated with the head-mounted system, wherein the virtual reality manager is further configured to display the current virtual image of a virtual environment to the user on the display device.

15. The virtual reality system of claim 13, wherein the current virtual image includes a virtual left hand and a virtual right hand in which the virtual left hand is an image representing a left hand of the user and the virtual right hand is an image representing a right hand of the user.

16. The virtual reality system of claim 12, wherein the virtual reality manager comprises:

the data coordinator configured to synchronize the hand position data and the head position data with respect to time, wherein the data coordinator comprises;

a hand data modulator configured to identify modified hand position data using left hand position data in the hand position data, right hand position data in the hand position data, and the image-based position data;

a head data modulator configured to identify modified head position data using the head position data and the current frame of reference;

the control data generator configured to generate the virtual image control data using the modified hand position data, the modified head position data, and a set of constraints; and

a constraint identifier configured to identify the set of constraints using user data, wherein the user data is based on a set of user images.

17. The virtual reality system of claim 16, wherein the data coordinator further comprises:

a feedback controller configured to identify the finger position error, the relative hand position error, and the head position error using the modified hand position data, the modified head position data, and the set of constraints.

18. The virtual reality system of claim 12, wherein the virtual reality system further comprises:

the hand system, wherein the hand system comprises;

a left glove configured to generate left hand position data, wherein the left glove is configured to substantially conform to a left hand of the user; and

a right glove configured to generate right hand position data, wherein the right glove is configured to substantially conform to a right hand of the user.

19. The virtual reality system of claim 12, wherein the virtual image control data is used to control a virtual image of a virtual environment in which the virtual environment is a simulation of an engineering environment selected from one of a design environment, a manufacturing environment, a computer environment, a testing environment, a data management environment, an inspection environment, and an operations environment.

20. A computer comprising:

a bus;

a non-transitory storage device connected to the bus, wherein the non-transitory storage device includes program code; and

a processor unit connected to the bus, wherein the processor unit is configured to execute the program code to;

receive hand position data for at least one hand of a user from a hand system;

receive head position data for a head of the user from a head-mounted system;

identify image-based position data and a current frame of reference for a current time using a target image corresponding to the current time;

generate virtual image control data for the current time using the hand position data, the head position data, the image-based position data, and the current frame of reference, wherein the virtual image control data is configured for use by a virtual image application; and

generate the virtual image control data with a desired level of accuracy using a feedback controller, a finger position error, a relative hand position error, and a head position error;

wherein finger position error is a difference between positions of fingers identified in the hand position data and positions of the fingers simulated in a current virtual image;

wherein relative hand position error is a difference between positions of hands identified in the hand position data and positions of virtual hands in the current virtual image; and

wherein head position error is a difference between position of the head identified in the head position data and position of the head in the current virtual image.

21. The computer of claim 20, wherein the processor unit is further configured to execute the program code to display a current virtual image for the current time, received from the virtual image application, on a display device associated with the head-mounted system.