SYSTEMS AND METHODS FOR COMBINING VIRTUAL AND REAL-TIME PHYSICAL ENVIRONMENTS

• US 20100182340A1
• Filed: 01/19/2009
• Published: 07/22/2010
• Est. Priority Date: 01/19/2009
• Status: Abandoned Application

• Current Assignee: Edward N. Bachelder, Noah Brickman
• Original Assignee: Edward N. Bachelder, Noah Brickman
• Inventors: Bachelder, Edward N., Brickman, Noah
• Application Number:
  US12/356,048
• Publication Number:
  US 20100182340A1
• Time in Patent Office: Days
• US Class Current: 345/633
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: A63F 2300/6009 for importing or creating game content, e.g. authoring tools during game development, adapting content to different platforms, use of a scripting language to create content
• CPC Class Codes: A63F 2300/69 Involving elements of the real world in the game world, e.g. measurement in live races, real video
• CPC Class Codes: G02B 2027/0112 comprising device for genereting colour display
• CPC Class Codes: G02B 2027/0118 comprising devices for improving the contrast of the display / brillance control visibility
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G09G 2340/0428 Gradation resolution change
• CPC Class Codes: G09G 2340/14 Solving problems related to the presentation of information to be displayed
• CPC Class Codes: G09G 3/003 to produce spatial visual effects
• CPC Class Codes: G09G 5/026 Control of mixing and/or overlay of colours in general G09G5/022 and G09G5/024 take precedence

Abstract

Systems, methods and structures for combining virtual reality and real-time environment by combining captured real-time video data and real-time 3D environment renderings to create a fused, that is, combined environment, including capturing video imagery in RGB or HSV/HSV color coordinate systems and processing it to determine which areas should be made transparent, or have other color modifications made, based on sensed cultural features, electromagnetic spectrum values, and/or sensor line-of-sight, wherein the sensed features can also include electromagnetic radiation characteristics such as color, infra-red, ultra-violet light values, cultural features can include patterns of these characteristics, such as object recognition using edge detection, and whereby the processed image is then overlaid on, and fused into a 3D environment to combine the two data sources into a single scene to thereby create an effect whereby a user can look through predesignated areas or “windows” in the video image to see into a 3D simulated world, and/or see other enhanced or reprocessed features of the captured image.

Claims

1. A virtual reality system comprising:

a head mounted camera and one or more fixed cameras adapted to receive images of a physical environment and to produce a frame of the physical environment that contains pixels representing at least a first range of electromagnetic spectrum values;

at least one range of predetermined target threshold electromagnetic spectrum values;

a means for processing the frame and rendering pixels having values within the target threshold range of values to be transparent pixels;

a means for generating a virtual image;

a means for combining the virtual image with the frame of the physical environment to form a combined image whereby the virtual image is visible at all positions of the transparent pixels; and

,

a means for displaying the combined image.

2. The system of claim 1 wherein the target threshold electromagnetic spectrum range of values includes values representative of one or more of hue, saturation or brightness.

3. The system of claim 1 further including a means for comparing the pixels of the frame to the target color under different predetermined electromagnetic conditions to determine whether a match with the target electromagnetic spectrum exists at any of the predetermined electromagnetic values.

4. The system of claim 1 further including a means for tracking locations of a physical object moving within a predetermined distance from the camera to produce a tracked object image and for replacing the tracked object image with a virtual image of the tracked object when the physical object moves beyond the predetermined distance from the camera.

5. A method for combining virtual reality environments and real-time physical environments for a user comprising:

identifying a target pixel depth;

providing the real-time physical environment, the real-time physical environment having a predesignated area to be overlaid with virtual reality images and having the target pixel depth corresponding a predetermined distance to an object in the predesignated area;

identifying the predesignated area via pixel depth, whereby the depth is determined by distance from the object in the predesignated area to a first distance sensor;

determining the distance from the first distance sensor to the object with at least one distance sensor selected from the group consisting of camera, laser, lidar, sonar and stereoscopy devices;

providing virtual reality images;

providing the user with a head mounted display adapted to have a perspective during use corresponding to a perspective of the user'"'"'s eyes along a predetermined line of sight;

providing a first depth sensor, the first depth sensor being a video camera and mounting the video camera at a location from which the video camera has a perspective substantially similar to the perspective of the user'"'"'s eyes;

providing a second depth sensor and a third depth sensor;

mounting the second depth sensor and mounting the third depth sensor at various locations around the user;

adapting the video camera, the second depth sensor and the third depth sensor to provide data representative of the distance from the object to the video camera along the line of sight;

operating the video camera to provide real video images of the real-time physical environment;

capturing the real video images in digital pixels;

identifying areas of the real video images to be overlaid with virtual video images by comparing the pixel depth of the digital pixels to a predetermined target pixel depth;

making transparent all digital pixels of the real video images whose pixel depth exceeds the predetermined target pixel depth;

overlaying the real video images onto the virtual video images to form a combined image; and

,

providing to the head mounted display the combined image.

6. A computer-implemented system for combining a virtual reality environment and a physical environment for a user comprising:

a computer;

a camera operatively connected to the computer and adapted to provide to the computer real-time physical environment video images in digital pixels using a hue, saturation and brightness color coordinate system, and adapted to be mounted on the head of a user at a location from which the camera has a view substantially similar to the view of the user'"'"'s eyes;

a physical object operatively connected to the computer and adapted to interact with the user and to provide input to the computer in response to interaction with the user;

a virtual image generator operatively connected to the computer and adapted to provide to the computer virtual video images in digital pixels;

a position detector operatively connected to the computer and adapted to be mounted on the head of the user and to provide to the computer three-dimensional, spatial information about the location and direction of the user'"'"'s head;

an image display operatively connected to the computer and adapted to be mounted on the head of the user and adapted to receive video images from the computer;

a real-time physical environment;

a pre-determined target masking color covering a predesignated area of the real-time physical environment;

the computer programmed to recognize the target masking color in the hue, saturation and brightness color coordinate system;

a range of predetermined target threshold values of hue corresponding to the pre-determined target masking color hue;

a range of predetermined target threshold values of saturation corresponding to the pre-determined target masking color saturation;

a range of predetermined target threshold values of brightness corresponding to the pre-determined target masking color brightness;

the computer programmed to make transparent areas of the real-time physical environment video images in which the color of the pixels of the real-time physical environment video images fall within the predetermined target threshold values of hue, of saturation and of brightness of the pixels of the target masking color;

the computer adapted to change the virtual video images in response to the input to the computer from the physical object;

the computer programmed to overlay the real-time physical environment video images onto the virtual video images to form combined video images; and

,

the computer programmed to provide to the image display the combined video images.

7. The system of claim 6 in which the pre-determined target masking color is magenta.

8. The system of claim 6 in which the real-time physical environment is representative of the inside of a vehicle.

9. The system of claim 6 in which the physical object is representative of any of a handle, an accelerator, a steering mechanism or a firing device.

10. The system of claim 6 in which the real-time physical environment is the inside of a helicopter.

11. The system of claim 8 in which the pre-designated area is representative of a window of the vehicle.

12. The system of claim 6 in which the pre-designated area is a flat surface.

13. The system of claim 6 in which the pre-designated area is a shallow dish.

14. The system of claim 6 wherein the camera includes a user-controlled exposure capability.

15. The system of claim 6 wherein the camera includes an automatic exposure capability.

16. The system of claim 6 wherein the pre-designated area is adapted to emit light to keep constant the target masking color in a variety of dynamically changing lighting conditions.

17. The system of claim 6 wherein the computer is programmed to make transparent all areas of the real video images in which the color of the pixels of the real video images match the target color in a variety of dynamically changing lighting conditions.

18. The system of claim 6 in which correspondence between a color in an R, G, B color coordinate system and the same color in a Hue (H), Saturation (S) and Brightness (V) color coordinate system is defined by a formula, in which R, G and B values are between 0.0 and 1.0, MAX equals a maximum value of each of the R, G, B values, respectively, MIN equals a minimum value of each of the R, G, B values, respectively, and

19. The system of claim 6 in which the real-time physical environment is a near-field environment and the virtual reality environment is a far-field environment.

20. The system of claim 19 wherein the computer is adapted to convert the video images of the real-time physical near-field environment and the video images of the virtual reality far-field environment into bitmaps.