System for combining virtual images with real-world scenes

• US 6,166,744 A
• Filed: 09/15/1998
• Issued: 12/26/2000
• Est. Priority Date: 11/26/1997
• Status: Expired due to Term

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Pathfinder Systems, Inc.
• Inventors: Jaszlics, Ivan J., Jaszlics, Sheila L.
• Primary Examiner(s): Zimmerman, Mark K.
• Assistant Examiner(s): Padmanabhan, Mano
• Application Number:
  US09/153,503
• Time in Patent Office: 833 Days
• Field of Search: 345/435, 345/421, 345/11, 434/11
• US Class Current: 345/629
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects

Abstract

A system for combining virtual images with images of the real world. A range scanner determines the shape and distance of real-world objects within a field of interest to the observer using the apparatus. Virtual masking objects, which are simplified computer models of real-world objects, are derived from the range data. Virtual entities that are not present in the real world are simulated in an electronic computer. Images of the virtual entities and the virtual masking objects are combined into masked virtual images. The masked virtual images show the portions of virtual entities that would be visible if these entities actually existed in the real world. The masked virtual images and images of the real world scene are combined in such a manner that the virtual images appear to be obscured, when appropriate for their simulated, virtual location by real-world objects. The resulting combined image is presented in an electronic or optical display.

Claims

1. A system for combining virtual images with a real-world scene within a field of interest for an observer, said system comprising:

a range scanner scanning the field of interest and generating range data indicating the distance of real-world objects within the field of interest;

a computer model simulating a virtual entity and producing a virtual image of said virtual entity at a location within the field of interest;

means for generating masked virtual objects from said range data and said virtual image indicating those portions of said virtual image that are visible in the field of interest;

means for combining said masked virtual objects and a real-world image of the field of interest to create a combined image in which said virtual image appears in the real-world image; and

display means for displaying said combined image to the observer.

2. The system of claim 1 wherein said display means comprises the imaging system of an armored vehicle and wherein the field of view comprises the field of view through said imaging system.

3. The system of claim 1 wherein said display means comprises the display of an aircraft simulator and wherein the field of view comprises the field of view from the cockpit of a simulated aircraft.

4. The system of claim 1 wherein said display means comprises an electronically-controlled visor and wherein the field of view comprises the field of view visible through said visor.

5. The system of claim 1 wherein said display means comprises the display system of a computing device and wherein the field of view comprises the field of view of a video camera transmitting images to said computing device.

6. The system of claim 1 further comprising means for generating virtual masking objects from said range data corresponding to real-world objects within the field of interest, and wherein said means for generating masked virtual objects combines said virtual masking objects with said virtual image to determine those portions of said virtual image that are visible in the field of interest.

7. The system of claim 6 wherein said virtual masking objects, said virtual images, and said masked virtual objects are defined in virtual reality modeling language (VRML).

8. The system of claim 1 further comprising control means for enabling the observer to change the field of view.

9. The system of claim 1 further comprising means for enabling the observer to interact with said computer model simulation and said virtual image.

10. The system of claim 1 wherein said range scanner periodically scans an array of ranging sectors in the field of view.

11. The system of claim 10 further comprising means for extrapolating changes in said masked virtual objects between said periodic scans.

12. A method for combining virtual images with a real-world scene within a field of interest for an observer, said method comprising:

scanning the field of interest to generate range data indicating the distance of real-world objects within the field of interest;

simulating a virtual entity to produce a virtual image of said virtual entity at a location within the field of interest;

generating masked virtual objects from said range data and said virtual image indicating those portions of said virtual image that are visible in the field of interest;

combining said masked virtual objects and a real-world image of the field of interest to create a combined image in which said virtual image appears in the real-world image; and

displaying said combined image to the observer.

13. The method of claim 12 wherein said combined image is displayed through the imaging system of an armored vehicle and wherein the field of view comprises the field of view through said imaging system.

14. The method of claim 12 wherein said combined image is displayed through the display of an aircraft simulator and wherein the field of view comprises the field of view from the cockpit of a simulated aircraft.

15. The method of claim 12 wherein said combined image is displayed through an electronically-controlled visor and wherein the field of view comprises the field of view visible through said visor.

16. The method of claim 12 wherein said combined image is displayed through the display of a computing device and wherein the field of view comprises the field of view of a video camera transmitting images to said computing device.

17. The method of claim 12 wherein said combined image is injected into the optical train of an optical instrument.

18. The method of claim 12 further comprising generating virtual masking objects from said range data corresponding to real-world objects within the field of interest, and wherein said step of generating masked virtual objects combines said virtual masking objects with said virtual image to determine those portions of said virtual image that are visible in the field of interest.

19. The method of claim 18 wherein said virtual masking objects, said virtual images, and said masked virtual objects are defined in virtual reality modeling language (VRML).

20. The method of claim 12 further comprising providing controls to enable the observer to change the field of view.

21. The method of claim 12 further comprising providing controls to enable the observer to interact with said simulation of said virtual entity.

22. The method of claim 12 wherein said step of scanning comprises periodically scanning an array of ranging sectors in the field of view.

23. The method of claim 22 further comprising extrapolating changes in said masked virtual objects between said periodic scans.

24. A method for combining virtual images with a real-world scene within a field of interest for an observer, said method comprising:

scanning the field of interest to generate range data for a two-dimensional array of range sectors indicating the distance of real-world objects within the field of interest;

generating virtual masking objects from said range data corresponding to real-world objects within the field of interest;

simulating a virtual entity to produce a virtual image of said virtual entity at a location within the field of interest;

combining said virtual image and said virtual masking objects to generate a masked virtual object indicating those portions of said virtual image that are visible in the field of interest;

combining said masked virtual objects and the real-world image of the field of interest to create a combined image in which said virtual image appears in the real-world image; and

displaying said combined image to the observer.

25. The method of claim 24 wherein said virtual masking objects, said virtual images, and said masked virtual objects are defined in virtual reality modeling language (VRML).

26. The method of claim 24 further comprising providing controls to enable the observer to change the field of view.

27. The method of claim 24 further comprising providing controls to enable the observer to interact with said simulation of said virtual entity.

28. The method of claim 24 wherein said step of scanning comprises periodically scanning an array of ranging sectors in the field of view.

29. The method of claim 28 further comprising extrapolating changes in said virtual masking objects between said scans.