Multi-user real-time augmented reality system and method

• US 6,317,127 B1
• Filed: 10/16/1996
• Issued: 11/13/2001
• Est. Priority Date: 10/16/1996
• Status: Expired due to Term

• Current Assignee: Hughes Electronics Corporation (AT&T, Inc.)
• Original Assignee: Hughes Electronics Corporation (AT&T, Inc.)
• Inventors: Payton, David W., Daily, Michael J.
• Primary Examiner(s): Cuchlinski, Jr., William A.
• Assistant Examiner(s): NGUYEN, THU V
• Application Number:
  US08/731,542
• Time in Patent Office: 1,854 Days
• Field of Search: 345/1-3, 345/7, 345/8, 345/425, 345/427, 345/419, 345/435, 348/442, 348/746, 348/113, 348/115, 348/143, 348/144
• US Class Current: 345/629
• CPC Class Codes: G06T 15/10 Geometric effects

Abstract

A high resolution real-time multi-user augmented reality system broadcasts a wide FOV video signal from a plurality of image sources over a distribution bus to a plurality of users that independently select and view different portions of the video signal such that the angular resolution of the displayed video signal is constant over the range of possible viewing orientations. This is accomplished by assuming that each user'"'"'s position is fixed at the center of a virtual sphere, the image sources'"'"' positions are known relative to the center of the virtual sphere, and that the users are looking at a portion of the inner surface of the sphere. As a result, the “flat” images generated by the image sources and the “flat” images viewed by the users can be efficiently mapped onto the virtual sphere and represented as index pairs (d,n). Thus, each user extracts those pixels on the virtual sphere corresponding to the user'"'"'s current FOV and remaps them to a flat display. The video signals can be augmented with synthetic point-of-interest data such as visual overlays or audio messages that are registered to the video.

Claims

1. An augmented reality system for displaying landmarks to at least one user, comprising:

at least one image source positioned at a known orientation with respect to a center of a virtual sphere in which points on said virtual sphere are represented as spherical coordinate pairs (d,n) that respectively denote segments and rings of said virtual sphere, said image source generating video signals that together form a wide field of view (FOV) video signal having pixels (s, x_s,y_s,i) in which s identifies said image source, (x_s, y_s) are cartesian coordinates of pixels on an image plane of said image source, and i is a pixel value;

a source processor that maps said pixels (s,x_s,y_s,i) into spherical coordinate triplets (d,n,i);

a storage device that stores landmark data including overlay symbols and audio tracks that correspond to said wide FOV video signal;

a video bus for broadcasting the triplets (d,n,i) to multiple users;

a system for obtaining the user'"'"'s viewpoint orientation;

at least one user display system, comprising;

a) a display for displaying a respective subimage of said wide FOV video signal with an image plane of said display represented in cartesian coordinates (x_u,y_u); and

b) a user controllable viewpoint selector for selecting said respective subimage in the wide FOV video signal wherein a center of said virtual sphere is defined as a fixed position of an user using said display; and

at least one local processor for;

mapping pixels (x_u,y_u) of said respective subimage to respective spherical coordinate pairs (d,n);

extracting from said video bus, a respective set of said spherical triplets (d,n,i) that correspond to said respective set of said spherical coordinate pairs (d,n), and

remapping said respective set of spherical triplets (d,n, i) to pixels (x_u,y_u,i) in said image plane, and

wherein said local processor, in response to user selection, is capable of mapping selected portions of said overlay symbols and audio tracks to cartesian coordinates (x_u, y_u) of said image plane.

2. The augmented reality system of claim 1, wherein said display and said user controllable viewpoint selector respectively comprise:

a head mounted display (HMD) that is worn by said user; and

a head tracker on said HMD that tracks head movements of said user.

3. The augmented reality system of claim 1, further comprising:

a distribution processor; and

wherein said system for obtaining the user'"'"'s viewpoint orientation is capable of obtaining the position and heading of the user, and said distribution processor is capable of extracting in response to said position and heading, overlay symbols and audio tracks that correspond to said wide FOV video signal.

4. The augmented reality system of claim 3, further comprising:

an overlay distribution bus for distribution of overlay symbols and audio tracks.

5. The augmented reality system of claim 1, wherein said positioning system includes a Global Positioning Satellite System.

6. The augmented reality system of claim 1, wherein said user display system further comprises a user controllable zoom mechanism for zooming in and out on said respective subimage.

7. A method of providing an augmented reality system for displaying landmarks to at least one user, comprising the steps of:

defining a virtual sphere in which points on said virtual sphere are represented as spherical coordinate pairs (d,n) that respectively denote segments and rings of said virtual sphere and a center of said virtual sphere is defined as a fixed position of said user;

generating at least one video signal from a known orientation in said virtual sphere that forms a wide field-of-view (FOV) video signal having pixels (s,x_s,y_s,i) in which s identities said video signal, (x_s, y_s) are cartesian coordinates of pixels within said video signal, and i is a pixel value;

mapping said pixels (s,x_s,y_s,i) into spherical coordinate triplets (d,n,i);

broadcasting said spherical coordinate triplets (d,n,i) over a video bus to said at least one user;

tracking a viewpoint orientation for said user that defines a subimage in said wide FOV video signal;

in response to said viewpoint orientation, mapping cartesian coordinates (x_u,y_u) of pixels in a user display to respective spherical coordinate pairs (d, n);

in response to said mapping steps, identifying a set of said spherical coordinate (d,n,i) triplets that corresponds to said subimage;

in response to said identifying step, remapping said set of spherical coordinate (d,n,i) triplets to pixels (x_u,y_u,i) in said user display;

storing landmark data including overlay symbols and audio tracks that correspond to said wide FOV video signal; and

in response to user selection, providing respective portions of said landmark data that are within said subimage to said user display.

8. The method of claim 7, wherein said defining step includes the step of generating said spherical coordinate pairs (d,n) with an angular resolution α

and further including a step of adjusting a focal length for said user to zoom in and out on said subimage.

9. The method of claim 7, further comprising the steps of:

obtaining the position and heading of said user; and

in response to said position and heading, providing respective portions of said overlay symbols and audio tracks to said user display that are within said subimage; and

wherein said providing step includes the steps of;

registering said overlay symbols and audio tracks to respective spherical coordinate pairs (d,n);

in response to said registering step, extracting said respective portions; and

mapping said respective portions to said user display.

10. The method of claim 9, wherein the step of providing respective portions of said overlay symbols and audio tracks comprises distributing said respective portions over an overlay distribution bus.

11. The method of claim 7, wherein said user display is a head mounted display and said tracking step includes the step of monitoring the head movement of said user to establish said viewpoint orientation.

12. The method of claim 7, wherein said obtaining step includes the step of employing a Global Positioning Satellite System.

13. An airplane fitted with an augmented reality system for displaying landmarks to the user, comprising:

a plurality of sensors each attached to the underside of the airplane and providing a video signal of the hemisphere below the airplane;

a server for digitizing the video signal provided by each of said plurality of sensors, mapping the video signal into spherical coordinate (d,n) pairs and multiplexing the video signal provided by each one of said plurality of sensors into a wide field of view (FOV) signal, wherein the server also broadcasts landmark data including overlay symbols and audio tracks corresponding to the wide FOV signal;

a video bus for distributing said wide FOV signal to a plurality of passengers on board said airplane;

a system for obtaining the viewpoint orientation of each one of said plurality of passengers;

user display systems, each said display system comprising;

a) a display for displaying a respective subimage of said wide FOV video signal with an image plane ot said display represented in cartesian coordinates (x_u,y_u); and

b) a user controllable viewpoint selector for selecting said respective subimage in the wide FOV video signal wherein a center of said virtual sphere is defined as a fixed position of the passenger using said display; and

local processors, wherein each local processor is for;

mapping pixels (x_u,y_u) of said respective subimage to respective spherical coordinate pairs (d,n);

extracting from said video bus, a respective set of spherical triplets (d,n,i) that correspond to said respective spherical coordinate pairs (d,n), and

remapping said respective set of spherical triplets (d,n,i) to pixels (x_u,y_u,i) in said image plane, and

wherein each said local processor, in response to user selection is capable of mapping selected portions of said overlay symbols and audio tracks to cartesian coordinates (x_u, y_u) of said image plane.

14. The airplane of claim 13, wherein said system for obtaining the viewpoint orientation includes a subsystem for obtaining the positioning and heading of the passenger.

15. The airplane of claim 14, wherein said subsystem is a Global Positioning System.