Automated frame of reference calibration for augmented reality

• US 9,508,146 B2
• Filed: 10/31/2012
• Issued: 11/29/2016
• Est. Priority Date: 10/31/2012
• Status: Active Grant

• Current Assignee: The Boeing Co.
• Original Assignee: The Boeing Co.
• Inventors: Davies, Paul R., Lee, David K.
• Primary Examiner(s): Zimmerman, Mark
• Assistant Examiner(s): Le, Sarah
• Application Number:
  US13/665,837
• Publication Number:
  US 20140118339A1
• Time in Patent Office: 1,490 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06T 15/10 Geometric effects
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 3/60 Rotation of whole images or parts thereof
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration

Abstract

One or more systems, methods, routines and/or techniques for automated frame of reference calibration for augmented reality are described. One or more systems, methods, routines and/or techniques may allow for calibration of an Augmented Reality (AR) system, for example, by automatically calibrating the frames of reference of virtual objects and/or a camera. One example calibration routine and/or technique may determine and/or calculate a mapping or transform from a frame of reference of a virtual object (e.g., a CAD model) to a coordinate frame associated with the tracking system. Another example calibration routine and/or technique may determine and/or calculate a mapping or transform from a camera lens frame of reference to a frame of reference of the whole camera as determined by a tracking system. These routines and/or techniques may calibrate an AR system to provide rapid, precise alignment between virtual content and a live camera view of a real scene.

Claims

1. A method for augmented reality executed by a data processing system having at least one processor, the method comprising:

receiving or establishing a tracking system coordinate frame associated with an object tracking system,

wherein the tracking system coordinate frame is aligned with a real 3D space, and

wherein the tracking system tracks the position and orientation in a real 3D space of a real object and of a camera;

receiving from the tracking system a first real object frame of reference for the real object,

wherein the first real object frame of reference indicates a position and orientation of the real object relative to the tracking system coordinate frame;

determining a second real object frame of reference for the real object,

wherein the second real object frame of reference indicates a position and orientation of the real object relative to the tracking system coordinate frame;

receiving a first virtual object frame of reference for a virtual object,

wherein the virtual object is modeled after the real object, and

wherein the first virtual object frame of reference is unrelated to the tracking system coordinate frame;

determining a real object origin by calculating a centroid of three or more real object non-collinear points;

determining a second virtual object frame of reference for the virtual object,

wherein the second virtual object frame of reference indicates a position and orientation of the virtual object relative to the tracking system coordinate frame, and wherein the second real object frame of reference is aligned with the second virtual object frame of reference;

determining a virtual object origin by calculating a centroid of three or more virtual object non-collinear points;

determining a virtual object mapping between the first virtual object frame of reference and the tracking system coordinate frame, wherein the virtual object mapping includes a transform matrix to transform between the first virtual object frame of reference and the tracking system coordinate frame; and

displaying an augmented scene including a view of the real 3D space, a view of the real object and one or more overlaid virtual items,

wherein the virtual object mapping is used to place the one or more overlaid virtual items in the augmented scene such that the one or more virtual items are aligned with the real object.

2. The method of claim 1, wherein determining the second real object frame of reference includes:

receiving or detecting three or more real object non-collinear points on the real object,

wherein the location of three or more real object non-collinear points are defined relative to the tracking system coordinate frame; and

determining a real object orientation that is related to the orientation of the first real object frame of reference.

3. The method of claim 2, wherein determining the second virtual object frame of reference includes:

receiving or indicating three or more virtual object non-collinear points on the virtual object,

wherein the location of three or more virtual object non-collinear points are defined relative to the tracking system coordinate frame; and

determining a virtual object orientation.

4. The method of claim 3, wherein the second real object frame of reference and the second virtual object frame of reference are aligned by ensuring that:

the three or more virtual object non-collinear points and the three or more real object non-collinear points are located at approximately the same location relative to the tracking system coordinate frame; and

the real object orientation and the virtual object orientation are approximately the same as each orientation relates to the tracking system coordinate frame.

5. The method of claim 1, wherein determining the virtual object mapping includes:

receiving or determining a first transform matrix that represents the first real object frame of reference;

determining a second transform matrix that represents the difference in location and orientation between the first real object frame of reference and the second real object frame of reference;

determining a third transform matrix that represents the difference in location and orientation between the first virtual object frame of reference and the second virtual object frame of reference; and

computing a fourth transform matrix that represents the virtual object mapping,

wherein the fourth transform matrix represents the difference in location and orientation between the first virtual object frame of reference and the tracking system coordinate frame, and

wherein the fourth transform matrix is computed by performing matrix multiplication between the first, second and third transform matrices.

6. The method of claim 1, wherein using the virtual object mapping to place the one or more overlaid virtual items in the augmented scene includes:

receiving a first virtual item frame of reference for a first overlaid virtual item,

wherein the first virtual item frame of reference is unrelated to the tracking system coordinate frame;

referencing the virtual object mapping to transform the first virtual item frame of reference to a transformed frame of reference that relates to the tracking system coordinate frame; and

placing the first overlaid virtual item in the augmented scene using the transform frame of reference.

7. The method of claim 1, further comprising:

receiving from the tracking system real-time information about the location and orientation of the real object;

updating the virtual object mapping based on the real-time information; and

updating the augmented scene by updating the placement of the one or more overlaid virtual items such that the one or more virtual items remain aligned with the real object.

8. A method for augmented reality executed by a data processing system having at least one processor, the method comprising:

receiving or establishing a tracking system coordinate frame associated with an object tracking system,

wherein the tracking system coordinate frame is aligned with a real 3D space, and

wherein the tracking system tracks the position and orientation in a real 3D space of a camera that captures the real 3D space and a printed marker;

receiving from the tracking system a camera frame of reference for the camera,

wherein the camera frame of reference indicates a position and orientation of the camera relative to the tracking system coordinate frame;

receiving or establishing a printed marker coordinate frame associated with the printed marker,

wherein the printed marker coordinate frame is aligned with the real 3D space, and

wherein the printed marker coordinate frame is aligned with the tracking system coordinate frame;

determining a printed marker origin by calculating a centroid of three or more printed marker non-collinear points;

determining a camera lens frame of reference for the lens of the camera,

wherein the camera lens frame of reference indicates a position and orientation of the camera lens relative to the printed marker coordinate frame;

determining a camera lens mapping between the camera frame of reference and the camera lens frame of reference, wherein the camera lens mapping includes a transform matrix to transform between the camera frame of reference and the camera lens frame of reference; and

displaying an augmented scene including a view of the real 3D space and one or more virtual items,

wherein the camera lens mapping is used to alter or distort the one or more virtual items in the augmented scene.

9. The method of claim 8, wherein determining the camera lens frame of reference includes receiving camera lens position and orientation information a software program associated with the printed marker.

10. The method of claim 8, wherein aligning the printed marker coordinate frame with the tracking system coordinate frame includes placing the printed marker at a location and orientation in the real 3D space such that:

an origin associated with the printed marker is located at approximately the same location in the 3D space as an origin associated with the tracking system; and

three axes associated with the printed marker are approximately aligned in the 3D space with three axes associated with the tracking system.

11. The method of claim 8, wherein determining the camera lens mapping includes:

receiving or determining a first transform matrix that represents the first real object frame of reference;

determining a second transform matrix that represents the difference in location and orientation between the camera lens frame of reference and the printed marker coordinate frame;

computing a third transform matrix that represents the camera lens mapping,

wherein the third transform matrix represents the difference in location and orientation between the camera frame of reference and the camera lens frame of reference, and

wherein the third transform matrix is computed by performing matrix division either between the first and second transform matrices or between the second and first transform matrices.

12. The method of claim 8, wherein using the camera lens mapping to alter or distort the one or more overlaid virtual items in the augmented scene includes:

placing a first overlaid virtual item in the augmented scene using a first virtual item frame of reference,

wherein the first virtual item frame of reference relates to the tracking system coordinate frame;

receiving from the tracking system real-time information about the location and orientation of the camera;

referencing the virtual object mapping to transform to update the camera lens frame of reference based on the real-time information; and

updating the augmented scene by altering or distorting the one or more overlaid virtual items based on the updated the camera lens frame of reference.

13. The method of claim 8, wherein altering or distorting the one or more overlaid virtual items in the augmented scene includes shrinking the overlaid virtual items in response to the camera moving further away from a real object in the 3D space.

14. The method of claim 8, wherein altering or distorting the one or more overlaid virtual items in the augmented scene includes enlarging the overlaid virtual items in response to the camera moving closer to a real object in the 3D space.

15. The method of claim 8, wherein altering or distorting the one or more overlaid virtual items in the augmented scene includes rotating the overlaid virtual items in response to the camera moving at an angle relative to a real object in the 3D space.

16. A system, comprising:

a camera that captures a view of a real 3D space including a real object;

a tracking system that tracks the position and orientation in a real 3D space of the real object and of the camera, wherein the tracking system is configured to establish a tracking system coordinate frame associated with the tracking system, wherein the tracking system coordinate frame is aligned with the real 3D space; and

a computer coupled to the camera and the tracking system, the computer having one or more memory units, the computer being configured with a virtual modeler,

wherein the virtual modeler is configured to receive from the tracking system a first real object frame of reference for the real object, wherein the first real object frame of reference indicates a position and orientation of the real object relative to the tracking system coordinate frame;

wherein the virtual modeler is further configured to compute a second real object frame of reference for the real object, wherein the second real object frame of reference indicates a position and orientation of the real object relative to the tracking system coordinate frame;

wherein the virtual modeler is further configured to compute the second real object frame of reference by;

receiving or detecting three or more real object non-collinear points on the real object, wherein the location of three or more real object non-collinear points are defined relative to the tracking system coordinate frame;

determining a real object origin by calculating a centroid of the three or more real object non-collinear points; and

determining a real object orientation that is related to the orientation of the first real object frame of reference;

wherein the virtual modeler is further configured to receive from the one or more memory units a first virtual object frame of reference for a virtual object, wherein the virtual object is modeled after the real object, and wherein the first virtual object frame of reference is unrelated to the tracking system coordinate frame;

wherein the virtual modeler is further configured to compute a second virtual object frame of reference for the virtual object, wherein the second virtual object frame of reference indicates a position and orientation of the virtual object relative to the tracking system coordinate frame, and wherein the second real object frame of reference is aligned with the second virtual object frame of reference;

wherein the virtual modeler is further configured to compute the second virtual object frame of reference by;

receiving or indicating three or more virtual object non-collinear points on the virtual object, wherein the location of three or more virtual object non-collinear points are defined relative to the tracking system coordinate frame;

determining a virtual object origin by calculating a centroid of the three or more virtual object non-collinear points; and

determining a virtual object orientation;

wherein the virtual modeler is further configured to compute a virtual object mapping between the first virtual object frame of reference and the tracking system coordinate frame, and wherein the virtual object mapping includes a transform matrix to transform between the first virtual object frame of reference and the tracking system coordinate frame; and

wherein the virtual modeler is further configured to generate and store in the one or more memory units an augmented scene including a view of the real 3D space, a view of the real object and one or more overlaid virtual items, wherein the virtual object mapping is used to place the one or more overlaid virtual items in the augmented scene such that the one or more virtual items are aligned with the real object.

17. The system of claim 16, further comprising a display coupled to the computer, wherein the virtual modeler is further configured to communicate the augmented scene to the display, and wherein the display is configured to show the augmented scene to a user.

18. The system of claim 16, wherein using the virtual object mapping to place the one or more overlaid virtual items in the augmented scene includes:

receiving a first virtual item frame of reference for a first overlaid virtual item,

wherein the first virtual item frame of reference is unrelated to the tracking system coordinate frame;

referencing the virtual object mapping to transform the first virtual item frame of reference to a transformed frame of reference that relates to the tracking system coordinate frame; and

placing the first overlaid virtual item in the augmented scene using the transform frame of reference.

19. The method of claim 16, wherein the virtual modeler is further configured to:

receive from the tracking system real-time information about the location and orientation of the real object;

update the virtual object mapping based on the real-time information; and

update the augmented scene by updating the placement of the one or more overlaid virtual items such that the one or more virtual items remain aligned with the real object.

20. A data processing system, comprising:

one or more memory units that store computer code; and

one or more processor units coupled to the one or more memory units, wherein the one or more processor units execute the computer code stored in the one or more memory units to;

receive or establish a tracking system coordinate frame associated with an object tracking system,

wherein the tracking system coordinate frame is aligned with a real 3D space, and

wherein the tracking system tracks the position and orientation in a real 3D space of a camera that captures the real 3D space and a printed marker;

receive from the tracking system a camera frame of reference for the camera,

wherein the camera frame of reference indicates a position and orientation of the camera relative to the tracking system coordinate frame;

receive or establish a printed marker coordinate frame associated with the printed marker,

wherein the printed marker coordinate frame is aligned with the real 3D space, and

wherein the printed marker coordinate frame is aligned with the tracking system coordinate frame;

wherein receiving or establishing the printed marker coordinate frame includes;

receiving or detecting three or more printed marker non-collinear points on the printed marker, wherein the location of three or more printed marker non-collinear points are defined relative to the tracking system coordinate frame;

determining a printed marker origin by calculating a centroid of the three or more printed marker non-collinear points; and

determining a printed marker orientation that is related to the orientation of the printed marker coordinate frame;

determine a camera lens frame of reference for the lens of the camera,

wherein the camera lens frame of reference indicates a position and orientation of the camera lens relative to the printed marker coordinate frame;

determine a camera lens mapping between the camera frame of reference and the camera lens frame of reference, wherein the camera lens mapping includes a transform matrix to transform between the camera frame of reference and the camera lens frame of reference; and

display an augmented scene including a view of the real 3D space and one or more virtual items,

wherein the camera lens mapping is used to alter or distort the one or more virtual items in the augmented scene.