AUGMENTED REALITY (AR) SYSTEM AND METHOD FOR TRACKING PARTS AND VISUALLY CUEING A USER TO IDENTIFY AND LOCATE PARTS IN A SCENE
First Claim
Patent Images
1. An augmented reality (AR) system, comprising:
- a camera for recording images within a field-of-view (FOV) over a larger field-of-regard (FOR) in a scene including parts;
a display;
one or more memory units;
templates for the parts stored in said one or more memory units;
a plurality of instructions stored in said one or more memory units, each said instruction including a list of parts and instructions for the use of those parts; and
a computer comprising;
a classifier module that processes images of the scene using the templates to locate and identify the parts in the scene over the FOR, said classified parts'"'"' identities and locations stored in said one or more memory units;
a pose module that processes data from images of the scene to estimate camera pose;
an instruction module that retrieves an instruction;
a tracker module that for each instruction determines from the estimated camera pose the list of parts and the locations of those parts that lie within the current FOV and those parts that lie outside the current FOV;
a scene-rendering module that displays the image of the scene within the current FOV on the display;
a parts rendering module that for each instruction renders parts icons registered to parts within the current FOV and renders parts icons with location indicators for parts outside the current FOV over the displayed images of the scene; and
an instruction-rendering module that renders the instructions for the use of those parts.
1 Assignment
0 Petitions
Accused Products
Abstract
An AR system both identifies and visually tracks parts for a user by maintaining spatial awareness of the user'"'"'s pose and provides instructions to the user for the use of those parts. Tracking the identified parts, both inside and outside the current Field of View (FOV), and any missing parts for use with the current instruction improves the effectiveness and efficiency of both novice and experienced user alike.
-
Citations
20 Claims
-
1. An augmented reality (AR) system, comprising:
-
a camera for recording images within a field-of-view (FOV) over a larger field-of-regard (FOR) in a scene including parts; a display; one or more memory units; templates for the parts stored in said one or more memory units; a plurality of instructions stored in said one or more memory units, each said instruction including a list of parts and instructions for the use of those parts; and a computer comprising; a classifier module that processes images of the scene using the templates to locate and identify the parts in the scene over the FOR, said classified parts'"'"' identities and locations stored in said one or more memory units; a pose module that processes data from images of the scene to estimate camera pose; an instruction module that retrieves an instruction; a tracker module that for each instruction determines from the estimated camera pose the list of parts and the locations of those parts that lie within the current FOV and those parts that lie outside the current FOV; a scene-rendering module that displays the image of the scene within the current FOV on the display; a parts rendering module that for each instruction renders parts icons registered to parts within the current FOV and renders parts icons with location indicators for parts outside the current FOV over the displayed images of the scene; and an instruction-rendering module that renders the instructions for the use of those parts. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13)
-
-
14. An augmented reality (AR) system for retrofitting a vehicle with a kit of parts including mechanical parts and tools, comprising:
-
a camera for recording images within a field-of-view (FOV) over a larger field-of-regard (FOR) in a scene including the vehicle and parts from the kit; a display; one or more memory units; templates for the parts stored in said one or more memory units; a plurality of instructions stored in said one or more memory units, each said instruction including a list of parts and instructions for assembling or disassembling those parts from the vehicle; and a computer comprising; a classifier module that processes images of the scene using the templates to locate and identify the parts in the scene over the FOR, said classified parts'"'"' identities and locations stored in said one or more memory units, said classifier module comprising a fiducial marker classifier that detects fiducial markers on parts and provides the part'"'"'s identity and location and a parts classifier that uses the stored templates to detect parts and provides the part'"'"'s identity and location; a pose module that processes data from images of the scene to estimate camera pose, said pose module comprising at least two sources selected from the fiducial marker classifier, said classifier detecting fiducial markers on non-part scene features and providing the scene features identity and location in addition to the parts identity and locations; the parts classifier that provides identities and locations of parts; a visual scene feature classifier that identifies and locates points of interest in the scene; and an inertial sensor that estimates camera motion; and a camera pose estimator that processes inputs from at least two said sources and generates the estimated camera pose; an instruction module that retrieves an instruction; a tracker module that for each instruction determines from the estimated camera pose the list of parts and the locations of those parts that lie within the current FOV and those parts that lie outside the current FOV; a scene-rendering module that displays the image of the scene within the current FOV on the display; a parts rendering module that for each instruction renders parts icons registered to parts within the current FOV and renders parts icons with location indicators for parts outside the current FOV over the displayed images of the scene; and an instruction-rendering module that renders the instructions for the use of those parts on the display registered to the vehicle.
-
-
15. A method of augmented reality (AR), comprising:
-
processing images of a scene from different Field-of-View (FOV) to locate and identify parts in the scene over a larger Field-of-Regard (FOR) and to store the parts'"'"' identities and locations; processing images of the scene from a current FOV to estimate a pose; and for each of a sequence of instructions, each said instruction including a list of parts and instructions for the use of those parts, using the estimated pose to determine the parts that lie within the current FOV and the parts that lie outside the current FOV from the list of parts associated with the execution of that instruction; rendering parts icons registered to the parts within the current FOV and rendering parts icons with location indicators for the parts outside the current FOV over a displayed image of the scene from the current FOV; and rendering the instructions for the use of those parts. - View Dependent Claims (16, 17, 18, 19, 20)
-
Specification