Method and apparatus for managing and controlling manned and automated utility vehicles
First Claim
1. A method of determining a coordinate position and rotational orientation of an object within a predefined coordinate space, the method comprising:
- a) providing a plurality of unique position markers having identifying indicia and positional reference indicia thereupon, the markers being arranged at predetermined known positional locations within the coordinate space, the known positional locations and known angular orientations being stored in a map, so that at least two position markers are always within view of the object;
b) using an image acquisition system mounted on the object, acquiring an image of the at least two position markers M1, M2 within view;
c) establishing a center point Ni, N2 of each respective marker M1, M2 and determining a line segment N1-N2 connecting the respective center points N1, N2;
d) determining a center point O of the field of view of the image acquisition system;
e) determining line segments O-N1, O-N2 respectively connecting point O with the centers N1, N2 of the respective position markers M1, M2;
f) using the lengths and directions of line segments O-N1, O-N2 to calculate the position of the vehicle relative to the known positions of the markers M1, M2, thereby determining the location of the vehicle within the coordinate space; and
g) using the direction of line segment N1-N2 to calculate the rotational orientation of the vehicle within the coordinate space.
9 Assignments
0 Petitions
Accused Products
Abstract
A method and apparatus for managing manned and automated utility vehicles, and for picking up and delivering objects by automated vehicles. A machine vision image acquisition apparatus determines the position and the rotational orientation of vehicles in a predefined coordinate space by acquiring an image of one or more position markers and processing the acquired image to calculate the vehicle'"'"'s position and rotational orientation based on processed image data. The position of the vehicle is determined in two dimensions. Rotational orientation (heading) is determined in the plane of motion. An improved method of position and rotational orientation is presented. Based upon the determined position and rotational orientation of the vehicles stored in a map of the coordinate space, a vehicle controller, implemented as part of a computer, controls the automated vehicles through motion and steering commands, and communicates with the manned vehicle operators by transmitting control messages to each operator.
216 Citations
40 Claims
-
1. A method of determining a coordinate position and rotational orientation of an object within a predefined coordinate space, the method comprising:
-
a) providing a plurality of unique position markers having identifying indicia and positional reference indicia thereupon, the markers being arranged at predetermined known positional locations within the coordinate space, the known positional locations and known angular orientations being stored in a map, so that at least two position markers are always within view of the object; b) using an image acquisition system mounted on the object, acquiring an image of the at least two position markers M1, M2 within view; c) establishing a center point Ni, N2 of each respective marker M1, M2 and determining a line segment N1-N2 connecting the respective center points N1, N2; d) determining a center point O of the field of view of the image acquisition system; e) determining line segments O-N1, O-N2 respectively connecting point O with the centers N1, N2 of the respective position markers M1, M2; f) using the lengths and directions of line segments O-N1, O-N2 to calculate the position of the vehicle relative to the known positions of the markers M1, M2, thereby determining the location of the vehicle within the coordinate space; and g) using the direction of line segment N1-N2 to calculate the rotational orientation of the vehicle within the coordinate space. - View Dependent Claims (2, 3)
-
-
4. A method of determining a coordinate position and rotational orientation of an object within a predefined coordinate space, the method comprising:
-
a) providing a plurality of unique position markers having identifying indicia and positional reference indicia thereupon, the markers being arranged at predetermined known positional locations within the coordinate space, the known positional locations and known angular orientations being stored in a map, the markers being spaced so that a plurality of position markers M1 M2, . . . , Mx are always within view of the object; b) using an image acquisition system mounted on the object, acquiring an image of the plurality of position markers M1, M2, . . . , Mx within a field of view of the image acquisition system; c) establishing a center point N1, N2, . . . , Nx of each respective marker M1, M2, . . . , Mx and determining line segments N1-N2, N1-N3, . . . , N1-Nx, . . . , N(x−
1)-Nx connecting the respective pairs of center points;d) determining a center point O of the field of view of the image acquisition system; e) determining line segments O-N1, O-N2, . . . , O-Nx respectively connecting point O with the centers N1, N2, . . . , Nx of the respective position markers M1, M2, . . . , Mx; f) using the lengths and directions of line segments O-N1, O-N2, . . . , O-Nx to calculate the position of the vehicle relative to each of the known positions of the markers M1, M2, . . . , Mx; g) calculating a mean value of the position of the vehicle within the coordinate space, thereby determining the location of the vehicle within the coordinate space; h) using the directions of line segments N1-N2, N1-N3, . . . , N1-Nx, . . . , N(x−
1)-Nx within the field of view to calculate the rotational orientation of the vehicle relative to the respective pairs M1, M2;
M1, M3;
. . . ;
M1, Mx;
. . . ;
M(x−
1), Mx of position markers M1, M2, . . . , Mx; andi) calculating a mean value of the rotational orientation of the vehicle relative to the respective pairs of position markers M1, M2, . . . , Mx;
thereby determining the rotational orientation of the vehicle within the coordinate space. - View Dependent Claims (5, 6, 7, 8)
-
-
9. An apparatus useful for determining a coordinate position and rotational orientation of an object within a predefined coordinate space, the apparatus comprising:
-
a) a plurality of unique position markers, each comprising a machine-readable code, arranged in predetermined positional locations within the coordinate space such that at least two position markers are always within view of the object; b) an image acquisition system, comprised of a machine vision system, the machine vision system comprising a camera, an optional light source, and image capture electronics, mounted on the object, for acquiring an image of the position markers within view; c) an image processing system for processing pixels in the acquired image to determine the identity of each position marker, the position of each position marker relative to the vehicle, and the rotational orientation of each position marker relative to the vehicle; d) a computer unit for calculating the position of the vehicle and the rotational orientation of the vehicle in the coordinate space; e) a system controller for receiving the object position and the rotational orientation of the object from the computer unit, and for transmitting object position and the rotational orientation to a host system, the controller having a memory for storing; i) a predetermined map of the coordinate space, and ii) the object position and rotational orientation. - View Dependent Claims (10)
-
-
11. A method for optically navigating an automated vehicle within a predefined coordinate space, the method comprising:
-
a) creating a map of the coordinate space, the map determining allowable travel routes and locations of obstacles within the coordinate space, and storing the map within a memory in a vehicle controller; b) establishing a destination for each automated vehicle within the coordinate space and storing the identity and destination of each vehicle within the memory in the vehicle controller; c) determining the coordinate position and the rotational orientation of each vehicle within the predefined coordinate space by; i) providing a plurality of unique position markers having identifying indicia, positional reference and angular reference indicia thereupon, the markers being arranged at predetermined known positional locations and known angular orientations within the coordinate space, the known positional locations and known angular orientations being stored in the map, so that at least one position marker is within view of the vehicle; ii) using an image acquisition system mounted on each vehicle; 1) acquiring an image of the at least one position marker within view; 2) processing the image to determine the identity, the position relative to the vehicle, and the rotational orientation relative to the vehicle of each position marker within view; and 3) calculating the position of the vehicle and the rotational orientation of the vehicle in the coordinate space and storing the position and rotational orientation information in a memory in the image acquisition system; d) transmitting the vehicle identity and the stored coordinate position and rotational orientation of that vehicle from the image acquisition system on each vehicle to the vehicle controller and storing the vehicle identity and coordinate position and rotational orientation in the memory within the vehicle controller; e) using the coordinate position and rotational orientation, the predetermined map of the coordinate space, and the destination stored within the memory in the vehicle controller, determining a desired path for each automated vehicle; f) transmitting motion and steering instructions to each automated vehicle; g) repeating steps c) through f) until each automated vehicle reaches the established destination for that vehicle. - View Dependent Claims (12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24)
-
-
25. An apparatus useful for optically navigating an automated vehicle within a predefined coordinate space, the apparatus comprising:
-
a) a plurality of unique position markers, each comprising a machine-readable code, arranged in predetermined positional locations within the coordinate space such that at least one position marker is within view of the automated vehicle; b) an image acquisition system, comprised of a machine vision system, the machine vision system comprising a camera, an optional light source, and image capture electronics, mounted on the vehicle, for acquiring an image of the position markers within view; c) an image processing system for processing pixels in the acquired image to determine the identity of each position marker, the position of each position marker relative to the vehicle, and the rotational orientation of each position marker relative to the vehicle; d) a computer unit for calculating the position of the vehicle and the rotational orientation of the vehicle in the coordinate space; and e) a vehicle controller for receiving destinations for the vehicle from an input, for receiving the vehicle position and the rotational orientation of the vehicle from the computer unit, and for transmitting motion and steering instructions to the automated vehicle, the controller having a memory for storing; i) a predetermined map of the coordinate space, ii) the vehicle position and rotational orientation, iii) the destination of the vehicle, the vehicle controller determining a desired path for the automated vehicle and for transmitting motion and steering instructions to the automated vehicle.
-
-
26. A method for picking up an object from a first, present, location and rotational orientation, transporting the object and delivering that object to a second, destination, location and rotational orientation within a predefined coordinate space by an optically navigated automated vehicle, the method comprising:
-
a) creating a map of the coordinate space, the map determining allowable travel routes and locations of obstacles within the coordinate space, and storing the map within a memory in a vehicle controller; b) identifying the object to be transported, the present location and rotational orientation of that object and a destination location and rotational orientation of that object within the coordinate space and storing the identity, the present location and rotational orientation and the destination location and rotational orientation of the object within the memory in the vehicle controller; c) designating an automated vehicle as the delivery vehicle for the transport and delivery; d) determining a coordinate position and a rotational orientation of the delivery vehicle and all other vehicles within the predefined coordinate space by; i) providing a plurality of unique position markers having identifying indicia, positional reference and angular reference indicia thereupon, the markers being arranged at predetermined known positional locations and known angular orientations within the coordinate space, the known positional locations and known angular orientations being stored in the map, so that at least one position marker is within view of the vehicle; ii) using an image acquisition system mounted on the delivery vehicle; 1) acquiring an image of the at least one position marker within view; 2) processing the image to determine the identity, the position relative to the delivery vehicle, and the rotational orientation relative to the delivery vehicle of each position marker within view; and 3) calculating the position of the delivery vehicle, the rotational orientation of the delivery vehicle, the positions of all other vehicles and the rotational orientation of all other vehicles in the coordinate space and storing the position and rotational orientation information in a memory in the image acquisition system; e) transmitting the delivery vehicle identity and the stored coordinate position and rotational orientation of that vehicle from the image acquisition system on the delivery vehicle to the vehicle controller and storing the delivery vehicle identity and coordinate position and rotational orientation in the memory within the controller, and transmitting the manned vehicle identities and the stored coordinate position and rotational orientation of the manned vehicles from the image acquisition system on each respective vehicle to the vehicle controller and storing the respective vehicle identities and coordinate positions and rotational orientations in the memory within the controller; f) using the predetermined map of the coordinate space, the identity, position location and rotational orientation of the object and the present position location and rotational orientation of the designated delivery vehicle stored within the memory in the vehicle controller, determining a desired path for the delivery vehicle to pick up the object; g) transmitting motion and steering instructions to the delivery vehicle; h) repeating steps d) through g) until the delivery vehicle reaches the location of the object at the rotational orientation of the object; i) transmitting motion, steering and fork control instructions to the delivery vehicle, causing the vehicle to pick up the object; j) using the predetermined map of the coordinate space, the present position location and rotational orientation of the designated delivery vehicle and the destination location and rotational orientation for that object stored within the memory in the vehicle controller, determining a desired path for the delivery vehicle to deliver the object; k) transmitting motion and steering instructions to the delivery vehicle to causing it to follow the desired path; l) repeating steps d), j) and k) until the delivery vehicle reaches the destination location and the destination rotational orientation of the object; m) transmitting motion, steering and fork control instructions to the delivery vehicle, causing the vehicle to deposit the object at the destination location; n) calculating the actual position and rotational orientation of the object when it has been deposited; o) transmitting the actual position and rotational orientation of the delivered object to a host system. - View Dependent Claims (27, 28, 29, 30, 31)
-
-
32. An apparatus useful for controlling an optically navigated automated vehicle within a predefined coordinate space for picking up an object from a first location and rotational orientation, transporting the object and delivering that object to a second, destination, location and rotational orientation, the apparatus comprising:
-
a) a plurality of unique position markers, each comprising a machine-readable code, arranged in predetermined positional locations within the coordinate space such that at least one position marker is within view of the automated vehicle; b) an image acquisition system, comprised of a machine vision system, the machine vision system comprising a camera, an optional light source, and image capture electronics, mounted on the vehicle, for acquiring an image of the position markers within view; c) an image processing system for processing pixels in the acquired image to determine the identity of each position marker, the position of each position marker relative to the vehicle, and the rotational orientation of each position marker relative to the vehicle; d) a computer unit for calculating the position of the vehicle and the rotational orientation of the vehicle in the coordinate space; and e) a vehicle controller for receiving destinations for the vehicle from an input, for receiving the vehicle position and the rotational orientation of the vehicle from the computer unit, and for transmitting motion and steering instructions to the automated vehicle, the controller having a memory for storing; i) a predetermined map of the coordinate space, ii) the vehicle position and rotational orientation, iii) the destination of the vehicle, f) a wireless data communication network for transmitting data between the computer unit and the vehicle controller; the vehicle controller;
determining a desired path for the automated vehicle to acquire the object at the first location and rotational orientation;
transmitting motion and steering instructions to the automated vehicle until the automated vehicle reaches the location and rotational orientation of the object;
transmitting fork control instructions to the automated vehicle, causing the automated vehicle to acquire the object;
determining a desired path for the automated vehicle to the destination of the object;
transmitting motion and steering instructions to the automated vehicle until the automated vehicle reaches the destination location and rotational orientation for the object;
transmitting fork control instructions to the automated vehicle, causing the automated vehicle to deposit the object at the destination location and rotational orientation.
-
-
33. A method of managing utility vehicles within a predefined coordinate space, where the vehicles may be manned (i.e., human operated) or automated, i.e., guided by machines by determining a coordinate position and rotational orientation of the vehicles and by navigating the automated vehicles, the method comprising:
-
a) creating a map of the coordinate space, the map determining allowable travel routes and locations of obstacles within the coordinate space, and storing the map within a memory in a vehicle controller; b) establishing a destination for each manned vehicle and each automated vehicle within the coordinate space and storing the identity and destination of each vehicle within the memory in the vehicle controller; c) determining the coordinate position and the rotational orientation of each vehicle within the predefined coordinate space by; i) providing a plurality of unique position markers having identifying indicia, positional reference and angular reference indicia thereupon, the markers being arranged at predetermined known positional locations and known angular orientations within the coordinate space, the known positional locations and known angular orientations being stored in the map, so that at least one position marker is within view of the vehicle; ii) using an image acquisition system mounted on each (manned and automated) vehicle; 1) acquiring an image of the at least one position marker within view; 2) processing the image to determine the identity, the position relative to the vehicle, and the rotational orientation relative to the vehicle of each position marker within view; and 3) calculating the coordinate position of the vehicle and the rotational orientation of the vehicle in the coordinate space and storing the coordinate position and rotational orientation information in a memory in the image acquisition system; d) transmitting the vehicle identity and the stored coordinate position and rotational orientation of that vehicle from the image acquisition system on each vehicle to the vehicle controller and storing the vehicle identity and coordinate position and rotational orientation in the memory within the vehicle controller; e) using the coordinate position of the vehicle and the rotational orientation of the vehicle, the predetermined map of the coordinate space and the destination for each vehicle stored within the memory in the vehicle controller, determining a desired path for each automated vehicle; f) transmitting motion and steering instructions to each automated vehicle; g) determining a predicted trajectory and a safety zone for each manned vehicle and each automated vehicle by calculating the velocity and direction of travel of each vehicle from coordinate positions at successive time intervals; h) determining any areas of intersection of the safety zone of each manned vehicle and each automated vehicle with the safety zones of other manned and OF other automated vehicles to predict a potential collision; i) transmitting instructions to reduce speed, turn, or stop, to any automated vehicle that has a safety zone intersecting any safety zone of any other vehicle to prevent the predicted collision; j) transmitting a warning to any manned vehicle that has a safety zone intersecting any safety zone of any other vehicle to alert the operator of such manned vehicle of a predicted potential collision, so that the operator can take appropriate action to avoid the predicted collision; and k) repeating steps c) through j) until each automated vehicle reaches the established destination for that vehicle. - View Dependent Claims (34, 35, 36)
-
-
37. A method of managing a mixed environment of manned vehicles and automated vehicles within a predefined coordinate space by determining a coordinate position and rotational orientation of manned vehicles and by optically navigating automated vehicles within the predefined coordinate space, the method comprising:
-
a) creating a map of the coordinate space, the map determining allowable travel routes and locations of obstacles within the coordinate space, and storing the map within a memory in a vehicle controller; b) establishing a destination for each manned vehicle and each automated vehicle within the coordinate space and storing the identity and destination of each vehicle within the memory in the vehicle controller; c) determining the coordinate position and the rotational orientation of each vehicle within the predefined coordinate space by; i) providing a plurality of unique position markers having identifying indicia, positional reference and angular reference indicia thereupon, the markers being arranged at predetermined known positional locations and known angular orientations within the coordinate space, the known positional locations and known angular orientations being stored in the map, so that at least two position markers are within view of the vehicle; ii) using an image acquisition system mounted on each manned and automated vehicle; 1) acquiring an image of the at least two position markers within view; 2) processing the image to determine the identity, the position relative to the vehicle, and the rotational orientation relative to the vehicle of each position marker within view; and 3) calculating the coordinate position of the vehicle and the rotational orientation of the vehicle in the coordinate space; wherein the coordinate position and rotational orientation of each vehicle is determined by; A) acquiring an image of at least two position markers M1, M2 within view; B) establishing a center point N1, N2 of each respective marker M1, M2 and determining a line segment N1-N2 connecting the respective center points N1, N2; C) determining a center point O of the field of view of the image acquisition system; D) determining line segments O-N1, O-N2 respectively connecting point O with the centers N1, N2 of the respective position markers M1, M2; E) using the lengths and directions of line segments O-N1, O-N2 to calculate the position of the vehicle relative to the known positions of the markers M1, M2, thereby determining the location of the vehicle within the coordinate space; and F) using the direction of line segment N1-N2 to calculate the rotational orientation of the vehicle within the coordinate space; and
storing the coordinate location and the rotational orientation of the vehicle in a memory in the image acquisition system;d) transmitting the vehicle identity and the stored coordinate position and rotational orientation of that vehicle from the image acquisition system on each vehicle to the vehicle controller and storing the vehicle identity and coordinate position and rotational orientation in the memory within the vehicle controller; e) using the coordinate position and rotational orientation of each vehicle, the predetermined map of the coordinate space and the destination for each vehicle stored within the memory in the vehicle controller, determining a desired path for each automated vehicle; f) transmitting motion and steering instructions to each automated vehicle; g) determining a predicted trajectory and a safety zone for each manned vehicle and each automated vehicle by calculating the velocity and direction of travel of each vehicle from coordinate positions at successive time intervals; h) determining any areas of intersection of the safety zone of each manned vehicle and each automated vehicle with the safety zones of other manned or automated vehicles to predict a potential collision; i) transmitting instructions to reduce speed, turn, or stop to any automated vehicle that has a safety zone intersecting any safety zone of any other vehicle to prevent the predicted collision; j) transmitting a warning to any manned vehicles that has a safety zone intersecting any safety zone of any other vehicle to alert the operators of such manned vehicles of a predicted potential collision, so that the operators can take appropriate action to avoid the predicted collision; and k) repeating steps c) through j) until each automated vehicle reaches the established destination for that vehicle. - View Dependent Claims (38, 39)
i) transmitting instructions to an automatically vehicle previously slowed or stopped to resume its previous velocity and direction of travel; or if no areas of intersection between a manned and another manned vehicle are subsequently determined; j) transmitting a signal canceling the warning to the manned vehicles to alert the operators that no potential collision is predicted.
-
-
39. The method of claim 37 further comprising a method of rerouting an automated vehicle within a coordinate space, the method comprising:
-
after determining a desired path for each automated vehicle, for each iteration of step c); c1) checking to determine if the desired path is blocked by a stationary vehicle; c2) if the desired path is blocked, selecting an alternate path; and c3) issuing new motion and steering instructions to each automated vehicle to follow the alternate path.
-
-
40. An apparatus useful for managing a mixed environment of manned vehicles and automated vehicles within a predefined coordinate space by determining a coordinate position and rotational orientation of each manned vehicle and by optically navigating each automated vehicle within the predefined coordinate space, the apparatus comprising:
-
a) a plurality of unique position markers, each comprising a machine-readable code, arranged in predetermined positional locations within the coordinate space such that at least one position marker is within view of the automated vehicle; b) an image acquisition system, comprised of a machine vision system, the machine vision system comprising a camera, an optional light source, and image capture electronics, mounted on each vehicle, for acquiring an image of the position markers within view; c) an image processing system for processing pixels in the acquired image to determine the identity of each position marker, the position of each position marker relative to the vehicle, and the rotational orientation of each position marker relative to each vehicle; d) a computer unit on each vehicle for calculating the position of that vehicle and the rotational orientation of that vehicle in the coordinate space; and e) a vehicle controller for receiving destinations for each automated vehicle from an input, for receiving the vehicle position and the rotational orientation of each vehicle from the computer unit on that vehicle, and for transmitting motion and steering instructions to each automated vehicle, the controller having a memory for storing; i) a predetermined map of the coordinate space, ii) the vehicle position and rotational orientation of each vehicle, iii) the destination of each automated vehicle, f) a wireless data communication network for transmitting data between the computer unit and the vehicle controller; the vehicle controller;
determining allowable travel routes and locations of obstacles within the coordinate space from the predetermined map;
determining a desired path for each automated vehicle using the allowable travel routes;
transmitting motion and steering instructions to the automated vehicle until each automated vehicle reaches its destination;
determining a predicted trajectory and a safety zone for each manned vehicle and each automated vehicle by calculating the velocity and direction of travel of each vehicle from coordinate positions at successive time intervals;
determining any areas of intersection of the safety zone of each manned vehicle and each automated vehicle with the safety zones of other manned or automated vehicles to predict a potential collision;
transmitting instructions to reduce speed, turn, or stop to any automated vehicle that has a safety zone intersecting any safety zone of any other vehicle to prevent the predicted collision; and
transmitting a warning to any manned vehicles that has a safety zone intersecting any safety zone of any other vehicle to alert the operators of such manned vehicles of a predicted potential collision, so that the operators can take appropriate action to avoid the predicted collision.
-
Specification