Operation estimating apparatus and related article of manufacture

• US 8,077,970 B2
• Filed: 12/04/2007
• Issued: 12/13/2011
• Est. Priority Date: 12/04/2006
• Status: Active Grant

• Current Assignee: DENSO Corporation
• Original Assignee: DENSO Corporation
• Inventors: Ishikawa, Takahiro
• Primary Examiner(s): Le, Brian Q
• Assistant Examiner(s): Park, Edward
• Application Number:
  US11/987,800
• Publication Number:
  US 20080130953A1
• Time in Patent Office: 1,470 Days
• Field of Search: 382/103, 382/104, 382/106, 382/107, 382/100, 382/153, 382/181, 382/231
• US Class Current: 382/181
• CPC Class Codes: G06V 10/245 by locating a pattern; Special marks for positioning
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition

Abstract

An operation estimating apparatus includes an image obtaining unit, a human body feature point specifying unit, and an operation estimating unit. The image obtaining unit repeatedly obtains images. The human body feature point specifying unit specifies a predetermined human body feature point of an operator in each of the images. The operation estimating unit estimates one of operations based on the human body feature points. The operation estimating unit compares an actual posture locus of the operator with a transitional estimation model for each of the operations by the operator to obtain a degree of approximation of the transitional estimation model to the actual posture locus. The operation estimating unit estimates that the operator is going to perform the one of the operations that corresponds to an estimated posture locus of the transitional estimation model having the degree of approximation that satisfies a predetermined threshold.

Claims

1. An operation estimating apparatus comprising:

an image obtaining means for repeatedly obtaining images, each of the images having a plurality of operational objects and an operator that is positioned to be able to perform operations on the plurality of operational objects;

a human body feature point specifying means for specifying a predetermined human body feature point of the operator in each of the images that are repeatedly obtained by the image obtaining means; and

an operation estimating means for estimating one of the operations, which the operator is going to perform, based on the human body feature points specified by the human body feature point specifying means in the images, wherein;

the operation estimating means compares an actual posture locus of the operator with a transitional estimation model for each of the operations by the operator to obtain a degree of approximation of the transitional estimation model to the actual posture locus, the transitional estimation model for the each of the operations being formed based on an estimated posture locus of the each of the operations, the operator being estimated to track the estimated posture locus to operate one of the plurality of operational objects that corresponds to the estimated posture locus, the actual posture locus of the operator being obtained based on the human body feature points specified by the human body feature point specifying means in the images;

the operation estimating means estimates that the operator is going to perform the one of the operations that corresponds to the estimated posture locus of the transitional estimation model having the degree of approximation that satisfies a predetermined threshold;

the operation estimating means compares the actual posture locus with (a) the transitional estimation model for the each of the operations, and with (b) a stationary estimation model for each of the plurality of operational objects to obtain the degree of approximation of the transitional estimation model to the actual posture locus and a degree of approximation of the stationary estimation model to the actual posture locus, the stationary estimation model for the each of the plurality of operational objects being formed based on an estimated posture point of the operator, the operator being estimated to take the estimated posture point to continuously operate one of the plurality of operational objects that corresponds to the estimated posture point; and

the operation estimating means estimates either;

(a) that the operator is going to perform the one of the operations that corresponds to the estimated posture locus of the transitional estimation model having the degree of approximation that satisfies a predetermined threshold;

or

(b) that the operator is operating the one of the plurality of operational objects that corresponds to the estimated posture point of the stationary estimation model having the degree of approximation that satisfies a predetermined threshold.

2. The operation estimating apparatus according to claim 1, further comprising:

an operation supporting means for supporting the operation on a corresponding one of the plurality of operational objects that corresponds to a result of estimation by the operation estimating means.

3. The operation estimating apparatus according to claim 2, wherein:

the operation supporting means turns on a lamp adapted for illuminating the corresponding one of the plurality of operational objects that corresponds to the result of estimation by the operation estimating means such that the operation supporting means illuminates the corresponding one of the plurality of operational objects.

4. The operation estimating apparatus according to claim 2, wherein:

the plurality of operational objects includes an enclosing operational object, which has a cover, and which encloses a matter by opening and closing the cover, the enclosing operational object having an opening/closing mechanism that opens and closes the cover; and

the operation supporting means controls the opening/closing mechanism of the enclosing operational object to open the cover such that the operation supporting means supports the operation on the enclosing operational object when the corresponding one of the plurality of operational objects that corresponds to the result of estimation by the operation estimating means is the enclosing operational object.

5. The operation estimating apparatus according to claim 1, wherein:

the operation estimating apparatus is mounted on a vehicle;

the plurality of operational objects includes at least one operational object of the vehicle; and

the operator exists in a seat of the vehicle.

6. The operation estimating apparatus according to claim 1, wherein the operation estimating means reads the transitional estimation model for the each of the operations stored in a storage, and compares the actual posture locus with the read transitional estimation model for the each of the operations.

7. The operation estimating apparatus according to claim 1, wherein the operation estimating means reads the stationary estimation model for the each of the plurality of operational objects stored in a storage, and compares the actual posture locus with the read stationary estimation model for the each of the plurality of operational objects.

8. An article of manufacture comprising:

a non-transitory computer readable medium readable by a computer system; and

program instructions carried by the non-transitory computer readable medium for causing the computer system to execute various procedures as all elements of the operation estimating apparatus according to claim 1.

9. An operation estimating apparatus comprising:

an image obtaining means for repeatedly obtaining images, each of the images having a plurality of operational objects and an operator that is positioned to be able to perform operations on the plurality of operational objects;

a human body feature point specifying means for specifying a predetermined human body feature point of the operator in each of the images that are repeatedly obtained by the image obtaining means;

an operation estimating means for estimating one of the operations, which the operator is going to perform, based on the human body feature points specified by the human body feature point specifying means in the images; and

an operation supporting means for supporting the operation on a corresponding one of the plurality of operational objects that corresponds to a result of estimation by the operation estimating means, wherein;

the operation estimating means compares an actual posture locus of the operator with a transitional estimation model for each of the operations by the operator to obtain a degree of approximation of the transitional estimation model to the actual posture locus, the transitional estimation model for the each of the operations being formed based on an estimated posture locus of the each of the operations, the operator being estimated to track the estimated posture locus to operate one of the plurality of operational objects that corresponds to the estimated posture locus, the actual posture locus of the operator being obtained based on the human body feature points specified by the human body feature point specifying means in the images;

the operation estimating means estimates that the operator is going to perform the one of the operations that corresponds to the estimated posture locus of the transitional estimation model having the degree of approximation that satisfies a predetermined threshold;

the plurality of operational objects includes an enclosing operational object, which has a cover, and which encloses a matter by opening and closing the cover, the enclosing operational object having an opening/closing mechanism that opens and closes the cover; and

the operation supporting means controls the opening/closing mechanism of the enclosing operational object to open the cover such that the operation supporting means supports the operation on the enclosing operational object when the corresponding one of the plurality of operational objects that corresponds to the result of estimation by the operation estimating means is the enclosing operational object; and

the operation supporting means controls the opening/closing mechanism of the enclosing operational object to close the cover when the corresponding one of the plurality of operational objects that corresponds to the result of estimation by the operation estimating means is different from the enclosing operational object and simultaneously when the cover of the enclosing operational object has already been opened.

10. An operation estimating apparatus comprising:

an image obtaining means for repeatedly obtaining images, each of the images having a plurality of operational objects and an operator that is positioned to be able to perform operations on the plurality of operational objects;

a human body feature point specifying means for specifying a predetermined human body feature point of the operator in each of the images that are repeatedly obtained by the image obtaining means;

an operation estimating means for estimating one of the operations, which the operator is going to perform, based on the human body feature points specified by the human body feature point specifying means in the images;

an approach detecting means for determining whether or not there is a matter that relatively approaches the vehicle; and

a first interruption notifying means for sending a notification to stop operating the corresponding one of the plurality of operational objects when the approach detecting means determines that the matter approaches the vehicle in a case, where the corresponding one of the plurality of operational objects that corresponds to the result of estimation by the operation estimating means is different from the steering wheel and the shift lever, wherein;

the operation estimating means compares an actual posture locus of the operator with a transitional estimation model for each of the operations by the operator to obtain a degree of approximation of the transitional estimation model to the actual posture locus, the transitional estimation model for the each of the operations being formed based on an estimated posture locus of the each of the operations, the operator being estimated to track the estimated posture locus to operate one of the plurality of operational objects that corresponds to the estimated posture locus, the actual posture locus of the operator being obtained based on the human body feature points specified by the human body feature point specifying means in the images;

the operation estimating means estimates that the operator is going to perform the one of the operations that corresponds to the estimated posture locus of the transitional estimation model having the degree of approximation that satisfies a predetermined threshold

the operation estimating apparatus is mounted on a vehicle;

the plurality of operational objects includes at least one operational object of the vehicle;

the operator exists in a seat of the vehicle; and

the at least one operational object includes a steering wheel and a shift lever of the vehicle.

11. An operation estimating apparatus comprising:

an image obtaining means for repeatedly obtaining images, each of the images having a plurality of operational objects and an operator that is positioned to be able to perform operations on the plurality of operational objects;

a human body feature point specifying means for specifying a predetermined human body feature point of the operator in each of the images that are repeatedly obtained by the image obtaining means; and

an operation estimating means for estimating one of the operations, which the operator is going to perform, based on the human body feature points specified by the human body feature point specifying means in the images;

a travel state detecting means for detecting a travel state of the vehicle;

a traffic information obtaining means for obtaining information about a traffic condition around the vehicle; and

a second interruption notifying means for sending a notification to stop operating the corresponding one of the plurality of operational objects when the travel state detected by the travel state detecting means indicates that the vehicle stops and simultaneously when the information obtained by the traffic information obtaining means indicates the traffic condition that requires the vehicle to start in a case, where the corresponding one of the plurality of operational objects that corresponds to the result of estimation by the operation estimating means is different from the steering wheel and the shift lever, wherein;

the operation estimating means compares an actual posture locus of the operator with a transitional estimation model for each of the operations by the operator to obtain a degree of approximation of the transitional estimation model to the actual posture locus, the transitional estimation model for the each of the operations being formed based on an estimated posture locus of the each of the operations, the operator being estimated to track the estimated posture locus to operate one of the plurality of operational objects that corresponds to the estimated posture locus, the actual posture locus of the operator being obtained based on the human body feature points specified by the human body feature point specifying means in the images;

the operation estimating means estimates that the operator is going to perform the one of the operations that corresponds to the estimated posture locus of the transitional estimation model having the degree of approximation that satisfies a predetermined threshold;

the operation estimating apparatus is mounted on a vehicle;

the plurality of operational objects includes at least one operational object of the vehicle;

the operator exists in a seat of the vehicle; and

the at least one operational object includes a steering wheel and a shift lever of the vehicle.

12. An operation estimating apparatus comprising:

an image obtaining means for repeatedly obtaining images, each of the images having a plurality of operational objects and an operator that is positioned to be able to perform operations on the plurality of operational objects;

a human body feature point specifying means for specifying a predetermined human body feature point of the operator in each of the images that are repeatedly obtained by the image obtaining means;

an operation estimating means for estimating one of the operations, which the operator is going to perform, based on the human body feature points specified by the human body feature point specifying means in the images;

an operation state monitoring means for detecting an operational state of at least one of the plurality of operational objects to determine that the at least one of the plurality of operational objects is operated; and

a stationary estimation model correcting means for correcting a stationary estimation model that corresponds to the at least one of the plurality of operational objects in accordance with an actual posture point at a time when the operation state monitoring means determines that the at least one of the plurality of operational objects is operated, the actual posture point being obtained based on the human body feature point specified by the human body feature point specifying means, the stationary estimation model being formed based on an estimated posture point of the operator, the operator being estimated to take the estimated posture point to continuously operate the at least one of the plurality of operational objects, wherein;

the operation estimating means compares an actual posture locus of the operator with a transitional estimation model for each of the operations by the operator to obtain a degree of approximation of the transitional estimation model to the actual posture locus, the transitional estimation model for the each of the operations being formed based on an estimated posture locus of the each of the operations, the operator being estimated to track the estimated posture locus to operate one of the plurality of operational objects that corresponds to the estimated posture locus, the actual posture locus of the operator being obtained based on the human body feature points specified by the human body feature point specifying means in the images; and

the operation estimating means estimates that the operator is going to perform the one of the operations that corresponds to the estimated posture locus of the transitional estimation model having the degree of approximation that satisfies a predetermined threshold.

13. The operation estimating apparatus according to claim 12, further comprising:

a stationary point storing means for storing the human body feature point into the storage every time the operation state monitoring means determines that the at least one of the plurality of operational objects is operated such that the stationary point storing means stores a plurality of human body feature points in the storage, the human body feature point being specified by the human body feature point specifying means at a time, when the operation state monitoring means determines that the at least one of the plurality of operational objects is operated, wherein;

the stationary estimation model correcting means corrects the stationary estimation model that corresponds to the at least one of the plurality of operational objects based on the plurality of human body feature points stored in the storage when a predetermined correction condition is satisfied.

14. The operation estimating apparatus according to claim 13, wherein:

the stationary estimation model correcting means calculates an averaged actual posture point based on a predetermined number of actual posture points, each of which is obtained based on a corresponding one of the plurality of human body feature points stored in the storage by the stationary point storing means; and

the stationary estimation model correcting means updates the stationary estimation model, which corresponds to the at least one of the plurality of operational objects, to make the stationary estimation model into an estimation model, which is formed by the averaged actual posture point.

15. The operation estimating apparatus according to claim 14, wherein:

when a number of the plurality of human body feature points stored in the storage by the stationary point storing means is less than the predetermined number, the stationary estimation model correcting means compensates a shortfall using a prepared human body feature point that corresponds to the at least one of the plurality of operational objects, to calculate the averaged actual posture point; and

the prepared human body feature point of the at least one of the plurality of operational objects is selected among a plurality of prepared human body feature points that are correspondingly prepared for the plurality of operational objects.

16. An operation estimating apparatus comprising:

an image obtaining means for repeatedly obtaining images, each of the images having a plurality of operational objects and an operator that is positioned to be able to perform operations on the plurality of operational objects;

a human body feature point specifying means for specifying a predetermined human body feature point of the operator in each of the images that are repeatedly obtained by the image obtaining means; and

an operation estimating means for estimating one of the operations, which the operator is going to perform, based on the human body feature points specified by the human body feature point specifying means in the images, wherein;

the operation estimating means compares an actual posture locus of the operator with a transitional estimation model for each of the operations by the operator to obtain a degree of approximation of the transitional estimation model to the actual posture locus, the transitional estimation model for the each of the operations being formed based on an estimated posture locus of the each of the operations, the operator being estimated to track the estimated posture locus to operate one of the plurality of operational objects that corresponds to the estimated posture locus, the actual posture locus of the operator being obtained based on the human body feature points specified by the human body feature point specifying means in the images;

the operation estimating means estimates that the operator is going to perform the one of the operations that corresponds to the estimated posture locus of the transitional estimation model having the degree of approximation that satisfies a predetermined threshold;

the transitional estimation model includes an estimation model that is formed based on the estimated posture locus tracked by the operator that performs a transitional operation as one of the operations; and

the operator finishes operating a certain operational object of the plurality of operational objects, and then, starts operating another operational object of the plurality of operational objects in the transitional operation, the operation estimating apparatus further comprising;

an operation state monitoring means for detecting an operational state of at least one of the plurality of operational objects to determine that the at least one of the plurality of operation objects is operated; and

a transitional estimation model correcting means for correcting the transitional estimation model that corresponds to the transitional operation based on the actual posture locus that is obtained based on the human body feature points specified by the human body feature point specifying means specified during an interval between (a) a time, at which the operation state monitoring means becomes unable to detect the operation on the certain operational object, which has been detected to be operated, and (b) a time, at which the operation state monitoring means detects the operation on the another operational object.

17. The operation estimating apparatus according to claim 16, further comprising:

a transitional locus storing means for storing in the storage a human body feature point group made of the human body feature points specified by the human body feature point specifying means during the interval, wherein;

the transitional estimation model correcting means corrects the transitional estimation model that corresponds to the transitional operation based on the actual posture locus obtained based on the human body feature point group stored in the storage when a predetermined correction condition is satisfied.

18. The operation estimating apparatus according to claim 17, wherein:

the transitional estimation model correcting means calculates an averaged actual posture locus based on a predetermined number of actual posture loci, each of which is obtained based on a corresponding one of the human body feature point groups stored in the storage; and

the transitional estimation model correcting means updates the transitional estimation model that corresponds to the transitional operation to make the transitional estimation model into an estimation model, which is formed based on the averaged actual posture locus.

19. The operation estimating apparatus according to claim 18, wherein:

when a number of the human body feature point groups stored in the storage is less than the predetermined number, the transitional estimation model correcting means compensates a shortfall using a prepared human body feature point group that corresponds to the transitional operation to calculate the averaged actual posture locus; and

the prepared human body feature point group of the transitional operation is selected among a plurality of prepared human body feature point groups that are correspondingly prepared for the operations.