IMAGE PROCESSING DEVICE, IMAGE PROCESSING METHOD, AND A COMPUTER-READABLE NON-TRANSITORY MEDIUM

• US 20140064557A1
• Filed: 07/19/2013
• Published: 03/06/2014
• Est. Priority Date: 08/31/2012
• Status: Active Grant

• Current Assignee: Fujitsu Limited
• Original Assignee: Fujitsu Limited
• Inventors: HARA, Nobuyuki, FUJIMOTO, Katsuhito, MURASE, Taichi, MOTEKI, Atsunori
• Granted Patent: US 9,542,003 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883

Abstract

An image processing method includes recognizing a first object that is included in the image data, calculating a positional variation amount of a feature point that corresponds to the first object and is moving in an image center direction so as to calculate a moving speed of the first object on the basis of the positional variation amount, determining whether or not the first object is a gaze target object of the user, in accordance with a behavior of the first object, the behavior being obtained on the basis of the positional variation amount of the first object among the plurality of image data, of which acquisition time is respectively different and whether or not the object continuingly exists in a second region, the second region being positioned in an inside of the first region and including a center point of the image data, for a predetermined period of time.

Claims

1. An image processing device, comprising:

an acquisition unit configured to acquire a plurality of image data, of which acquisition time is respectively different, in a sight direction of a user;

a storage unit configured to store data related to a feature point of an object;

an object recognition unit that extracts data corresponding to the feature point of the object from a first region in the image data and associates the data that is extracted with the data related to the feature point of the object, the data related to the feature point being stored in the storage unit, so as to recognize a first object that is included in the image data;

a movement estimation unit that calculates a positional variation amount of the data related to the feature point, the data corresponding to the first object and being moving in an image center direction, on the basis of the plurality of image data and calculates a moving speed of the first object on the basis of the positional variation amount; and

a gaze determination unit that determines whether or not the first object is a gaze target object of the user, in accordance with a behavior of the first object, the behavior being obtained on the basis of the positional variation amount of the first object among the plurality of image data, and whether or not the object continuingly exists in a second region, the second region being positioned in an inside of the first region and including a center point of the image data, for a predetermined period of time.

2. An image processing device according to claim 1 the device further comprising:

a motion amount calculation unit that calculates a positional variation amount of the feature point that corresponds to the first object on the basis of the plurality of image data, of which acquisition time is respectively different, when the first object is determined as the gaze target object of the user; and

a motion determination unit that estimates time that is taken for a motion of the user on the basis of the moving speed and the positional variation amount of the first object and determines a motion of the user on the basis of the time that is estimated.

3. The image processing device according to claim 2, wherein the movement estimation unit estimates a moving speed of the first object on the basis of a positional variation amount and a moving direction of a feature point, the feature point corresponding to the first object and being moving in an outside of the second region including the center point of the image data toward an image center direction, among feature points that correspond to the first object.

4. The image processing device according to claim 2, further comprising:

a storage unit that stores a user motion association table that is associated with a moving speed of the gaze target object;

wherein

the motion determination unit selects the user motion association table on the basis of the moving speed and the moving direction of the first object and refers to the user motion association table that is selected, so as to determine a motion of the user on the basis of the moving speed and the moving direction of the first object and a time average of motion vectors.

5. The image processing device according to claim 2, wherein

the motion amount calculation unit accumulates the motion vectors among frames, of which acquisition time is respectively different, in a predetermined amount so as to generate a temporal change pattern of the motion vectors,

a model temporal change pattern, in which a motion of the user and a motion vector are associated with each other, is preliminarily stored in the storage unit for every moving direction of the object and every moving speed of the object, and

the motion determination unit selects the model temporal change pattern on the basis of the moving speed and the moving direction of the first object and compares a degree of similarity with respect to a temporal change pattern of the motion vector so as to determine a motion of the user.

6. An image processing method, comprising:

acquiring image data in a sight direction of a user;

extracting a feature point from a first region in the image data and associating the feature point that is extracted with a feature point of an object, the feature point of the object being stored in a storage unit, so as to recognize a first object that is included in the image data;

calculating a positional variation amount of a feature point that corresponds to the first object and is moving in an image center direction, among feature points corresponding to the first object, on the basis of a plurality of image data, of which acquisition time is respectively different, so as to calculate a moving speed of the first object on the basis of the positional variation amount; and

determining whether or not the first object is a gaze target object of the user, in accordance with a behavior of the first object, the behavior being obtained on the basis of the positional variation amount of the first object among the plurality of image data, of which acquisition time is respectively different, and whether or not the object continuingly exists in a second region, the second region being positioned in an inside of the first region and including a center point of the image data, for a predetermined period of time.

7. An image processing method according to claim 6, the method further comprising:

calculating a positional variation amount of the feature point that corresponds to the first object on the basis of the plurality of image data, of which acquisition time is respectively different, when the first object is determined as the gaze target object of the user; and

estimating time that is taken for a motion of the user on the basis of the moving speed and the positional variation amount of the first object so as to determine a motion of the user on the basis of the time that is estimated.

8. A computer-readable, non-transitory medium storing therein an image processing program that causes a computer to execute a procedure, the procedure comprising:

acquiring image data in a sight direction of a user;

extracting a feature point from a first region in the image data and associating the feature point that is extracted with a feature point of an object, the feature point of the object being stored in a storage unit, so as to recognize a first object that is included in the image data;

calculating a positional variation amount of a feature point that corresponds to the first object and is moving in an image center direction, among feature points corresponding to the first object, on the basis of a plurality of image data, of which acquisition time is respectively different, so as to calculate a moving speed of the first object on the basis of the positional variation amount;

determining whether or not the first object is a gaze target object of the user, in accordance with a behavior of the first object, the behavior being obtained on the basis of the positional variation amount of the first object among the plurality of image data, of which acquisition time is respectively different, and whether or not the object continuingly exists in a predetermined region, the predetermined region being positioned in an inside of the first region and including a center point of the image data, for a predetermined period of time;

calculating a positional variation amount of a feature point that corresponds to the first object on the basis of the plurality of image data, of which acquisition time is respectively different, when the first object is determined as the gaze target object of the user; and

estimating time that is taken for a motion of the user on the basis of the moving speed and the positional variation amount of the first object so as to determine a motion of the user on the basis of the time that is estimated.