Image processing device, image processing method, and a computer-readable non-transitory medium

• US 9,542,003 B2
• Filed: 07/19/2013
• Issued: 01/10/2017
• Est. Priority Date: 08/31/2012
• Status: Active Grant

• Current Assignee: Fujitsu Limited
• Original Assignee: Fujitsu Limited
• Inventors: Hara, Nobuyuki, Fujimoto, Katsuhito, Murase, Taichi, Moteki, Atsunori
• Primary Examiner(s): Bali, Vikkram
• Assistant Examiner(s): Mangialaschi, Tracy
• Application Number:
  US13/946,546
• Publication Number:
  US 20140064557A1
• Time in Patent Office: 1,271 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883

Abstract

An image processing method includes recognizing a first object that is included in the image data, calculating a positional variation amount of a feature point that corresponds to the first object and is moving in an image center direction so as to calculate a moving speed of the first object on the basis of the positional variation amount, determining whether or not the first object is a gaze target object of the user, in accordance with a behavior of the first object, the behavior being obtained on the basis of the positional variation amount of the first object among the plurality of image data, of which acquisition time is respectively different and whether or not the object continuingly exists in a second region, the second region being positioned in an inside of the first region and including a center point of the image data, for a predetermined period of time.

Claims

1. An image processing device configured to execute a process in accordance with a type of gesture performed by a head of a user equipped with a camera, the image processing device, comprising:

a memory configured to store;

feature point information to separately identify at least two objects,

user motion association information in which the type of gesture and a combination of a moving direction of an object, a moving speed of the object, and a motion vector corresponding to the object are associated with each other, and

display information indicating a relationship between the type of gesture and collateral information indicating detail information of a plurality of objects existing around the user; and

a processor coupled to the memory and configured to;

extract a plurality of feature points positioned within a predetermined peripheral region in an image from a plurality of images captured by the user wearing the camera at different timings,

specify an object corresponding to the plurality of feature points by referring to the feature point information,

determine whether the object moving from the predetermined peripheral region to a central region positioned in an inside of the predetermined peripheral region and a coordinate variation of the object after reaching the central region remains within a predetermined range,

determine that the object is a target object indicating a gaze target of the user, in response to determining that the object is moving from the predetermined peripheral region to the central region and that the coordinate variation of the object remains within the predetermined range,

calculate a combination of a moving direction of the target object, a moving speed of the target object, and a motion vector corresponding to the target object, by using a coordinate variation of the target object before reaching the central region, on the plurality of images,

extract the type of gesture corresponding to the calculated combination from the user motion association information,

extract the collateral information of the target object corresponding to the extracted type of gesture from the display information, and

display the extracted collateral information in a manner to be superposed on the target object.

2. The image processing device according to claim 1, further comprising a glasses-like casing,

wherein the memory, the processor and the camera are disposed over the glasses-like casing.

3. The image processing device according to claim 2,

wherein the camera is disposed on a center between both eyes when wearing the glasses-like casing.

4. An image processing method executed by an image processing device, the image processing method comprising:

storing feature point information to separately identify at least two objects;

storing user motion association information in which a type of gesture performed by a head of a user and a combination of a moving direction of an object, a moving speed of the object, and a motion vector corresponding to the object are associated with each other;

storing display information indicating a relationship between the type of gesture and collateral information indicating detail information of a plurality of objects existing around the user;

extracting a plurality of feature points positioned within a predetermined peripheral region in an image from a plurality of images captured by the user wearing a camera at different timings;

specifying an object corresponding to the plurality of feature points by referring to the feature point information;

determining whether the object moving from the predetermined peripheral region to a central region positioned in an inside of the predetermined peripheral region and a coordinate variation of the object after reaching the central region remains within a predetermined range;

determining that the object is a target object indicating a gaze target of the user, in response to determining that the object is moving from the predetermined peripheral region to the central region and that the coordinate variation of the object remains within the predetermined range;

calculating a combination of a moving direction of the target object, a moving speed of the target object, and a motion vector corresponding to the target object, by using a coordinate variation of the target object before reaching the central region, on the plurality of images;

extracting the type of gesture corresponding to the calculated combination from the user motion association information;

extracting the collateral information of the target object corresponding to the extracted type of gesture from the display information; and

displaying the extracted collateral information in a manner to be superposed on the target object.

5. The image processing method according to claim 4, wherein the calculating of the motion vector of the calculated combination includes:

calculating a plurality of variation amounts of one or more feature points for each of acquiring intervals of the plurality of images;

obtaining a plurality of averaged variation amounts by averaging the plurality of variation amounts which are calculated for each of the acquiring intervals;

calculating a number of frames, based on an average speed of the target object and a predetermined number of frames obtained when a positional relation between the user and the target object is invariant;

calculating the motion vector by averaging the plurality of averaged variation amounts by the calculated number of frames.

6. The image processing method according to claim 5, wherein the calculating of the number of frames includes:

calculating a product of a constant value and the average speed of the target object; and

calculating a difference between the predetermined number of frames and the calculated product.

7. The image processing method according to claim 5, further comprising:

determining whether the plurality of averaged variation amounts satisfies both of a start condition and an end condition of an outward way of motion of the user;

obtaining a cumulative value of the plurality of averaged variation amounts from a start image to an end image of the outward way among the plurality of acquired images, and

wherein the specifying a head motion of the user corresponding to the calculated combination includes specifying the head motion of the user based on the cumulative value and the user motion information.

8. The image processing method according to claim 5, further comprising:

setting an upper limit of the plurality of variation amounts; and

determining whether the plurality of variation amounts is valid based on the upper limit.

9. The image processing method according to claim 5, wherein

parts within a predetermined length from upper and lower ends of each of the plurality of images are set as the predetermined peripheral region when the plurality of feature points are moving vertically;

and

parts within a predetermined length from the right and left ends of each of the plurality of images are set as the predetermined peripheral region when an object is moving horizontally.

10. The image processing method according to claim 5, further comprising

determining that the coordinate variation of the object after reaching the central region remains within the predetermined range, when a coordinate of the object remains in the central region for a predetermined number of frames continuingly after the coordinate of the object enters the central region.

11. The image processing method according to claim 4, wherein

the combination stored in the user motion association information further includes a tolerance range of the motion vector corresponding to the object, and

the extracting of the type of the gesture includes acquiring, from the user motion association information, a motion vector similar to the calculated motion vector, based on the tolerance range of the motion vector corresponding to the object.

12. The image processing method according to claim 4,

wherein the central region and the predetermined peripheral region are separated by an intermediate region within the image.

13. The image processing method according to claim 4,

wherein the user motion association information are tables which are different for every moving direction of an object and every moving speed of the object.