Image processing device and method

• US 9,213,897 B2
• Filed: 10/12/2012
• Issued: 12/15/2015
• Est. Priority Date: 12/07/2011
• Status: Active Grant

• Current Assignee: Fujitsu Limited
• Original Assignee: Fujitsu Limited
• Inventors: Nakai, Takehiro, Moteki, Atsunori, Fujimoto, Katsuhito, Hara, Nobuyuki, Ozawa, Noriaki, Murase, Taichi
• Primary Examiner(s): Yentrapati, Avinash
• Application Number:
  US13/650,711
• Publication Number:
  US 20130148849A1
• Time in Patent Office: 1,159 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06V 10/255 Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 10/443 by matching or filtering
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 30/142 using hand-held instruments; Constructional details of the instruments

Abstract

An image processing device that accesses a storage unit that stores a feature point of a recognition-target object, the device includes an obtaining unit mounted with a user and configured to obtain image data in a direction of a field of view of the user; a recognizing unit configured to recognize the recognition-target object included in the image data by extracting a feature point from the image data and associating the extracted feature point and the feature point of the recognition-target object stored in the storage unit with each other; a calculating unit configured to calculate a location change amount of the feature point corresponding to the recognition-target object recognized by the recognizing unit from a plurality of the image data obtained at different times and calculate a motion vector of the recognition-target object from the location change amount; and a determining unit configured to determine a movement.

Claims

1. An image processing device that accesses a storage that stores a feature point of a recognition-target object, the device comprising:

a camera wearable by a user and when worn by the user, configured to obtain image data in a direction of a field of view of the user;

a processor; and

a memory which stores a plurality of instructions, which when executed by the processor, cause the processor to execute;

generating a model histogram;

recognizing the recognition-target object in the image data by extracting a feature point from the image data and associating the extracted feature point with the stored feature point of the recognition-target object,

calculating a location change amount of the feature point of the recognition-target object from a plurality of the image data obtained at different times,

calculating a motion vector of the recognition-target object from the location change amount,

determining a movement of the user based on the calculated motion vector and a user movement correspondence table in which motion vectors and user movements are in an opposite direction of the motion vectors respectively;

dividing the user movement into an outward path and a return path at a point where the motion vector is switched from a positive direction to a negative direction;

calculating a histogram of the motion vectors in consideration of the outward path and the return path; and

determining the movement of the user by comparing the histogram of the motion vectors to the model histogram in consideration of the outward path and the return path.

2. The image processing device according to claim 1,

wherein an object the user is gazing at via the image processing device is used in calculation of a location change amount of the recognition-target object or an area change amount of the recognition-target object from the plurality of image data.

3. The image processing device according to claim 1,

wherein the recognizing specifies a location of the recognition-target object included in the image data from the feature point of the recognition-target object and the calculating calculates the location change amount of the feature point and a location change amount of at least four endpoints containing the recognition-target object the location of which is specified.

4. The image processing device according to claim 1,

wherein the storage stores a model histogram in which a movement of the user and a motion vector are associated with each other, and the determining determines the movement of the user by comparing similarity between a histogram of motion vectors generated by accumulating a predetermined amount of motion vectors calculated and the model histogram.

5. The image processing device according to claim 1,

wherein the storage stores auxiliary information regarding the recognition-target object associated with the movement of the user.

6. The image processing device according to claim 5, wherein the image processing device displays the auxiliary information regarding the recognition-target object associated with the movement of the user determined.

7. An image processing method comprising:

generating a model histogram;

obtaining, from a device wearable by a user, image data in a direction of a field of view of the user when the device is worn by the user;

extracting a feature point from the image data;

reading a stored feature point of a recognition-target object stored in a storage of the device;

recognizing an object included in the image data by associating the extracted feature point with the stored feature point of the recognition-target object;

calculating, by a processor, a location change amount of the feature point of the recognized recognition-target object from a plurality of the image data obtained at different times and calculating a motion vector of the recognition-target object from the location change amount;

determining a movement of the user based on the calculated motion vector and a user movement correspondence table in which motion vectors and user movements are in an opposite direction of the motion vectors respectively;

dividing the user movement into an outward path and a return path at a point where the motion vector is switched from a positive direction to a negative direction;

calculating a histogram of the motion vectors in consideration of the outward path and the return path; and

determining the movement of the user by comparing the histogram of the motion vectors to the model histogram in consideration of the outward path and the return path.

8. The image processing method according to claim 7,

wherein, in the recognizing, a location change amount of the recognition-target object or an area change amount of the recognition-target object is calculated from the plurality of image data and, based on the location change amount or the area change amount, an object the user is gazing at is recognized.

9. The image processing method according to claim 7,

wherein, in the calculating, a location of the recognition-target object included in the image data is specified from the feature point of the recognition-target object, and a location change amount of at least four endpoints containing the recognition-target object the location of which is specified is calculated as the location change amount of the feature point.

10. The image processing method according to claim 7,

wherein, in the determining, the movement of the user is determined by comparing similarity between a model histogram in which a movement of the user and a motion vector are associated with each other and a histogram of motion vectors generated by accumulating a predetermined amount of motion vectors calculated in the calculating.

11. The image processing method according to claim 7,

wherein the storage stores auxiliary information regarding the recognition-target object associated with the movement of the user.

12. The image processing method according to claim 11, further comprising:

displaying the auxiliary information regarding the recognition-target object associated with the movement of the user determined in the determining.

13. A non-transitory computer-readable storage medium storing an image processing program that causes a computer to execute a process comprising:

generating a model histogram;

obtaining, from a device wearable by a user, image data in a direction of a field of view of the user when the device is worn by the user;

extracting a feature point from the image data;

reading a stored feature point of a recognition-target object stored in a storage;

recognizing an object included in the image data by associating the extracted feature point and the stored feature point of the recognition-target object;

calculating a location change amount at the feature point of the recognized recognition-target object from a plurality of the image data obtained at different times and calculating a motion vector of the recognition-target object from the location change amount; and

determining a movement of the user based on the calculated motion vector and a user movement correspondence table in which motion vectors and user movements are in an opposite direction of the motion vectors respectively;

dividing the user movement into an outward path and a return path at a point where the motion vector is switched from a positive direction to a negative direction;

calculating a histogram of the motion vectors in consideration of the outward path and the return path; and

determining the movement of the user by comparing the histogram of the motion vectors to the model histogram in consideration of the outward path and the return path.

14. The non-transitory computer-readable storage medium according to claim 13,

wherein, in the recognizing, a location change amount of the recognition-target object or an area change amount of the recognition-target object is calculated from the plurality of image data and, based on the location change amount or the area change amount, an object the user is gazing at is recognized.

15. The non-transitory computer-readable storage medium according to claim 13,

wherein, in the calculating, a location of the recognition-target object included in the image data is specified from the feature point of the recognition-target object, and a location change amount of at least four endpoints containing the recognition-target object the location of which is specified is calculated as the location change amount of the feature point.

16. The non-transitory computer-readable storage medium according to claim 13,

wherein, in the determining, the movement of the user is determined by comparing similarity between a model histogram in which a movement of the user and a motion vector are associated with each other and a histogram of motion vectors generated by accumulating a predetermined amount of motion vectors calculated in the calculating.

17. The non-transitory computer-readable storage medium according to claim 13,

wherein the storage stores auxiliary information regarding the recognition-target object associated with the movement of the user.

18. The non-transitory computer-readable medium according to claim 17, causes the computer to execute the process comprising:

displaying the auxiliary information regarding the recognition-target object associated with the movement of the user determined in the determining.

19. An image processing device that accesses a storage that stores a feature point of a recognition-target object, the device comprising:

a camera wearable by a user and when worn by the user, configured to obtain image data in a direction of a field of view of the user;

a processor; and

a memory which stores a plurality of instructions, which when executed by the processor, cause the processor to,

generate a model histogram;

recognize the recognition-target object included in the image data by extracting a feature point from the image data and associating the extracted feature point and a stored feature point of the recognition-target object,

calculate a location change amount of the feature point corresponding to the recognition-target object recognized by the recognizing from a plurality of the image data obtained at different times,

calculate a motion vector of the recognition-target object from the location change amount,

determine a movement of the user based on the calculated motion vector and a user movement correspondence table in which motion vectors and user movements are in an opposite direction of the motion vectors respectively;

divide the user movement into an outward path and a return path at a point where the motion vector is switched from a positive direction to a negative direction;

calculate a histogram of the motion vectors in consideration of the outward path and the return path; and

determine the movement of the user by comparing the histogram of the motion vectors to the model histogram in consideration of the outward path and the return path.