Information processing apparatus and control method and control program thereof, and communication terminal and control method and control program thereof

• US 9,418,314 B2
• Filed: 01/25/2013
• Issued: 08/16/2016
• Est. Priority Date: 01/30/2012
• Status: Active Grant

• Current Assignee: NEC Corporation
• Original Assignee: NEC Corporation
• Inventors: Nomura, Toshiyuki, Yamada, Akio, Iwamoto, Kota, Mase, Ryota
• Primary Examiner(s): Mehta, Bhavesh
• Assistant Examiner(s): Mistry, Oneal R
• Application Number:
  US14/375,452
• Publication Number:
  US 20150003704A1
• Time in Patent Office: 1,299 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06V 10/42 Global feature extraction by analysis of the whole pattern, e.g. using frequency domain transformations or autocorrelation
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 10/758 Involving statistics of pixels or of feature values, e.g. histogram matching
• CPC Class Codes: G06V 20/66 Trinkets, e.g. shirt buttons or jewellery items recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 2201/034 of medical instruments
• CPC Class Codes: G16H 30/20 for handling medical images, e.g. DICOM, HL7 or PACS
• CPC Class Codes: G16H 40/20 for the management or administration of healthcare resources or facilities, e.g. managing hospital staff or surgery rooms
• CPC Class Codes: G16H 40/67 for remote operation

Abstract

A medical article and m-number of first local features which are respectively constituted by a feature vector of 1 dimension to i dimensions of m-number of feature points in an image of the medical article are stored in association with each other, n-number of feature points are extracted from an image in a captured video, n-number of second local features respectively constituted by a feature vector of 1 dimension to j dimensions are generated, a smaller number of dimensions among the number of dimensions i and the number of dimensions j is selected, and an existence of the medical article in the image in the video is recognized when it is determined that a prescribed ratio or more of the m-number of first local features up to the selected number of dimensions corresponds to the n-number of second local features up to the selected number of dimensions.

Claims

1. An information processing system, comprising:

first local feature storing unit that stores, in association with each other, a medical article and m-number of first local features which are respectively constituted by a feature vector of 1 dimension to i dimensions and which are generated for each of m-number of local areas including each of m-number of feature points in an image of the medical article;

second local feature generating unit that extracts n-number of feature points from an image of a video captured by imaging unit, and that generates n-number of second local features respectively constituted by a feature vector of 1 dimension to j dimensions for n-number of local areas including each of the n-number of feature points; and

recognizing unit that selects a smaller number of dimensions among the number of dimensions i of the feature vector of the first local feature and the number of dimensions j of the feature vector of the second local feature, and that recognizes that the medical article exists in the image in the video when determining that a prescribed ratio or more of the m-number of first local features constituted by a feature vector up to the selected number of dimensions corresponds to the n-number of second local features constituted by a feature vector up to the selected number of dimensions.

2. The information processing system according to claim 1, further comprising informing unit that informs a recognition result obtained from the recognizing unit.

3. The information processing system according to claim 2, comprising a communication terminal carried by a user, and an information processing apparatus that communicates with the communication terminal, wherein

the communication terminal includes the imaging unit, the second local feature generating unit, and the informing unit, and the n-number of second local features are transmitted from the communication terminal to the information processing apparatus, and

the information processing apparatus includes the first local feature storing unit and the recognizing unit, and the recognition result obtained from the recognizing unit is transmitted from the information processing apparatus to the communication terminal.

4. The information processing system according to claim 1, wherein

the first local feature storing unit stores the m-number of first local features generated from images of a plurality of medical articles in association with each of the medical articles, and

the recognizing unit recognizes a plurality of medical articles included in the image captured by the imaging unit, and includes arrangement determining unit that determines an arrangement of the plurality of medical articles in the image captured by the imaging unit based on an alignment of the n-number of second local features.

5. The information processing system according to claim 4, wherein

the medical article is a medical device and an image captured by the imaging unit is of an examination room, a hospital room, or an operation room,

the arrangement determining unit recognizes an arrangement of the medical device in the examination room, the hospital room, or the operation room,

the second local feature generating unit includes accuracy adjusting unit that adjusts accuracy of the second local feature, and

the recognizing unit further recognizes a mistake, a defect, or a state of the medical device based on a second local feature generated by the second local feature generating unit by adjusting to a higher accuracy.

6. The information processing system according to claim 4, wherein

the medical article is a medical instrument and an image captured by the imaging unit is of a tray on which the medical instrument is arranged,

the arrangement determining unit recognizes an arrangement of the medical instrument on the tray,

the second local feature generating unit includes accuracy adjusting unit that adjusts accuracy of the second local feature, and

the recognizing unit further recognizes a mistake, a defect, or a state of the medical instrument based on a second local feature generated by the second local feature generating unit by adjusting to a higher accuracy.

7. The information processing system according to claim 4, wherein

the medical article is a pharmaceutical product and an image captured by the imaging unit is of a medicine shelf or a medicine tray,

the arrangement determining unit recognizes an arrangement of the pharmaceutical product on the medicine shelf or the medicine tray,

the second local feature generating unit includes accuracy adjusting unit that adjusts accuracy of the second local feature, and

the recognizing unit further recognizes a mistake, a defect, or a state of the pharmaceutical product based on a second local feature generated by the second local feature generating unit by adjusting to a higher accuracy.

8. The information processing system according to claim 7, further comprising managing unit that performs inventory based on an arrangement of the plurality of pharmaceutical products recognized by the arrangement determining unit.

9. The information processing system according to claim 1, wherein the first local feature and the second local feature are each generated by dividing a local area including a feature point extracted from an image into a plurality of sub-areas and generating a feature vector of a plurality of dimensions constituted by a histogram in a gradient direction in the plurality of sub-areas.

10. The information processing system according to claim 9, wherein the first local feature and the second local feature are each generated by selecting a dimension at which a correlation between adjacent sub-areas is lower among the generated feature vector of a plurality of dimensions.

11. The information processing system according to claim 9, wherein the plurality of dimensions of the feature vector are arranged to circle the local area once for every predetermined number of dimensions so that dimensions can be selected starting from a first dimension in a descending order of contributions to the feature point and in accordance with an increase in accuracy that is required with respect to the local feature.

12. The information processing system according to claim 11, wherein the second local feature generating unit generates the second local feature corresponding to a correlation of the medical articles so that the second local feature with a larger number of dimensions is generated for a medical article that has a higher correlation with another medical article.

13. The information processing system according to claim 11, wherein the first local feature storing unit stores the first local feature corresponding to a correlation of the medical articles so that the first local feature with a larger number of dimensions is stored for a medical article that has a higher correlation with another medical article.

14. An information processing method in an information processing system including first local feature storing unit that stores, in association with each other, a medical article and m-number of first local features which are respectively constituted by a feature vector of 1 dimension to i dimensions and which are generated for each of m-number of local areas including each of m-number of feature points in an image of the medical article, the method comprising the steps of:

extracting n-number of feature points from an image in a captured video and generating n-number of second local features respectively constituted by a feature vector of 1 dimension to j dimensions for n-number of local areas including each of the n-number of feature points; and

selecting a smaller number of dimensions among the number of dimensions i of the feature vector of the first local feature and the number of dimensions j of the feature vector of the second local feature, and recognizing that the medical article exists in the image in the video when determining that a prescribed ratio or more of the m-number of first local features constituted by a feature vector up to the selected number of dimensions corresponds to the n-number of second local features constituted by a feature vector up to the selected number of dimensions.

15. An information processing apparatus, comprising:

first local feature storing unit that stores, in association with each other, a medical article and m-number of first local features which are respectively constituted by a feature vector of 1 dimension to i dimensions and which are generated for each of m-number of local areas including each of m-number of feature points in an image of the medical article;

second receiving unit that extracts n-number of feature points from an image of a video captured by a communication terminal and that receives, from the communication terminal, n-number of second local features respectively constituted by a feature vector of 1 dimension to j dimensions for n-number of local areas including each of the n-number of feature points;

recognizing unit that selects a smaller number of dimensions among the number of dimensions i of the feature vector of the first local feature and the number of dimensions j of the feature vector of the second local feature, and that recognizes that the medical article exists in the image in the video when determining that a prescribed ratio or more of the m-number of first local features constituted by a feature vector up to the selected number of dimensions corresponds to the n-number of second local features constituted by a feature vector up to the selected number of dimensions; and

second transmitting unit that transmits information indicating the recognized medical article to the communication terminal.

16. A control method of an information processing apparatus including first local feature storing unit that stores, in association with each other, a medical article and m-number of first local features which are respectively constituted by a feature vector of 1 dimension to i dimensions and which are generated for each of m-number of local areas including each of m-number of feature points in an image of the medical article, the method comprising the steps of:

extracting n-number of feature points from an image of a video captured by a communication terminal and receiving, from the communication terminal, n-number of second local features respectively constituted by a feature vector of 1 dimension to j dimensions for n-number of local areas including each of the n-number of feature points;

selecting a smaller number of dimensions among the number of dimensions i of the feature vector of the first local feature and the number of dimensions j of the feature vector of the second local feature, and recognizing that the medical article exists in the image in the video when determining that a prescribed ratio or more of the m-number of first local features constituted by a feature vector up to the selected number of dimensions corresponds to the n-number of second local features constituted by a feature vector up to the selected number of dimensions; and

transmitting information indicating the recognized medical article to the communication terminal.

17. A non-transitory computer-readable storage medium with a control program of an information processing apparatus including first local feature storing unit that stores, in association with each other, a medical article and m-number of first local features which are respectively constituted by a feature vector of 1 dimension to i dimensions and which are generated for each of m-number of local areas including each of m-number of feature points in an image of the medical article, the program causing a computer to execute the steps of:

extracting n-number of feature points from an image of a video captured by a communication terminal and receiving, from the communication terminal, n-number of second local features respectively constituted by a feature vector of 1 dimension to j dimensions for n-number of local areas including each of the n-number of feature points;

selecting a smaller number of dimensions among the number of dimensions i of the feature vector of the first local feature and the number of dimensions j of the feature vector of the second local feature, and recognizing that the medical article exists in the image in the video when determining that a prescribed ratio or more of the m-number of first local features constituted by a feature vector up to the selected number of dimensions corresponds to the n-number of second local features constituted by a feature vector up to the selected number of dimensions; and

transmitting information indicating the recognized medical article to the communication terminal.