Article management system, information processing apparatus, and control method and control program of information processing apparatus

• US 10,438,084 B2
• Filed: 02/17/2014
• Issued: 10/08/2019
• Est. Priority Date: 03/04/2013
• Status: Active Grant

• Current Assignee: NEC Corporation
• Original Assignee: NEC Corporation
• Inventors: Nomura, Toshiyuki, Iwamoto, Kota, Higa, Kyota, Ohashi, Keishi, Hattori, Wataru
• Primary Examiner(s): Vu, Kim Y
• Assistant Examiner(s): Vanchy, Jr., Michael J
• Application Number:
  US14/772,524
• Publication Number:
  US 20150379366A1
• Time in Patent Office: 2,059 Days
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 18/23 Clustering techniques
• CPC Class Codes: G06K 17/00 Methods or arrangements for effecting co-operative working between equipments covered by two or more of main groups G06K1/00 - G06K15/00, e.g. automatic card files incorporating conveying and reading operations
• CPC Class Codes: G06K 7/10366 the interrogation device being adapted for miscellaneous applications
• CPC Class Codes: G06Q 10/08 Logistics, e.g. warehousing, loading or distribution; Inventory or stock management
• CPC Class Codes: G06Q 10/087 Inventory or stock management, e.g. order filling, procurement or balancing against orders
• CPC Class Codes: G06Q 10/0875 Itemisation or classification of parts, supplies or services, e.g. bill of materials
• CPC Class Codes: G06T 2207/30242 Counting objects in image
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 2201/10 Recognition assisted with metadata

Abstract

An apparatus of this invention is directed to an information processing apparatus that effectively counts, on a type basis, articles of a plurality of types displayed in a depth direction on a display shelf. The information processing apparatus includes a display count acquirer that acquires display count information of articles using article presence/absence sensors provided on the display shelf on which the articles are placed, an article identifier that acquires article identification information capable of identifying the types of articles based on an image acquired by capturing the display shelf, and a display recognizer that recognizes, based on the display count information and the article identification information, display count of each type of the articles.

Claims

1. An information processing apparatus comprising:

a shelf allocation storage that stores a positional arrangement of sensor groups corresponding to a shelf allocation in a display shelf on which articles are placed, each sensor group comprising a plurality of article sensors configured to sense the presence, absence, or both of an article;

a memory storing instructions; and

at least one processor configured to process the instructions to;

acquire display count information of articles using the article sensors provided on the display;

identify types of articles, names of the articles, and an array of the articles based on an image acquired by capturing the display shelf, to generate a result of identifying;

recognize, by linking the display count information and the result of identifying in reference to the positional arrangement of the sensor groups stored in the shelf allocation storage, display count of each type of the articles;

compare the identified array of the articles and the recognized display count of each type of the articles with a history of article array, to generate a result of comparing; and

discriminate a change in article array from an arrangement error based on the result of comparing.

2. The information processing apparatus according to claim 1, wherein

the at least one processor is configured to acquire the display count information of articles by processing instructions to acquire the display count information of articles included in each article column on the display shelf, in which articles of the same type are arranged from a near side to a far side, using the article sensors, and

the at least one processor is configured to identify the types of the articles by processing instructions to identify the type of an article in a front of the article column based on the image obtained by capturing a front of the display shelf.

3. The information processing apparatus according to claim 1, wherein each of the article sensors comprises:

RFID tags, each having an identifier, which are placed at display positions of articles;

a response receiver that has an antenna for generating, in a region including the RFID tags, an electromagnetic field a strength of which is changed by presence of article, and receives response signals from the RFID tags; and

a presence/absence determiner that determines presence/absence of articles based on strengths of the response signals from the RFID tags.

4. The information processing apparatus according to claim 1, wherein the at least one processor is configured to identify the types of articles, the names of the articles, and the array of the articles by processing instructions to:

generate local features of a plurality of feature points from the image acquired by capturing the display shelf;

cluster the plurality of feature points based on coordinates of the plurality of feature points; and

collate the local features of the clustered feature points with a local feature generated based on an image of an article.

5. The information processing apparatus according to claim 1,

wherein a sheet identification mark unique to each of the sensor groups is attached to each of the sensor groups on the front of the display shelf, and

wherein the at least one processor is configured to:

acquire the display count information by further processing instructions to acquire the display count information of each article column from the plurality of article sensors together with information of the sheet identification mark;

identify the types of articles, the names of the articles, and the array of the articles by further processing instructions to identify the sheet identification mark included in the captured image and acquire the information of the sheet identification mark; and

recognize the display count by processing instructions to link the display count information and the result of identifying further using the information of the sheet identification mark.

6. The information processing apparatus according to claim 1, wherein a plurality of display shelves exist, and a unique shelf identification mark is attached to the front of each display shelf, and wherein the at least one processor is configured to:

identify the types of articles, the names of the articles, and the array of the articles by further processing instructions to recognize the shelf identification mark included in the captured image and acquire the information of the shelf identification mark; and

recognize the display count by processing instructions to link the display count information and the result of identifying further using the information of the shelf identification mark.

7. The information processing apparatus according to claim 1, wherein a plurality of display shelves exist, and said information processing apparatus further comprises an article array storage that stores an array of the plurality of articles in at least part of the display shelf, wherein the at least one processor is configured to:

identify the types of articles, the names of the articles, and the array of the articles by further processing instructions to specify the display shelf based on the array of the articles; and

recognize the display count by processing instructions to link the display count information and the result of identifying in the specified display shelf.

8. The information processing apparatus according to claim 1, wherein the at least one processor is configured to recognize the display count by further processing instructions to

collate a first article presence/absence pattern identified based on a captured image and a second article presence/absence pattern acquired from the article sensors, and

link the result of identifying and the article presence/absence information at a display position of the display shelf in which the first article presence/absence pattern matches the second article presence/absence pattern.

9. The information processing apparatus according to claim 1, wherein the at least one processor is further configured to process instructions to:

collate presence/absence information of a plurality of articles acquired from the article sensors and the types of the plurality of articles identified in the captured image, to generate a result of collating and

determine an incorrect arrangement of an article in the article column based on the result of collating.

10. The information processing apparatus according to claim 1, wherein the at least one processor is further configured to process instructions to:

determine whether the articles have been identified;

determine whether the display count is zero; and

adjust at least one of a parameter used to identify the types of the articles and a parameter used to sense presence/absence of the articles in response to at least one of;

(1) determining that the articles have not been identified and determining that the display count is not zero and (2) determining that the articles have been identified and determining that the display count is zero.

11. The information processing apparatus according to claim 1, wherein the at least one processor is further configured to process instructions to:

display the recognized display count of the articles.

12. A control method of an information processing apparatus which comprises a shelf allocation storage that stores a positional arrangement of sensor groups corresponding to a shelf allocation in a display shelf on which articles are placed, each sensor group comprising a plurality of article sensors configured to sense the presence and/or absence of an article, the control method comprising:

acquiring display count information of articles using the article sensors provided on the display shelf;

identifying types of articles, name of the articles, and an array of the articles based on an image acquired by capturing the display shelf, to generate a result of identifying;

recognizing, by linking the display count information and the result of identifying in reference to the positional arrangement of the sensor groups stored in the shelf allocation storage, display count of each type of the articles;

comparing the identified array of the articles and the recognized display count of each type of the articles with a history of article array, to generate a result of comparing; and

discriminating a change in article array from an arrangement error based on the result of comparing.

13. A non-transitory computer readable medium storing a control program of an information processing apparatus, which comprises a shelf allocation storage that stores a positional arrangement of sensor groups corresponding to a shelf allocation in a display shelf on which articles are placed, each sensor group comprising a plurality of article sensors configured to sense the presence and/or absence of an article, for causing a computer to execute a method comprising:

acquiring display count information of articles using the article sensors provided on the display shelf;

identifying types of articles, name of the articles, and an array of the articles based on an image acquired by capturing the display shelf, to generate a result of identifying;

recognizing, by linking the display count information and the result of identifying in reference to the positional arrangement of the sensor groups stored in the shelf allocation storage, display count of each type of the articles;

comparing the identified array of the articles and the recognized display count of each type of the articles with a history of article array, to generate a result of comparing; and

discriminating a change in article array from an arrangement error based on the result of comparing.

14. An article management system comprising:

article sensors configured to sense the presence, absence, or both of an article and provided on a display shelf on which articles are placed;

an image capturer that captures the display shelf;

a shelf allocation storage that stores a positional arrangement of sensor groups corresponding to a shelf allocation in the display shelf, each sensor group comprising a plurality of article sensors;

a memory storing instructions; and

at least one processor configured to process the instructions to;

acquire display count information of the articles using the article sensors;

identify types of articles, names of the articles, and an array of the articles based on an image of the display shelf acquired by the image capturer, to generate a result of identifying;

recognize, by linking the display count information and the result of identifying in reference to the positional arrangement of the sensor groups stored in the shelf allocation storage, display count of each type of the articles;

compare the identified array of the articles and the recognized display count of each type of the articles with a history of article array, to generate a result of comparing; and

discriminate a change in article array from an arrangement error based on the result of comparing.