Information processing system, information processing method, information processing device and control method and control program thereof, and communication terminal and control method and control program thereof

• US 9,754,169 B2
• Filed: 12/07/2012
• Issued: 09/05/2017
• Est. Priority Date: 12/16/2011
• Status: Active Grant

• Current Assignee: NEC Corporation
• Original Assignee: NEC Corporation
• Inventors: Nomura, Toshiyuki, Yamada, Akio, Iwamoto, Kota, Mase, Ryota
• Primary Examiner(s): Fujita, Katrina
• Application Number:
  US14/365,008
• Publication Number:
  US 20150098610A1
• Time in Patent Office: 1,733 Days
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06Q 10/06 Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
• CPC Class Codes: G06Q 10/087 Inventory or stock management, e.g. order filling, procurement or balancing against orders
• CPC Class Codes: G06Q 10/10 Office automation; Time management
• CPC Class Codes: G06Q 50/10 Services
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69

Abstract

Merchandise management is implemented by recognizing a piece of merchandise in an image on a video in real time. A piece of merchandise and m-number of first local feature which are respectively 1-dimensional to i-dimensional feature vectors are stored after being associated with each other, n-number of feature points are extracted from an image on a video captured by an imaging unit, n-number of second local feature which are respectively 1-dimensional to j-dimensional feature vectors are generated, a smaller number of dimensions of the number of dimensions i and the number of dimensions j is selected, and a recognition that the merchandise exists in the image on the video is made when it is determined that a prescribed proportion or more of the m-number of first local feature of the selected number of dimensions correspond to the n-number of second local feature of the selected number of dimensions.

Claims

1. An information processing system, comprising:

a local feature storage which stores a piece of merchandise and m-number of first local feature which are respectively 1-dimensional to i-dimensional feature vectors and which are generated for each of m-number of local areas including each of m-number of feature points of an image of the merchandise after associating the piece of merchandise and m-number of first local feature with each other;

an imaging unit;

a first processor that executes the steps of;

extracting n-number of feature points from an image on a video captured by the imaging unit, and

generating n-number of second local feature which are respectively 1-dimensional to j-dimensional feature vectors for n-number of local areas including each of the n-number of feature points; and

a second processor that executes the steps of;

selecting a smaller number of dimensions among the number of dimensions i of the feature vectors of the first local feature and the number of dimensions j of the feature vectors of the second local feature, and

recognizing that the merchandise exists in the image on the video when determination is made that a prescribed proportion or more of the m-number of first local feature which are feature vectors up to the selected number of dimensions correspond to the n-number of second local feature which are feature vectors up to the selected number of dimensions,

wherein the first local feature and the second local feature are generated by dividing a local area including a feature point extracted from an image into a plurality of sub areas and generating feature vectors of a plurality of dimensions constituted by a gradient direction histogram in the plurality of sub areas, and

wherein the first local feature and the second local feature are generated by deleting dimensions with a higher correlation between adjacent sub areas from the generated feature vectors of the plurality of dimensions.

2. The information processing system according to claim 1, wherein the first processor further executes the step of notifying a recognition result of the recognizing step.

3. The information processing system according to claim 2, wherein

the information processing system comprises a communication terminal carried by a user and an information processing device that communicates with the communication terminal,

the communication terminal includes the imaging unit and the first processor and transmits the m-number of second local feature from the communication terminal to the information processing device, and

the information processing device includes the local feature storage and the second processor and transmits a recognition result of the recognizing step from the information processing device to the communication terminal.

4. A communication terminal in the information processing system according to claim 3,

the communication terminal comprising:

an imaging unit;

a processor that executes the steps of extracting n-number of feature points from an image on a video captured by the imaging unit and generating n-number of second local feature which are respectively 1-dimensional to j-dimensional feature vectors for n-number of local areas including each of the n-number of feature points;

a transmitter which transmits the m-number of second local feature to an information processing device that recognizes merchandise included in the captured image based on comparison with local feature; and

a receiver which receives information representing merchandise included in the captured image from the information processing device.

5. A control method of a communication terminal in the information processing system according to claim 3,

the control method comprising:

an imaging step of performing imaging;

a second local feature generating step of extracting n-number of feature points from an image on a video captured in the imaging step and generating n-number of second local feature which are respectively 1-dimensional to j-dimensional feature vectors for n-number of local areas including each of the n-number of feature points;

a first transmitting step of transmitting the m-number of second local feature to an information processing device that recognizes merchandise included in the captured image based on comparison with local feature; and

a first receiving step of receiving information representing merchandise included in the captured image from the information processing device.

6. A non-transitory computer-readable storage medium storing thereon a control program of a communication terminal in the information processing system according to claim 3,

the control program causing a computer to execute:

an imaging step of performing imaging;

a second local feature generating step of extracting n-number of feature points from an image on a video captured by the imaging and generating n-number of second local feature which are respectively 1-dimensional to j-dimensional feature vectors for n-number of local areas including each of the n-number of feature points;

a first transmitting step of transmitting the m-number of second local feature to an information processing device that recognizes merchandise included in the captured image based on comparison with local feature; and

a first receiving step of receiving information representing merchandise included in the captured image from the information processing device.

7. An information processing device including a local feature storage that stores a piece of merchandise and m-number of first local feature which are respectively 1-dimensional to i-dimensional feature vectors and which are generated for each of m-number of local areas including each of m-number of feature points of an image of the merchandise after associating the piece of merchandise and m-number of first local feature with each other in the information processing system according to claim 3,

the information processing device comprising:

a receiver which receives, from the communication terminal, n-number of second local feature which are respectively 1-dimensional to j-dimensional feature vectors for n-number of local areas including each of n-number of feature points extracted from an image on a video captured by the imaging unit;

a second processor that executes the steps of selecting a smaller number of dimensions among the number of dimensions i of the feature vectors of the first local feature and the number of dimensions j of the feature vectors of the second local feature, and recognizing that the merchandise exists in the image on the video when determination is made that a prescribed proportion or more of the m-number of first local feature which are feature vectors up to the selected number of dimensions correspond to the n-number of second local feature which are feature vectors up to the selected number of dimensions, and

a transmitter which transmits information representing the recognized merchandise to the communication terminal.

8. A control method of an information processing device including a local feature storage that stores a piece of merchandise and m-number of first local feature which are respectively 1-dimensional to i-dimensional feature vectors and which are respectively generated for m-number of local areas including each of m-number of feature points of an image of the merchandise after associating the piece of merchandise and m-number of first local feature with each other in the information processing system according to claim 3,

the control method comprising:

a second receiving step of receiving, from the communication terminal, n-number of second local feature which are respectively 1-dimensional to j-dimensional feature vectors for n-number of local areas including each of n-number of feature points extracted from an image on a video captured by the imaging unit;

a recognizing step of selecting a smaller number of dimensions among the number of dimensions i of the feature vectors of the first local feature and the number of dimensions j of the feature vectors of the second local feature, and recognizing that the merchandise exists in the image on the video when determination is made that a prescribed proportion or more of the m-number of first local feature which are feature vectors up to the selected number of dimensions correspond to the n-number of second local feature which are feature vectors up to the selected number of dimensions; and

a second transmitting step of transmitting information representing the recognized merchandise to the communication terminal.

9. A non-transitory computer-readable storage medium storing thereon a control program of an information processing device including a local feature storage that stores a piece of merchandise and m-number of first local feature which are respectively 1-dimensional to i-dimensional feature vectors and which are generated each of for m-number of local areas including each of m-number of feature points of an image of the merchandise after associating the piece of merchandise and m-number of first local feature with each other in the information processing system according to claim 3,

the control program causing a computer to execute:

a second receiving step of receiving, from the communication terminal, n-number of second local feature which are respectively 1-dimensional to j-dimensional feature vectors for n-number of local areas including each of n-number of feature points extracted from an image on a video captured by the imaging unit;

a recognizing step of selecting a smaller number of dimensions among the number of dimensions i of the feature vectors of the first local feature and the number of dimensions j of the feature vectors of the second local feature, and recognizing that the merchandise exists in the image on the video when determination is made that a prescribed proportion or more of the m-number of first local feature which are feature vectors up to the selected number of dimensions correspond to the n-number of second local feature which are feature vectors up to the selected number of dimensions; and

a second transmitting step of transmitting information representing the recognized merchandise to the communication terminal.

10. The information processing system according to claim 1, wherein

the local feature storage stores, in association with each of a plurality of pieces of merchandise, the m-number of first local feature generated from an image of each piece of merchandise, and

the second processor executes the steps of recognizing a plurality of pieces of merchandise included in the image captured by the imaging unit and recognizing an arrangement of the plurality of pieces of merchandise included in the image captured by the imaging unit based on an arrangement of the n-number of second local feature.

11. The information processing system according to claim 10, wherein the second processor performs the step of performing inventory or shelf allocation based on the arrangement of the plurality of pieces of merchandise recognized by the second processor.

12. The information processing system according to claim 1, wherein

the image of the merchandise includes an image of a packing container containing a plurality of pieces of merchandise, and

the second processor executes the step of recognizing the packing container.

13. The information processing system according to claim 12, wherein the second processor performs the step of performing distribution management or stock management of the merchandise based on the recognition of the packing container by the second processor.

14. The information processing system according to claim 1, wherein the first processor executes the step of generating, in correspondence to a correlation of the merchandise, the second local feature having a larger number of dimensions with respect to merchandise having a higher correlation with other merchandise.

15. The information processing system according to claim 1, wherein the local feature storage stores, in correspondence to a correlation of the merchandise, the first local feature having a larger number of dimensions with respect to merchandise having a higher correlation with other merchandise.

16. The information processing system according to claim 1, wherein

the second processor executes the step of determining a direction of the merchandise and recognizes a position of an identifier attached to the merchandise based on the direction, the identifier identifying the merchandise.

17. The information processing system according to claim 1, wherein the merchandise is a book in a library.

18. The information processing system according to claim 1, wherein the merchandise is a work of art in an art museum.

19. The information processing system according to claim 1, wherein the merchandise is an exhibit in a museum.

20. An information processing method in an information processing system including a local feature storage that stores a piece of merchandise and m-number of first local feature which are respectively 1-dimensional to i-dimensional feature vectors and which are generated for each of m-number of local areas including each of m-number of feature points of an image of the merchandise after associating the piece of merchandise and m-number of first local feature with each other,

the information processing method comprising:

an imaging step of performing imaging;

a second local feature generating step of extracting n-number of feature points from an image on a video captured in the imaging step and generating n-number of second local feature which are respectively 1-dimensional to j-dimensional feature vectors for n-number of local areas including each of the n-number of feature points; and

a recognizing step of selecting a smaller number of dimensions among the number of dimensions i of the feature vectors of the first local feature and the number of dimensions j of the feature vectors of the second local feature, and recognizing that the merchandise exists in the image on the video when determination is made that a prescribed proportion or more of the m-number of first local feature which are feature vectors up to the selected number of dimensions correspond to the n-number of second local feature which are feature vectors up to the selected number of dimensions,

wherein the first local feature and the second local feature are generated by dividing a local area including a feature point extracted from an image into a plurality of sub areas and generating feature vectors of a plurality of dimensions constituted by a gradient direction histogram in the plurality of sub areas, and

wherein the first local feature and the second local feature are generated by deleting dimensions with a higher correlation between adjacent sub areas from the generated feature vectors of the plurality of dimensions.