Video processing system, video processing method, video processing device for mobile terminal or server and control method and control program thereof

• US 9,355,317 B2
• Filed: 12/04/2012
• Issued: 05/31/2016
• Est. Priority Date: 12/14/2011
• Status: Active Grant

• Current Assignee: NEC Corporation
• Original Assignee: NEC Corporation
• Inventors: Nomura, Toshiyuki, Yamada, Akio, Iwamoto, Kota, Mase, Ryota
• Primary Examiner(s): MARIAM, DANIEL G
• Application Number:
  US14/362,561
• Publication Number:
  US 20140376815A1
• Time in Patent Office: 1,274 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 16/783 using metadata automatically derived from the content
• CPC Class Codes: G06F 18/213 Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
• CPC Class Codes: G06V 10/44 Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
• CPC Class Codes: G06V 10/7715 Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames

Abstract

The present invention is to notify a recognition result with respect to a recognition object in a video in real time while maintaining recognition accuracy. A recognition object and m-number of first local characteristic quantities which are respectively 1-dimensional to i-dimensional characteristic vectors are stored in association with each other, and n-number of second local characteristic quantities which are respectively 1-dimensional to j-dimensional characteristic vectors are generated for n-number of local areas respectively including n-number of characteristic points from an image in a video. In addition, a smaller dimension is selected from the dimension i and the dimension j, a recognition that the recognition object exists in an image in the video is made when it is determined that a prescribed proportion or more of the m-number of first local characteristic quantities which are characteristic vectors up to the selected number of dimensions correspond to the n-number of second local characteristic quantities which are characteristic vectors up to the selected number of dimensions, and information representing the recognition object is displayed in superposition on an image in which the recognition object exists in the video.

Claims

1. A video processing device, comprising:

a first local characteristic quantity storing unit that stores a recognition object and m-number of first local characteristic quantities which are respectively 1-dimensional to i-dimensional characteristic vectors and which are generated for each of m-number of local areas including each of m-number of characteristic points in an image of the recognition object, with the recognition object and m-number of first local characteristic quantities being associated with each other;

a second local characteristic quantity generating unit that extracts n-number of characteristic points from an image in a video and that generates n-number of second local characteristic quantities which are respectively 1-dimensional to j-dimensional characteristic vectors for n-number of local areas including each of the n-number of characteristic points;

a recognizing unit that selects a smaller number of dimensions among the number of dimensions i of the characteristic vectors of the first local characteristic quantities and the number of dimensions j of the characteristic vectors of the second local characteristic quantities, and recognizes that the recognition object exists in the image in the video when determining that a prescribed proportion or more of the m-number of first local characteristic quantities which are characteristic vectors up to the selected number of dimensions correspond to the n-number of second local characteristic quantities which are characteristic vectors up to the selected number of dimensions, and

a displaying unit that displays information representing the recognition object recognized by the recognizing unit in an image in which the recognition object exists in the video.

2. The video processing device according to claim 1, further comprising

a related information storing unit that stores, in association with the recognition object, related information that is related to the recognition object,

wherein the displaying unit further displays the related information in superposition on an image in which the recognition object exists in the video.

3. The video processing device according to claim 1, further comprising:

a link information storing unit that stores, in association with the recognition object, link information for accessing related information that is related to the recognition object,

wherein the displaying unit further displays the link information in superposition on an image in which the recognition object exists in the video.

4. The video processing device according to claim 1, further comprising:

a link information storing unit that stores, in association with the recognition object, link information for accessing related information that is related to the recognition object; and

a downloading unit that accesses the related information in accordance with the link information,

wherein the displaying unit further displays the related information in superposition on an image in which the recognition object exists in the video.

5. The video processing device according to claim 1, wherein

the first local characteristic quantity storing unit stores a tuple constituted by the m-number of first local characteristic quantities and position coordinates of the m-number of characteristic points in an image of the recognition object,

the second local characteristic quantity generating unit stores a tuple constituted by the n-number of second local characteristic quantities and position coordinates of the n-number of characteristic points in an image in the video, and

the recognizing unit recognizes that the recognition object exists in the image in the video when determining that a set of a tuple of the n-number of second local characteristic quantities and position coordinates thereof and a set of a prescribed proportion or more of a tuple of the m-number of first local characteristic quantities and position coordinates thereof have a linear conversion relationship.

6. The video processing device according to claim 1, wherein the first local characteristic quantities and the second local characteristic quantities are generated by dividing a local area including a characteristic point extracted from an image into a plurality of sub areas and generating characteristic vectors of a plurality of dimensions constituted by a gradient direction histogram in the plurality of sub areas.

7. The video processing device according to claim 6, wherein the first local characteristic quantities and the second local characteristic quantities are generated by deleting dimensions with a higher correlation between adjacent sub areas from the generated characteristic vectors of the plurality of dimensions.

8. The video processing device according to claim 6, wherein the first local characteristic quantities and the second local characteristic quantities are generated by deleting a characteristic point determined to have low importance from the plurality of characteristic points extracted from an image.

9. The video processing device according to claim 6, wherein the plurality of dimensions of the characteristic vectors are arranged to circumnavigate the local area for every prescribed number of dimensions so that the plurality of dimensions of the characteristic vectors can be selected sequentially from the dimensions contributing to a characteristic of the characteristic point and sequentially from a 1st dimension in accordance with an increase in accuracy required with respect to the local characteristic quantities.

10. A control method of a video processing device including a first local characteristic quantity storing unit that stores a recognition object and m-number of first local characteristic quantities which are respectively 1-dimensional to i-dimensional characteristic vectors and which are generated for each of m-number of local areas including each of m-number of characteristic points in an image of the recognition object, with the recognition object and m-number of first local characteristic quantities being associated with each other,

the control method of a video processing device comprising:

a second local characteristic quantity generating step of extracting n-number of characteristic points from an image in a video and generating n-number of second local characteristic quantities which are respectively 1-dimensional to j-dimensional characteristic vectors for n-number of local areas including each of the n-number of characteristic points;

a recognizing step of selecting a smaller number of dimensions among the number of dimensions i of the characteristic vectors of the first local characteristic quantities and the number of dimensions j of the characteristic vectors of the second local characteristic quantities, and recognizing that the recognition object exists in the image in the video when determining that a prescribed proportion or more of the m-number of first local characteristic quantities which are characteristic vectors up to the selected number of dimensions correspond to the n-number of second local characteristic quantities which are characteristic vectors up to the selected number of dimensions, and

a displaying step of displaying information representing the recognition object recognized in the recognizing step in superposition on an image in which the recognition object exists in the video.

11. A non-transitory computer-readable storage medium having a control program in a video processing device including a first local characteristic quantity storing unit that stores m-number of first local characteristic quantities which are respectively 1-dimensional to i-dimensional characteristic vectors and which are generated for each of m-number of local areas including each of m-number of characteristic points in an image of a recognition object, with the m-number of first local characteristic quantities being associated with the recognition object, the control method comprising:

a second local characteristic quantity generating step of extracting n-number of characteristic points from an image in a video and generating n-number of second local characteristic quantities which are respectively 1-dimensional to j-dimensional characteristic vectors for n-number of local areas including each of the n-number of characteristic points;

a recognizing step of selecting a smaller number of dimensions among the number of dimensions i of the characteristic vectors of the first local characteristic quantities and the number of dimensions j of the characteristic vectors of the second local characteristic quantities, and recognizing that the recognition object exists in the image in the video when determining that a prescribed proportion or more of the m-number of first local characteristic quantities which are characteristic vectors up to the selected number of dimensions correspond to the n-number of second local characteristic quantities which are characteristic vectors up to the selected number of dimensions, and

a displaying step of displaying information representing the recognition object recognized in the recognizing step in superposition on an image in which the recognition object exists in the video.

12. A video processing system including a video processing device for a mobile terminal and a video processing device for a server which are connected to each other via a network,

the video processing system comprising:

a first local characteristic quantity storing unit that stores m-number of first local characteristic quantities which are respectively which are 1-dimensional to i-dimensional characteristic vectors and which are generated for each of m-number of local areas including each of m-number of characteristic points in an image of a recognition object, with the m-number of first local characteristic quantities being associated with the recognition object;

a second local characteristic quantity generating unit that extracts n-number of characteristic points from an image in a video and that generates n-number of second local characteristic quantities which are respectively 1-dimensional to j-dimensional characteristic vectors for n-number of local areas including each of the n-number of characteristic points;

a recognizing unit that selects a smaller number of dimensions among the number of dimensions i of the characteristic vectors of the first local characteristic quantities and the number of dimensions j of the characteristic vectors of the second local characteristic quantities, and recognizes that the recognition object exists in the image in the video when determining that a prescribed proportion or more of the m-number of first local characteristic quantities which are by characteristic vectors up to the selected number of dimensions and which are stored in the first local characteristic quantity storing unit correspond to the n-number of second local characteristic quantities which are characteristic vectors up to the selected number of dimensions and which are generated by the second local characteristic quantity generating unit; and

a displaying unit that displays information representing the recognition object recognized by the recognizing unit in superposition on an image in which the recognition object exists in the video.

13. The video processing system according to claim 12, wherein

the video processing device for a mobile terminal comprises:

the second local characteristic quantity generating unit;

a second local characteristic quantity transmitting unit which encodes the n-number of second local characteristic quantities and which transmits the encoded n-number of second local characteristic quantities to the video processing device for a server via the network;

a recognition object receiving unit which receives information representing a recognition object recognized by the video processing device for a server from the video processing device for a server; and

a displaying unit that displays information representing the received recognition object in superposition on an image in which the recognition object exists in the video, and

the video processing device for a server comprises;

the first local characteristic quantity storing unit;

second local characteristic quantity receiving unit which receives the encoded n-number of second local characteristic quantities from the video processing device for a mobile terminal and decodes the encoded n-number of second local characteristic quantities;

the recognizing unit; and

a recognition object transmitting unit which transmits information representing the recognition object recognized by the recognizing unit to the video processing device for a mobile terminal via the network.

14. The video processing system according to claim 12, further comprising:

a link information storing unit which stores, in association with the recognition object, link information for accessing related information that is related to the recognition object, wherein

the displaying unit further displays the link information in superposition on an image in which the recognition object exists in the video.

15. The video processing system according to claim 12, further comprising:

a link information storing unit which stores, in association with the recognition object, link information for accessing related information that is related to the recognition object; and

a downloading unit which accesses the related information in accordance with the link information, wherein

the displaying unit further displays the related information in superposition on an image in which the recognition object exists in the video.

16. A video processing device for a mobile terminal in the video processing system according to claim 12,

the video processing device for a mobile terminal comprising:

a second local characteristic quantity generating unit that extracts n-number of characteristic points from an image in a video and that generates n-number of second local characteristic quantities which are respectively 1-dimensional to j-dimensional characteristic vectors for n-number of local areas including each of the n-number of characteristic points;

a second local characteristic quantity transmitting unit that encodes the n-number of second local characteristic quantities and which transmits the encoded n-number of second local characteristic quantities to a video processing device for a server via a network;

a recognition object receiving unit that receives information representing a recognition object recognized by the video processing device for a server from the video processing device for a server; and

a displaying unit that displays information representing the received recognition object in superposition on an image in which the recognition object exists in the video.

17. A control method of a video processing device for a mobile terminal in the video processing system according to claim 12, comprising:

a second local characteristic quantity generating step of extracting n-number of characteristic points from an image in a video and generating n-number of second local characteristic quantities which are respectively 1-dimensional to j-dimensional characteristic vectors for n-number of local areas including each of the n-number of characteristic points;

a second local characteristic quantity transmitting step of encoding the n-number of second local characteristic quantities and transmitting the encoded n-number of second local characteristic quantities to a video processing device for a server via a network;

a recognition object receiving step of receiving information representing a recognition object recognized by the video processing device for a server from the video processing device for a server; and

a displaying step of displaying information representing the received recognition object in superposition on an image in which the recognition object exists in the video.

18. A control program of a video processing device for a mobile terminal in the video processing system according to claim 12, the control program causing a computer to execute:

a second local characteristic quantity generating step of extracting n-number of characteristic points from an image in a video and generating n-number of second local characteristic quantities which are respectively 1-dimensional to j-dimensional characteristic vectors for n-number of local areas including each of the n-number of characteristic points;

a second local characteristic quantity transmitting step of encoding the n-number of second local characteristic quantities and transmitting the encoded n-number of second local characteristic quantities to a video processing device for a server via a network;

a recognition object receiving step of receiving information representing a recognition object recognized by the video processing device for a server from the video processing device for a server; and

a displaying step of displaying information representing the received recognition object in superposition on an image in which the recognition object exists in the video.

19. A video processing device for a server in the video processing system according to claim 12,

the video processing device for a server comprising:

a first local characteristic quantity storing unit that stores a recognition object and m-number of first local characteristic quantities which are respectively 1-dimensional to i-dimensional characteristic vectors and which are generated for each of m-number of local areas including each of m-number of characteristic points in an image of the recognition object, with the recognition object and m-number of first local characteristic quantities being associated with each other;

a second local characteristic quantity receiving unit that receives the encoded n-number of second local characteristic quantities from a video processing device for a mobile terminal and decodes the encoded n-number of second local characteristic quantities;

a recognizing unit that selects a smaller number of dimensions among the number of dimensions i of the characteristic vectors of the first local characteristic quantities and the number of dimensions j of the characteristic vectors of the second local characteristic quantities, and recognizes that the recognition object exists in the image in the video when determining that a prescribed proportion or more of the m-number of first local characteristic quantities which are characteristic vectors up to the selected number of dimensions correspond to the n-number of second local characteristic quantities which are characteristic vectors up to the selected number of dimensions, and

a recognition object transmitting unit that transmits information representing the recognition object recognized by the recognizing unit to the video processing device for a mobile terminal via a network.

20. A control method of a video processing device for a server in the video processing system according to claim 12, the video processing device including a first local characteristic quantity storing unit that stores a recognition object and m-number of first local characteristic quantities which are respectively 1-dimensional to i-dimensional characteristic vectors and which are generated for each of m-number of local areas including each of m-number of characteristic points in an image of the recognition object, with the recognition object and m-number of first local characteristic quantities being associated with each other,

the control method comprising:

a second local characteristic quantity receiving step of receiving the encoded n-number of second local characteristic quantities from a video processing device for a mobile terminal and decoding the encoded n-number of second local characteristic quantities;

a recognizing step of selecting a smaller number of dimensions among the number of dimensions i of the characteristic vectors of the first local characteristic quantities and the number of dimensions j of the characteristic vectors of the second local characteristic quantities, and recognizing that the recognition object exists in the image in the video when determining that a prescribed proportion or more of the m-number of first local characteristic quantities which are characteristic vectors up to the selected number of dimensions correspond to the n-number of second local characteristic quantities which are characteristic vectors up to the selected number of dimensions, and

a recognition object transmitting step of transmitting information representing the recognition object recognized in the recognizing step to the video processing device for a mobile terminal via a network.

21. A non-transitory computer-readable storage medium having a control program of a video processing device for a server in the video processing system according to claim 12, the video processing device including a first local characteristic quantity storing unit that stores a recognition object and m-number of first local characteristic quantities which are respectively 1-dimensional to i-dimensional characteristic vectors and which are generated for each of m-number of local areas including each of m-number of characteristic points in an image of the recognition object, with the recognition object and m-number of first local characteristic quantities being associated with each other,

the control program causing a computer to execute:

a second local characteristic quantity receiving step of receiving the encoded n-number of second local characteristic quantities from a video processing device for a mobile terminal and decoding the encoded n-number of second local characteristic quantities;

a recognizing step of selecting a smaller number of dimensions among the number of dimensions i of the characteristic vectors of the first local characteristic quantities and the number of dimensions j of the characteristic vectors of the second local characteristic quantities, and recognizing that the recognition object exists in the image in the video when determining that a prescribed proportion or more of the m-number of first local characteristic quantities which are characteristic vectors up to the selected number of dimensions correspond to the n-number of second local characteristic quantities which are characteristic vectors up to the selected number of dimensions, and

a recognition object transmitting step of transmitting information representing the recognition object recognized in the recognizing step to the video processing device for a mobile terminal via a network.

22. A video processing method in a video processing system which includes a video processing device a for a mobile terminal and a video processing device for a server which are connected to each other via a network and which includes a first local characteristic quantity storing unit that stores a recognition object and m-number of first local characteristic quantities which are respectively 1-dimensional to i-dimensional characteristic vectors and which are generated for each of m-number of local areas including each of m-number of characteristic points in an image of the recognition object, with the first local characteristic quantities being associated with the recognition object,

the video processing method comprising:

a second local characteristic quantity generating step of extracting n-number of characteristic points from an image in a video and generating n-number of second local characteristic quantities which are respectively 1-dimensional to j-dimensional characteristic vectors for n-number of local areas including each of the n-number of characteristic points;

a recognizing step of selecting a smaller number of dimensions among the number of dimensions i of the characteristic vectors of the first local characteristic quantities and the number of dimensions j of the characteristic vectors of the second local characteristic quantities, and recognizes that the recognition object exists in the image in the video when determining that a prescribed proportion or more of the m-number of first local characteristic quantities which are characteristic vectors up to the selected number of dimensions and which are stored in the first local characteristic quantity storing unit correspond to the n-number of second local characteristic quantities which are characteristic vectors up to the selected number of dimensions and which are generated by the second local characteristic quantity generating unit; and

a displaying step of displaying information representing the recognition object recognized by the recognizing unit in superposition on an image in which the recognition object exists in the video.