Information display apparatus that superposes information on an imaged image and displays the information and the imaged image, and related information display system, control method and program

• US 10,410,607 B2
• Filed: 06/22/2017
• Issued: 09/10/2019
• Est. Priority Date: 12/29/2012
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Naganawa, Yoshihiro
• Primary Examiner(s): Wang, Samantha (Yuehan)
• Application Number:
  US15/630,510
• Publication Number:
  US 20170287445A1
• Time in Patent Office: 810 Days
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G09G 2320/106 Determination of movement vectors or equivalent parameters within the image
• CPC Class Codes: G09G 5/377 for mixing or overlaying two or more graphic patterns G09G5/02, G09G5/397 take precedence

Abstract

An information display apparatus includes at least one processor programmed to cause the apparatus, among other things, to superpose, on an imaged image, information related to an object when an object of one or more objects is in a target area, and not to superpose, on the imaged image, the information related to an object when the object of the one or more objects is outside of the target area. In a case in which a percentage of the one or more objects existing in the imaged image does not exceed a threshold value, the whole imaged image is determined to be the target area without using a detected movement direction, the one or more objects recognizable in the imaged image are recognized, and the information related to the recognized one or more objects in the imaged image is superposed on the imaged image.

Claims

1. An information display apparatus that superposes information on an imaged image and displays the information and the imaged image, the apparatus comprising:

at least one processor programmed to cause the apparatus;

(a) to detect, with a sensor, a movement direction of the imaged image relative to an imaging target;

(b) to determine a location of a portion of a target area within the imaged image, based on the detected movement direction, the location of the portion of the target area being shifted within the imaged image based on the detected movement direction;

(c) to recognize an object of one or more objects in the imaged image; and

(d) to superpose, on the imaged image, information related to an object when the object of the one or more objects is in the target area, and not to superpose, on the imaged image, the information related to an object when the object of the one or more objects is outside of the target area,

wherein, in a case in which a percentage of the one or more objects existing in the imaged image does not exceed a threshold value;

(i) the whole imaged image is determined to be the target area without using the detected movement direction;

(ii) the one or more objects recognizable in the imaged image are recognized; and

(iii) the information related to the recognized one or more objects in the imaged image is superposed on the imaged image.

2. The information display apparatus according to claim 1, wherein the at least one processor is further programmed to cause the apparatus (e) to detect a speed of movement of the imaged image, and, in a case in which the detected speed is equal to or less than a first threshold value, the location of the portion of the target area is determined to be a first area that is located near a center of the imaged image, and, in a case in which the detected speed exceeds the first threshold value, the location of the portion of the target area is determined to be a second area that is shifted within the imaging target in the detected movement direction.

3. The information display apparatus according to claim 2, wherein, in a case in which the detected speed exceeds a second threshold value that is greater than the first threshold value, the portion of the target area is determined to be a third area that is narrower than the second area.

4. The information display apparatus according to claim 1, wherein the object in the imaged image existing in the portion of the target area at a changed position within the imaged image is recognized.

5. An information display apparatus that superposes information on an imaged image and displays the information and the imaged image, the apparatus comprising:

at least one processor programmed to cause the apparatus;

(a) to detect, with a sensor, a movement direction of the imaged image relative to an imaging target;

(b) to determine a location of a portion of a target area within the imaged image, based on the detected movement direction, the location of the portion of the target area being shifted within the imaged image based on the detected movement direction;

(c) to recognize an object of one or more objects in the imaged image;

(d) to preferentially superpose, in a case in which a number of the one or more objects existing in the imaged image exceeds a threshold value, information related to the object being in the target area on the imaged image, and to suppress superposition of the information related to the object being outside of the target area on the imaged image; and

(e) to determine, in a case in which the number of the one or more objects existing in the imaged image does not exceed the threshold value, the whole imaged image to be the target area without using the detected movement direction, and to superpose the information related to the object recognizable in the whole imaged image on the imaged image.

6. The information display apparatus according to claim 5, wherein the at least one processor is further programmed to cause the apparatus (f) to detect a speed of movement of the imaged image, and, in a case in which the detected speed is equal to or less than a first threshold value, the location of the portion of the target area is determined to be a first area that is located near a center of the imaged image, and, in a case in which the detected speed exceeds the first threshold value, the location of the portion of the target area is determined to be a second area that is shifted within the imaging target in the detected movement direction.

7. The information display apparatus according to claim 6, wherein, in a case in which the detected speed exceeds a second threshold value that is greater than the first threshold value, the portion of the target area is determined to be a third area that is narrower than the second area.

8. The information display apparatus according to claim 5, wherein the object in the imaged image existing in the portion of the target area at a changed position within the imaged image is recognized.

9. An information display apparatus that superposes information on an image and displays the information and the imaged image, the apparatus comprising:

at least one processor programmed to cause the apparatus;

(a) to detect a movement direction of the image relative to a target;

(b) to determine a location of a portion of a target area within the image, based on the detected movement direction, the location of the portion of the target area being shifted within the image in the detected movement direction;

(c) to recognize an object of one or more objects in the image;

(d) to preferentially superpose, in a case in which the number of the one or more objects existing in the image exceeds a threshold value, information related to the object being in the target area on the imaged image, and to suppress superposition of the information related to the object being outside of the target area on the image; and

(e) to determine, in a case in which the number of the one or more objects existing in the image does not exceed the threshold value, the whole image to be the target area without using the detected movement direction, and to superpose the information related to the object recognizable in the whole image on the image.

10. The information display apparatus according to claim 9, wherein the at least one processor is further programmed to cause the apparatus (f) to detect a speed of movement of the image, and, in a case in which the detected speed is equal to or less than a first threshold value, the location of the portion of the target area is determined to be a first area that is located near a center of the imaged image, and, in a case in which the detected speed exceeds the first threshold value, the location of the portion of the target area is determined to be a second area that is shifted within the imaging target in the detected movement direction.

11. The information display apparatus according to claim 10, wherein, in a case in which the detected speed exceeds a second threshold value that is greater than the first threshold value, the portion of the target area is determined to be a third area that is narrower than the second area.

12. The information display apparatus according to claim 9, wherein the object in the imaged image existing in the target area at a changed position within the image is recognized.

13. An information display apparatus that superposes information on an image and displays the information and the image, the apparatus comprising:

at least one processor programmed to cause the apparatus;

(a) to detect a movement speed of the image relative to a target;

(b) to determine a first area located near a center of the image to be a location of a target area in a case in which the movement speed is equal to or less than a first threshold value, the location of the target area being shifted within the image based on a movement of the image;

(c) to recognize an object of one or more objects in the image;

(d) to preferentially superpose, in a case in which a number of the one or more objects existing in the image exceeds a threshold value, information related to the object being in the target area on the image, and to suppress superposition of the information related to the object being outside of the target area on the image; and

(e) to determine, in a case in which the number of the one or more objects existing in the image does not exceed the threshold value, the whole image to be the target area without using the detected movement speed, and to superpose the information related to the object recognizable in the whole image on the image.

14. The information display apparatus according to claim 13, wherein, in a case in which the movement speed exceeds the first threshold value, a second area that is narrower than the first area and located near the center of the image is determined to be the location of the target area.

15. An information display method that superposes information on an imaged image and displays the information and the imaged image, the method comprising:

detecting, with a sensor, a movement direction of the imaged image relative to an imaging target;

determining a location of a portion of a target area within the imaged image, based on the detected movement direction, the location of the portion of the target area being shifted within the imaged image based on the detected movement direction;

recognizing an object of one or more objects in the imaged image; and

superposing, on the imaged image, information related to an object when the object is in the target area, and not superposing, on the imaged image, the information related to an object when the object is outside of the target area,

wherein, in a case in which a percentage of the one or more objects existing in the imaged image does not exceed a threshold value;

(i) the whole imaged image is determined to be the target area without using the detected movement direction;

(ii) the one or more objects recognizable in the imaged image are recognized; and

(iii) the information related to the recognized one or more objects in the imaged image is superposed on the imaged image.

16. An information display method that superposes information on an imaged image and displays the information and the imaged image, the method comprising:

detecting, with a sensor, a movement direction of the imaged image relative to an imaging target;

determining a location of a portion of a target area within the imaged image, based on the detected movement direction, the location of the portion of the target area being shifted within the imaged image based on the detected movement direction;

recognizing an object of one or more objects in the imaged image;

preferentially superposing, in a case in which a number of the one or more objects existing in the imaged image exceeds a threshold value, information related to the object being in the target area on the image, and suppressing superposition of the information related to the object being outside of the target area on the imaged image; and

determining, in a case in which the number of the one or more objects existing in the imaged image does not exceed the threshold value, the whole imaged image to be the target area without using the detected movement direction, and superposing the information related to the object recognizable in the whole imaged image on the imaged image.

17. An information display method that superposes information on an image and displays the information and the image, the method comprising:

detecting a movement direction of the image relative to a target;

determining a location of a portion of a target area within the image, based on the detected movement direction, the location of the portion of the target area being shifted within the image in the detected movement direction;

recognizing an object of one or more objects in the image;

preferentially superposing, in a case in which a number of the one or more objects existing in the image exceeds a threshold value, information related to the object being in the target area on the image, and suppressing superposition of the information related to the object being outside of the target area on the image; and

determining, in a case in which the number of the one or more objects existing in the image does not exceed the threshold value, the whole image to be the target area without using the detected movement direction, and superposing the information related to the object recognizable in the whole image on the image.

18. An information display method that superposes information on an image and displays the information and the image, the method comprising:

detecting a movement speed of the image relative to a target;

determining a first area located near a center of the image to be a location of a target area in a case in which the movement speed is equal to or less than a first threshold value, wherein the location of the target area is shifted within the image based on a movement of the image;

recognizing an object of one or more objects in the image;

preferentially superposing, in a case in which a number of the one or more objects existing in the image exceeds a threshold value, information related to the object being in the target area on the image, and suppressing superposition of the information related to the object being outside of the target area on the image; and

determining, in a case in which the number of the one or more objects existing in the image does not exceed the threshold value, the whole image to be the target area without using the detected movement speed, and superposing the information related to the object recognizable in the whole image on the image.

19. A non-transitory computer-readable storage medium that stores a computer-executable program to achieve an information display method that superposes information on an imaged image and displays the information and the image, the method comprising:

detecting, with a sensor, a movement direction of the imaged image relative to an imaging target;

determining a location of a portion of a target area within the imaged image, based on the detected movement direction, the location of the portion of the target area being shifted within the imaged image based on the detected movement direction;

recognizing an object of one or more objects in the imaged image;

preferentially superposing, in a case in which a number of the one or more objects existing in the imaged image exceeds a threshold value, information related to the object being in the target area on the imaged image, and suppressing superposition of the information related to the object being outside of the target area on the imaged image; and

determining, in a case in which the number of the one or more objects existing in the imaged image does not exceed the threshold value, the whole imaged image to be the target area without using the detected movement direction, and superposing the information related to the object recognizable in the whole imaged image on the imaged image.