Information displaying apparatus and information displaying method

• US 9,204,050 B2
• Filed: 12/22/2009
• Issued: 12/01/2015
• Est. Priority Date: 12/25/2008
• Status: Active Grant

• Current Assignee: Sinkitech Limited
• Original Assignee: Panasonic Intellectual Property Management Co., Ltd. (Panasonic Holdings Corporation)
• Inventors: Emura, Koichi, Inatomi, Yasuaki, Ueda, Junko, Kanehara, Toshiki
• Primary Examiner(s): McDowell, Jr., Maurice L
• Assistant Examiner(s): Isanians, Raffi
• Application Number:
  US13/140,889
• Publication Number:
  US 20110254861A1
• Time in Patent Office: 2,170 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04N 21/41407 embedded in a portable device, e.g. video client on a mobile phone, PDA, laptop constructional details of equipment or arrangements specially adapted for portable computer application G06F1/1626; arrangements specially adapted for mobile receivers in broadcast systems H04H20/57
• CPC Class Codes: H04N 21/42202 environmental sensors, e.g. for detecting temperature, luminosity, pressure, earthquakes
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/4316 for displaying supplemental content in a region of the screen, e.g. an advertisement in a separate window
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/4622 Retrieving content or additional data from different sources, e.g. from a broadcast channel and the Internet web site content organization and management for information retrieval from the Internet G06F16/958; transmission by internet of broadcast information H04H60/82; stock exchange data over packet-switching network H04L12/1804; push services including data channel over packet-switching network H04L12/1859
• CPC Class Codes: H04N 21/4725 using interactive regions of the image, e.g. hot spots details of information retrieval from the Internet by using URLs G06F16/955; processing chained hypermedia data for information retrieval G06F16/94
• CPC Class Codes: H04N 23/62 Control of parameters via user interfaces
• CPC Class Codes: H04N 23/632 for displaying or modifying preview images prior to image capturing, e.g. variety of image resolutions or capturing parameters
• CPC Class Codes: H04N 23/633 for displaying additional information relating to control or operation of the camera
• CPC Class Codes: H04N 23/67 Focus control based on electronic image sensor signals
• CPC Class Codes: H04N 5/445 for displaying additional information H04N5/50 takes precedence

Abstract

An information displaying apparatus is capable of providing an easy-to-see display of information that the user wants to know out of information related to objects seen in a captured real-world image. The information displaying apparatus comprises an image input unit for inputting a captured real-world image. A distance acquiring unit acquires the distance between an object seen in the input captured real-world image and a view point of the captured real-world image. An object superimposing unit outputs an image in which information related to the object is superimposed on the captured real-world image. An information output control unit controls the information output by the object superimposing unit. The information output control unit decides, in accordance with the acquired distance, how to present the information outputted by the object superimposing unit.

Claims

1. An information displaying apparatus, comprising:

an image input section that receives, as an input, a captured real-world image;

a distance acquisition section that measures and acquires, for each of a plurality of objects shown in the captured real-world image, a horizontal distance in the real-world between the object and a viewpoint of the captured real-world image, the horizontal distance negating an elevation of the object with respect to the viewpoint in the real-world;

a subject position and orientation acquisition section that acquires a position of the viewpoint in the real-world;

an object positioning section that calculates and acquires, for each object of the objects, a real-world position of the object, the real-world position calculated based on the position of the viewpoint, the horizontal distance, a direction, and an angle of elevation of the object with reference to the viewpoint;

an object information search section that searches, for each object of the objects, for a piece of information relating to the object using the real-world position of the object as a search key, and acquires the piece of information;

an object superimposition section that outputs an image in which pieces of information corresponding to the objects are superimposed on the captured real-world image; and

an information output control section that controls, for each of the objects, the piece of information output from the object superimposition section,

wherein, the shorter the acquired horizontal distance is, the greater an information amount for the piece of information which is displayed in the image output from the object superimposition section is to be decided, for each of the objects, by the information output control section, and

the information amount for the piece of information, for each of the objects, is determined based on a number of the objects and a proportion of the captured real-world image occupied by object areas of the objects, the piece of information corresponding to each object being displayed in an area of the captured real-world image other than the object areas.

2. The information displaying apparatus according to claim 1, wherein, the shorter the horizontal distance is, the more an emphasized presentation mode for the piece of information is to be decided, for each of the objects, by the information output control section.

3. The information displaying apparatus according to claim 1, further comprising:

an image recognition section that recognizes from the captured real-world image each of the objects shown in the captured real-world image,

wherein the information output control section makes the object superimposition section output the piece of information relating to each of the objects.

4. The information displaying apparatus according to claim 1, wherein the object positioning section acquires the real-world position of each of the objects based on an area in the captured real-world image of each of the objects and a viewpoint and a field of view of the captured real-world image.

5. The information displaying apparatus according to claim 2, further comprising:

a lens control section that changes a zoom factor of the captured real-world image input by the image input section,

wherein, the larger a zoom factor of the captured real-world image is, the greater information amount for the piece of information is to be decided by the information output control section.

6. The information displaying apparatus according to claim 1, wherein the information output control section decides upon the objects that are information display objects from among the objects shown in the captured real-world image.

7. The information displaying apparatus according to claim 6, wherein the information output control section decides upon the objects that are the information display objects by being within a range of a predetermined number of the objects.

8. The information displaying apparatus according to claim 6, further comprising:

a lens control section that changes a focus position of the captured real-world image input by the image input section,

wherein the information output control section decides upon the objects near the focus position as the information display objects.

9. The information displaying apparatus according to claim 8,

wherein the lens control section receives a predetermined operation and fixes the focus position, and

the object superimposition section changes the information amount for the piece of information which is displayed when the focus position has been fixed.

10. The information displaying apparatus according to claim 6, further comprising:

a lens control section that changes a depth of field of the captured real-world image input by the image input section,

wherein the information output control section decides upon the objects positioned within a range of the depth of field as the information display objects.

11. The information displaying apparatus according to claim 1, wherein the object superimposition section displays the piece of information, associated with an area in the captured real-world image of the corresponding object.

12. The information displaying apparatus according to claim 3, further comprising:

an information position decision section that decides display positions of the pieces of information of the objects as positions corresponding to real-world positional relationships of the objects.

13. The information displaying apparatus according to claim 12, wherein the information position decision section detects a depth direction of the captured real-world image, and decides the display positions of the pieces of information of the objects so as to be aligned in the depth direction.

14. The information displaying apparatus according to claim 1, further comprising:

an operation input section that receives a user operation; and

an operation control section that switches among and sets a mode in which the user operation is reflected in the captured real-world image, a mode in which the user operation is reflected in the pieces of information output by the object superposition section, and a mode in which the user operation is reflected in the captured real-world image and the pieces of information are changed in conjunction with the captured real-world image.

15. The information displaying apparatus according to claim 1, further comprising:

an operation input section that receives a user operation via a touch panel operation for the image output by the object superimposition section.

17. The information displaying apparatus according to claim 1, wherein the distance acquisition section acquires, for each of the objects, an actual distance in the real-world between the object and the viewpoint, and converts the actual distance to the horizontal distance based on an angle of elevation of the object with respect to the viewpoint in the real-world.

18. The information displaying apparatus according to claim 1, wherein the pieces of information are displayed on a perspective line that extends radially from a vanishing point of the captured real-world image.

16. An information displaying method, comprising:

receiving, as an input, a captured real-world image;

measuring and acquiring, for each of a plurality of objects shown in the captured real-world image, a horizontal distance in the real-world between the object and a viewpoint of the captured real-world image, the horizontal distance negating an elevation of the object with respect to the viewpoint in the real-world;

acquiring a position of the viewpoint in the real-world;

calculating and acquiring, by a processor for each object of the objects, a real-world position of the object, the real-world position based on the position of the viewpoint, the horizontal distance, a direction, and an angle of elevation of the object with reference to the viewpoint;

searching, for each object of the objects, for a piece of information relating to the object using the real-world position of the object as a search key, and acquiring the piece of information;

deciding, for each object of the objects, a greater information amount for the piece of information relating to the object and to be displayed as the shorter the acquired horizontal distance is; and

outputting pieces of information corresponding to the objects in the decided information amount superimposed on the captured real-world image, wherein the information amount for the piece of information, for each of the objects, is determined based on a number of the objects and a proportion of the captured real-world image occupied by object areas of the objects, the piece of information corresponding to each object being displayed in an area of the captured real-world image other than the object areas.

19. An information displaying apparatus, comprising:

a display;

a processor; and

a memory including a set of instructions that, when executed by the processor, cause the processor to perform operations including;

receiving, as an input, a captured real-world image;

measuring and acquiring, for each of a plurality of objects shown in the captured real-world image, a horizontal distance in the real-world between the object and a viewpoint of the captured real-world image, the horizontal distance negating an elevation of the object with respect to the viewpoint in the real-world;

acquiring a position of the viewpoint in the real-world;

calculating and acquiring, for each object of the objects, a real-world position of the object, the real-world position calculated based on the position of the viewpoint, the horizontal distance, a direction, and an angle of elevation of the object with reference to the viewpoint;

searching, for each object of the objects, for a piece of information relating to the object using the real-world position of the object as a search key, and acquiring the piece of information;

outputting an image on the display in which pieces of information corresponding to the objects are superimposed on the captured real-world image; and

controlling, for each of the objects, the piece of information output from the object superimposition section,

wherein, the shorter the acquired horizontal distance is, the greater an information amount for the piece of information which is displayed in the image is to be decided, for each of the objects, and

the information amount for the piece of information, for each of the objects, is determined based on a number of the objects and a proportion of the captured real-world image occupied by object areas of the objects, the piece of information corresponding to each object being displayed in an area of the captured real-world image other than the object areas.