Image providing system and image providing method

• US 8,531,514 B2
• Filed: 09/18/2008
• Issued: 09/10/2013
• Est. Priority Date: 09/20/2007
• Status: Active Grant

• Current Assignee: NEC Corporation
• Original Assignee: NEC Corporation
• Inventors: Shimizu, Motoaki, Imaoka, Hitoshi, Chono, Keiichi, Adachi, Fumihiro, Nakazawa, Satoshi, Ishiguro, Shin, Yuguchi, Masayuki
• Primary Examiner(s): Kelley, Christopher S
• Assistant Examiner(s): MESSMORE, JONATHAN R
• Application Number:
  US12/679,277
• Publication Number:
  US 20100214400A1
• Time in Patent Office: 1,818 Days
• Field of Search: 348/77, 348113-172, 348180-376
• US Class Current: 348/77
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06V 20/63 Scene text, e.g. street names
• CPC Class Codes: H04N 1/00244 with a server, e.g. an internet server fax-servers or the like for store and forward H04N1/324
• CPC Class Codes: H04N 1/00307 with a mobile telephone apparatus constructional features of portable telephone sets H04M1/0202; mobile radio systems H04W84/00
• CPC Class Codes: H04N 1/00323 with a measuring, monitoring or signaling apparatus, e.g. for transmitting measured information to a central location
• CPC Class Codes: H04N 1/32776 using an interactive, user-operated device, e.g. a computer terminal, mobile telephone H04N1/32782, H04N1/32784 take precedence
• CPC Class Codes: H04N 21/21805 enabling multiple viewpoints, e.g. using a plurality of cameras
• CPC Class Codes: H04N 21/25841 involving the geographical location of the client retrieval from the Internet by querying based on geographical locations G06F16/9537; arrangements for identifying locations of receiving stations in broadcast systems H04H60/51; location of the user terminal in data switching networks H04L67/52; services making use of the location of users or terminals in wireless networks H04W4/02; locating users or terminals in wireless networks H04W64/00
• CPC Class Codes: H04N 21/41407 embedded in a portable device, e.g. video client on a mobile phone, PDA, laptop constructional details of equipment or arrangements specially adapted for portable computer application G06F1/1626; arrangements specially adapted for mobile receivers in broadcast systems H04H20/57
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 2201/0084 Digital still camera
• CPC Class Codes: H04N 7/173 with two-way working, e.g. subscriber sending a programme selection signal

Abstract

A terminal has a visual axis by using which a user points to a target object. An information generating server acquires an internal image or an external image of the target object pointed by a user, and generates a display image which is displayed on the terminal.

Claims

1. An image providing system comprising:

a terminal which has a visual axis by using which a user points a target object to be identified from a plurality of objects and is configured to identify said target object pointed by said visual axis from said plurality of objects;

an information generating server configured to acquire an internal image as an image of an inside of said target object or an external image as an image of external surroundings, which is taken from said target object, and generate a display image as an image to be outputted by said terminal based on said internal image or said external image, wherein said external surroundings can not be seen from said user with eyes and said display image is an image to be seen along a direction of the terminal which is a direction of said visual axis or a direction close to said direction of said visual axis; and

a positional information server configured to store object positional information as three-dimensional positional information of each of said plurality of objects and retrieval information as information for accessing to information generating server corresponding to said target object such that said object positional information and said retrieval information are respectively correlated to said each object,

wherein said terminal includes;

a position sensor configured to detect terminal positional information indicating a position of said terminal,

a direction sensor configured to detect terminal directional information indicating said direction of said terminal,

a user image acquisition camera for acquiring an image of a user,

an identifying unit configured to detect a face and both eyes from an image of a face of a user taken by said user image acquisition camera, set a user position indicating a position of said user at a center point between said both eyes based on said image of said face and said terminal positional information, and identify said target object based on said user position, said terminal positional information and said object positional information,

an information acquiring unit configured to access said information generating server corresponding to said target object based on said retrieval information, and acquire said display image of said target object, and

an output unit configured to display said display image.

2. The image providing system according to claim 1, wherein said information generating server includes:

an image acquiring unit including one or more image acquisition cameras for acquiring said internal image or said external image;

a display image generating unit configured to generate said display image based on said internal image or said external image; and

an information generating unit configured to order said output unit to generate information based on a request from said information acquiring unit.

3. The image providing system according to claim 2, wherein said one or more image acquisition cameras of said image acquiring unit further includes:

a camera position sensor configured to detect camera positional information indicating a position of a camera; and

a camera direction sensor configured to detect camera directional information indication a direction of a camera,

said display image generating unit acquires said terminal positional information, said terminal directional information, or both of said terminal positional information and said terminal directional information,

said display image generating unit, based on said camera positional information, said camera directional information, or both of said camera positional information and said camera directional information and based on said terminal positional information, said terminal directional information, or both of said terminal positional information and said terminal directional information, identifies from said one or more image acquisition cameras, a close camera as said image acquisition camera closest to said visual axis of said terminal, and

said display image generating unit generates said display image based on said internal image or said external image acquired by said close camera.

4. The image providing system according to claim 2, wherein said information generating server further includes an information storage unit that stores relevant information as information related to said target object,

said information storage unit stores as said relevant information, said internal image or said external image acquired by said image acquiring unit, and

said display image generating unit generates said display image based on said internal image or said external image stored in said information storage unit.

5. An image providing system comprising:

a terminal which has a visual axis by using which a user points a target object to be identified from a plurality of objects and is configured to identify said target object pointed by said visual axis from said plurality of objects;

an information generating server configured to acquire an internal image as an image of an inside of said target object, and generate a display image as an image to be outputted by said terminal based on said internal image, wherein said internal image can not be seen from said user with eyes and said display image is an image to be seen along a direction of the terminal which is a direction of said visual axis or a direction close to said direction of said visual axis; and

a positional information server configured to store object positional information as three-dimensional positional information of each of said plurality of objects and retrieval information as information for accessing to information generating server corresponding to said target object such that said object positional information and said retrieval information are respectively correlated to said each object,

wherein said terminal identifies said target object pointed by said visual axis based on a position of said terminal and the direction of said terminal, accesses said information generating server corresponding to said target object based on said retrieval information of said identified target object, acquires said display image of said target object from said information generating server, and displays said display image, said terminal including;

a position sensor configured to detect terminal positional information indicating the position of said terminal;

a direction sensor configured to detect terminal directional information indicating said direction of said terminal;

an identifying unit configured to identify said target object based on said terminal positional information, said terminal directional information and said object positional information;

an output unit configured to display said display image; and

an information acquiring unit configured to access said information generating server corresponding to said target object based on said retrieval information, acquire said display image of said target object, and output said display image to said output unit,

wherein said information generating server includes;

an image acquiring unit including one or more image acquisition cameras for acquiring said internal image;

a display image generating unit configured to estimate a three-dimensional shape of the inside of said target object based on said internal image, and generate said display image, an arbitrary view point image in which said three-dimensional shape is seen from said position indicated by said terminal positional information along said direction indicated by said terminal directional information; and

an information generating unit configured to order said output unit to generate information based on a request from said information acquiring unit.

6. The image providing system according to claim 5, wherein the inside of said target object is divided into small blocks as one or more spatial regions,

said positional information server further stores in a database, small block information as three-dimensional positional information of said small blocks and retrieval information as information for accessing an information generating server corresponding to said small blocks,

said identifying unit identifies a target small block from said small blocks,

said information acquiring unit accesses said information generating server corresponding to said target small block identified by said identifying unit and acquires a small block display image of said target small block, and

said output unit outputs said small block display image.

7. An image providing method comprising:

identifying a target object pointed by a visual axis which is set to a terminal in order to point said target object as an object which a user intends to identify from a plurality of objects;

accessing an information generating server corresponding to said identified target object; and

displaying on an output unit of said terminal, an internal image as an image of an inside of said target object,

wherein said internal image can not be seen from said user with eyes,

said internal image is an image to be seen along a direction of said visual axis or a direction close to said direction of said visual axis, and

said identifying said target object includes;

acquiring said terminal positional information as a position of said terminal;

acquiring object positional information as positional information of said target object;

detecting a face and both eyes of user from an image acquired by a user image acquisition camera of said terminal;

setting a user position indicating a position of said user at a center point of a line connecting said both eyes of said user based on an image of said face and said terminal positional information; and

identifying said target object based on said user position, said terminal positional information, and said object positional information.

8. The image providing method according to claim 7, wherein said identifying said target object further includes:

detecting an object identification instruction;

acquiring terminal directional information as a direction of said terminal; and

identifying said target object based on said terminal positional information, said terminal directional information, and said object positional information.

9. The image providing method according to claim 8, wherein said accessing said information generating server includes:

acquiring retrieval information as information for accessing said information generating server corresponding to said target object based on object positional information of said target object;

accessing said information generating server based on said retrieval information;

acquiring an object information list as an output display for a selection of information to be acquired by said user; and

displaying said object information list on said output unit of said terminal.

10. The image providing method according to claim 9, wherein said displaying on said output unit includes:

acquiring said internal image from one or more image acquisition cameras installed to said target object;

generating based on said internal image, an display image as an image to be outputted to said terminal; and

displaying said display image on said output unit of said terminal.

11. The image providing method according to claim 10, wherein said acquiring said internal image includes:

identifying a close camera as said image acquisition camera closest to said visual axis of said terminal among said image acquisition cameras based on camera positional information indicating positions of said image acquisition cameras, camera directional information indicating directions of said image acquisition cameras, or both of said camera positional information and said camera directional information and based on said terminal positional information, said terminal directional information, or both of said terminal positional information and said terminal directional information; and

acquiring said internal image from said close camera.

12. The image providing method according to claim 10, wherein said generating based on said internal image includes:

estimating a three-dimensional shape of the inside of said target object based on said internal image acquired from said image acquisition camera;

generating as said display image, an arbitrary view point image in which said three-dimensional shape is seen from said position of said terminal indicated by said terminal positional information along said direction of said terminal indicated by said terminal directional information; and

displaying said arbitrary view point image on said output unit of said terminal.

13. The image providing method according to claim 8, wherein the inside of said target object is divided into small blocks as one or more spatial regions,

said acquiring said object positional information includes acquiring small block positional information as three-dimensional positional information of said small blocks, and

said identifying said target object based on said terminal positional information, said terminal directional information, and said object positional information includes identifying a target small block based on said terminal positional information, said terminal directional information, and said small block positional information.

14. The image providing method according to claim 10, wherein said acquiring said internal image includes storing said internal image, and

said generating based on said internal image includes generating said display image based on said stored internal image.

15. A non-transitory computer-readable recording medium which records a program that when executed causes a terminal to perform a method comprising:

identifying a target object pointed by a visual axis which is set to said terminal in order to point said target object as an object which a user intends to identify from a plurality of objects;

accessing an information generating server corresponding to said identified target object; and

displaying on an output unit of said terminal, an internal image as an image of an inside of said target object,

wherein said internal image can not be seen from said user with eyes,

said internal image is an image to be seen along a direction of said visual axis or a direction close to said direction of said visual axis, and

said identifying said target object includes;

acquiring said terminal positional information as a position of said terminal;

acquiring object positional information as positional information of said target object;

detecting a face and both eyes of user from an image acquired by a user image acquisition camera of said terminal;

setting a user position indicating a position of said user at a center point of a line connecting said both eyes of said user based on an image of said face and said terminal positional information; and

identifying said target object based on said user position, said terminal positional information, and said object positional information.