Information processor, processing method and program for displaying a virtual image

• US 8,751,969 B2
• Filed: 06/30/2010
• Issued: 06/10/2014
• Est. Priority Date: 07/21/2009
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Matsuda, Kouichi, Fukuchi, Masaki
• Primary Examiner(s): Vu, Kieu
• Assistant Examiner(s): Chung, Andrew
• Application Number:
  US13/383,511
• Publication Number:
  US 20120124509A1
• Time in Patent Office: 1,441 Days
• Field of Search: 715856-862, 345 7- 9
• US Class Current: 715/856
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2219/2004 Aligning objects, relative positioning of parts

Abstract

The present invention provides a configuration that allows for a cursor or other object that has moved outside the display section to be displayed as a virtual object and observed. A cursor or object lying in an area outside the area of the display section of a PC or other device is displayed as a virtual object. For example, the display of goggles worn by the user displays a display device such as a PC and the area outside the display device. The three-dimensional position of the cursor or object that has probably moved in response to user operation is calculated, after which the cursor or object is displayed as a virtual object at the calculated position. Further, object information for the object specified by the cursor is acquired and presented.

Claims

1. An information processor comprising:

a coordinate processing module adapted to determine whether a position of a cursor displayed on a first display section, is located in or outside an area of the first display section and output cursor position information to a virtual object management section if the cursor is located outside the area of the first display section;

a camera adapted to capture an image made up of a real object including the first display section;

a three-dimensional information analysis section adapted to analyze a three-dimensional position of the real object included in the camera-captured image;

a second display section adapted to display the camera-captured image;

a virtual object management section adapted to generate a virtual object different from the real object included in the camera-captured image and generate a composite image including the generated virtual object and the real object so as to display the composite image on the second display section; and

an application execution section adapted to process a specified object specified by the cursor,

wherein the virtual object management section calculates a three-dimensional position of the cursor based on the cursor position information supplied from the coordinate processing module so as to display, on the second display section, the composite image in which a virtual cursor is placed at a first calculated position;

wherein the application execution section determines whether the specified object is moved by the cursor to locate in or outside the area of the first display section and outputs object position information of the specified object to the virtual object management section if the specified object is moved to locate outside the area of the first display section; and

wherein the virtual object management section calculates a three-dimensional position of the specified object based on the object position information of the specified object so as to display, on the second display section, a composite image in which a virtual specified object is placed at a second calculated position.

2. The information processor of claim 1,

wherein if the three-dimensional position of the specified object includes the display area of the first display section, the virtual object management section displays, on the second display section, a composite image with an object area image overlapping the display area of the first display section.

3. The information processor of claim 1 further comprising:

an object information acquisition section adapted to acquire image data of a real object specified by the cursor placed as the virtual object and search data based on the acquired image data so as to acquire object information,

wherein the object information acquisition section outputs the acquired object information to the first display section as display data.

4. The information processor of claim 3, wherein

the object information acquisition section accesses a database or a server storing real object image data and object information associated with each other so as to acquire object information through a search based on the real object image data.

5. The information processor of claim 1, wherein

the virtual object management section calculates a plane including the display surface of the first display section based on three-dimensional position information of components making up the first display section included in the camera-captured image and calculates the three-dimensional position of the cursor so that the cursor position is placed on the plane.

6. The information processor of claim 1, wherein

the cursor is a mouse cursor that moves by mouse operation, and

the coordinate processing module receives mouse cursor displacement information resulting from the mouse operation and determines whether the mouse cursor is located in or outside the area of the first display section.

7. An information processing method performed by an information processor, the information processing method comprising:

determining, by the information processor, whether a position of a cursor displayed on a first display section, is located in or outside an area of the first display section and outputting cursor position information if the cursor is located outside the area of the first display section;

analyzing a three-dimensional position of a real object in a camera-captured image, the camera-captured image including the first display section;

generating a virtual object different from the real object included in the camera-captured image;

generating a composite image including the generated virtual object and the real object so as to display the composite image on a second display section;

calculating a three-dimensional position of the cursor based on the cursor position information so as to display, on the second display section, a composite image in which a virtual cursor is placed at a first calculated position;

determining whether a specified object specified by the cursor is moved by the cursor to locate in or outside the area of the first display section and outputting object position information of the specified object to the virtual object management section if the specified object is moved to locate outside the area of the first display section; and

calculating a three-dimensional position of the specified object based on the object position information of the specified object so as to display, on the second display section, a composite image in which a virtual specified object is placed at a second calculated position.

8. A non-transitory computer-readable medium storing a program which, when executed by an information processor, causes the information processor to:

determine whether a position of a cursor displayed on a first display section, is located in or outside an area of the first display section and output cursor position information to a virtual object management section if the cursor is located outside the area of the first display section;

analyze a three-dimensional position of the real object included in the camera-captured image, the camera-captured image including the first display section;

generate a virtual object different from the real object included in the camera-captured image and generate a composite image including the generated virtual object and the real object so as to display the composite image on the second display section;

calculate a three-dimensional position of the cursor based on the cursor position information supplied from the coordinate processing module so as to display, on the second display section, a composite image in which a virtual cursor is placed at the calculated position;

determine whether a specified object specified by the cursor is moved by the cursor to locate in or outside the area of the first display section and output object position information of the specified object to the virtual object management section if the specified object is moved to locate outside the area of the first display section; and

calculate a three-dimensional position of the specified object based on the object position information of the specified object so as to display, on the second display section, a composite image in which a virtual specified object is placed at a second calculated position.