Information processing apparatus, information processing method, and program for extending or expanding a viewing area of content displayed on a 2D workspace into a 3D virtual display screen

• US 8,994,721 B2
• Filed: 03/17/2010
• Issued: 03/31/2015
• Est. Priority Date: 04/23/2009
• Status: Expired due to Fees

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Matsuda, Kouichi
• Primary Examiner(s): Zarka, David
• Assistant Examiner(s): Vu, Khoa
• Application Number:
  US13/264,133
• Publication Number:
  US 20120032955A1
• Time in Patent Office: 1,840 Days
• Field of Search: None
• US Class Current: 345/419
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04N 13/156 Mixing image signals

Abstract

A configuration is realized which enables data not displayed on a display section to be always observed as a virtual object. A configuration is realized with which, in various kinds of display apparatus such as a PC and a portable telephone having a display section, non-displayed data that extends off the area of the display section is displayed as a virtual object at a position contiguous to the display data of the display section. This configuration allows the user to always observe and view not only data displayed on a limited display area of a PC or the like but also data that has gone off the display section, thereby enhancing data processing efficiency.

Claims

1. An information processing apparatus comprising:

at least one processor;

a physical display unit comprising a two dimensional physical display screen; and

a virtual display unit comprising a three dimensional virtual display screen, wherein the three dimensional virtual display screen is configured to at least one of extend or expand a viewing area of a content window displayed on the two dimensional physical display screen onto the three dimensional virtual display screen using the at least one processor, wherein the displayed content window spans between the two dimensional physical display screen and the three dimensional virtual display screen;

a three-dimensional information analysis section configured to analyze a three-dimensional position of a real object including the physical display section using the at least one processor;

a virtual object management section that takes input of the virtually displayed data and the positional information of the virtually displayed data which are stored in the memory using the at least one processor;

wherein the virtual object management section transforms coordinate data, which is the positional information of the virtually displayed data stored in the memory, into a camera coordinate system serving as an indicator of a display position in a display image of the second display section, and determines the display position of the virtual object by applying the transformed coordinate data using the at least one processor,

wherein the virtual object management section judges whether or not an intersection occurs between the display position of the virtual object transformed into the camera coordinate system, and a plane forming the real object included in the camera-captured image, and if the intersection occurs, executes a coordinate re-transformation process so as to position the transformed coordinate data in the plane forming the real object, and determines the display position of the virtual object by applying the re-transformed coordinate data using the at least one processor.

2. The information processing apparatus according to claim 1, wherein:

the information processing apparatus includes

a first apparatus having the application executing section, and

a second apparatus having the three-dimensional information analysis section, and the virtual object management section; and

data stored into a first memory in the first apparatus by the application executing section is transferred to the second apparatus via a communication section, and in the second apparatus, the three-dimensional information analysis section and the virtual object management section perform processing using received data using the at least one processor.

5. The information processing apparatus of claim 1, comprising an application executing section configured to store virtually displayed data configured to be viewed from the virtual display unit and positional information of the virtually displayed data into a memory in data processing using a first display section of the physical display unit as at least one reference point for the positional information using the at least one processor.

6. The information processing apparatus of claim 5 where the virtually displayed data is displayed contiguous to display data displayed by the physical display unit.

7. The information processing apparatus of claim 1 wherein the real object is captured by a camera.

8. The information processing apparatus of claim 1, wherein the three-dimensional positional information of the physical display section is analyzed by the three-dimensional information analysis section and is configured to:

generate a composite image formed by the real object included in a camera-captured image and a virtual object formed by the virtually displayed data using the at least one processor; and

display the composite image on a second display section using the at least one processor.

9. The information processing apparatus of claim 8, wherein the virtual object management section is configured to set a display position of the virtual object on the virtual display section at a position located in a direction of extension from a boundary of the display data of the physical display section displayed on the physical display section which is outside of the physical display section using the at least one processor.

3. An information processing method which is executed in an information processing apparatus, comprising:

an application executing step of an application executing section storing virtual displayed data and positional information of the virtual displayed data into a memory in data processing using a first display section, the virtual displayed data being contiguous to physically display data of the first display section, wherein a physical display unit comprising a two dimensional physical display screen and a virtual display unit comprising a three dimensional virtual display screen, wherein the three dimensional virtual display screen is configured to at least one of extend or expand a viewing area of a content window displayed on the two dimensional physical display screen onto the three dimensional virtual display screen, wherein the displayed content window spans between the two dimensional physical display screen and the three dimensional virtual display screen;

a three-dimensional information analysis step of a three-dimensional information analysis section analyzing a three-dimensional position of a real object including the first display section included in a camera-captured image; and

a virtual object management step of a virtual object management section taking input of the virtually displayed data and the positional information of the physically displayed data which are stored in the memory, and three-dimensional positional information of the first display section analyzed by the three-dimensional information analysis section, generating a composite image formed by the real object included in the camera-captured image and a virtual object formed by the virtually displayed data, and displaying the composite image on a second display section,

wherein the virtual object management step has a step of setting a display position of the virtual object on the second display section, at a position located in a direction of extension from a boundary of the display data of the first display section displayed on the second display section to an outside of the first display section,

wherein the virtual object management step transforms coordinate data, which is the positional information of the virtually displayed data stored in the memory, into a camera coordinate system serving as an indicator of a display position in a display image of the second display section, and determines the display position of the virtual object by applying the transformed coordinate data, and

wherein the virtual object management step judges whether or not an intersection occurs between the display position of the virtual object transformed into the camera coordinate system, and a plane forming the real object included in the camera-captured image, and if the intersection occurs, executes a coordinate re-transformation process so as to position the transformed coordinate data in the plane forming the real object, and determines the display position of the virtual object by applying the re-transformed coordinate data.

4. A non-transitory computer-readable medium having embodied thereon a program, which when executed by a computer, causes information processing to be executed in an information processing apparatus, comprising:

an application executing step of causing an application executing section to store virtually displayed data and positional information of the virtually displayed data into a memory in data processing using a first display section, the virtually displayed data being contiguous to display data of the first display section;

a three-dimensional information analysis step of causing a three-dimensional information analysis section to analyze a three-dimensional position of a real object including the first display section included in a camera-captured image; and

a virtual object management step of causing a virtual object management section to take input of the virtually displayed data and the positional information of the virtually displayed data which are stored in the memory, and three-dimensional positional information of the first display section analyzed by the three-dimensional information analysis section, generate a composite image formed by the real object included in the camera-captured image and a virtual object formed by the virtually displayed data, and display the composite image on a second display section, wherein a physical display unit comprising a two dimensional physical display screen and a virtual display unit comprising a three dimensional virtual display screen, wherein the three dimensional virtually displayed data is configured to at least one of extend or expand a viewing area of a content window displayed on the two dimensional physical display screen onto the three dimensional virtual display screen, wherein the displayed content window spans between the two dimensional physical display screen and the three dimensional virtual display screen,

wherein the virtual object management step has a step of causing a display position of the virtual object on the second display section to be set, at a position located in a direction of extension from a boundary of the display data of the first display section displayed on the second display section to an outside of the first display section,

wherein the virtual object management step transforms coordinate data, which is the positional information of the virtually displayed data stored in the memory, into a camera coordinate system serving as an indicator of a display position in a display image of the second display section, and determines the display position of the virtual object by applying the transformed coordinate data, and

wherein the virtual object management step judges whether or not an intersection occurs between the display position of the virtual object transformed into the camera coordinate system, and a plane forming the real object included in the camera-captured image, and if the intersection occurs, executes a coordinate re-transformation process so as to position the transformed coordinate data in the plane forming the real object, and determines the display position of the virtual object by applying the re-transformed coordinate data.