Method of automatically forming one three-dimensional space with multiple screens

• US 8,988,343 B2
• Filed: 03/29/2013
• Issued: 03/24/2015
• Est. Priority Date: 03/29/2013
• Status: Active Grant

• Current Assignee: Panasonic Intellectual Property Management Co., Ltd. (Panasonic Holdings Corporation)
• Original Assignee: Panasonic Intellectual Property Management Co., Ltd. (Panasonic Holdings Corporation)
• Inventors: Fei, Yue, Melle, Andrea, Kryze, David
• Primary Examiner(s): Dharia, Prabodh M
• Application Number:
  US13/853,767
• Publication Number:
  US 20140293017A1
• Time in Patent Office: 725 Days
• Field of Search: 345/8, 345101-107, 345/156, 345/173, 345/214, 345/420, 345/581, 345/589, 345629-639, 345/685, 345/690, 359/844, 715/850, 715/854, 348/118, 348/14.01, 382/103, 382/165, 382/154, 382/255, 455/414.1, 455/450, 455/556.1
• US Class Current: 345/156
• CPC Class Codes: G02F 1/13336 Combining plural substrates to produce large-area displays, e.g. tiled displays
• CPC Class Codes: G06F 3/01 Input arrangements or combined input and output arrangements for interaction between user and computer G06F3/16 takes precedence
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/0325 using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image
• CPC Class Codes: G06F 3/1446 display composed of modules, e.g. video walls
• CPC Class Codes: G09F 9/3026 Video wall, i.e. stackable semiconductor matrix display modules video wall control systems G06F3/1446, G09G2300/026
• CPC Class Codes: G09G 2320/0261 in the context of movement of objects on the screen or movement of the observer relative to the screen
• CPC Class Codes: G09G 2356/00 Detection of the display position w.r.t. other display screens
• CPC Class Codes: H04N 13/106 Processing image signals for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/327 Calibration thereof
• CPC Class Codes: H04N 13/332 Displays for viewing with the aid of special glasses or head-mounted displays [HMD]
• CPC Class Codes: H04N 13/366 using viewer tracking
• CPC Class Codes: H04N 13/398 Synchronisation thereof; Control thereof

Abstract

Multiple electronic displays are arranged to define an ensemble of the user'"'"'s choosing. The processor generates and displays a uniquely encoded emblem encoding a unique identifier for that display. A camera captures at least one image of the ensemble so that the encoded emblem for each display is captured. The captured images are then processed to extract the following attributes: a) unique identifier of the display; b) position of the encoded emblem relative to the reference coordinate system; and c) pointing direction of the encoded emblem relative to the reference coordinate system. The processor collectively processes the attributes for each of the emblem images to compute for each display a transformation matrix that selects a portion of the image data for display on that monitor and performs pointing direction correction so that the information presented on that display appears spatially consistent with the information presented on the other displays.

Claims

1. A method for presenting information across multiple electronic displays, comprising:

arranged to define a non-coplanar ensemble;

generating and displaying on each display a uniquely encoded emblem of predetermined size and shape that encodes a unique identifier for that display;

optically capturing at least one digital image of the ensemble so that the encoded emblem for each display is captured from a vantage point spaced apart from the ensemble, the optically captured image having a captured size that varies from the predetermined size based on distance from vantage point to ensemble and having a captured shape that has foreshortened dimensions across the line of sight between vantage point and ensemble;

defining a reference coordinate system corresponding to the space occupied by the ensemble of electronic displays;

processing each of the captured encoded emblem images to extract the following attributes;

a) unique identifier of the display;

b) position of the encoded emblem relative to the reference coordinate system; and

c) three-dimensional orientation of the encoded emblem relative to the reference coordinate system, where the three-dimensional orientation is computed using the captured size and shape;

supplying the information to be presented as image data to be displayed across the ensemble of displays; and

collectively processing the attributes for each of the encoded emblem images to compute for each display a transformation matrix that selects a portion of the image data for display on that monitor and that performs three-dimensional orientation correction so that the information presented on that display appears spatially consistent with the information presented on the other displays.

2. The method of claim 1 wherein said generating and displaying on each display a uniquely encoded emblem step is performed by generating and displaying an encoded emblem of known size and shape.

3. The method of claim 1 wherein said unique identifier is expressed using a machine-readable code.

4. The method of claim 1 wherein said unique identifier is expressed using a QR code.

5. The method of claim 1 wherein said step of capturing at least one digital image is performed using a still camera.

6. The method of claim 1 wherein said step of capturing at least one digital image is performed using a video camera.

7. The method of claim 1 wherein said step of capturing at least one digital image is performed by using a processor programmed to control a camera.

8. The method of claim 1 wherein the position attribute is extracted using graphical processing to locate the center of the emblem.

9. The method of claim 1 wherein the three-dimensional orientation attribute is extracted using graphical processing to locate and measure angles defined by edges of the emblem.

10. The method of claim 1 wherein the position attribute is extracted using graphical processing to locate the center of the emblem and using this center as x,y coordinates that are projected into said reference coordinate system.

11. The method of claim 1 further comprising generating and displaying on each display a uniquely encoded emblem of predetermined fixed size and shape.

12. The method of claim 11 further comprising:

using a camera to capture at least one image of the encoded emblem;

using graphical processing to determine the size of the captured encoded emblem;

comparing the size of the captured encoded emblem with said predetermined fixed size to determine a parameter indicative of distance from the camera to the displayed encoded emblem; and

using said parameter indicative of distance as a z coordinate of said position attribute.

13. The method of claim 1 further comprising:

using a camera to capture multiple images of the encoded emblem for each display, where at least some of the multiple images are taken from different vantage points relative to the space occupied by the ensemble; and

processing the multiple images by triangulation to determine the position attribute in the reference coordinate system.

14. The method of claim 1 wherein the reference coordinate system is a three-dimensional reference system.

15. The method of claim 1 wherein said step of supplying information to be presented comprises generating different information for each of a viewer'"'"'s left and right eyes.

16. The method of claim 15 further comprising providing said displays with at least one associated camera that captures information indicative of the location of a user while viewing said ensemble and using said user location information to modify said data streams to account for the position of the user'"'"'s left and right eyes.

17. The method of claim 15 further comprising providing at least a plurality of said displays each with an associated camera that captures information indicative of the location of a user while viewing said ensemble and combining the information indicative of the location of a user from said associated cameras into a single user head-tracking estimate.

18. The method of claim 17 wherein said combining step is performed by generating weighting factors based on head-tracking confidence from each associated camera and applying said weighting factors when combining the information indicative of user location from each of said associated cameras.

19. The method of claim 1 wherein said step of supplying information to be presented comprises generating different information data streams for each of a viewer'"'"'s left and right eyes;

and applying said transformation matrix separately to each of said data streams.

20. A method for ascertaining the positions and orientations of a plurality of electronic displays arranged in a non-coplanar ensemble, comprising:

generating and displaying on each display a uniquely encoded emblem of predetermined size and shape that encodes a unique identifier for that display;

optically capturing at least one digital image of the ensemble so that the encoded emblem for each display is captured from a vantage point spaced apart from the ensemble, the optically captured image having a captured size that varies from the predetermined size based on distance from vantage point to ensemble and having a captured shape that has foreshortened dimensions across the line of sight between vantage point and ensemble;

defining a reference coordinate system corresponding to the space occupied by the ensemble of electronic displays;

processing each of the captured encoded emblem images to extract the following attributes;

a) unique identifier of the display;

b) position of the encoded emblem relative to the reference coordinate system; and

c) three-dimensional orientation of the encoded emblem relative to the reference coordinate system, where the three-dimensional orientation is computed using the captured size and shape; and

supplying the information to be presented as image data to be displayed across the ensemble of displays.

21. An apparatus for presenting information across multiple electronic displays that are arranged as a non-coplanar ensemble, comprising:

a processor programmed to generate a uniquely encoded emblem of predetermined size and shape that encodes a unique identifier for each display, the processor having associated graphics circuitry to send the encoded emblem to the display for visual presentation;

a camera positioned to optically capture at least one digital image of the ensemble so that the encoded emblem for each display is captured from a vantage point spaced apart from the ensemble, the optically captured image having a captured size that varies from the predetermined size based on distance from vantage point to ensemble and having a captured shape that has foreshortened dimensions across the line of sight between vantage point and ensemble;

the processor being programmed to define a reference coordinate system corresponding to the space occupied by the ensemble of electronic displays;

the processor being further programmed to process each of the captured encoded emblem images to extract the following attributes;

a) unique identifier of the display;

b) position of the encoded emblem relative to the reference coordinate system; and

c) three-dimensional orientation of the encoded emblem relative to the reference coordinate system, where the three-dimensional orientation is computed using the captured size and shape;

the processor having an input port to receive the information to be presented as image data to be displayed across the ensemble of displays; and

the processor being programmed to collectively process the attributes for each of the encoded emblem images to compute for each display a transformation matrix that selects a portion of the image data for display on that monitor and that performs three-dimensional orientation correction so that the information presented on that display appears spatially consistent with the information presented on the other displays.