Spatially mediated augmentations of and interactions among distinct devices and applications via extended pixel manifold

• US 10,529,302 B2
• Filed: 07/06/2017
• Issued: 01/07/2020
• Est. Priority Date: 07/07/2016
• Status: Active Grant

• Current Assignee: Oblong Industries, Inc. (Oblong, Inc.)
• Original Assignee: Oblong Industries, Inc. (Oblong, Inc.)
• Inventors: Underkoffler, John Stephen, Sparrell, Carlton J., Harvey, Brandon Lee Walter, Hawkes, Peter Joseph
• Primary Examiner(s): Chang, Kent W
• Assistant Examiner(s): Bibbee, Chayce R
• Application Number:
  US15/643,264
• Publication Number:
  US 20180012567A1
• Time in Patent Office: 915 Days
• CPC Class Codes: G06F 2203/04803 Split screen, i.e. subdividing the display area or the window area into separate subareas
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/14 Digital output to display device ; Cooperation and interconnection of the display device with other functional units
• CPC Class Codes: G06F 3/1446 display composed of modules, e.g. video walls
• CPC Class Codes: G09G 2320/10 Special adaptations of display systems for operation with variable images
• CPC Class Codes: G09G 2340/0464 Positioning
• CPC Class Codes: G09G 2340/10 Mixing of images, i.e. displayed pixel being the result of an operation, e.g. adding, on the corresponding input pixels
• CPC Class Codes: G09G 2354/00 Aspects of interface with display user
• CPC Class Codes: G09G 2370/20 Details of the management of multiple sources of image data
• CPC Class Codes: G09G 5/12 Synchronisation between the display unit and other units, e.g. other display units, video-disc players
• CPC Class Codes: G09G 5/14 Display of multiple viewports
• CPC Class Codes: H04L 67/141 Setup of application sessions admission control or resource allocation in data switching networks H04L47/70
• CPC Class Codes: H04W 4/025 using location based information parameters

Abstract

Systems and methods for display. Program instructions of a client application are accessed. The client application is an application of a client device that is located within a viewable region of a display device. Display output of a first display system application is generated by executing the accessed program instructions. A first absolute position of the client device is determined. A display area for the display output of the display system application is determined. A first pixel region of the display device is determined. The first display region corresponds to the determined display area and the first absolute position of the client device. The display output of the display system application is displayed by using the determined first pixel region.

Claims

1. A method comprising:

at a display system computing system that is communicatively coupled to a display device;

accessing first program instructions of a first client application of a first client device that is located within a first viewable region of the display device;

generating display output of a first display system application by executing the accessed first program instructions;

determining a first absolute three-dimensional (3D) position of the first client device in a three-dimensional coordinate space by using a 3D tracking system;

determining a first pixel width and a first pixel height for the display output of the first display system application;

determining a first pixel region, of the display device, that corresponds to the determined first pixel width and first pixel height and that corresponds to the first absolute 3D position of the first client device in the three-dimensional coordinate space; and

displaying the display output of the first display system application by using the determined first pixel region,

wherein the computing system uses the 3D tracking system to spatially track absolute 3D position of the first client device and a second client device in real time,

wherein the computing system communicatively couples the first display system application with a second display system application, of the second client device, based on proximity of the first client device to the second client device as determined by the tracking of the first client device and the second client device, and

wherein the computing system controls the display device to display a visual indicator that indicates that the first display system application and the second display system application are communicatively coupled.

2. The method of claim 1, further comprising:

the computing system using the 3D tracking system to spatially track absolute 3D position of the first client device in real time;

responsive to a change of position of the first client device from the first absolute 3D position to a second absolute 3D position, the computing system;

determining a second pixel region of the display device that corresponds to the determined first pixel width and first pixel height and the second absolute 3D position of the first client device; and

displaying the display output of the first display system application by using the determined second pixel region.

3. The method of claim 2, further comprising:

the computing system using the tracking system to spatially track absolute 3D position and orientation of a first user input device in real time;

the computing system generating user input information based on the spatial tracking of the 3D position and orientation of a first user input device in real time;

the computing system determining whether the generated user input information corresponds to the display output of the first display system application; and

responsive to a determination that the generated user input information corresponds to the display output of the first display system application, the computing system;

updating the first display system application by using the user input information, and

providing the user input information to the first client application.

4. The method of claim 3, further comprising:

at the computing system;

accessing second program instructions of a second client application of a second client device that is located within the first viewable region;

generating display output of a second display system application by executing the accessed second program instructions;

determining a first absolute 3D position of the second client device;

determining a second pixel width and a second pixel height for the display output of the second display system application;

determining a second pixel region of the display device that corresponds to the determined second pixel width and second pixel height for the second display system application and the first absolute 3D position of the second client device; and

displaying the display output of the second display system application by using the determined second pixel region for the second display system application,

wherein the computing system uses the tracking system to spatially track absolute 3D position of the second client device in real time.

5. The method of claim 1, further comprising:

the computing system transferring data from the first display system application to the communicatively coupled second display system application.

6. The method of claim 3, wherein the first user input device is an input device of a user of the first client device.

7. The method of claim 3, wherein the first user input device is an input device of a user of a second client device.

8. The method of claim 1, further comprising:

responsive to changed program data of the first client application, the computing system updating the first display system application by using the received changed program data of the first client application.

9. The method of claim 3,

wherein the display device is a multi-location multi-region display device,

wherein the first viewable region is a first viewable sub-region of a first sub-display device of the display device at a first location,

wherein the output of the first display system application is displayed at a display region of the first sub-display device;

wherein the output of a second display system application of a second client device located in a second viewable sub-region of a second sub-display device is displayed by a display region of the second sub-display device, and

wherein the second sub-display device is located at a second location.

10. The method of claim 9, wherein the first sub-display device displays a first visual indicator and the second sub-display device displays a second visual indicator, wherein the first visual indicator indicates presence of the second client device within the second viewable sub-region, and wherein the second visual indicator indicates presence of the first client device within the first viewable sub-region.

11. The method of claim 1, further comprising:

the computing system determining that the first client device is located within the first viewable region based on information received from the first client device.

12. The method of claim 1, further comprising:

the computing system determining that the first client device is located within the first viewable region based on information received from the tracking system that is communicatively coupled to the computing system.

13. The method of claim 1, further comprising:

the computing system augmenting functionality of the first display system application with respect to functionality of the first client application at the first client device.

14. The method of claim 3,

wherein determining whether the generated user input information corresponds to the display output of the first display system application comprises:

determining a cursor position of the first user input device based on the user input information, wherein the cursor position is a position of a cursor displayed by the display device;

determining whether the cursor position is a position within display output of the first display system application; and

in a case where the cursor position is a position within display output of the first display system application, the computing system determining that the generated user input information corresponds to the display output of the first display system application.

15. The method of claim 1, wherein determining the first pixel region of the display device comprises:

determining at least a first pixel having an absolute 3D position that corresponds to the first absolute 3D position of the first client device; and

determining a pixel region that has the first pixel width and the first pixel height, and that has the first pixel as a center pixel of the pixel region,

wherein the display system computing system receives the first program instructions of the first client application from the first client device, and

wherein each absolute 3D position tracked by the tracking system is represented by X, Y and Z, coordinates within the three-dimensional coordinate space tracked by the 3D tracking system.

16. The method of claim 15,

wherein determining at least a first pixel having an absolute 3D position that corresponds to the first absolute 3D position of the first client device comprises:

determining at least a first pixel having an absolute 3D position that is proximate to the first absolute 3D position of the first client device, and

wherein the computing system determines the absolute 3D position of the first pixel based on absolute 3D positions stored for each pixel of the display device.

17. The method of claim 15,

wherein determining at least a first pixel having an absolute 3D position that corresponds to the first absolute 3D position of the first client device comprises:

determining at least a first pixel having an absolute 3D position that is proximate to the first absolute 3D position of the first client device, and

wherein the computing system computes the absolute 3D position of the first pixel based on stored absolute 3D position information for at least one pixel of the display device and information indicating relative position of the first pixel within the display device.

18. The method of claim 1,

wherein the tracking system is communicatively coupled to the computing system,

wherein the computing system includes at least one display system computing device,

wherein the absolute 3D position of each pixel of the display device is an absolute 3D position within the three-dimensional coordinate space,

wherein the display device is a multi-region display device and the first pixel region includes pixels of at least one display region of the multi-region display device, and

wherein the computing system determines the first pixel region by accessing stored absolute 3D position information for at least one pixel of the display device.

19. The method of claim 18,

wherein a multi-region display system includes the display system computing system, the tracking system, and the multi-region display device,

wherein the first display system application is a spatial application,

wherein the first display system application is an application of a first spatial application display session of the first client device,

wherein the second display system application is a spatial application,

wherein the second display system application is an application of a second spatial application display session of the second client device,

wherein the computing system establishes the first spatial application display session with the first client application of the first client device,

wherein the computing system establishes the second spatial application display session with the second client application of the second client device, and

wherein the computing system communicatively couples the first display system application with the second display system application by merging the second spatial application display session with the first spatial application display session.