Operating environment with gestural control and multiple client devices, displays, and users

• US 9,740,293 B2
• Filed: 12/31/2013
• Issued: 08/22/2017
• Est. Priority Date: 04/02/2009
• Status: Active Grant

• Current Assignee: Oblong Industries, Inc. (Oblong, Inc.)
• Original Assignee: Oblong Industries, Inc. (Oblong, Inc.)
• Inventors: Kramer, Kwindla Hultman, Underkoffler, John, Sparrell, Carlton, Singh, Navjot, Hollenbach, Kate, Yarin, Paul
• Primary Examiner(s): Shankar, Vijay
• Application Number:
  US14/145,016
• Publication Number:
  US 20150077326A1
• Time in Patent Office: 1,330 Days
• Field of Search: 345156-178
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/02 Input arrangements using manually operated switches, e.g. using keyboards or dials
• CPC Class Codes: G06F 3/0236 using selection techniques to select from displayed items
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0325 using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/03545 Pens or stylus
• CPC Class Codes: G06F 3/04812 Interaction techniques based on cursor appearance or behaviour, e.g. being affected by the presence of displayed objects
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06V 10/245 by locating a pattern; Special marks for positioning
• CPC Class Codes: G06V 10/50 by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis
• CPC Class Codes: G06V 40/107 Static hand or arm
• CPC Class Codes: G06V 40/113 Recognition of static hand signs
• CPC Class Codes: H04L 67/025 for remote control or remote monitoring of applications
• CPC Class Codes: H04M 3/567 Multimedia conference systems
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q
• CPC Class Codes: H04N 7/15 Conference systems

Abstract

Embodiments described herein includes a system comprising a processor coupled to display devices, sensors, remote client devices, and computer applications. The computer applications orchestrate content of the remote client devices simultaneously across the display devices and the remote client devices, and allow simultaneous control of the display devices. The simultaneous control includes automatically detecting a gesture of at least one object from gesture data received via the sensors. The detecting comprises identifying the gesture using only the gesture data. The computer applications translate the gesture to a gesture signal, and control the display devices in response to the gesture signal.

Claims

1. A system comprising:

a processor coupled to a display system comprising a plurality of display devices;

a plurality of remote client devices coupled to the processor, wherein each remote client device of the plurality of remote client devices includes content of a session workflow; and

a plurality of applications coupled to the processor, wherein the plurality of applications integrate the content of each of the plurality of remote client devices simultaneously in a session workflow hosted at the display system, and allow simultaneous control of the content at the display system, wherein the simultaneous control comprises receiving event data from source devices of the plurality of remote client devices and controlling the session workflow with the content at the display system in response to the event data.

2. The system of claim 1, wherein a remote client device of the plurality of remote client devices is configured to detect an event of a source device, and generate at least one data sequence comprising device event data specifying the event and state information of the event.

3. The system of claim 2, wherein the device event data and state information are type-specific data having a type corresponding to an application of the source device.

4. The system of claim 3, wherein the remote client device is configured to form a data capsule to include the at least one data sequence.

5. The system of claim 4, wherein the data capsule comprises a data structure including an application-independent representation of the at least one data sequence.

6. The system of claim 5, wherein the remote client device is configured to transfer the data capsule to a repository coupled to the plurality of display devices.

7. The system of claim 6, wherein the data capsule is configured to maintain intact the at least one data sequence of the data capsule during the transfer.

8. The system of claim 6, wherein the processor is configured to detect a second event of the display system and search the repository for data capsules corresponding to the second event.

9. The system of claim 8, wherein the processor is configured to identify a correspondence between the data capsule and the second event of the display system and in response extract the data capsule from the repository.

10. The system of claim 9, wherein the processor is configured to execute on behalf of the display system a processing operation corresponding to the second event in response to contents of the data capsule.

11. The system of claim 10, wherein the source device corresponds to an application of a first type and the display system corresponds to a second application of a second type.

12. The system of claim 6, wherein the repository is coupled to a plurality of applications running on the processor, the repository including a plurality of data capsules corresponding to the plurality of applications, the repository providing access to the plurality of data capsules by the plurality of applications, wherein at least two applications of the plurality of applications are different applications.

13. The system of claim 6, wherein the repository provides state caching of a plurality of data capsules.

14. The system of claim 6, wherein the repository provides linear sequencing of a plurality of data capsules.

15. The system of claim 6, wherein the generating of the at least one data sequence comprises:

generating a first respective data set that includes first respective device event data;

generating a second respective data set that includes second respective state information; and

forming a first data sequence to include the first respective data set and the second respective data set.

16. The system of claim 15, wherein the generating of the first respective data set includes forming the first respective data set to include identification data of the source device, the identification data including data identifying the source device.

17. The system of claim 15, wherein the generating of the at least one data sequence comprises:

generating a first respective data set that includes first respective device event data;

generating a second respective data set that includes second respective state information; and

forming a second data sequence to include the first respective data set and the second respective data set.

18. The system of claim 17, wherein the generating of the first respective data set includes generating a first respective data set offset, wherein the first respective data set offset points to the first respective data set of the second data sequence.

19. The system of claim 17, wherein the generating of the second respective data set includes generating a second respective data set offset, wherein the second respective data set offset points to the second respective data set of the second data sequence.

20. The system of claim 15, wherein the first respective data set is a description list, the description list including a description of the data.

21. The system of claim 15, wherein the device event data is a tagged byte-sequence representing typed data.

22. The system of claim 21, wherein the device event data includes a type header and a type-specific data layout.

23. The system of claim 15, wherein the state information is a tagged byte-sequence representing typed data.

24. The system of claim 23, wherein the state information includes a type header and a type-specific data layout.

25. The system of claim 15, comprising:

generating at least one offset; and

forming the data capsule to include the at least one offset.

26. The system of claim 25, comprising:

generating a first offset having a first variable length;

wherein the first offset points to the device event data of a first data sequence of the at least one data sequence.

27. The system of claim 25, comprising:

generating a second offset having a second variable length;

wherein the second offset points to the state information of a first data sequence of the at least one data sequence.

28. The system of claim 25, comprising:

forming a first code path through the data capsule using a first offset of the at least one offset;

forming a second code path through the data capsule using a second offset of the at least one offset;

wherein the first code path and the second code path are different paths.

29. The system of claim 25, wherein at least one of the first offset and the second offset include metadata, the metadata comprising context-specific metadata corresponding to a context of the event data.

30. The system of claim 15, comprising:

generating a header that includes a length of the data capsule;

forming the data capsule to include the header.

31. The system of claim 15, wherein the data structure is untyped.

32. The system of claim 15, wherein the data structure of the data capsule provides a platform-independent representation of the event data and the state information.

33. The system of claim 15, wherein the data structure of the data capsule provides platform-independent access to the event data and the state information.

34. The system of claim 15, wherein the event comprises a user interface event.

35. The system of claim 15, wherein the event comprises a graphics event.

36. The system of claim 15, wherein the event comprises depositing of state information.