Control system for navigating a principal dimension of a data space

• US 9,471,147 B2
• Filed: 09/09/2013
• Issued: 10/18/2016
• Est. Priority Date: 02/08/2006
• Status: Active Grant

• Current Assignee: Oblong Industries, Inc. (Oblong, Inc.)
• Original Assignee: Oblong Industries, Inc. (Oblong, Inc.)
• Inventors: Underkoffler, John S., Kramer, Kwindla H.
• Primary Examiner(s): Shankar, Vijay
• Application Number:
  US14/021,628
• Publication Number:
  US 20140225822A1
• Time in Patent Office: 1,135 Days
• Field of Search: 345156-169, 345173-179
• US Class Current: 1/1
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0325 using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image
• CPC Class Codes: G06F 3/1423 controlling a plurality of local displays, e.g. CRT and flat panel display
• CPC Class Codes: G06F 3/1454 involving copying of the display data of a local workstation or window to a remote workstation or window so that an actual copy of the data is displayed simultaneously on two or more displays, e.g. teledisplay
• CPC Class Codes: G06T 7/00 Image analysis
• CPC Class Codes: G06V 10/245 by locating a pattern; Special marks for positioning
• CPC Class Codes: G06V 40/107 Static hand or arm
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training
• CPC Class Codes: G09G 2354/00 Aspects of interface with display user
• CPC Class Codes: G09G 5/00 Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators image data processing or generation, in general G06T
• CPC Class Codes: G09G 5/08 Cursor circuits

Abstract

Systems and methods are described for navigating through a data space. The navigating comprises detecting a gesture of a body from gesture data received via a detector. The gesture data is absolute three-space location data of an instantaneous state of the body at a point in time and physical space. The detecting comprises identifying the gesture using the gesture data. The navigating comprises translating the gesture to a gesture signal, and navigating through the data space in response to the gesture signal. The data space is a data-representational space comprising a dataset represented in the physical space.

Claims

1. A system comprising:

a detector configured to receive gesture data that represents a gesture made by a body; and

a processor coupled to the detector, the processor configured to detect the gesture from the gesture data, wherein the gesture data comprises at least one of position and orientation of the body, wherein the processor is configured to translate the gesture to a gesture signal, wherein the processor is configured to control navigation through a data space in response to the gesture signal, wherein the data space is a data-representational space comprising a dataset represented in the physical space.

2. The system of claim 1, wherein the processor is configured to align a parameter-control axis of the dataset with a dimension of the physical space.

3. The system of claim 2, wherein the dimension is a depth dimension.

4. The system of claim 2, wherein the dimension is a horizontal dimension.

5. The system of claim 2, wherein the dimension is a vertical dimension.

6. The system of claim 2, wherein the dimension is a lateral dimension.

7. The system of claim 2, wherein the navigating comprises motion along the dimension to effect a data-space translation along the parameter-control axis.

8. The system of claim 2, wherein the navigating comprises navigating to quantized parameter spaces of the data space.

9. The system of claim 1, wherein the detecting comprising identifying the gesture using only the gesture data.

10. The system of claim 1, wherein the gesture data is absolute three-space location data of an instantaneous state of the body at a point in time and physical space.

11. The system of claim 1, wherein the detecting includes detecting an evolving position of the body.

12. The system of claim 1, wherein the detecting includes detecting an evolving orientation of the body.

13. The system of claim 1, wherein the detecting includes detecting an evolving pose of the body, wherein the pose is a geometric disposition of a part of the body relative to at least one other part of the body.

14. The system of claim 1, wherein the detecting includes detecting evolving motion of the body.

15. The system of claim 1, wherein the detecting includes detecting at least one of an evolving position of the body, orientation of the body, pose of the body, and motion of the body.

16. The system of claim 1, wherein the processor is configured to analyze the gesture into a sequence of gestural events.

17. The system of claim 16, wherein the processor is configured to identify the gesture.

18. The system of claim 17, wherein the identifying of the gesture includes identifying at least one of an evolving position of the body, orientation of the body, pose of the body, and motion of the body.

19. The system of claim 17, wherein the processor is configured to generate a representation of the gestural events of the sequence of gestural events.

20. The system of claim 19, wherein the processor is configured to distribute the representation of the gestural events to at least one control component coupled to the data space.

21. The system of claim 20, wherein the processor is configured to synchronize the representation of the gestural events with a graphical depiction of the data space.

22. The system of claim 21, wherein the processor is configured to synchronize the representation of the gestural events with a graphical depiction of the navigating through the data space.

23. The system of claim 20, wherein the processor is configured to synchronize the representation of the gestural events with an aural depiction of the data space.

24. The system of claim 23, wherein the processor is configured to synchronize the representation of the gestural events with an aural depiction of the navigating through the data space.

25. The system of claim 1, wherein the dataset represents spatial information.

26. The system of claim 25, wherein the dataset represents spatial information of at least one of phenomena, events, measurements, observations, and structure.

27. The system of claim 1, wherein the dataset represents non-spatial information.

28. The system of claim 1, wherein the gesture comprises linear spatial motion.

29. The system of claim 1, wherein the navigating comprises linear verging through the data space.

30. The system of claim 1, wherein the processor is configured to:

render the dataset in a plurality of data frames that are graphical depictions of a plurality of regions of the data space;

display each data frame as a visible frame on a display.

31. The system of claim 30, wherein a size and an aspect ratio of the data frame coincide with the size and the aspect ratio of the display.

32. The system of claim 30, wherein a center and a normal vector of the data frame coincide with the center and the normal vector of the display.

33. The system of claim 30, wherein a position and an orientation of the data frame coincide with the position and the orientation of the display.

34. The system of claim 30, wherein each data frame comprises graphical data elements representing elements of the dataset.

35. The system of claim 34, wherein the graphical data elements are static elements.

36. The system of claim 34, wherein the graphical data elements are dynamic elements.

37. The system of claim 30, wherein the data frame is a two-dimensional construct.

38. The system of claim 37, wherein the data frame is resident in a three-dimensional graphics rendering environment having a coordinate system that coincides with coordinates that describe a local environment that includes the body.

39. The system of claim 30, wherein the navigating through the data space comprises navigating through the plurality of data frames.

40. The system of claim 30, wherein the processor is configured to:

detect a first pose of the body;

activate pushback interaction in response to detecting the first pose.

41. The system of claim 40, wherein the processor is configured to:

record a first position at which the first pose is entered, wherein the first position is a three-space hand position;

set the first position as an origin, wherein subsequent detected body positions are reported as relative offsets to the origin.

42. The system of claim 41, wherein the processor is configured to detect a gesture by detecting movement of the body.

43. The system of claim 42, wherein the detecting comprises detecting a first movement of the body, wherein the first movement is movement along an axis toward the display, wherein the axis is defined as an axis normal to a view surface of the display.

44. The system of claim 43, wherein, in response to the first movement of the body, the navigating comprises displacing the plurality of data frames along the axis, wherein more of the plane in which the data frames lie becomes visible, wherein a first visible frame rendered on the display is seen to recede from the display and neighboring data frames of the first data frame become visible.

45. The system of claim 43, wherein, in response to the first movement of the body, the navigating comprises displacing the plurality of data frames along the axis from a current data frame, wherein data frames translate downward, wherein the current data frame is the data frame currently rendered on the display.

46. The system of claim 45, wherein an appearance of each data frame varies in response to the displacing, wherein data frames positioned below the current data frame take on a relatively high degree of transparency and blur, wherein the current data frame transitions from visible and begins taking on a degree of transparency and blur, wherein data frames positioned above the current data frame take on a relatively lower degree of transparency and blur and become visible.

47. The system of claim 43, wherein the detecting comprises detecting a second movement of the body, wherein the second movement is movement along the axis away from the display.

48. The system of claim 47, wherein, in response to the second movement of the body, the navigating comprises displacing the plurality of data frames along the axis, wherein less of the plane in which the data frames lie becomes visible, wherein the first visible frame rendered on the display is seen to verge from the display and neighboring data frames of the first data frame become less visible.

49. The system of claim 47, wherein, in response to the second movement of the body, the navigating comprises displacing the plurality of data frames along the axis from a current data frame, wherein the data frames translate upward, wherein the current data frame is the data frame currently rendered on the display.

50. The system of claim 49, wherein an appearance of each data frame varies in response to the displacing, wherein data frames positioned above the current data frame take on a relatively high degree of transparency and blur, wherein the current data frame transitions from visible and begins taking on a degree of transparency and blur, wherein data frames positioned below the current data frame take on a relatively lower degree of transparency and blur and start to become visible.

51. The system of claim 42, wherein the detecting comprises detecting a third movement of the body, wherein the third movement is movement in a plane parallel to a view surface of the display.

52. The system of claim 51, wherein, in response to the third movement of the body, the navigating comprises displacing the plurality of data frames in a direction corresponding to the third movement.

53. The system of claim 52, wherein a current data frame rendered on the display is seen to slide in the direction and an adjacent data frame to the current data frame slides into view from a side of the display opposite the direction, wherein the current data frame is the data frame currently rendered on the display.

54. The system of claim 42, wherein the navigating comprises continuously updating a displacement of the plurality of data frames in direct response to the movement of the body.

55. The system of claim 54, wherein the processor is configured to:

detect a second pose of the body;

terminate pushback interaction in response to detecting the second pose, wherein the terminating comprises displaying a data frame of the plurality of data frames as coplanar with the display.

56. The system of claim 42, comprising a mapping between a physical distance of displacement of the body during a gesture and logical distance between successive points in the data space during the navigating, wherein the mapping controls at least one of relative displacement of the plurality of data frames in the data space and an appearance of each discrete data frame.

57. The system of claim 42, wherein the processor is configured to form a gestural interaction space comprising an active zone and a dead zone, wherein the active zone is adjacent the display and the dead zone is adjacent the active zone.

58. The system of claim 57, wherein the navigating through the data space in response to the gesture signal is activated in response to the gesture when the gesture is detected in the active region.

59. The system of claim 57, comprising a feedback indicator rendered on the display.

60. The system of claim 59, wherein the feedback indicator displays feedback indicating the body is in one of the active zone and the dead zone.

61. The system of claim 59, wherein the feedback indicator displays feedback indicating a physical offset of the body from the origin.

62. The system of claim 1, wherein the detecting includes at least one of detecting a location of the body, detecting an orientation of the body, and detecting motion of the body.

63. The system of claim 1, wherein the processor is configured to identify the gesture, wherein the identifying includes identifying a pose and an orientation of a portion of the body.

64. The system of claim 1, wherein the detecting includes detecting at least one of a first set of appendages and a second set of appendages of the body.

65. The system of claim 1, wherein the detecting includes dynamically detecting a position of at least one tag coupled to the body.

66. The system of claim 65, wherein the detecting includes detecting position of a set of tags coupled to the body.

67. The system of claim 66, wherein each tag of the set of tags includes a pattern, wherein each pattern of each tag of the set of tags is different than any pattern of any remaining tag of the plurality of tags.

68. The system of claim 1, wherein the detecting includes dynamically detecting and locating a marker on the body.

69. The system of claim 68, wherein the detecting includes detecting position of a set of markers coupled to the body.

70. The system of claim 68, wherein the set of markers form a plurality of patterns on the body.

71. The system of claim 68, wherein the detecting includes detecting position of a plurality of appendages of the body using a set of markers coupled to each of the appendages.

72. The system of claim 1, wherein the translating comprises translating information of the gesture to a gesture notation.

73. The system of claim 72, wherein the gesture notation represents a gesture vocabulary, and the gesture signal comprises communications of the gesture vocabulary.

74. The system of claim 73, wherein the gesture vocabulary represents in textual form instantaneous pose states of kinematic linkages of the body.

75. The system of claim 73, wherein the gesture vocabulary represents in textual form an orientation of kinematic linkages of the body.

76. The system of claim 73, wherein the gesture vocabulary represents in textual form a combination of orientations of kinematic linkages of the body.

77. The system of claim 73, wherein the gesture vocabulary includes a string of characters that represent a state of kinematic linkages of the body.

78. The system of claim 77, wherein the kinematic linkage is at least one first appendage of the body.

79. The system of claim 78, comprising assigning each position in the string to a second appendage, the second appendage connected to the first appendage.

80. The system of claim 79, comprising assigning characters of a plurality of characters to each of a plurality of positions of the second appendage.

81. The system of claim 80, wherein the plurality of positions is established relative to a coordinate origin.

82. The system of claim 81, wherein the processor is configured to establish the coordinate origin using a position selected from a group consisting of an absolute position and orientation in space, a fixed position and orientation relative to the body irrespective of an overall position and heading of the body, and interactively in response to an action of the body.

83. The system of claim 80, wherein the processor is configured to assign characters of the plurality of characters to each of a plurality of orientations of the first appendage.

84. The system of claim 78, wherein the detecting comprises detecting when an extrapolated position of the body intersects virtual space, wherein the virtual space comprises space depicted on a display device coupled to the computer.

85. The system of claim 84, wherein the processor is configured to control a virtual object in the virtual space when the extrapolated position intersects the virtual object.

86. The system of claim 85, wherein the controlling comprises controlling a position of the virtual object in the virtual space in response to the extrapolated position in the virtual space.

87. The system of claim 85, wherein the controlling comprises controlling attitude of the virtual object in the virtual space in response to the gesture.

88. The system of claim 1, wherein the processor is configured to control scaling of the detecting and controlling to generate coincidence between virtual space and the physical space, wherein the virtual space comprises space depicted on a display, wherein the physical space comprises space inhabited by the body.

89. The system of claim 88, wherein the processor is configured to control at least one virtual object in the virtual space in response to movement of at least one physical object in the physical space.