Systems and methods of interfacing with a machine

• US 7,755,608 B2
• Filed: 01/23/2004
• Issued: 07/13/2010
• Est. Priority Date: 01/23/2004
• Status: Active Grant

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Samadani, Ramin, Chang, Nelson Liang An
• Primary Examiner(s): Shalwala; Bipin
• Assistant Examiner(s): Holton; Steven E
• Application Number:
  US10/764,423
• Publication Number:
  US 20050166163A1
• Time in Patent Office: 2,363 Days
• Field of Search: 345/169, 345/173, 345/175, 345 7- 9, 345/156
• US Class Current: 345/157
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

Systems and methods of interfacing with a machine are described. In one aspect, sets of contemporaneous images of an interactive space are acquired from multiple respective fields of view. An input target is detected in the acquired images. Coordinates of the input target detected in the acquired images are computed. A spatiotemporal input data structure linking input target coordinates computed from contemporaneous images to respective reference times is constructed. The spatiotemporal input data structure is processed to identify an input instruction. The identified input instruction is executed on a machine. In another aspect, an image is displayed at a display location disposed between a viewing space and an interactive space, wherein the displayed image is viewable from a perspective in the viewing space. Images of the interactive space are acquired from at least one field of view. An input target is detected in the acquired images. Coordinates of the input target detected in the acquired images are computed. An input instruction is identified based on the computed input coordinates. The identified input instruction is executed on a machine.

Claims

1. A method of interfacing with a machine, comprising:

at each of multiple capture times, contemporaneously acquiring a respective image from each of multiple fields of view defining an interactive space to create a respective set of contemporaneous images;

detecting an input target in the acquired images;

computing two-dimensional coordinates of the input target detected in the acquired images;

constructing a spatiotemporal input data structure linking each of the capture times to the computed two-dimensional coordinates of the input target in each of the contemporaneously acquired images in a respective one of the sets of contemporaneous images;

processing the spatiotemporal input data structure to identify an input instruction; and

executing the identified input instruction on the machine.

2. The method of claim 1, wherein images of the interactive space are acquired from at least one stereoscopic pair of fields of view directed along substantially parallel axes intersecting the interactive space.

3. The method of claim 1, wherein images of the interactive space are acquired from at least three different fields of view.

4. The method of claim 1, wherein detecting the input target in the acquired images comprises comparing values of pixels in the acquired images to at least one threshold pixel value.

5. The method of claim 4, wherein computing coordinates of the input target comprises computing coordinates of centroids of respective groups of pixels in the acquired images with values greater than the at least one threshold pixel value.

6. The method of claim 4, wherein detecting the input target in the acquired images comprises segmenting foreground pixels from background pixels in the acquired images.

7. The method of claim 1, further comprising computing calibration parameters for the multiple fields of view.

8. The method of claim 7, wherein computing coordinates of the detected input target comprises computing three-dimensional coordinates of the input target in the interactive space based on the computed two-dimensional coordinates and the computed calibration parameters.

9. The method of claim 8, wherein the spatiotemporal input data structure additionally links each of the capture times to respective three-dimensional coordinates of the input target computed from the computed two-dimensional coordinates of the input target in each of the contemporaneously acquired images in a respective one of the sets of contemporaneous images.

10. The method of claim 1, further comprising acquiring color values of the detected input target in the acquired images.

11. The method of claim 10, wherein the spatiotemporal input data structure additionally links each of the capture times to respective color values determined from the contemporaneously acquired images in a respective one of the sets of contemporaneous images.

12. The method of claim 1, wherein the spatiotemporal input data structure is constructed in the form of a linked list of data records.

13. The method of claim 1, wherein processing the spatiotemporal input data structure comprises identifying traces of the input target in the interactive space, each trace being defined by a respective set of connected data items in the spatiotemporal input data structure.

14. The method of claim 13, wherein identifying traces comprises detecting state change events and segmenting traces based on detected state change events.

15. The method of claim 13, wherein identifying traces comprises computing coordinates of bounding regions encompassing respective traces.

16. The method of claim 15, wherein the computed bounding region coordinates are two-dimensional coordinates of areas in the acquired images.

17. The method of claim 15, wherein the computed bounding region coordinates are three-dimensional coordinates of volumes in the interactive space.

18. The method of claim 13, wherein the spatiotemporal input data structure is processed to interpret the identified input target traces.

19. The method of claim 18, further comprising comparing an identified trace to a predefined representation of an input gesture corresponding to a respective input instruction.

20. The method of claim 19, wherein processing the spatiotemporal input data structure comprises translating the trace into a predefined alphanumeric character.

21. The method of claim 18, further comprising comparing an identified trace to a location in the interactive space corresponding to a virtual interactive object.

22. The method of claim 21, wherein the virtual interactive object corresponds to a virtual machine instruction input.

23. The method of claim 22, wherein the virtual machine instruction input is predefined.

24. The method of claim 22, further comprising constructing the virtual machine instruction input in response to processing of at least one identified input target trace.

25. The method of claim 22, wherein the virtual machine instruction input corresponds to a respective mode of interpreting traces.

26. The method of claim 1, wherein executing the identified input instruction comprises displaying an image in accordance with the identified input instruction.

27. The method of claim 26, wherein the displaying comprises displaying a combination of image data generated based on the acquired images and machine-generated virtual image data.

28. The method of claim 26, further comprising displaying a sequence of images at the display location showing a virtual object being manipulated in accordance with one or more identified input instructions.

29. The method of claim 1, further comprising interpolating between ones of the images contemporaneously acquired from ones of the fields of view to generate a synthetic view of the interactive space.

30. A method of interfacing with a machine, comprising:

acquiring sets of contemporaneous images of an interactive space from multiple respective fields of view;

detecting an input target in the acquired images;

computing coordinates of the input target detected in the acquired images;

constructing a spatiotemporal input data structure linking input target coordinates computed from contemporaneous images to respective reference times;

processing the spatiotemporal input data structure to identify an input instruction; and

executing the identified input instruction on the machine; and

interpolating between fields of view to generate a synthetic view of the interactive space.

31. A system for interfacing with a machine, comprising:

multiple imaging devices configured to contemporaneously acquire, at each of multiple capture times, a respective image from each of multiple fields of view defining an interactive space to create a respective set of contemporaneous images; and

a processing module configured to detect an input target in the acquired images, compute two-dimensional coordinates of the input target detected in the acquired images, construct a spatiotemporal input data structure linking each of the capture times to the computed two-dimensional coordinates of the input target in each of the contemporaneously acquired images in a respective one of the sets of contemporaneous images, process the spatiotemporal input data structure to identify an input instruction, and execute the identified input instruction on the machine.

32. A machine-readable medium storing machine-readable instructions for causing a machine to:

at each of multiple capture times, contemporaneously acquire a respective image from each of multiple fields of view defining an interactive space to create a respective set of contemporaneous images;

detect an input target in the acquired images;

compute two-dimensional coordinates of the input target detected in the acquired images;

construct a spatiotemporal input data structure linking each of the capture times to the computed two-dimensional coordinates of the input target in each of the contemporaneously acquired images in a respective one of the sets of contemporaneous images;

process the spatiotemporal input data structure to identify an input instruction; and

execute the identified input instruction on the machine.

33. A method of interfacing with a machine, comprising:

displaying an image at a display location disposed between a viewing space and an interactive space, wherein the displayed image is viewable from a perspective in the viewing space;

acquiring images of the interactive space from a field of view directed toward the interactive space along an optical axis intersecting a central area of the display location;

detecting an input target in the acquired images;

computing coordinates of the input target detected in the acquired images;

identifying an input instruction based on the computed input coordinates; and

executing the identified input instruction on the machine.

34. The method of claim 33, wherein the display location corresponds to a display area of a portable electronic device.

35. The method of claim 33, wherein the display location corresponds to a display area embedded in a desktop surface.

36. The method of claim 33, wherein displaying the image comprises projecting the image onto a surface.

37. The method of claim 33, wherein acquiring images comprises acquiring images of the interactive space from at least one field of view disposed between the display location and the interactive space.

38. The method of claim 33, wherein acquiring images comprises acquiring images of the interactive space from multiple fields of view.

39. The method of claim 38, further comprising interpolating between fields of view to display an image at the display location corresponding to a synthetic view of the interactive space.

40. The method of claim 38, further comprising computing calibration parameters for the multiple fields of view.

41. The method of claim 40, wherein computing coordinates of the detected input target comprises computing three-dimensional coordinates of the input target in the interactive space based on the computed calibration parameters.

42. The method of claim 33, wherein detecting the input target in the acquired images comprises comparing values of pixels in the acquired images to at least one threshold pixel value.

43. The method of claim 42, wherein computing coordinates of the input target comprises computing coordinates of centroids of respective groups of pixels in the acquired images with values greater than the threshold.

44. The method of claim 42, wherein detecting the input target in the acquired images comprises segmenting foreground pixels from background pixels in the acquired images.

45. The method of claim 33, wherein computing coordinates of the detected input target comprises computing two-dimensional coordinates of the input target detected in the acquired images.

46. The method of claim 33, wherein identifying an input instruction comprises identifying traces of the input target in the interactive space.

47. The method of claim 46, wherein identifying traces comprises detecting state change events and segmenting traces based on detected state change events.

48. The method of claim 46, wherein identifying traces comprises computing coordinates of bounding regions encompassing respective traces.

49. The method of claim 48, wherein the computed bounding region coordinates are two-dimensional coordinates of areas in the acquired images.

50. The method of claim 48, wherein the computed bounding region coordinates are three-dimensional coordinates of volumes in the interactive space.

51. The method of claim 46, wherein identifying the input instruction comprises interpreting the identified input target traces.

52. The method of claim 51, further comprising comparing an identified trace to a predefined representation of an input gesture corresponding to a respective input instruction.

53. The method of claim 52, wherein processing the spatiotemporal input data structure comprises translating the trace into a predefined alphanumeric character.

54. The method of claim 51, further comprising comparing an identified trace to a location in the interactive space corresponding to a virtual interactive object.

55. The method of claim 54, wherein the virtual interactive object corresponds to a virtual machine instruction input.

56. The method of claim 55, wherein the virtual machine instruction input is predefined.

57. The method of claim 55, further comprising constructing the virtual machine instruction input in response to processing of at least one identified input target trace.

58. The method of claim 55, wherein the virtual machine instruction input corresponds to a respective mode of interpreting traces.

59. The method of claim 33, wherein executing the identified input instruction comprises displaying an image at the display location in accordance with the identified input instruction.

60. The method of claim 59, wherein the displayed image comprises a combination of image data generated based on the acquired images and machine-generated virtual image data.

61. The method of claim 59, further comprising displaying a sequence of images at the display location showing a virtual object being manipulated in accordance with one or more identified input instructions.

62. A system of interfacing with a machine, comprising:

a display configured to present an image at a display location disposed between a viewing space and an interactive space, wherein the displayed image is viewable from a perspective in the viewing space;

at least one imaging device configured to acquire images of the interactive space from a field of view directed toward the interactive space along an optical axis intersecting a central area of the display location; and

a processing system configured to detect an input target in the acquired images, compute coordinates of the input target detected in the acquired images, identify an input instruction based on the computed input coordinates, and execute the identified input instruction on the machine.

63. A machine-readable medium storing machine-readable instructions for causing a machine to:

display an image at a display location disposed between a viewing space and an interactive space, wherein the displayed image is viewable from a perspective in the viewing space;

acquire images of the interactive space from a field of view directed toward the interactive space along an optical axis intersecting a central area of the display location;

detect an input target in the acquired images;

compute coordinates of the input target detected in the acquired images;

identify an input instruction based on the computed input coordinates; and

execute the identified input instruction on the machine.