Passive touch system and method of detecting user input

• US 20060034486A1
• Filed: 10/13/2005
• Published: 02/16/2006
• Est. Priority Date: 07/05/2000
• Status: Abandoned Application

• Current Assignee: Gerald Morrison, Manvinder Singh
• Original Assignee: Gerald Morrison, Manvinder Singh
• Inventors: Morrison, Gerald, Singh, Manvinder
• Application Number:
  US11/248,166
• Publication Number:
  US 20060034486A1
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: G06F 3/042 by opto-electronic means
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06F 3/0428 by sensing at the edges of the touch surface the interruption of optical paths, e.g. an illumination plane, parallel to the touch surface which may be virtual sensing beam interruptions in a planar beam grid of an optical touch-screen G06F3/0421

Abstract

A passive touch system includes a passive touch surface and at least two cameras associated with the touch surface. The at least two cameras acquire images of the touch surface from different locations and have overlapping fields of view. A processor receives and processes images acquired by the at least two cameras to detect the existence of a pointer therein and to determine the location of the pointer relative to the touch surface. Actual pointer contact with the touch surface and pointer hover above the touch surface can be determined.

Claims

1-18. -18. (canceled)

19. A camera based position tracking system comprising:

a frame;

a region of interest at least partially within said frame;

at least two cameras associated with said frame, said at least two cameras acquiring images of said region of interest from different locations and having overlapping fields of view;

a digital signal processor associated with each camera, the digital signal processors selecting pixel subsets of images acquired by said at least two cameras and processing pixel data acquired by the selected pixel subsets to generate pointer characteristic data when a pointer exists in said acquired images; and

a master digital signal processor in communication with said digital signal processors, said master digital signal processor receiving pointer characteristic data from said digital signal processors and triangulating the pointer_-characteristic data to determine the location of said pointer relative to said at least two cameras.

20. A method of tracking an object of interest, the method comprising:

acquiring a first image and a second image representing different viewpoints of the object of interest;

processing the first image into a first image data set and the second image into a second image data set;

processing the first image data set and the second image data set to generate a background data set associated with a background;

generating a first difference map by determining differences between the first image data set and the background data set and a second difference map by determining differences between the second image data set and the background data set;

detecting a first relative position of the object of interest in the first difference map and a second relative position of the object of interest in the second difference map; and

producing an absolute position of the object of interest from the first and second relative positions of the object of interest.

21. The method of claim 20 wherein processing the first image into the first image data set and the second image into the second image data set includes determining an active image region for each of the first and second images, and extracting an active image data set from the first and second images contained within the active image region.

22. The method of claim 21 wherein extracting the active image data set includes cropping the first and second images.

23. The method of claim 21 wherein extracting the active image data set includes rotating the first and second images.

24. The method of claim 21 wherein extracting the active image data set includes shearing the first and second images.

25. The method of claim 21 wherein extracting the active image data set includes arranging the active image data set into an image pixel array having rows and columns.

26. The method of claim 25 wherein extracting the active image data set further includes:

identifying the maximum pixel value within each column of the image pixel array; and

generating data sets having one row whereby the identified maximum pixel value for each column represents that column.

27. The method of claim 20 wherein processing the first image into a first image data set and the second image into a second image data set includes filtering the first and second images.

28. The method of claim 27 wherein filtering includes extracting the edges in the first and second images.

29. The method of claim 27 wherein filtering further includes processing the first image data set and the second image data set to emphasize differences between the first image data set and the background data set and to emphasize differences between the second image data set and the background data set.

30. The method of claim 20 wherein processing the first image data set and the second image data set to generate the background data set includes generating a first set of one or more background data sets associated with the first image data set, and generating a second set of one or more background data sets associated with the second image data set.

31. The method of claim 30 wherein generating the first set of background data sets includes sampling the first image data set, and generating the second set of background data sets includes sampling the second image data set.

32. The method of claim 31 wherein generating the first set of one or more background data sets includes maintaining multiple samples of the first image data set within each background data set, and generating the second set of one or more background data sets includes maintaining multiple samples of the second image data set within each background data set.

33. The method of claim 32 wherein generating each first background data set includes selecting from the multiple samples one value that is representative of the background for each element within the first image data set, and generating each second background data set includes selecting from the multiple samples one value that is representative of the background for each element within the second image data set.

34. The method of claim 31 wherein sampling occurs automatically at predefined time intervals, where each sample may include data that is not associated with the background.

35. The method of claim 20 wherein generating includes comparing the first image data set to a subset of the background data set, and comparing the second image data set to a subset of the background data set.

36. The method of claim 20 wherein generating a first difference map further includes representing each element in the first image data set as one of two states, and generating a second difference map further includes representing each element in the second image data set as one of two states, where the two states represent whether the value is consistent with the background.

37. The method of claim 20 wherein detecting includes identifying a cluster in each of the first and second difference maps, each cluster having elements whose state within its associated difference map indicates that the elements are inconsistent with the background.

38. The method of claim 37 wherein identifying the cluster further includes reducing the difference map to one row by counting the elements within a column that are inconsistent with the background.

39. The method of claim 38 wherein identifying the cluster further includes identifying the column as being within the cluster and classifying nearby columns as being within the cluster.

40. The method of claim 39 wherein identifying the column as being within the cluster includes identifying the median column.

41. The method of claim 37 wherein identifying the cluster further includes identifying a position associated with the cluster.

42. The method of claim 41 wherein identifying the position associated with the cluster includes calculating the weighted mean of elements within the cluster.

43. The method of claim 37 wherein detecting further includes classifying the cluster as the object of interest.

44. The method of claim 43 wherein classifying the cluster further includes counting the elements within the cluster and classifying the cluster as the object of interest only if that count exceeds a predefined threshold.

45. The method of claim 43 wherein classifying the cluster further includes counting the elements within the cluster and counting a total number of elements classified as inconsistent within the background within the difference map, and classifying the cluster as the object of interest only if the ratio of the count of elements within the cluster over the total number of elements exceeds a predefined threshold.

46. The method of claim 37 wherein detecting further includes identifying a sub-cluster within the cluster that represents a pointing end of the object of interest and identifying a position of the sub-cluster.

47. The method of claim 20 wherein the object of interest is a user'"'"'s hand.

48. The method of claim 20 further including controlling an application program using the absolute position of the object of interest.

49. The method of claim 20 further comprising:

acquiring a third image and a fourth image representing different viewpoints of the object of interest;

processing the third image into a third image data set and the fourth image into a fourth image data set;

processing the third image data set and the fourth image data set to generate the background data set associated with the background;

generating a third difference map by determining differences between the third image data set and the background data set, and a fourth difference map by determining differences between the fourth image data set and the background data set;

detecting a third relative position of the object of interest in the third difference map and a fourth relative position of the object of interest in the fourth difference map; and

producing the absolute position of the object of interest from the first, second, third and fourth relative positions of the object of interest.

50. The method of claim 49 wherein the object of interest is a user'"'"'s hand.

51. The method of claim 49 further including controlling an application program using the absolute position of the object of interest.

52. A method of tracking an object of interest controlled by a user to interface with a computer, the method comprising:

acquiring images from at least two viewpoints;

processing the acquired images to produce an image data set for each acquired image;

comparing each image data set to one or more background data sets to produce a difference map for each acquired image;

detecting a relative position of an object of interest within each difference map;

producing an absolute position of the object of interest from the relative positions of the object of interest; and

using the absolute position to allow the user to interact with a computer application.

53. The method of claim 52 further including:

mapping the absolute position of the object of interest to screen coordinates associated with the computer application; and

using the mapped position to interface with the computer application.

54. The method of claim 52 further including:

recognizing a gesture associated with the object of interest by analyzing changes in the absolute position of the object of interest; and

combining the absolute position and the gesture to interface with the computer application.

55. A multiple camera tracking system for interfacing with an application program running on a computer, the tracking system comprising:

two or more video cameras arranged to provide different viewpoints of a region of interest and operable to produce a series of video images;

a processor operable to receive the series of video images and detect objects appearing in the region of interest, the processor executing a process to;

generate a background data set from the video images;

generate an image data set for each received video image;

compare each image data set to the background data set to produce a difference map for each image data set;

detect a relative position of an object of interest within each difference map; and

produce an absolute position of the object of interest from the relative positions of the object of interest and map the absolute position to a position indicator associated with the application program.

56. The multiple camera tracking system of claim 55 wherein the object of interest is a human hand.

57. The multiple camera tracking system of claim 55 wherein the region of interest is defined to be in front of a video display associated with the computer, and wherein the processor is further operable to map the absolute position of the object of interest to the position indicator such that the location of the position indicator on the video display is aligned with the object of interest.

58. The multiple camera tracking system of claim 55 wherein the region of interest is defined to be any distance in front of a video display associated with the computer, and wherein the processor is further operable to map the absolute position of the object of interest to the position indicator such that the location of the position indicator on the video display is aligned to a position pointed to by the object of interest.

59. The multiple camera tracking system of claim 55 wherein the region of interest is defined to be any distance in front of a video display associated with the computer, and wherein the processor is further operable to map the absolute position of the object of interest to the position indicator such that movements of the object of interest are scaled to larger movements of the location of the position indicator on the video display.

60. The multiple camera tracking system of claim 55 wherein the processor is further configured to emulate a computer mouse function.

61. The multiple camera tracking system of claim 55 wherein the processor is further configured to emulate controlling buttons of a computer mouse using gestures derived from the motion of the object of interest.

62. The multiple camera tracking system of claim 55 wherein a position of the object of interest triggers a selection action within the application program.

63. The multiple camera tracking system of claim 55 wherein the processor is further configured to emulate controlling buttons of a computer mouse based on a position of the object of interest within the region of interest.

64. The multiple camera tracking system of claim 55 wherein a position of the object of interest within the bounds of an interactive display region triggers a selection action within the application program.

65. The multiple camera tracking system of claim 55 wherein the processor is further configured to emulate controlling buttons of a computer mouse based on a position of the position indicator within the bounds of an interactive display region.

66. The multiple camera tracking system of claim 55 wherein the background data set includes data points representing at least a portion of a stationary structure.

67. The multiple camera tracking system of claim 66 wherein at least a portion of the stationary structure includes a patterned surface that is visible to the video cameras.

68. The multiple camera tracking system of claim 66 wherein the stationary structure is a frame.

69. A multiple camera tracking system for interfacing with an application program running on a computer, the tracking system comprising:

two or more video cameras arranged to provide different viewpoints of a region of interest and operable to produce a series of video images;

a processor operable to receive the series of video images and detect objects appearing in the region of interest, the processor executing a process to;

generate a background data set from the video images;

generate an image data set for each received video image;

compare each image data set to the background data set to produce a difference map for each image data set;

detect a relative position of an object of interest within each difference map;

produce an absolute position of the object of interest from the relative positions of the object of interest;

define sub regions within the region of interest;

identify a sub region occupied by the object of interest;

associate an action with the identified sub region that is activated when the object of interest occupies the identified sub region; and

apply the action to interface with the application program.

70. The multiple camera tracking system of claim 69 wherein the object of interest is a human hand.

71. The multiple camera tracking system of claim 69 wherein the action associated with the identified sub region emulates the activation of keys of a keyboard associated with the application program.

72. The multiple camera tracking system of claim 69 wherein a position of the object of interest in any sub region triggers the action.