Multiple camera control system

• US 7,058,204 B2
• Filed: 09/26/2001
• Issued: 06/06/2006
• Est. Priority Date: 10/03/2000
• Status: Expired due to Term

• Current Assignee: Qualcomm, Inc.
• Original Assignee: GestureTek Incorporated (Winning Brands Corp.)
• Inventors: MacDougall, Francis, Hildreth, Evan
• Primary Examiner(s): COUSO, YON JUNG
• Application Number:
  US09/962,612
• Publication Number:
  US 20020064382A1
• Time in Patent Office: 1,714 Days
• Field of Search: 345/863, 348/42, 348/47, 348/48, 348/143, 348/153, 348/159, 382/103, 382/106, 382/107, 382/154, 382/286, 382/291, 382/292
• US Class Current: 382/103
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/20132 Image cropping
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/174 involving the use of two or more images
• CPC Class Codes: G06T 7/254 involving subtraction of images
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06V 40/107 Static hand or arm

Abstract

A multiple camera tracking system for interfacing with an application program running on a computer is provided. The tracking system includes two or more video cameras arranged to provide different viewpoints of a region of interest, and are operable to produce a series of video images. A processor is operable to receive the series of video images and detect objects appearing in the region of interest. The processor executes a process to generate a background data set from the video images, generate an image data set for each received video image, compare each image data set to the background data set to produce a difference map for each image data set, detect a relative position of an object of interest within each difference map, and produce an absolute position of the object of interest from the relative positions of the object of interest and map the absolute position to a position indicator associated with the application program.

Claims

1. A method of tracking an object of interest, the method comprising:

acquiring a first image and a second image representing different viewpoints of the object of interest;

processing the first image into a first image data set and the second image into a second image data set;

processing the first image data set and the second image data set to generate a background data set associated with a background;

generating a first difference map by determining differences between the first image data set and the background data set and a second difference map by determining differences between the second image data set and the background data set;

detecting a first relative position of the object of interest in the first difference map and a second relative position of the object of interest in the second difference map; and

producing an absolute position of the object of interest from the first and second relative positions of the object of interest.

2. The method of claim 1 wherein processing the first image into the first image data set and the second image into the second image data set includes determining an active image region for each of the first and second images, and extracting an active image data set from the first and second images contained within the active image region.

3. The method of claim 2 wherein extracting the active image data set includes cropping the first and second images.

4. The method of claim 2 wherein extracting the active image data set includes rotating the first and second images.

5. The method of claim 2 wherein extracting the active image data set includes shearing the first and second images.

6. The method of claim 2 wherein extracting the active image data set includes arranging the active image data set into an image pixel array having rows and columns.

7. The method of claim 6 wherein extracting the active image data set further includes:

identifying the maximum pixel value within each column of the image pixel array; and

generating data sets having one row whereby the identified maximum pixel value for each column represents that column.

8. The method of claim 1 wherein processing the first image into a first image data set and the second image into a second image data set includes filtering the first and second images.

9. The method of claim 8 wherein filtering includes extracting the edges in the first and second images.

10. The method of claim 8 wherein filtering further includes processing the first image data set and the second image data set to emphasize differences between the first image data set and the background data set and to emphasize differences between the second image data set and the background data set.

11. The method of claim 1 wherein processing the first image data set and the second image data set to generate the background data set includes generating a first set of one or more background data sets associated with the first image data set, and generating a second set of one or more background data sets associated with the second image data set.

12. The method of claim 11 wherein generating the first set of one or more background data sets includes generating a first background set representing a maximum value of data within the first image data set representative of the background, and generating the second set of one or more background data sets includes generating a second background set representing a maximum value of data within the second image data set representative of the background.

13. The method of claim 12 wherein generating further includes, for the first and second background sets representing the maximum value of data representative of the background, increasing the values contained within the first and second background sets by a predetermined value.

14. The method of claim 11 wherein generating the first set of one or more background data sets includes generating a first background set representing a minimum value of data within the first image data set representative of the background, and generating the second set of one or more background data sets includes generating a second background set representing a minimum value of data within the second image data set representative of the background.

15. The method of claim 14 wherein generating further includes, for the first and second background sets representing the minimum value of data representative of the background, decreasing the values contained within the first and second background sets by a predetermined value.

16. The method of claim 11 wherein generating the first set of background data sets includes sampling the first image data set, and generating the second set of background data sets includes sampling the second image data set.

17. The method of claim 16 wherein generating the first set of one or more background data sets includes maintaining multiple samples of the first image data set within each background data set, and generating the second set of one or more background data sets includes maintaining multiple samples of the second image data set within each background data set.

18. The method of claim 17 wherein generating each first background data set includes selecting from the multiple samples one value that is representative of the background for each element within the first image data set, and generating each second background data set includes selecting from the multiple samples one value that is representative of the background for each element within the second image data set.

19. The method of claim 18 wherein selecting includes selecting the median value from all sample values in each of the background data sets.

20. The method of claim 16 wherein sampling occurs automatically at predefined time intervals, where each sample may include data that is not associated with the background.

21. The method of claim 1 wherein generating includes comparing the first image data set to a subset of the background data set, and comparing the second image data set to a subset of the background data set.

22. The method of claim 1 wherein generating a first difference map further includes representing each element in the first image data set as one of two states, and generating a second difference map further includes representing each element in the second image data set as one of two states, where the two states represent whether the value is consistent with the background.

23. The method of claim 1 wherein detecting includes identifying a cluster in each of the first and second difference maps, each cluster having elements whose state within its associated difference map indicates that the elements are inconsistent with the background.

24. The method of claim 23 wherein identifying the cluster further includes reducing the difference map to one row by counting the elements within a column that are inconsistent with the background.

25. The method of claim 24 wherein identifying the cluster further includes identifying the column as being within the cluster and classifying nearby columns as being within the cluster.

26. The method of claim 25 wherein identifying the column as being within the cluster includes identifying the median column.

27. The method of claim 23 wherein identifying the cluster further includes identifying a position associated with the cluster.

28. The method of claim 27 wherein identifying the position associated with the cluster includes calculating the weighted mean of elements within the cluster.

29. The method of claim 23 wherein detecting further includes classifying the cluster as the object of interest.

30. The method of claim 29 wherein classifying the cluster further includes counting the elements within the cluster and classifying the cluster as the object of interest only if that count exceeds a predefined threshold.

31. The method of claim 29 wherein classifying the cluster further includes counting the elements within the cluster and counting a total number of elements classified as inconsistent within the background within the difference map, and classifying the cluster as the object of interest only if the ratio of the count of elements within the cluster over the total number of elements exceeds a predefined threshold.

32. The method of claim 23 wherein detecting further includes identifying a sub-cluster within the cluster that represents a pointing end of the object of interest and identifying a position of the sub-cluster.

33. The method of claim 1 wherein the object of interest is a user'"'"'s hand.

34. The method of claim 1 further including controlling an application program using the absolute position of the object of interest.

35. The method of claim 1 further comprising:

acquiring a third image and a fourth image representing different viewpoints of the object of interest;

processing the third image into a third image data set and the fourth image into a fourth image data set;

processing the third image data set and the fourth image data set to generate the background data set associated with the background;

generating a third difference map by determining differences between the third image data set and the background data set, and a fourth difference map by determining differences between the fourth image data set and the background data set;

detecting a third relative position of the object of interest in the third difference map and a fourth relative position of the object of interest in the fourth difference map; and

producing the absolute position of the object of interest from the first, second, third and fourth relative positions of the object of interest.

36. The method of claim 35 wherein the object of interest is a user'"'"'s hand.

37. The method of claim 35 further including controlling an application program using the absolute position of the object of interest.

38. A method of tracking an object of interest controlled by a user to interface with a computer, the method comprising:

acquiring images from at least two viewpoints;

processing the acquired images to produce an image data set for each acquired image;

comparing each image data set to one or more background data sets to produce a difference map for each acquired image;

detecting a relative position of an object of interest within each difference map;

producing an absolute position of the object of interest from the relative positions of the object of interest; and

using the absolute position to allow the user to interact with a computer application.

39. The method of claim 38 further including:

mapping the absolute position of the object of interest to screen coordinates associated with the computer application; and

using the mapped position to interface with the computer application.

40. The method of claim 38 further including:

recognizing a gesture associated with the object of interest by analyzing changes in the absolute position of the object of interest; and

combining the absolute position and the gesture to interface with the computer application.

41. A multiple camera tracking system for interfacing with an application program running on a computer, the tracking system comprising:

two or more video cameras arranged to provide different viewpoints of a region of interest and operable to produce a series of video images;

a processor operable to receive the series of video images and detect objects appearing in the region of interest, the processor executing a process to;

generate a background data set from the video images;

generate an image data set for each received video image;

compare each image data set to the background data set to produce a difference map for each image data set;

detect a relative position of an object of interest within each difference map; and

produce an absolute position of the object of interest from the relative positions of the object of interest and map the absolute position to a position indicator associated with the application program.

42. The multiple camera tracking system of claim 41 wherein the object of interest is a human hand.

43. The multiple camera tracking system of claim 41 wherein the region of interest is defined to be in front of a video display associated with the computer, and wherein the processor is further operable to map the absolute position of the object of interest to the position indicator such that the location of the position indicator on the video display is aligned with the object of interest.

44. The multiple camera tracking system of claim 41 wherein the region of interest is defined to be any distance in front of a video display associated with the computer, and wherein the processor is further operable to map the absolute position of the object of interest to the position indicator such that the location of the position indicator on the video display is aligned to a position pointed to by the object of interest.

45. The multiple camera tracking system of claim 41 wherein the region of interest is defined to be any distance in front of a video display associated with the computer, and wherein the processor is further operable to map the absolute position of the object of interest to the position indicator such that movements of the object of interest are scaled to larger movements of the location of the position indicator on the video display.

46. The multiple camera tracking system of claim 41 wherein the processor is further configured to emulate a computer mouse function.

47. The multiple camera tracking system of claim 41 wherein the processor is further configured to emulate controlling buttons of a computer mouse using gestures derived from the motion of the object of interest.

48. The multiple camera tracking system of claim 41 wherein a sustained position of the object of interest for a predetermined time period triggers a selection action within the application program.

49. The multiple camera tracking system of claim 41 wherein the processor is further configured to emulate controlling buttons of a computer mouse based on a sustained position of the object of interest for a predetermined time period.

50. The multiple camera tracking system of claim 41 wherein sustaining a position of the object of interest within the bounds of an interactive display region for a predetermined time period triggers a selection action within the application program.

51. The multiple camera tracking system of claim 41 wherein the processor is further configured to emulate controlling buttons of a computer mouse based on a sustained position of the position indicator within the bounds of an interactive display region for a predetermined time period.

52. The multiple camera tracking system of claim 41 wherein the background data set includes data points representing at least a portion of a stationary structure.

53. The multiple camera tracking system of claim 52 wherein at least a portion of the stationary structure includes a patterned surface that is visible to the video cameras.

54. The multiple camera tracking system of claim 52 wherein the stationary structure is a window frame.

55. The multiple camera tracking system of claim 52 wherein the stationary structure includes a strip of light.

56. A multiple camera tracking system for interfacing with an application program running on a computer, the tracking system comprising:

two or more video cameras arranged to provide different viewpoints of a region of interest and operable to produce a series of video images;

a processor operable to receive the series of video images and detect objects appearing in the region of interest, the processor executing a process to;

generate a background data set from the video images;

generate an image data set for each received video image;

compare each image data set to the background data set to produce a difference map for each image data set;

detect a relative position of an object of interest within each difference map;

produce an absolute position of the object of interest from the relative positions of the object of interest;

define sub regions within the region of interest;

identify a sub region occupied by the object of interest;

associate an action with the identified sub region that is activated when the object of interest occupies the identified sub region; and

apply the action to interface with the application program.

57. The multiple camera tracking system of claim 56 wherein the object of interest is a human hand.

58. The multiple camera tracking system of claim 56 wherein the action associated with the identified sub region emulates the activation of keys of a keyboard associated with the application program.

59. The multiple camera tracking system of claim 56 wherein sustaining a position of the object of interest in any sub region for a predetermined time period triggers the action.