Multiple camera control system

• US 7,555,142 B2
• Filed: 10/31/2007
• Issued: 06/30/2009
• Est. Priority Date: 10/03/2000
• Status: Expired due to Term

• Current Assignee: Qualcomm, Inc.
• Original Assignee: GestureTek Incorporated (Winning Brands Corp.)
• Inventors: MacDougall, Francis, Hildreth, Evan
• Primary Examiner(s): COUSO, YON JUNG
• Application Number:
  US11/932,869
• Publication Number:
  US 20080056536A1
• Time in Patent Office: 608 Days
• Field of Search: 382/103, 382/106, 382/107, 382/154, 382/286, 382/291, 382/292, 348/42, 348/47, 348/48, 348/143, 348/153, 348/159, 345/863
• US Class Current: 382/103
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/20132 Image cropping
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/174 involving the use of two or more images
• CPC Class Codes: G06T 7/254 involving subtraction of images
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06V 40/107 Static hand or arm

Abstract

A multiple camera tracking system for interfacing with an application program running on a computer is provided. The tracking system includes two or more video cameras arranged to provide different viewpoints of a region of interest, and are operable to produce a series of video images. A processor is operable to receive the series of video images and detect objects appearing in the region of interest. The processor executes a process to generate a background data set from the video images, generate an image data set for each received video image, compare each image data set to the background-data set to produce a difference map for each image data set, detect a relative position of an object of interest within each difference map, and produce an absolute position of the object of interest from the relative positions of the object of interest and map the absolute position to a position indicator associated with the application program.

Claims

1. A computer-implemented method comprising:

accessing, using a processor, first and second images each capturing a different view of an object and a background;

detecting, using the processor, for each of the first and second images, a difference between a filtered luminance characteristic and a background model;

determining, using the processor, a position of the object based on the detected differences; and

controlling, using the processor, an application based on the determined position of the object.

2. The method of claim 1, wherein the background is a controlled background.

3. The method of claim 2, wherein the controlled background comprises a strip of light.

4. The method of claim 2, wherein the controlled background comprises a surface, wall or window frame having arbitrary luminance, textures or edges.

5. The method of claim 2, wherein the controlled background exhibits a predetermined pattern or texture.

6. The method of claim 5, wherein the predetermined pattern is a stripe pattern.

7. The method of claim 5, wherein detecting, for each of the first and second images, the occlusion of the background further comprises:

detecting, for each of the first and second images, a region that does not exhibit the predetermined pattern or texture; and

outputting, for each of the first and second images, data associated with the region as the occlusion.

8. The method of claim 1, wherein the first and second images are acquired using first and second cameras, respectively, and wherein the determined position of the object is a relative position with respect to the first and second cameras, respectively.

9. The method of claim 1, wherein the determined position of the objection is an absolute position.

10. The method of claim 1, further comprising detecting a gesture of the object, wherein the application is controlled further based on the gesture.

11. The method of claim 1, wherein detecting, for each of the first and second images, the difference between the filtered luminance characteristic and the background model further comprises filtering the first and second images to extract the luminance characteristic of the first and second images where the background and the object differ, but are otherwise invariant within the background over time.

12. The method of claim 1, further comprising processing the first and second images to exclude pixels outside bounds of the background.

13. The method of claim 1, wherein the first and second images each comprise a single row of pixels.

14. The method of claim 1, wherein detecting, for each of the first and second images, the difference between the luminance characteristic and the background model further comprises:

tagging a pixel of the first or second image to a first state if the pixel is consistent with the background model; and

tagging the pixel to a second state if the pixel is inconsistent with the background model.

15. The method of claim 1, further comprising providing, as the background, a strip of light that is capable of being detected in the first and second images.

27. The method of claim 1, wherein accessing first and second images comprises acquiring the first and second images from at least one camera configured to transfer data to the processor.

16. An apparatus comprising:

one or more processing devices and a computer-readable medium coupled to the one or more processing devices, the medium storing instructions which, when executed by the one or more processing devices, cause the one or more computers to;

acquire first and second images each capturing a different view of an object and a background;

detect, for each of the first and second images, a difference between a filtered luminance characteristic and a background model; and

means for determining a position of the object based on the detected differences and controlling an application based on the determined position of the object.

17. A device comprising:

a camera configured to acquire first and second images each capturing a different view of an object and a background; and

a processor configured to;

detect, for each of the first and second images, a difference between a filtered luminance characteristic and a background model,

determine a position of the object based on the detected differences, and

control an application based on the determined position of the object.

18. A computer readable medium encoded with a computer program comprising instructions that, when executed, operate to cause a computer to;

acquire first and second images each capturing a different view of an object and a background;

detect, for each of the first and second images, a difference between a filtered luminance characteristic and a background model;

determine a position of the object based on the detected differences; and

control an application based on the determined position of the object.

19. A computer readable medium encoded with a computer program comprising instructions that, when executed, operate to cause a computer to;

acquire first and second images each capturing a different view of an object;

filter a luminance characteristic of each of the first and second images;

determine, based on filtering the luminance characteristic, a position of the object; and

control an application based on the determined position of the object.

20. A computer readable medium encoded with a computer program comprising instructions that, when executed, operate to cause a computer to;

acquire first and second images each capturing a different view of an object;

detect a difference between the first and second images;

determine a position of the object based on the difference; and

control an application based on the determined position of the object.

21. A computer readable medium encoded with a computer program comprising instructions that, when executed, operate to cause a computer to;

acquire first and second images each capturing a different view of an object and a background, the background being oriented parallel to a reference plane defined with respect to an imaging device configured to acquire at least one of the first and second images;

filter a luminance characteristic of each of the first and second images;

determine, based on the filtered luminance characteristic, a position of the object along an axis that is perpendicular to the reference plane; and

control an application based on the determined position of the object.

29. The medium of claim 21, further comprising instructions to:

determine a second position of the object along the axis; and

detect, based on the position and the second position, a hovering of the object with respect to the reference plane.

22. A computer readable medium encoded with a computer program comprising instructions that, when executed, operate to cause a computer to;

define first and second planes oriented parallel with a reference plane defined with respect to an imaging device configured to acquire at least one of the first and second images;

acquire first and second images each capturing a different view of an object and a background;

filter a luminance characteristic of each of the first and second images;

determine, based on the filtered luminance characteristic, a state of the object based on a position of the object along an axis that is perpendicular to the reference plane in relation to the first and second planes; and

generate first or second user inputs to an application if the object is determined to be in first or second states, respectively.

30. The medium of claim 22, wherein instructions to determine the state of the object comprises instructions to determine a gesture made by the object.

31. The medium of claim 30, wherein the gesture comprises a poking motion made by the object.

23. A computer readable medium encoded with a computer program comprising instructions that, when executed, operate to cause a computer to acquire an image of a view of an object;

generate a background model, further comprising;

adding the image to a buffer configured to store image data associated with the view;

selecting a value representative of a background for each portion of the view;

outputting, as the background model, the selected values;

determine a position of the object based on comparing the acquired image with the background model; and

control an application based on the determined position of the object.

32. The medium of claim 23, wherein the state of the object is a motion of the object.

24. A computer readable medium encoded with a computer program comprising instructions that, when executed, operate to cause a computer to;

acquire first through fourth images, the first and second images each capturing different views of an object at a first position, and the third and fourth images each capturing different views of the object at a second position;

filter a luminance characteristic of each of the first through fourth images;

determine, based on the filtered luminance characteristic, a motion of the object; and

control an application based on the determined motion.

33. The medium of claim 24, wherein the motion is a gesture.

25. A computer-implemented method of tracking an object of interest, the method comprising:

accessing a first image and a second image representing different viewpoints of the object of interest, the first and second images being accessed b a processor coupled to a computer, the processor further performing operations comprising;

processing the first image into a first image data set and the second image into a second image data set;

processing the first image data set and the second image data set to generate a background data set associated with a background;

generating a first difference map by determining differences between the first image data set and the background data set and a second difference map by determining differences between the second image data set and the background data set;

detecting a first relative position of the object of interest in the first difference map and a second relative position of the object of interest in the second difference map; and

producing an absolute position of the object of interest from the first and second relative positions of the object of interest.

28. The method of claim 25, wherein accessing first and second images comprises acquiring the first and second images from at least one camera configured to transfer data to the processor.

26. A computer readable medium encoded with a computer program comprising instructions that, when executed, operate to cause a computer to;

acquire a first image capturing a view of an object and a background;

detect, for the first image, an occlusion of the background;

determine a position of the object based on the detected occlusion; and

control an application based on the determined position of the object and further based on a filtered luminance characteristic of the first image.

34. A computer readable medium encoded with a computer program comprising instructions that, when executed, operate to cause a computer to:

access a first image and a second image representing different viewpoints of the object of interest;

process the first image into a first image data set and the second image into a second image data set;

process the first image data set and the second image data set to generate a background data set associated with a background;

generate a first difference map by determining differences between the first image data set and the background data set and a second difference map by determining differences between the second image data set and the background data set;

detect a first relative position of the object of interest in the first difference map and a second relative position of the object of interest in the second difference map; and

produce an absolute position of the object of interest from the first and second relative positions of the object of interest.

35. A system comprising:

one or more computers; and

a computer-readable medium coupled to the one or more computers having instructions stored thereon which, when executed by the one or more computers, cause the one or more computers to perform operations comprising;

accessing a first image and a second image representing different viewpoints of the object of interest;

processing the first image into a first image data set and the second image into a second image data set;

processing the first image data set and the second image data set to generate a background data set associated with a background;

generating a first difference map by determining differences between the first image data set and the background data set and a second difference map by determining differences between the second image data set and the background data set;

detecting a first relative position of the object of interest in the first difference map and a second relative position of the object of interest in the second difference map; and

producing an absolute position of the object of interest from the first and second relative positions of the object of interest.