Haptic interaction with video and image data

• US 6,650,338 B1
• Filed: 11/24/1998
• Issued: 11/18/2003
• Est. Priority Date: 11/24/1998
• Status: Expired due to Term

• Current Assignee: Hanger Solutions, LLC (IP Investments Group LLC)
• Original Assignee: Interval Research Corporation
• Inventors: Saunders, Steven E., Kolarov, Krasimir D.
• Primary Examiner(s): NGUYEN, THU V
• Application Number:
  US09/199,040
• Time in Patent Office: 1,820 Days
• Field of Search: 345/619, 345/441, 382/190, 382/199, 382/203, 434/113, 434/114, 434/117, 434/262, 434/266-268, 434/272, 434/275
• US Class Current: 345/619
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G09B 23/285 for injections, endoscopy, bronchoscopy, sigmoidscopy, insertion of contraceptive devices or enemas

Abstract

The methods and systems of the present invention receive video or image data, for example via television broadcast, cable, or satellite television signal, or directly from a video player, and process the data to determine the geometry and material properties of various objects represented in the video image. Geometry and material property information may be deciphered from a video image by reference to various fixed data sources and/or the application of known methods for elucidating such information from video or image data, such as shape-from-motion or shape-from-shading techniques. Alternatively, all or some of this information may be provided to the system together with the video or image data via a preconfigured side channel. The system coordinates and assembles the image data with the data relating to geometry and material properties of the objects and provides a realistic haptic interaction with those objects to the user.

Claims

1. A method of haptically interacting with a video image, comprising:

receiving a video image;

displaying the video image;

determining geometry of one or more objects represented in the video image; and

interacting with said one or more objects represented in the video image using a haptic display device;

wherein interacting with said one or more objects includes sending haptic data to the haptic display device and receiving a force delivered via the haptic display device; and

said one or more objects represented in the video image are visually seen in the video image to respond to the force delivered via the haptic display device.

2. The method of claim 1, further comprising:

determining material properties of said one or more objects represented in the video image prior to said haptic interaction.

3. The method of claim 2, wherein said step of determining material properties of said one or more objects represented in the video image comprises use of a model recognizer.

4. The method of claim 3, wherein said model recognizer accesses a haptic model of a human.

5. The method of claim 1, further comprising:

receiving side information including data defining characteristics of objects represented in the video image; and

using the side information to assist in determining at least one of class, location, geometry, and material properties of said one or more objects represented in the video image.

6. The method of claim 5, wherein said side information is received as part of a television signal.

7. The method of claim 5, wherein said side information is received from a video player.

8. The method of claim 1, further comprising caching information determined for an object represented in a video image.

9. The method of claim 1, wherein said step of determining geometry of one or more objects represented in the video image comprises use of shape-from-motion techniques.

10. The method of claim 1, wherein said step of determining geometry of one or more objects represented in the video image comprises use of a shape filter.

11. The method of claim 1, wherein said video image is paused, and determinations of geometry and material properties obtained in one or more previous frames for one or more objects represented in the video are used as a basis for haptic interaction with said one or more objects represented in the paused video image.

12. The method of claim 1, wherein said video image is received from a television signal.

13. The method of claim 1, wherein said television signal has a source selected from at least broadcast, cable, and satellite.

14. The method of claim 1, wherein said video image is received from a video player.

15. The method of claim 1, wherein said geometry is three-dimensional.

16. A method of claim 1, wherein said one or more objects represented in the video image are capable of responding to the force delivered via the haptic display device haptically.

17. A method of claim 1, further including determining locations of said one or more objects within the video image.

18. A method of claim 1, further including determining locations of said one or more objects relative to each other.

19. A method of haptically interacting with a video image, comprising:

receiving a video image;

displaying the video image;

determining three-dimensional geometry of one or more objects represented in the video image;

determining material properties of said one or more objects represented in the video image;

elucidating a three-dimensional model for said one or more objects represented in the video image; and

haptically interacting with said model;

wherein haptically interacting with said model includes sending haptic data to a haptic display device and receiving a force delivered via the haptic display device; and

said one or more objects represented in the video image are visually seen in the video image to respond to the force delivered via the haptic display device.

20. A system for haptically interacting with a video image, comprising:

a video display module configured to receive and display a video image;

an image understanding module coupled to the video display module for processing the video image;

a model recognizer module coupled to the image understanding module for recognizing a model; and

a haptic display module configured to provide interactions with one or more objects represented in the video image using haptic data sent from the image understanding module and a force applied by a user;

wherein the one or more objects represented in the video image are visually seen in the video image to respond to the force delivered via the haptic display device.

21. The system of claim 20, wherein said video display module comprises a television.

22. The system of claim 20, wherein said image understanding module comprises a microprocessor configured to operate video data processing software.

23. The system of claim 22, wherein said video data processing software comprises shape-from-motion protocols.

24. The system of claim 20, wherein said image understanding module comprises one or more sources of fixed data including shape data.

25. The system of claim 20, wherein said image understanding module comprises a shape finder.

26. The system of claim 20, wherein said model recognizer comprises one or more sources of fixed data including geometric and material property data for one or more objects.

27. The system of claim 26, wherein said one or more fixed data sources comprises one or more libraries of haptic models.

28. The system of claim 27, wherein said libraries of haptic models comprises a haptic model of a human.

29. The system of claim 20, wherein said haptic display module comprises at least one of a joystick, one-point probe, haptic glove, and hexapod device.

30. The system of claim 20, further comprising a video data signal decoder.

31. A system for haptically interacting with a video image, comprising:

means for displaying a video image;

means for determining the geometry of one or more objects represented in the video image;

means for assigning material properties to said one or more objects represented in the video image; and

means for haptically interacting with said one or more objects represented in the video image by sending haptic data to a haptic display device and receiving a force applied by a user via the haptic display-device;

wherein the one or more objects represented in the video image are visually seen in the video image to respond to the force applied by the user via the haptic display device.