Automatic Haptic Generation Based on Visual Odometry

• US 20180165926A1
• Filed: 12/14/2016
• Published: 06/14/2018
• Est. Priority Date: 12/14/2016
• Status: Active Grant

• Current Assignee: Immersion Corporation
• Original Assignee: Immersion Corporation
• Inventors: Wu, Liwen, Saboune, Jamal, Raynes, Paige
• Granted Patent: US 10,600,290 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 7/269 using gradient-based methods
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G08B 6/00 Tactile signalling systems, e.g. personal calling systems
• CPC Class Codes: H04N 5/145 Movement estimation for video coding H04N19/51

Abstract

Examples of devices, systems, and methods to automatically generate haptics based on visual odometry are disclosed. In one example, a video having a plurality of frames is received and an optical flow estimate between a first frame from the plurality of frames and a second frame from the plurality of frames is created. In this example, the second frame is subsequent to the first frame. An apparent movement of a stationary object between the first frame and the second frame is detected based at least in part on the optical flow estimate in this example and at least one haptic effect corresponding to the apparent movement of the stationary object is generated based at least in part on the optical flow estimate. The generated haptic effect(s) may be output to a haptic file or a haptic output device, or both.

Claims

1. A method of producing a haptic effect comprising:

receiving a video having a plurality of frames;

determining an optical flow estimate between a first frame and a second frame, the second frame subsequent to the first frame;

detecting an apparent movement of a stationary object between the first frame and the second frame based at least in part on the optical flow estimate;

generating a haptic effect corresponding to the apparent movement of the stationary object based at least in part on the optical flow estimate; and

producing the haptic effect via a haptic output device.

2. The method of claim 1, wherein the video comprises at least one of a video file, a real-time video, or a point-of-view video.

3. The method of claim 1, wherein the first frame and the second frame are consecutive frames in the plurality of frames.

4. The method of claim 1, wherein the second frame is separated from the first frame by at least one frame in the plurality of frames and wherein the second frame is near the first frame.

5. The method of claim 1, wherein the second frame is selected from the plurality of frames based at least in part on a frame rate of the video.

6. The method of claim 1, wherein determining the optical flow estimate comprises:

determining a plurality of feature pixels within a partial area of the first frame; and

generating a motion vector corresponding to the plurality of feature pixels within the partial area of the first frame and a corresponding plurality of feature pixels within partial area of the second frame that indicates the apparent movement of the stationary object between the first frame and the second frame.

7. The method of claim 6, wherein detecting the apparent movement of the stationary object comprises detecting a same direction or a substantially same direction of the motion vector.

8. The method of claim 6, wherein the plurality of feature pixels within the partial area is determined based at least in part on a detection of the stationary object and wherein the determined partial area surrounds the stationary object.

9. The method of claim 6, wherein the plurality of feature pixels within the partial area correspond with a plurality of feature pixels within a prior partial area determined in creating a prior optical flow estimate corresponding to a frame other than the first frame in the plurality of frames.

10. The method of claim 6, wherein the optical flow estimate comprises motion vectors that correspond to only feature pixels within the partial area.

11. The method of claim 6, wherein the partial area comprises a top-center view corresponding to the first frame and the second frame.

12. The method of claim 6, wherein a size of the partial area is based at least in part on a number of detected stationary objects.

13. The method of claim 1, wherein the apparent movement of the stationary object is detected by determining a cluster of motion vectors within the optical flow estimate having a same direction or a substantially same direction.

14. The method of claim 13, wherein the generated haptic effect comprises an intensity parameter corresponding with a magnitude of at least one motion vector within the cluster of motion vectors.

15. The method of claim 1, wherein the haptic effect is output to a haptic track.

16. A non-transitory computer-readable medium comprising one or more software applications configured to be executed by a processor, the one or more software applications configured to:

receive a video having a plurality of frames;

determine an optical flow estimate between a first frame and a second frame, the second frame subsequent to the first frame;

detect an apparent movement of a stationary object between the first frame and the second frame based at least in part on the optical flow estimate;

generate a haptic effect corresponding to the apparent movement of the stationary object based at least in part on the optical flow estimate; and

produce the haptic effect via a haptic output device.

17. The non-transitory computer-readable medium of claim 16, wherein the optical flow estimate comprises a plurality of motion vectors and the apparent movement of the stationary object is detected based on a cluster of the motion vectors pointing in a same or a substantially same direction.

18. The non-transitory computer-readable medium of claim 16, wherein the haptic effect comprises an intensity parameter based at least in part on a magnitude of a motion vector from the cluster of motion vectors that corresponds to an approximate movement of a camera that recorded the video.

19. A device configured to produce a haptic effect, comprising:

a camera;

a display;

a haptic output device; and

a processor in communication with the camera, the display, and the haptic output device, wherein the processor is configured to;

receive a video having a plurality of frames from the camera;

determine an optical flow estimate between a first frame and a second frame, the second frame subsequent to the first frame;

detect an apparent movement of a stationary object between the first frame and the second frame based at least in part on the optical flow estimate;

generate a haptic signal configured to cause the haptic output device to output a haptic effect corresponding to the apparent movement of the stationary object based at least in part on the optical flow estimate;

output the first frame and the second frame to the display; and

output the haptic signal to the haptic output device such that the haptic output device produces the haptic effect as the second frame is displayed on the display.

20. The device of claim 19, wherein the first frame and the second frame are consecutive frames in the plurality of frames.