I have tested this and at the moment it can give unexpected results it is recommended you use the C alternative cvCalcOpticalFlowPyrLK.