FMP Issues with Wobbly Footage

In this video I was playing around with the wobbly footage that I had to see if I didn't have to use a still image. I have managed to use some tracking before, so I know I can do it. I had to use it for the green screened reporter scenes. Since the camera on set moves, I had to make the actor move with the camera. Therefore, I know it is do-able and within my abilities. For some reason, my guess is to do with the luma matte I had used, it did not want to track. I did ask for still footage but I was given it too late and had to get on with the main edit along with the other project I am working on.