So I'm just done fiddling with this for a couple of hours. I kinda got it better, but it didn't get rid of the problem. After alot of adjusting and playing with the worm gear, I'm 90% sure it's a damaged bearing or some dirt. When passing a certain point, it almost feels like it's "grinding" and uneven for a small part.
I have decided to just go for the stripdown, and also switch out all the bearings for something higher quality (SKF/FAG etc.).
Do you know what kind of grease is best to use here? The store I get the bearings from also sells SKF LGMT 2, mineral oil based, lithium soap thickened grease. Should be good enough?