After 30 plus years of overlocking I faced all sorts of problems, and maybe able to help you with yours. I am always happy to take your call, or you are welcome to visit our set-up (just phone first)
There are some yarns that just don’t stitch well. They are to lumpy, stretchy or sticky Just bin them!
POLYPROPOLENE will generally run the smoothest as it is even in thickness smooth and does not stretch.
NYLON is stretchy and sometimes sticky it will require the thread tension to be loosened and sometimes a spray of silicon will help, when tensions right it makes a good job.
WOOL gives you the best coverage of them all, but not all wools are suitable. an even thickness and two ply
wool is best. For odd colours I have used 8ply knitting wool from the local wool shop(100gram will overlock 10 metres)
YARN FEED Make sure your yarn is mounted on a wall or stand away from the machine (at least 2 mts) with the cone on the horizontal and up high enough that the yarn is travelling down hill the weight of lifting the yarn upsets the tension and will create scalloping.
THREADS We have a 3 thread machine and only use quality threads. My first choice is Poly-cotton or Bonded Nylon 20 denier.