You are not attaching the Koverite properly. Correctly applied, it will not come loose. I apply it with nitrate dope to the structure. Several coats to the wood, light sanding, then more dope through the fabric to adhere it to the wood. Additional dope to seal the fabric, then paint.

How are you attaching it? Whatever you are doing is not bonding the fabric to the wood.

Edit: I reread your post and see you are using CA to adhere the fabric. That doesn't work. Read the directions on the Koverite, and you will see the easy way to have success with this covering. Good luck.