In addition to the above suggestions, add another block inside the OUTER half of the pant, with a hole the diameter of the axel drilled into the block . Center the hole so the axel plugs into it when the pant is secured using one of the above suggestions. This arrangement should eliminate much of the up-and-down flexing of the pant which has been part of your problem.

Jack