Hi Aptor,
I would make some sort of neutral reference mark inside the fuselage at the servo arm with everything cold. Then run it up until the trim shift occurs, and quickly remove the hatch and see if the servo arm still lines up with the reference mark. This would narrow the problem down to servo drift or linkage shift.
Roger