No expert here too. You will DEFINITELY have issues if the pin does not bear well on BOTH sides in a high load situation. There will be unwanted slop or worse. The solution is simple. I would simply remove a few thousands off of the thickness of that servo arm, locally where you need it. But leave full thickness at the root (where the max moment is) and be certain that there is no abrubt thickness change in the transition (too sharp a change would be a high stress concentration).