A tail heavy plane will do exactly as you describe.
Trimmed for upright level flight it will carry some down elevator trim. When inverted, the down trim will make the plane climb. When hovering, the down trim will make the plane pull to the belly.