Once any swinging stops, the plumb points in the direction of that vertical line, and the airplane CG is located somewhere along than line.
True. But you don't know where along that line the center of mass is. So, to determine the balance point, you want the plane level (i.e. horizontal stab level). It isn't that the balance point changes when the plane is tilted, it's that you can't tell for sure where the balance point is with the plane tilted.