Engine balance is covered in sophomore engineering Kinematics classes. Pretty standard calculations, illustrated with static or dynamic models.

Tires are balanced the same way, statically, and sometimes tested dynamically, and you might be referring to that type of operation.

Before a newly designed engine is spun, the balance issues are addressed, and the success is always verified with the engine in operation (or, "dynamically".) This does not require a structural dynamicist, but one is often involved in the validation procedure.

If that's what you had in mind, you might start first with a kinematics book.

Best wishes,
Dave Olson