there's always going to be a point or points where you go too light and you have a poorly made plane. Which is why aerodynamics comes into play the faster and lighter you go. Also your hardware and it's ability to adjust to these changes comes into play. It's a balance of all these factors that decide how light your boat can be.

Example of a poorly designed light boat. My stupid billings jet boat. The boat is so wide that when loaded and in the water, the impeller isnt more than 10% in the water. This is bad, takes forever to prime, takes forever to recover from cavitating and cant get to WOT, because it turns into a skipping stone. The only way to fix the problem is to add a lot of weight. Which then leads to one more fix, dump the boat. Sometimes it's the only way. the example is a bit off topic, but shows even a "properly" setup boat can be too light right from the start.