I was browsing through the NOVAK website and came across the
Novak GTB Esc.

It says that it has no brushed motor limit. That I can comprehend,
but all it says for BL was that it is compatible with all Novak 540-size sensored brushless motors.

So does that mean that I can only use a Novak sensored BL motorwith the ESC, regardless of the number of turns?

If so, is there any way for me to "tweak" so that the Esc can be compatible with other BL motor?

i believe you should be able to run any sensored 540 brushless motor, not necessarily a novak. but i would use the novak motors. will you be racing?

I doubt I'll be racing, I prefer to hit the tracks with a couple of friends, having the
occassional bumping on one another and laughing it off... but maybe in time to come I will,
who knows? Lol...

if you're not going to be racing, novaks get kind of expensive for just messing around. if you're just hitting the tracks with friends and there's no restrictions on what you can run, as in you don't have to be "legal", i would check out the mamba max systems. on a 2s lipo or even 6 cell nimh with a 4600 or 5700 motor, i think you'd be really happy with it.

what's the system going in anyways?

Yes the GTB will run other brands of sensored motors, but for brushless the turn limit is 3.5 turns.