Clutch/brake fluid is hygroscopic, it absorbs water. Even a small amount of water in brake fluid reduces the boiling point by quite an amount, that is why its good to flush your brake fluid every 2 years at least.
A cooling system full of brake fluid would be very messy I'd imagine, any leak will destroy your paintwork.
I don't know what the specific heat capacity of brake fluid is but I'd be surprised if its any better than water.
Anyway running an engine too cool is bad for it. A cold engine has excessive running clearances, the ecu will overfuel coz it thinks its in the warmup cycle and anyway the hotter an engine can run the more thermally efficient it is. Thats why modern engines actually run hotter than older ones, they're designed that way.