I think J78 and FTF said it well.
Every car I've ever had went as fast or faster with no backpressure. You do typically need to jet richer though.
An engine that's too lean WILL be slower uncorked - until you re-jet.

The thing about velocity is key too, as long as that length of pipe is part of the tuning of total length vs pulses at a given RPM. That said, better too big than too small.

Primary header tubes that are too big can really hurt an engine, but beyond that, exhaust pipes that are "too big" don't hurt very much - but pipes that are "too small" will really hurt, in a v8 they can kill off ~100hp on a really hot engine.

I suspect JimW's old rusted out system was acting kind of like "open exhaust", and leaned things out too much so it was slower - and also shortened the overall length which took low end away.