If I chose to remain “independent”, what sort of downside can I expect?

Downsides include fewer referrals and responsibility for all aspects of your business including scheduling, billing and possibly lower gross income.