I have often argued the opposite to your current held view. It is my experience we have lots more than 3500 fans, with around 2000 who come every week and another 3000 or so who pick and choose.
Dont forget, when we beat crewe (2-1 I think) about six years ago on a tuesday nite, SC was moaning about a crowd of over 5500 for what he called a 'derby'. There was about 300 from crewe.