I poked around on the internet and found a lot of sites use the Pauling electronegativity scale (link 1, link 2). For carbon the value is 2.55 and for bromine 2.96. Not quite a 0.5 difference, but closer - and this "rule" is somewhat arbitrary to begin with. Also the dipole moment of bromobenzene is around 1.7 D (link 3), certainly not insignificant. I too would have said bromobenzene is polar. It is not strongly polar, whether it is "a little" polar, "mildly" polar, "somewhat" polar, etc. seems to be the question - but that's just semantics. Maybe the reason many people consider the $\ce{C-Br}$ bond to be polar is because it has a very observable effect on electrophilic aromatic substitution reaction rates. Specifically, why should the relative rate for electrophilic aromatic substitution at the meta position in bromobenzene be slower that the rate in benzene itself unless you invoke the inductive effects caused by a polar $\ce{C-Br}$ bond?