Honour the allow-ip option even with noauth