Usually this saying is repeated as: "Whoever is not with us is against us." In this way the overwhelming majority of people are enrolled among our enemies. Here two different sayings of Christ are confused: the one we have just read, and "Whoever is not with Me is against Me" (Matt. 12:30).
Where Jesus speaks about Himself, this is simply a statement of the fact that in relation to Christ one cannot be neutral; failure to receive Him inevitably turns into hostility. In Christian spirituality there is no state "between Christ and Belial." Jesus explains further: "Whoever does not gather with Me scatters." In Christian life there are no stops: a stone that has stopped flying upward is already falling.
But in relationships between people Jesus proposes a completely different logic. We are called to find the possibility of friendship, peace, and love with all people who do not feel unequivocal hostility toward us. Besides, we are called to love enemies too.