Have we ceased being "fishers of men" ?

Jesus said to the fishermen, "Follow Me and I will make you fishers of men..."

Notice that the second part of his invitation naturally followed the first - almost like a consequence.

If we have ceased being "fishers of men" or at least sincerely loving and showing love to others, does it mean that we have ceased to follow Jesus?

And if we have indeed ceased to follow Him, what is the cross we are unwilling to take up?

And if there is a cross we have refused to take up, what is keeping us from doing so? What worldly distraction are we not denying ourselves?

I know what keeps me from obeying the command, "Follow Me..." Honestly, it scares me. However, I am comforted that as long as I feel that I ought to surrender to God and earnestly wish that I could turn over a new leaf, God has NOT given up on me yet.