"If you don’t want a specific person to follow you, or you want to disconnect from a specific follower, you can block them at any time."
So, you need to block him. https://www.facebook.com/help/168009843260943 teach users how to block someone.
Also, Facebook mentioned that:
So,
1. If you turn off follow setting and then turn it back on, your followers will be restored.
2. If you deactivate and then reactivate account, your followers will be restored.
However, there's some conceptual missing here:
1. If you block your follower and then unblock him, both following connection is cut off. That's means you are no longer following him, and also, he/she no longer to following you. Because of security concerns, re-block someone only allow after 48 hours.
For those who are curious why 48 hours restriction, this security concerns is:
When user A block user B, then user B can't block user A. So every time user A want to visit user B profile, user A just unblock user B. that's why 48 hours restriction to minimize the stalking.
Is that's all?
No, it's not. there's one more way User A can force user B to unfollow him without blocking process:
The real story:
1. User A and user B is not friend. User B follow user A.
2. User A don't want user B follow him. But also, user A don't want to block user B. Seem like that's no official option to remove user B from following.
3. User A go to https://www.facebook.com/USER_B_USER_ID, click "Add Friend" option.
4. After Friend request sent, user A immediately click "Cancel request" option before user B able to notice the notification.
5. And now, the magic happen, user B is automatically unfollow user A, and vice vesa. Followers tab is gone.