Whether you remember it or not, you may have once granted Facebook permission to store the contacts in your phone's address book. If you did, Facebook is continuously uploading these contacts to its servers.
You can find out if your phone is doing this, and stop it pretty easily.
Here's how to do that on Facebook for Android: