Your contacts represent your social graph, that is, the links that tie you to your friends and family. This information is invaluable for intelligence agencies and for advertising. Here is the link to the contact management interface in gmail:
If you are concerned about your privacy, I advise you to
- Regularly remove contacts that you do not need and
- Cleanup your mailbox to delete all emails older than 2 years. To do that, see the related post: how to delete all old emails in gmail.