If a user is assigned as a follower they are able to view the contact.
Currently with granular permissions enabled & 'view only assigned data' users can only view a contact if they are a contact owner. Followers are unable to view the data.