Currently, it seems that contributors logged in can see and use the "INVITE" button in the top right corner of Community page and can invite new members.
Many groups are designed to be private, for example only for people who have purchased a product or for people who have taken a particular action. These groups will not want all members to be able to invite other people.
So, this INVITE button should not be visible by default and should be up to the community admin(s) as to whether contributors can invite new members or not.