Right now the options are only for Contact, Assigned User, Guest, Additional Email. I think it would be best to have a designated section for the Appointment or Calendar owner since we don't necessarily need to notify the assigned user for the other appointment from different calendars.