The directors shall hold office, one-third for one year, one-third for two years and one-third for three years from the first of June following their appointment, and at their first regular meeting shall cast lots for their respective terms; and annually thereafter the mayor shall, before the first of June of each year, appoint as before three directors, who shall hold office for three years and until their successors are appointed. The mayor may, by and with the consent of the legislative branch of the municipal government, remove any director for misconduct or neglect of duty.