The governor shall have the power to remove from office at any time any member of the board for continued neglect of duty, for incompetency, or for unprofessional conduct as that term is defined in this act.