Section 19. If, upon written complaint to the mayor or to the selectmen, it shall appear, after notice and hearing, that the city or town clerk, when a member of the board of registrars, and two registrars are of the same political party, the mayor or selectmen, as the case may be, shall remove from office the one of such two registrars having the shorter term remaining. If, upon like proceedings, it shall appear, after notice and hearing, that a registrar of voters, other than the city or town clerk, has ceased to act with the political party which he was appointed to represent, the mayor or selectmen, as the case may be, shall remove him from office.