Section 96. If a collector becomes incapacitated by reason of mental illness, absconds or removes from the town or in the judgment of the aldermen or selectmen is about to so remove or is otherwise unable to discharge his duty, or if he refuses on demand to exhibit to the aldermen or selectmen his books, vouchers and accounts of collections as provided in this chapter, the aldermen or selectmen may remove him from office.