The town clerk shall preserve such ballots in the sealed envelopes for a period of one hundred eighty days after the election. However, in the case of a contested election, either party to such action may request the court to order that the sealed envelopes containing challenged ballots be delivered to the board of admissions by the town clerk together with any memorandum or remarks which were attached to the election returns or required to be so attached. If so ordered, the board of admissions shall then convene and consider each challenged ballot and rule as to which ballots shall be counted. The results thereof shall be added to the vote totals. Federal offices shall not be counted on a challenged ballot that was issued to a person who was also issued a provisional ballot.