When the misconduct complained of is on the part of the judges of election, it shall not be held sufficient to set aside the election, unless the vote of the precinct, township or ward would change the result as to that office.