Section 123. The expense of removing a prisoner from one jail or house of correction to another shall be paid by the county from which he is removed. The expense of removing a prisoner to or from any of the correctional institutions of the commonwealth by order of the commissioner shall be paid upon bills approved by him, out of the appropriation for the removal of prisoners, except that when a removal of a prisoner is made under section one hundred and seventeen or one hundred and eighteen, the expense thereof shall be borne by the institution from which the prisoner is removed. The expense of removing a prisoner to the Bridgewater state hospital or to a state hospital shall be paid by the prison from which the prisoner is removed. The expense of removing, under section one hundred and seventeen or one hundred and eighteen, a person under commitment to a department for defective delinquents, shall be paid by the institution from which such person is removed; and all hospital expenses incurred under either of said sections in connection with a prisoner or a person under commitment as aforesaid shall be paid by the prison or institution from which such person is removed.