Damages for withholding the property recovered shall not exceed the fair value of the use of the property, exclusive of the use of improvements made by the defendant, for a period not exceeding six years; and, when permanent improvements have been made by a defendant, or those under whom the defendant claims, holding under color of title adversely to the claims of the plaintiff, in good faith, the value thereof shall be allowed as a setoff against the damages of the plaintiff.