It is unlawful for any person to injure, destroy, or remove any hedge, shade or ornamental trees, shrubbery, or crops, planted, cultivated, and grown or improvement made upon or along any portion of any state highway under permit from the department or otherwise, or to injure, destroy, or remove any fence erected under any such permit or otherwise. However, nothing in this section may be construed to prevent any person with the department to do so or the officers of the state charged with the duty of constructing and maintaining any such state highway, from removing any hedges, trees, shrubbery, or crops planted or improvements or fences built under permit, where in their judgment they interfere with or are detrimental to, the use of the state highway for public travel, or such removal is necessary for the construction, alteration, repair, improvement, or maintenance of the state highway.