I have 2 opinions on this:
1.) It was simply abandoned. Even in OoT, the temple is the most quiet place in the castle town with the exception of stray dogs roaming there at night. The lack of physical human life regarding maintenance of the temple explains the environment and ample plant life thriving within. The temple was never forgotten and was still considered an unbelievably important monument to Hyrule's history, so something or someone planted 2 non-human, fully physically capable guards there to protect the temple. Only a select few individuals were ever deemed worthy of having true use of the temple, most notably Link, so the appearance and abandonment was the least of anyone's concerns if they remembered the legend.
2.) Chaos or war broke out, causing immense damage to the temple's structure. In a panic, something or someone, as said before, planted 2 fully capable guards within to protect the temple from unworthy, future invaders.
As you can see in these pictures you notice that TP's temple is missing the altar and the door.