Natural fibers (Cotton, linen, silk, wool, cashmere, hemp) utilize far less energy to create and will eventually decompose as they are made from plant materials. Due to this, they won't end up in the throat of a fish, the nest of a bird, or the sushi you are eating.
(I've attached a decent introduction to why fast fashion and new clothing is detrimental to our environment below.)
https://www.newsweek.com/2016/09/09/old-clothes-fashion-waste-crisis-494824.html