Do bands that realize their potential early tend to go downhill from the start?

It seems like a lot of bands that make impressive debuts, usually opt to "dumb" their sound down over time. Often times it seems like they set the standard really high for themselves, but then don't even try to top their previous work? That really annoys me when a band doesn't try to put out anything that compliments their previous work, and they just do something else entirely and not very well either. Fans sit their and call it "growing as a band", while obviously it's more like "not living up to potential". This applies to so many bands I've listened to. I really think bands should quit being so lazy and realize when you put something good out, fans want more of that goodness before you start trailing off into something else like a cop out.