I guess the main problem are Agile zealots, who received their 'Scrum Master' certifications and now are trying to apply Agile onto anything - and that's where the problem is.
You cannot use Agile for everything - impossible. Agile is great if you are adding a feature to an existing product and can be shipped with minimum viable functionality and then iterate and add more.
If you need to build an API that needs to be specced, secure and simply just work out of the box, forget Agile. You need good old waterfall - at least for the core functionality and then use Agile to plaster on top bits.
If you're starting fresh, I would go every single day with Lean instead of Agile - because you have no idea yet what's going to work and what isn't.
Yes, those of us who have been in development for some years can see where it fits, zealots don't - they treat it as a silver bullet for everything.