Why would you be cautious? Not saying there's no downsides, but if anything I consider it to be an "upside" that the device is simpler. Now I don't need to worry about linux, or making sure it's a read only filesystem with an image all setup, etc.
I look at it this way: If you want a remote device that measures or controls just one or two things, but does very little if any calculations - use a esp8366/32 (or anything with low power requirements). If you need to process data (video stream), or do other calculations, potentially while controlling many different sensors/things - you'll likely want a raspberrpi/similar