jrcplus

We have IoT sensor data currently flowing Device → Telegraf → InfluxDB → Grafana (TIG stack). This is fine for simple queries and visualization. We want to keep using Grafana, but I'm looking to move to a more proper data infrastructure for 1) SQL-based analytics and ML applications, 2) pub-sub e.g. MQTT to decouple sources from sinks, 3) long-term storage.

We're talking on the order of dozens/hundreds of devices posting data every min or so, not petabytes.

Should I be following a data lake architecture and doing MQTT → S3 / Redshift → ??? → PostgreSQL → Grafana? That seems like a lot of work / maintenance cost / overkill (?).

Or just write a MQTT → PostgreSQL bridge? Why isn't there an off-the-shelf solution, or am I missing something? I feel like this must be a common scenario?

Or should I be looking more closely at enterprise solutions e.g. Snowflake?

Known constraints: • Grafana supports PostgreSQL as a data source (but not Amazon Redshift). • Kinesis Firehose supports S3 and Redshift as destinations (but not RDS / PostgreSQL). • Telegraf doesn't (officially) support PostgreSQL as an output.

Bottom line: I am not a data engineer and want to avoid being one :) Maybe someday we'll be able to afford one, but in the meantime, I want to set things up so it's at least pointing in the right direction, without significant time/money cost.