shlosky

How do you collect application metrics? For example, how do you keep track of the count of jobs your backend does?

I have all sorts of metrics I would like to count and later query. For example I have a lambda that processes stuff from a queue, and for each batch I would like to save a count like this:

    {
      "processes_count": 6,
      "timestamp": 1695422215,
      "count_by_type": {
        "type_a": 4,
        "type_b": 2
      }
    }

I would like to dump these pieces somewhere and later have the ability to query how many were processed within a time range.

So these are the options I considered: 1. write the json to the logs, and later have a component (beats?) that processed these logs and send to a timeseries db. 2. in the end of each execution send it directly to a timeseries db (like elasticearch).

What is better in terms of cost / scalability? Are there more options I should consider?