hekike

Open source has always been a big part of my career. I worked and maintained multiple Node.js libraries, joined the Node.js Diagnostics WG, and am now building OpenMeter. So I'm probably biased, but I think we reached a new inflection point with open source. The infrastructure layer, coding languages, and frameworks are all de facto open - what's next? Services that help teams build ‘applications’, like analytics and billing. Let me explain why I think this way:

Let’s look at billing as a use case, as this is what I’ve recently focused on. Billing is like choosing a cloud provider. It’s for life, hard to change, and dictates what kind of sales deals you can execute. Building requires lots of code and integration, and you lose money when it breaks. This is why you need to be strategic about it.

These three issues must be carefully considered when deciding if you should go with open-source:

1. Your use cases will evolve, and so will your dependencies

For example, every startup goes through an evolution of product packaging and billing requirements for their self-service credit card sales:

a. Early-stage startups usually hardcode pricing into their app to get started quickly b. During the growth stage, requirements for pricing models change, and now code is scattered between the billing platform and your product code c. When you become an enterprise, your sales team wants more flexibility to quickly structure custom deals and visibility in the CRM, like Salesforce

But as your company grows, you have now gone through all three stages, and your product catalog is all over the place: in your code, billing platform, and Salesforce. This is why you should build on extendable open-source components.

2. Beware of vendor lock-in

For example, billing is for life. This code becomes the revenue lifeblood of your company and is not easy to replace. That calls for no vendor lock-in and maximum business flexibility. Big vendors tend to push up prices at renewal time. It’s not a risk with open-source applications.

3. Mission-critical, and therefore strategic, components should be Open Source

As you evolve your pricing and sales motion, you produce large amounts of new code tied to the billing solution. This is problematic because open source will always outpace and out-innovate closed-source vendors over time. But now you are stuck with proprietary integrations, which become an expensive challenge over time.

—

This sounds like infrastructure, right? As an industry, we have already agreed that infrastructure should be open-source. Are more things becoming infrastructure as applications become more complex, or is it a progression that open-source evolves to include application services? Time will tell.

One thing is for sure: open source reduces cost and vendor lock-in while increasing customization options and talent density. This is helpful in every layer of your business stack and an insurance policy against poor vendor behavior.

AI is transforming software engineering, and many people wonder if developer jobs are going away or just changing. Here are my two cents:

For context, my journey as a developer over the years followed the path of many others: during university, I started as an intern in Eastern Europe at a small company, 100% of my time writing code, and later became a staff engineer at big tech (Netflix, Stripe), mainly focusing on alignment and architecture. Today, I’m running a startup (OpenMeter) and doing more product, sales, etc., than coding. This would be all irrelevant, except I don’t think this path I followed will work in the future for others. You and I, as developers, need to think about what’s next.

I often think about this AI-driven developer transformation for selfish reasons: Will I still have a job in the future? And as a founder, I have to be strategic about hiring. Let’s break down what’s happening.

I have seen this analog about AI changing developer jobs somewhere that resonated a lot with me. It goes like this: Copywriters' job, before the invention of the typewriter, was mainly focused on the craftsmanship of typography as their writings had to be easy to read. Their job changed when machines came out and shifted towards being grammatically correct, which later also changed with the invention of computers. It will change again with AI as it’s easier to write cohesive narratives with it. But you know what didn’t change? They must still develop a good story, make it easy to read, and sell it to their audience.

People who say developer jobs are going away only see the act of writing code previously inaccessible to most people. Suddenly, anyone can do it (or at least that’s the promise). But hold on, isn’t that writing code what most developers do? Not really, only early on in their careers. For example, I wrote almost zero code at Big Tech as a staff engineer. My job was to identify big technical problems holding back the entire company, understand why and what we can do about them, build alignments with leadership about prioritization, recommend a solution, design the architecture, and delegate the implementation to developers focused on coding. The only problem was there were never enough developers writing code precisely what AI is changing now. Some of you may say this sounds terrible or straight of pissed that this was my job as a Staff Engineer but believe me, it was both fun, and the impact on business was measurable, not talking about the salary. Still, the reality is simple: I was more of a product manager with deep technical knowledge than a developer. But you are probably getting the point; there is plenty to do as a technical person, and developers are just having a typewriter moment; writing code will become a commodity, and being a developer will stay, but the required skills will shift into:

- Good communication skills

- Product management skills

- Architecture knowledge

- Being a generalist

My two cents are that the roles of product managers and developers will merge, and future products will be mindblowing because all that energy and creativity will be used to solve customer problems instead of writing and maintaining code.

You probably noticed that the title of this post mentions two future developer types, but I only talked about one so far. I’m less confident about this second type, but I think this will go away last from how developer jobs are perceived today. No matter how good AI is today or will be, someone has to understand the code for mission-critical applications. Many businesses like critical infrastructure and fintech must be correct, compliant, and auditable; you can’t just ask ChatGPT to implement billing and hope it’s accurate. I think for a long time, even if they are just about reviewing pull requests, a new type of developer who is highly focused on correctness will be paid a lot to ensure the critical systems work flawlessly.

Comment to agree or disagree.

Hi HN, at OpenMeter, we switched to using ClickHouse as our database to pre-aggregate and query usage.

Additionally, our initial choice of the streaming processor, ksqlDB, revealed limitations when scaling for small to medium-sized producers. The constraint was around ksqlDB's limited capacity to run persisted queries per instance (20 on Confluent Cloud) and its lack of support for clusterization.

With ClickHouse, the new processing pipeline looks as: API -> Kafka -> Kafka Connect -> ClickHouse

Under the hood, we use the ClickHouse Kafka Connect Sink plugin to ensure consistent data movement. This plugin guarantees exactly-once delivery between Kafka topics and ClickHouse tables, which is critical, as Kafka Connect tasks are only aware of the latest topic offset acknowledged by the consumer. For example, consumers can fail to acknowledge a processed offset due to a network error or an exception. This is great as exactly-once inserts prevent dropping or double-inserting usage, leading to incorrect billing.

In OpenMeter, we pre-aggregate usage events into one-minute tumbling windows to reduce the number of rows we need to scan at query time. To do this, with ClickHouse, we use the AggregatingMergeTree table engine that enables incremental data aggregation when combined with MaterializedView. In ClickHouse, MaterializedViews are trigger-based and update when new records are inserted into the source table. Consequently, the corresponding materialized views are updated whenever Kafka Connect transfers a batch of events to ClickHouse. This also means inserts can fail when the view cannot process a record at trigger. We send failed events into the Dead Letter Queue topic for later processing.

To help ClickHouse with hot topics, we will consider adding an extra streaming aggregation step for high-producers, but this time with a more horizontally scalable stream processor like Arroyo. This would reduce ClickHouse insert batch sizes. Based on our tests, ClickHouse works best if batch sizes are 50-100k and less frequent than per second.

To see it in action, check out our open-source repo: https://github.com/openmeterio/openmeter

I recently looked into how to build out robust usage reporting with Stripe. This comes basically from the set of challenges around:

- Workaround Stripe API Rate Limits

- Handle billing cycles and minimize usage drift into the next billing cycle

- Deliver usage exactly once to avoid under and over-billing

- Stripe requires to report usage on Subscription Items

You can read the full blog post here:

https://openmeter.io/blog/how-to-report-usage-to-stripe

I'm curious about what people do to build our usage reporting. Do you use message queues? Workflow management software like Temporal? Simple CRON job?

I'm researching the best ways to collect the execution duration of containers/serverless workloads for billing and chargeback use cases. Per second granularity, like AWS CPU-Hours. I'm finding that most engineers' journeys start with listening to start/stop events and calculating the delta between those lifecycle events. But over time, they find it challenging to handle lost events or long-running processes. It's also not always fair to customers when a container takes longer to start (cold start) or stop.

So far, I have heard two primary ways: 1. Lifecycle events like start/stop, 2. Heartbeat-style pinging. I summarized them in this blog post: https://openmeter.io/blog/how-to-meter-workload-execution-duration

I would love to iterate on the post based on your thoughts and include other ways if any. How do you meter container/serverless execution duration for billing?