undefined | Better HN

0 pointsdcminter7mo ago0 comments

This could not possibly go wrong...

You're braver than me if you're willing to trust the LLM here - fine if you're ready to properly review all the relevant docs once you have code in hand, but there are some very expensive risks otherwise.

0 comments

awongh7mo ago

This is LLM as semantic search- so it's way way easier to start from the basic example code and google to confirm that it's correct than it is to read the docs from scratch and piece together the basic example code. Especially for things like configurations and permissions.

dcminterOP7mo ago

Sure, if you do that second part of verifying it. If you just get the LLM to spit it out then yolo it into production it is going to make you sad at some point.

simianwords7mo ago

There’s nothing brave in this. It generally works the way it should and even if it doesn’t - you just go back to see what went wrong.

I take code from stack overflow all the time and there’s like a 90% chance it can work. What’s the difference here?

jcattle7mo ago

However on AWS the difference between "generally working the way it should and not working the way it should" can be a 30,000$ cloud bill racked up in a few hours with EC2 going full speed ahead mining bitcoin.

simianwords7mo ago

For those high stakes cases maybe you can be more careful. You can still use an LLM to search and get references to the appropriate place and do your own verification.

But for low stakes LLM works just fine - not everything is going to blow up to a 30,000 bill.

In fact I'll take the complete opposite stance - verifying your design with an LLM will help you _save_ money more often than not. It knows things you don't and has awareness of concepts that you might have not even read about.

dcminterOP7mo ago

Well, the "accidentally making the S3 bucket public" scenario would be a good one. If you review carefully with full understanding of what e.g. all your policies are doing then great, no problem.

If you don't do that will you necessarily notice that you accidentally leaked customer data to the world?

The problem isn't the LLM it's assuming its output is correct just the same as assuming Stack Overflow answers are correct without verifying/understanding them.

simianwords7mo ago

I agree but its about the extent. I'm willing to accept the risk of ocassionally making S3 public but getting things done much faster, much like I don't meticulously read documentation when I can get the answer from stackoverflow.

If you are comparing with stackoverflow then I guess we are on the same page - most people are fine with taking stuff from stackoverflow and it doesn't count as "brave".

2 more replies

j / k navigate · click thread line to collapse

0 comments

awongh7mo ago

dcminterOP7mo ago

Sure, if you do that second part of verifying it. If you just get the LLM to spit it out then yolo it into production it is going to make you sad at some point.

simianwords7mo ago

There’s nothing brave in this. It generally works the way it should and even if it doesn’t - you just go back to see what went wrong.

I take code from stack overflow all the time and there’s like a 90% chance it can work. What’s the difference here?

jcattle7mo ago

simianwords7mo ago

For those high stakes cases maybe you can be more careful. You can still use an LLM to search and get references to the appropriate place and do your own verification.

But for low stakes LLM works just fine - not everything is going to blow up to a 30,000 bill.

dcminterOP7mo ago

Well, the "accidentally making the S3 bucket public" scenario would be a good one. If you review carefully with full understanding of what e.g. all your policies are doing then great, no problem.

If you don't do that will you necessarily notice that you accidentally leaked customer data to the world?

The problem isn't the LLM it's assuming its output is correct just the same as assuming Stack Overflow answers are correct without verifying/understanding them.

simianwords7mo ago

If you are comparing with stackoverflow then I guess we are on the same page - most people are fine with taking stuff from stackoverflow and it doesn't count as "brave".

2 more replies

j / k navigate · click thread line to collapse