AI chatbots were tasked to run a tech company (opens in new tab)

(businessinsider.com)

4 pointskmote002y ago5 comments

5 comments

Summary: Researchers in a new study tasked an AI-powered tech company (nicknamed "ChatDev") with developing 70 different programs. They found AI could develop software in under seven minutes for less than $1 in costs, on average. AI bots were assigned roles and were able to talk, make logical decisions, and troubleshoot bugs.

kmote00OP2y ago

Original paper: https://arxiv.org/pdf/2307.07924v3.pdf

az09mugen2y ago

"The study said 86.66% of the generated software systems were "executed flawlessly.""

What's that susupposed to mean ?

Unit tests were written to have this percentage ? By human or AI itself ?

86.66% of how many LOC ?

How long will it take for a human (because AI can badly do) to debug the code ?

What was the purpose of generated code ?

Lack of useful context here. Maybe I missed information fast-reading the article.

IMHO, it looks like just another rant on how "good" LLMs can "write code".

EDIT : Sorry OP, I didn't see the arxiv link.

I'm on mobile and read a 10 Mo pdf isn't worth it. I'll try to read it on a computer though, looks interesting.

az09mugen2y ago

I don't want to undermine the work of the researchers because they did a good one with a solid protocol.

But the problem to me is the specification, basically one of the tasks was : "design a basic Gomoku game". Which implies a set of pre-existing rules, something already known and well documented on the web. For example on github you can find 1.2k repos in python with the tag 'Gomoku'.

In a company you will be less likely asked to code a Gomoku from scratch, maybe for an interview, at most.

So in my POV, it's just code synthesis from already existing stuff. And reviewed.

I think my job is safe.

archo2y ago

https://archive.is/ku2xx

j / k navigate · click thread line to collapse

5 comments

kmote00OP2y ago

Original paper: https://arxiv.org/pdf/2307.07924v3.pdf

az09mugen2y ago

"The study said 86.66% of the generated software systems were "executed flawlessly.""

What's that susupposed to mean ?

Unit tests were written to have this percentage ? By human or AI itself ?

86.66% of how many LOC ?

How long will it take for a human (because AI can badly do) to debug the code ?

What was the purpose of generated code ?

Lack of useful context here. Maybe I missed information fast-reading the article.

IMHO, it looks like just another rant on how "good" LLMs can "write code".

EDIT : Sorry OP, I didn't see the arxiv link.

I'm on mobile and read a 10 Mo pdf isn't worth it. I'll try to read it on a computer though, looks interesting.

az09mugen2y ago

I don't want to undermine the work of the researchers because they did a good one with a solid protocol.

In a company you will be less likely asked to code a Gomoku from scratch, maybe for an interview, at most.

So in my POV, it's just code synthesis from already existing stuff. And reviewed.

I think my job is safe.

archo2y ago

https://archive.is/ku2xx

j / k navigate · click thread line to collapse