iirc the team managed to bring it to a lever an order of magnitude lower than off-the-shelf models
I think it's also a bit scary, because 8 weeks is very little time for testing, tuning, and validation of something as opaque as a machine learning model. If it worked right the first time, that's great. But there is still a lot of inherent uncertainty in ML projects. Decision makers need to take that uncertainty into account when planning.
That, or, the 8 weeks only covers the final training runs and the implementation/deployment, and doesn't include time spent developing and tuning proof-of-concept prototype models.
To me (programmer in Sweden) the largest single team I've been on was 14 people and that was _very_ large (indeed the largest in the tech department). We actually broke ourselves up into two more informal groups since we thought that was a more manageable team size.
can i get a clarification - when it says "in-browser" i hear "on-device" as in it doesnt call back to replit to get the predictions. i assume that's inaccurate?
for cost/compute purposes i'm wondering how small models have to get in order to run "truly in browser"
But as long as these are tiny parts of codebase (which will most probably be the case), I doubt anything can be done with that. No one will go to court because of a few generic functions.
I've gotta let you know that people copy code snippets from all sorts of codebases with little regard for licenses anyway, because they're toothless in 99% of cases, AI or not. It's a nice illusion that anyone respects licenses, but it's just not true.
It's been proved in court that AI does not infringe on copyright or licenses since it generates things from an understanding of the whole, instead of directly stealing, just like the human brain does.
Hypocrisy dressing up as progress in the machine learning field has been one of the most embarrassing scenes in software engineering recently. The genie may be out of the bottle but the fact is that a bunch of software engineers with a “move fast ethics later” attitude are the ones who let it out and they shouldn’t get to shrug it off for free.
Do you have a source for that?
This SF Conservancy article[0] says that's not true:
>Consider GitHub’s claim that “training ML systems on public data is fair use”. We have not found any case of note — at least in the USA — that truly contemplates that question.
The first major court case I know about is the class-action case Matthew Butterick is trying to build.[1]
[0] https://sfconservancy.org/blog/2022/feb/03/github-copilot-co...
If you ever plan to steal someone's code and justify it with "my brain is able to learn, therefore copyright doesn't exist" I warn you right now this will not fly.
If the “new” code is close enough to be considered a derived work then you will need a license.
Fully respecting the licenses this code was published under, one would hope?
For a simpler example than code-generating ML, if I write a program to recognize a directory of source code vs non-source code, and my logic is if (unbalanced parenthesis count in all files > X) { return "not source code"; } else { return "source code"}.
And then I compute X by scanning over the linux kernel source and counting the amount of unbalanced parens, have I just committed a GPL violation if I don't GPL my source code recognizer?
Unlicensed source code - the default - is still protected by copyright law. If it's hosted and served from a different jurisdiction where no exception exists for training data models.
Then there are also licenses that explicitly prohibit commercial usage to consider.
What it comes down to, as it always does, is that a small group of (practically) untouchable people are making money by abusing and thereby irreparably damaging the trust and good will of the collective.
It's a complex topic eh
"Where's the line between two separate programs, and one program with two parts? This is a legal question, which ultimately judges will decide. "[0]
[0]: https://www.gnu.org/licenses/gpl-faq.en.html#MereAggregation
That's the main difference from a human learning from copyrighted code (which is totally legal). If they have a good memory they might be able to reproduce copyrighted snippets, but they would usually (probably not always!) know they are doing that.
It blew my mind half of the times. It was like it knew what I was going to do.
The other times it was dumber than the standard auto complete. It doesn't have any awareness of already defined variables and doesn't use them to complete halfwritten variables. Hope this gets better soon.
Overall, I like my Jetbrains IDE but Replit is coming with appealing features for the side projects specially. Easy to use auth, db, analytics, deployment...
I wish either Replit levels up its IDE game or Jetbrains(or community) build plugins to match the state of art/joyful experience of programming.
ref: https://github.blog/2022-09-14-8-things-you-didnt-know-you-c...
- What areas/languages/tasks is it good at and what is it bad at?
- How often is it generating code with bugs?
- How often is the code that gets generated used as is vs immediately edited?
- What are the new frustrations that this causes that existing IDE code completion doesn't?
I work with trained models daily and I know that their failure cases are unintuitive, unexpected, and exasperating. I'd like to know as much about the failure cases of this model as possible before diving in.
- Ghostwriter is especially good at Python and JS, but supports up to 16 languages (to varying degrees of effectiveness) you can read more here: https://docs.replit.com/ghostwriter/faq#which-programming-la...
- As for tasks, it's great for reducing the amount of boilerplate code you need to write (Complete Code), writing React components (Complete + Generate Code), explaining code in plain English (Explain Code), translating code between languages (Transform Code), writing exhaustive tests (Complete Code)
- No stats on how often its generating code with bugs or how often the code gets generated used as is vs immediately edited, we're interested in getting both to help improve Ghostwriter
- Like any LLM, it can get stuck in a long-tail of repetitive loops; we're working pretty hard to improve and mitigate these issues, but, especially for new users, the repetition and hallucination type problems can be distracting.
- Developers who work at a company (e.g., as employee) and need to spit out features every sprint? Velocity is important, so I imagine these kind of developers need to squeeze every minute they are in front of the screen in order to produce working code?
- Developers who think of written code as one way to solve (tech) problems, so they don't really care much about the process of creating code, but mainly about the output (i.e., does the running program solves the issue at hand?)
- Senior developers who don't like to write boilerplate code?
I don't see myself as the target audience of Copilot or Ghostwriter. I do work as an employee, but I'm not a "feature machine". Usually the hardest part about my job is solving problems while communicating with other people. I don't need to write code "fast", and by the time I hit the keyboard to start coding, I don't really need that much help (granted, I'm not working on code that goes into space rockets... just normal e-commerce stuff)
I like to work on side projects and learn new technologies. When I was starting with programming, as part of the learning I liked to write boilerplate code (actually, that's how I learnt programming. I remember writing C boilerplate code by reading "The C Programming Language". Skipping the "boring" parts wouldn't have helped me in my learning).
If any, Copilot and similar tools take away all the joy of actually writing code (because, when I work on side projects, 50% of the satisfaction comes from actually writing code for the sake of writing code. The other 50% comes from the ability to solve a problem). So, yeah, maybe for the people like me who does find the act of writing code for the sake of writing code (you know like painting or taking photographs), Copilot seems like an unneeded tool?
I like to challenge my own beliefs, and I reluctantly tried it out. At least I would have a basis for my criticism, so I thought. I'm about 10 hours into using it, maybe.
If anything it has increased the joy of writing code for me. It eliminates the mundane busy work and lets me focus on solving problems. For me, the "real" coding happens in my head, putting it into an editor is just process. I still also have to check it's work whenever I use it, so I'm still deeply embedded in the coding process.
I believe it's akin to an easel vs Krita/Photoshop. Some people enjoy interacting with the physical medium, others enjoy the creative process.
I would strongly recommend trying it out in anger (i.e. a reasonably real codebase), at least for 30 minutes. Form a better opinion after that (which may well be the same as yours right now).
For reference: I've been coding since I was 8 (almost 30 years).
My primary UX issue with copilot is that it is trying too hard to be helpful, often suggesting code that I don't need. You also can't trust it with more complex cases but that's actually pretty reassuring : - )
So we have been considering using Codex or something for generating the code in a more streamlined version, the key reason of it being a benefit is we are a small team with each person owning more than one large repositories. It's gotten very annoying and our pace is far slower than what we would like, here something like this makes quite a lot of sense.
Though the problem with such specific tools is they can't generate any customized code for our codebase, we can finetune other codegen models and that's what we plan to do down the line, but this specific tool just not really useful if it can't specialized for our codebase.
For me, same as writing actually, the thinking of what I want to do is everything, and the doing it is nearly trivial. I don't picture having copilot write nontrivial code for me and then reviewing it would be different than writing it, even if I didn't know the exact syntax and had to look it up. So I agree, it feels like a solution to a problem I don't have, like is solves something that I don't spend time on.
Cynically, like GPT-3 probably help write content farm stuff, copilot probably helps write some junk code for something, but there are probably domain specific low code tools that do that better as well
See it as a better autocomplete for people who don't want to or can't learn by diligently doing the boring parts.
Plus, Ghostwriter is integrated with the Replit platform, meaning you get all the benefits Replit offers as a portable development environment you can take with you anywhere you go and host instantly.
I can't count the number of times I've copy pasted a chunk, commented it out, then put a comment describing what I'm after. Copilot will get it exactly right ~40% of the time, ~40% of the time it gives me a good starting place, and the other 20% I just scrap.
If you are doing something new, you need to let people see what it will do, before they will give you money.
At the bottom of the original blog post. He put his project back online and it is still up: https://riju.codes/
> When someone shows you who they are, believe them the first time.
- Maya Angelou
