I'm trying to figure out whether copilot in its current form is a tool that will be useful to me in my job. (I'd be able to do this evaluation properly if they'd just let me on the damned beta.)

Nearly right isn't good enough for this afaics. In fact, I expect there to be a slightly paradoxical effect where nearly-right is worse than obviously-wrong. An analysis of a piece of code like I did above is time consuming and cognitively taxing. An obviously wrong solution I can just reject immediately. An almost-right (or at least vaguely plausible) one like these takes thought to reject. Much more thought, in this case (for me, at least) than just writing the thing myself in the first place.

Edit: BTW, I don't get what you're saying with

"The first example is almost correct, conditioned off a sentence description. The second example is the right idea, it just bit off more than it could chew when slicing it all together."

The first one is completely (if subtly) wrong. It's supposed to swap two bits but it sets them to the same value. There's no interpretation of the description in which that's correct.

The second one is definitely not "the right idea". It tries to do it with string manipulations, which (regardless of the fact that it does so incorrectly) is completely the wrong approach. This one is actually "better" than the other in the paradoxical sense I mentioned above, because I could reject it the moment I saw it convert the number to a string.

That's what is happening here. There is no intelligence, just regurgitation. Randomization and maximum likelihood completion.

Just like with the competitive programming example, we're asking it to produce solutions that it has seen in its training set. If you ask for a nontrivial twist on one of those solutions, it fails.