I'll add one other data race goof: atomic.Value. Look at the implementation. Unlike pretty much every other language I've seen, atomic.Value isn't really atomic, since the concrete type can't ever change after being set. This stems from fact that interfaces are two words rather than one, and they can't be (hardware) atomically set. To fix it, Go just documents "hey, don't do that", and then panics if you do.
Generics might save us from the simple, mechanical flaws. Expect to see `Locker<T>` and `Atomic<T>` types cropping up. And unbounded buffered thread-safe queues backing channels. Etc. I'm very, very much looking forward to it.
--- edited to rant more ---
I also really wonder where all these "go makes concurrency a first-class concept" claims come from, because I see it quite a few places, and I feel like it's making some very strong implied claims that absolutely do not exist.
Go has channels and select. That's neat. But on the other hand it has threads... but no thread handles. It has implicit capturing of closures. It has ambiguous value vs pointer semantics. It (style- and ergonomic-wise) encourages field references, which have no way to enforce mutexes or atomics. It has had crippled lock APIs that effectively force use of channels for... I don't know, philosophical reasons?
Go is abnormally dangerous when it comes to concurrency IMO. The race detector does an amazing job helping you discover it, but it's very easy to not use it or not take full advantage of it (i.e. non-parallel tests), and few run their production services with the race detector enabled. Because if they did, it would crash all the time, because there are an absurd amount of races in nearly all of the popular libraries (and in common use of those libraries, because concurrency is not a first-class citizen and you can't tell when it's happening / when it shouldn't happen).
Given that some of the main architects behind Go had K&R C as background I wouldn't be surprised if "first-class" just meant that the language defines both a memory model and primitives for threading. C had neither until it basically adopted both from C++11.
"go makes concurrency a first-class concept" I think it usually refers to goroutines being built in the language.
"Go is abnormally dangerous when it comes to concurrency IMO". Personnally, it has not been my experience with Go concurrency. However I have hit some issues when trying to ocrhestrate tasks via channels and ended up resorting to atomics to do the job.
This is fud, I ran the race detector with a lot of popular lib and I never found issues like that.
But since you're claiming there are issues everywhere, do you have examples?
> The sync/atomic package defines new atomic types Bool, Int32, Int64, Uint32, Uint64, Uintptr, and Pointer. These types hide the underlying values so that all accesses are forced to use the atomic APIs. Pointer also avoids the need to convert to unsafe.Pointer at call sites. Int64 and Uint64 are automatically aligned to 64-bit boundaries in structs and allocated data, even on 32-bit systems.
Go 1.19 is expected to release in August.
How does this mean it's non-atomic? As far as I know you can still never Load() a partial Store(). (Also, even if it was possible, this would never be a good idea...)
In fact, it's even worse than that. If the Store() caller goes to sleep between setting the type and storing the pointer, it causes every Goroutine that calls Load() to block. They can't make forward progress if the store caller hangs.
* Just don’t be an idiot. Worse is better.
1. Closures and concurrency really don't mix well. The loop variable capture in particular is very pernicious. There's an open issue to change this behavior in the language: https://github.com/golang/go/issues/20733.
2. Yep. I've seen this problem in our codebase. I've grown to just be very deliberate with data that needs to be shared. Put it all in a struct that's passed around by its pointer.
3. This issue is caught fairly easily by the race detector. Using a sync.Map or a lock around a map is pretty easy to communicate with other Go devs.
4. This should be documented better, but the convention around structs that should not be passed around by value is to embed a noCopy field inside. https://github.com/golang/go/issues/8005#issuecomment-190753... This will get caught by go vet, since it'll treat it like a Locker.
5 & 6. Go makes it pretty easy to do ad-hoc concurrency as you see fit. This makes it possible for people to just create channels, waitgroups, and goroutines willy-nilly. It's really important to design upfront how you're gonna do an operation concurrently, especially because there aren't many guardrails. I'd suggest that many newcomers stick with x/sync.ErrGroup (which forces you to use its Go method, and can now set a cap on the # of goroutines), and use a *sync.Mutex inside a struct in 99% of cases.
7. Didn't encounter this that often, but sharing a bunch of state between (sub)tests should already be a red flag. Either there's something global that you initialized at the very beginning (like opening a connection), or that state should be scoped and passed down to that individual test, so it can't really infect everything around it.
I run my Go development server with the -race flag as a default. If it affects performance I'll turn it off but that's very rare in practice. Unfortunately a lot of applications don't run tests against their HTTP endpoints (like only internal library stuff) which is bad bad bad, but the -race flag at least helps mitigate.
To anyone reading who cares:
1) Always run your tests with the -race flag!
2) Always write tests for your HTTP handling code too!
3) Run your dev server with -race for a week and see what happens.
This will hard crash your Go program and there is nothing you can do about it. You can't recover(). Go vet will not catch anything. The -race flag will!
package main
import "time"
func main() {
m := map[int]int{}
go poop(m)
go poop(m)
time.Sleep(5 * time.Second)
}
func poop(m map[int]int) {
for i := 0; i < 1e10; i++ {
m[i] = i
}
}which is… uhh ok maybe?
But go is really trying to keep backwards compatibility so they won’t just change this
I'm surprised by some of them. For example, go vet nominally catches misuses of mutexes, so it's surprising that even a few of those slipped through. I wonder if those situations are a bit more complicated than the example.
Obviously, the ideal outcome is that static analysis can help eliminate as many issues as possible, by restricting the language, discouraging bad patterns, or giving the programmer more tools to detect bugs. gVisor, for example, has a really interesting tool called checklocks:
https://github.com/google/gvisor/tree/master/tools/checklock...
While it definitely has some caveats, ideas like these should help Go programs achieve a greater degree of robustness. Obviously, this class of error would be effectively prevented by borrow checking, but I suppose if you want programming language tradeoffs more tilted towards robustness, Rust already has a lot of that covered.
Does anyone not want robustness of their language to cover their mistakes?
At cost? … depends on what you’re charging me, and how much I’m getting
So now I'm hearing that Go, a garbage collected language, doesn't guarantee data race freedom? I guess it's garbage collected but not "managed" by a runtime or something?
Why go to all that effort to get off of C++ just to stop 30% short? These are C-like concurrency bugs, and you still have to use C-like "if not nil" error handling.
Why do people keep adopting this language? Where's the appeal?
> two or more threads in a single process access the same memory location concurrently, and at least one of the accesses is for writing.
Rust provides compile time protection against data races with the borrow checker. Go provides good but imperfect runtime detection of data races with the race detector. Like most things in engineering, either approach requires a trade off involving language complexity, safety, compile time speed, runtime speed, and tooling.
As this post demonstrates, data races are trivial and common in Go.
Because many of the core types (interface, slice, map) are non-thread-safe multiword structures, they also break memory safety: https://blog.stalkr.net/2015/04/golang-data-races-to-break-m...
If it is chock-full of race conditions, it is not trivial.
Java programs aren't guaranteed to be free of data race. Java spec guarantees that if that happens, there will be no undefined behavior (like in C++).
Now that I think about it, this must be the case, right? You have to get `synchronized` right in Java or else you won't get what you expect.
and no out-of-thin-air values.
There're many aspects to consider when evaluating a tool. To me, Go has one of the best overall packages:
- std lib - tooling - performance - concurrency - relatively easy to get devs - reliable - mature
Also, Go has no substantial drawback. I personally consider an external runtime a drawback, for example.
I also use Rust personally. This discussion shows the value of Rust in terms of correctness. But for my professional projects Rust lacks the ecosystem guarantees that Go has with its great and useful standard lib. Looking at the Cargo dependencies of a mid size Rust web service is scary and reminds me of NPM. A large fraction of essential libs are maintained by a single person. Or unfortunately unmaintained. Rust with Go's std lib would be truly great.
Garbage collectors solve double-free bugs and usually memory leaks due to cyclic references.
(As an example, image how much simpler Rust would be, if they went with garbage collection. Or how much more machinery Haskell would need, if they went with Rust's memory management strategies.)
That said, people ragging on rust pushing that trope are basically just making stuff up to hate on it. Anyone who looks into the language and views programming languages as tools and understands these issues gets why someone might use rust.
But yea, it's ironic... Especially seeing how many times I've seen smart colleagues get go concurrency wrong.
A side topic: this is really not something to be proud of. There used to be more people than quantity of work in Uber and engineers fought for credits by building bogus decomposed services, and the sheer number of services seems indicate it's still so.
I’d also be curious if the 50m lines of code included generated code.
Taking an existing service and making it into 2 new microservices is a "thing you did". Suddenly, you have "impact" and can claim the new service as "yours". Everyone wants to be a king of their little kingdom.
We’ve started to use it (temporal) a bit for general automations, and it’s pretty great. Monorepo with a lot of different activities (“microservices”) makes sense.
The activites are orchestrated in workflows (much like DDD “sagas”) and scheduled via temporal. This gives awesome introspection and observability.
"The key point here is our programmers... They’re not capable of understanding a brilliant language... So, the language that we give them has to be easy for them to understand"
I don't think go lang will die out because it does get some things right. Unfortunately, there's still a bit of things going wrong.
https://go.dev/doc/articles/race_detector
Edit: at the _end_ of the post, they mention that this is the second of two blog posts talking about this, and in the first post they explain that they caught these by deploying the default race detector and why they haven't been running it as part of CI (tl;dr it's slower and more resource-expensive and they had a large backlog).
https://eng.uber.com/dynamic-data-race-detection-in-go-code/
Everything in Go is passed by value, including slices.
Go has no reference types, but a lot of people think it does, and that’s a problem. An example of the much bigger problem of low barrier to entry programming, where lots of folks write code but have no deep understanding of the tools they use.
If there’s something that Go proves, it’s that one can’t make a language “idiot proof”. Rust has the same problem in terms of folks creating mess after mess with it, except it’s higher barrier to entry and gets a better caliber of people using it.
The trick is, what programs do we even want to exist? There's no need to be able to write all the programs you didn't want. In Rust, such programs get consigned to unsafe, which means that yes, sometimes to do general purpose programming (and especially e.g. in Rust's own stdlib) you must use unsafe Rust. But it already means we can constrain "idiots" (or more reasonably, new programmers) to safe Rust and rule out all those problems that aren't in the reduced domain of safe Rust.
You can go much further than Rust. WUFFS isn't a general purpose language at all. While a Rust compiler written entirely in Rust isn't a priority it'll likely happen sooner or later, but a WUFFS compiler written in WUFFS is nonsense, WUFFS doesn't even have strings. WUFFS is for, well, Wrangling Untrusted File Formats Safely, hence the name. Notice not files just the file format. WUFFS has no idea what a file is, no file APIs, since it doesn't know what strings are it couldn't easily name files anyway. But inside its domain WUFFS gets to be 100% safe while also being faster than code you'd actually write in other languages.
Take buffer overflow buffer[n]. In a language like C++ direct access isn't bounds checked and so overflows are common when n is too large, too dangerous. OK, in a language like (safe) Rust this access is bounds checked, now the overflow is prevented when n is too large but the bounds check cost CPU cycles, a little slower.
WUFFS doesn't do either, in WUFFS that variable n was used to index into buffer therefore n is constrained to be 0 <= n < buffer size. If the compiler can see any way that n might exceed this constraint your program does not compile. As a result at runtime there's no overflow and no bounds checking.
A complete idiot's WUFFS GIF decoder might be wrong - it could report spurious decoding errors, it could decode a blue dog as a pink roller skate, render images upside down or even decode JPEG instead of GIF - whatever, but it can't escape the limits of WUFFS itself. It can't go off piste and send your password database to a remote HTTP server or delete all your logs, or send spam emails or run some machine code it found inside the supposed GIF file.
Here's a simple question that's stumped me for some time: if multiple go routines are popping values out of a channel, does the channel need a mutex? Why do the "fan-out, fan-in?" examples in the "Pipelines and Cancellation" post on the Go blog not require mutex locks? Link here: https://go.dev/blog/pipelines
Stuff like that, along with the ambiguity of intializing stuff by value vs using make, the memory semantics of some of the primitives (slices, channels, etc). None of it was like "of course". If something is a reference, I'd rather the language tell me it's a reference. Maybe I'm still too new to the language.
Go doesn't do anything to help you with memory safety around concurrency. And the design of the language is also not helping you avoid logical bugs.
After using Rust, all other imperative languages feel like using an angle grinder with a wood saw blade and no guard. Sure you can do really good if you are careful. But thing will go sideways remarkably quickly. And with the constant urgency of shipping for yesterday. It makes sense most programs look like the aftermath of The Boys show.
What the hell is that company doing?
Try to imagine an ERD or DFD of their day-to-day operations. 2,100 unique services…
This isn't open source, correct?
Uber has adopted Go (Golang for long)
(†famous last words, I know)
It's not just data races--it's also logical races, which are near-impossible to detect or prevent without something like transactional memory.
Given that Go eschewed generics for the longest time, I can sort of see why they left out immutability markers:
To keep your sanity, you'd want some functions to take (and return!) both mutable and immutable data, as the situation requires. But some other functions should only take mutable data or only immutable data.
Thus ideally you'd need some kind of 'generic' (im-)mutability handling in your type system.
(Rust's borrow checker is basically one way to really deal with this (im-)mutability genericity.
For example, a function to do binary search on a sorted array doesn't change the array; thus it could take either a mutable or immutable version. But if the array changes (via another thread) while the function is running, then you might get into trouble.)
And if you feel that Erlang's lack of type safety is an issue, then Gleam has you covered.
Not even immutability, isolation.
Though obviously immutability makes things less weird, the real gain in terms of concurrency is that you can’t touch any data other than your own (process’s), and for the most part erlang doesn’t “cheat” either: aside from binaries, terms are actually copied over when sent, each process having its own heap.
Sequential erlang could be a procedural language based around mutability and it wouldn’t much alter its reliability guarantees (assuming binaries remain immutable, or become COW).
But even with Erlang, concurrency is hard. Any single process's data is immutable, but if you split a process in twain, the resulting union can behave as if it had mutable state.
And let's not forget about ETS (term storage), which is basically a mutable hash table that you often have to use to get anything done.
In any case, I agree that Go did _not_ improve on Erlang.
By system do you mean a process or a tool that detects these?
Edit: oh I see it highlights red and underlines every keyword. I find that incredibly distracting, so much so I assumed their highlighter was broken, but also just realized they are screenshots.
How is that possible and what do they do???
for i := 0; i < 10; i++ {
i := i
...
}
The example is brilliant. Thanks.
https://medium.com/@scott_white/concurrency-in-go-is-not-mag...
tl;dr Go doesn't magically solve data races and blaming the language itself isn't well supported by the examples/data.
The "Slices" example is just nasty! Like, this is just damning for Go's promise of "_relatively_ easy and carefree concurrency".
Think about it for a second or two,
>> The reference to the slice was resized in the middle of an append operation from another async routine.
What exactly happens in these cases? How can I trust myself, as a fallible human being, to reason about such cases when I'm trying to efficiently roll up a list of results. :-/
Compared to every other remotely mainstream language, perhaps even C++, these are extremely subtle and sharp.. nigh, razor sharp edges. Yuck.
One big takeaway is this harsh realization: Golang guarantees are scant more than what is offered by pure, relatively naïve and unadulterated BASH shell programming. I still will use it, but with newfound fear.
As a multi-hundred-kloc-authoring-gopher: I love Go, and this article is killing me inside. Go appears extremely sloppy at the edges of the envelope and language boundaries, moreso than even I had ever realized prior to now.
Full-disclosure: I am disgusted by the company that is Uber, but I'm grateful to the talented folks who've cast a light on this cesspool region of Golang. Thank you!
p.s. inane aside: I never would've guessed that in 2022, Java would start looking more and more appealing in new ways. Until now I've been more or less "all-in" on Go for years.
I don't quite understand the hatred (to the point of shouting "using Java? Over my dead body), especially in startups, towards Java. I mean, it's a language, big deal. Java's ecosystem more than enough offsets whatever inefficiencies in the language itself, at least for building many of the internal CRUD services. Besides, people like Martin Thompson shows us how to build low-latency applications with ease too. Libraries like JCTools beat the shit out of many new languages when it comes to concurrency for productivity, performance, and reliability. How many engineers in startups claim that they hate Elasticsearch because "Java sucks"? Yet how many can really build a platform as versatile as ES or a Lucene replacement with economical advantages? How many people in startups openly despise Spark or Flink and set out to build a replace because "Java is slow and ugly". Yeah, I've seen a few. And a payment company insists that Rust is the best language because "GC is inefficient and ugly", even though they are still in the phase of product iteration and all their services simply wrap around payment gateways? What's the point?
Disclaimer: I use Go in work. It's not like I have skin in the game for speaking about Java.
That's a bit of a stretch. Surely, you can build low-latency apps, but I'd be very careful with the "with ease" bit. Low-latency Java often means zero heap allocations, aggressive object avoidance / reuse, heavy use of primitive types everywhere, so it is very much low-level like C, only with no tools that even plain old C offers, e.g. no true stack-allocated structs and no pointers. And forget about all the high-level zero-cost abstractions that C++ and Rust offer.
The other thing is that I tend to write little programs where simple deployment on a low-resource machine is desirable.
Go can handle that. Java kind of does the job with Graal now.
The JVM is incredible, though, and I love Clojure. I’m hoping that Loom + Graal helps to kickstart more competition in the “concurrent, parallel, simple to deploy” space.
* Died to me; obviously they’re both alive and well in the broad world.
Java's got problems. The biggest one is the framework laden ecosystem and that some of the frameworks are all or nothing. But the language and runtime are rock solid. I don't get the hate.
That said... when programming in programming languages without a slice type, I always want to have one. And though it's confusing at times, the design does actually make sense; without a doubt, it's hard to think of how you would improve on the actual underlying design.
I really wish that Go's container types were persistent immutable or some-such. It wouldn't solve everything, but it feels to me like if they could've managed to do that, it would've been a lot easier to reason about.
Go slices are absolutely the worst type in Go, because out of laziness they serve as both slices and vectors rather than have a separate, independent, and opaque vector types.
This schizophrenia is the source of most if not all their traps and issues.
> I really wish that Go's container types were persistent immutable or some-such.
That would go against everything Go holds dear, since it's allergic to immutability and provides no support whatsoever for it (aside from simple information hiding).
I'm biased as a Rust fan in general, but I think Rust pretty much nails this. Rust distinguishes between a borrowing view of variable length (a slice, spelled &[T]) and an owned allocation of variable length (usually a Vec<T>). Go uses the same type for both, which makes the language smaller, but it leads to confusion about who's pointing to what when a slice is resized.
This makes the stated reason for the delay of generics hard to understand. They didn't wait to get list/vector/array/slice right.
I think ranges are part of D's design they got right, and I think a similar abstraction would be in line with golang's general design ethos, GC design, etc, other than perhaps some folks might pattern match it as "this is like STL therefor bad burn it with fire etc" without actually thinking about it in detail.
> What exactly happens in these cases?
Go's append looks like this:
mySlice = append(mySlice, newItem)
To me, this makes it very clear that 1) mySlice pointer can now point to someplace entirely different in memory, and 2) there maybe new allocation.
I write both Java and Go. For personal projects, I always choose go.
For me: minimize shared mutable data. If I really can’t get rid of some shared mutable data, I mutex it or use atomics or similar. This works very well—I almost never run into data races this way, but it is a discipline rather than a technical control, so you might have to deal with coworkers who lack this particular discipline.
Reminds me of programming in Javascript (it's extreme example, but the similarity is there).
Do you mean you always use `a[x:y:y]` in order to ensure there is no extra capacity and any append will have to copy the slice?
Is append guaranteed to create a new slice (and copy over the data) if the parameter is at capacity? Because if it could realloc internally then I don't think this trick is safe.
I've got to say I'm not entirely clear on what they talk about specifically.
Is it simply that the `results` inside the goroutine will be desync'd from `myResults` (and so the call to myAppend will interact oddly with additional manipulations of results), or is it that the copy can be made mid-update, and `result` itself could be incoherent?
This is problematic because even though you copy it, you're still pointing at the same backing array.
Therefore, a backing array with data like [1,2,3,4,5] could be pointed at by 2 slice headers (slice metadata) looking like
A: {len: 2, cap: 10} [1,2] B: {len:5, cap: 10} [1,2,3,4,5]
So any append operations on slice A will mess up the data in that backing array.
Now, sometimes your append will resize the slice, in which case the data is copied and a slice with a new larger backing array is returned. If this was happening concurrently then you'd lose the data in racing appends.
If the append doesn't need to resize the slice, then you'll overwrite the data in the backing array. And so you'll corrupt the data in the slice.
Here's an example I threw together: https://go.dev/play/p/qRUKUwIf3vx
Although the code in the post doesn't actually look like it has an issue. Their tooling just flagged it up as it potentially has an issue if the copy was actually used in the function. But the `safeAppend` function targets the correct slice each time.
Below is what might be what they have meant. This code snippet is racy because an unsafe read of myResults is done to pass it to the goroutine and then that version of myResults is passed to safeAppend:
func ProcessAll(uuids []string) {
var myResults []string
var mutex sync.Mutex
safeAppend := func(results []string, res string) {
mutex.Lock()
myResults = append(myResults, res)
mutex.Unlock()
}
for _, uuid := range uuids {
go func(id string, results []string) {
res := Foo(id)
safeAppend(myResults, id)
}(uuid, myResults) # <<< unsafe read of myResults
}
}
They talk about the "meta fields" of a slice. Is the problem that these "meta fields" (e.g. slice length and capacity) are passed by value, and that by copying them, they can get out of sync between coroutines?