Tracking issue for improving std::sync::{Mutex, RwLock, Condvar}

[m-ou-se]

A few weeks ago, on January 12th, the @rust-lang/libs team discussed a plan to improve std::sync::Mutex, std::sync::Condvar, and std::sync::RwLock.

On most platforms, these structures are currently wrappers around their pthread equivalent, such as pthread_mutex_t. These types are not movable, however, forcing us to wrap them in a Box, resulting in an allocation and indirection for our lock types. This also gets in the way of a const constructor for these types, which makes static locks more complicated than necessary.

@Amanieu presented his library, parking_lot, which implements the 'parking lot' structure from WebKit. Rather than letting the operating system handle any waiting queues, this manages the queues in user space in a global hash table indexed by the address of the lock. This allows for flexible 1-byte mutexes with almost no restrictions.

There has been an attempt at merging parking_lot into std, as a replacement for the implementation of std's locking primitives. However, many different blockers came up and that PR did not get merged. (See also my RustConf talk for some of this history.) Many of these blockers have been resolved since.

One of the problems with replacing std's lock implementations by parking_lot is that parking_lot allocates memory for its global hash table. A Rust program can define its own custom allocator, and such a custom allocator will likely use the standard library's locks, creating a cyclic dependency problem where you can't allocate memory without locking, but you can't lock without first allocating the hash table.

A possible solution is to use a static table with a fixed number of entries, which does not scale with the number of threads. This could work well in many situations, but can become problematic in highly parallel programs and systems. Another possible solution is to skip the global allocator and directly allocate this hash table with a native system API such as mmap on Unix. (See this thread for some discussion.)

In our meeting, we discussed what's natively available on various operating systems/platforms:

• On Windows, we have already switched to Windows SRW locks, which do not require allocation as they can be moved. They are small (32-bit), efficient, and const constructible. Just for Windows, there does not seem much advantage to switching to parking_lot, although there might be some (small) performance difference.

• On both Linux and some BSDs there's a native futex() syscall, which provides functionality similar (but more primitive) than a parking lot. Nearly equivalent functionality is available in Wasm. While not trivial, it's possible to implement our synchronization primitives directly based on such a futex. This makes all the primitives small (32-bit), efficient, and const constructible.

• On macOS and some other Unixes there doesn't seem to be anything that's similar futex syscall. (parking_lot uses pthread for its thread parking operations on those platforms.)

• Some smaller platforms that are (partially) supported by the standard library have their own non-trivial implementation of the standard locks, which are hard to maintain and have not gotten much validation.

After some discussion, the consensus was to providing the locks as 'thinnest possible wrapper' around the native lock APIs as long as they are still small, efficient, and const constructible. This means SRW locks on Windows, and futex-based locks on Linux, some BSDs, and Wasm. And for other platforms without suitable native APIs, a parking_lot-based implementation using one of the possible workarounds for the allocation problem.

This means that on platforms like Linux and Windows, the operating system will be responsible for managing the waiting queues of the locks, such that any kernel improvements and features like debugging facilities in this area are directly available for Rust programs.

It also means that porting the standard library to a new platform will be easier, and maintainance of the std implementation for the less common platforms will become easier. These platforms will now only have to implement a thread parker and can rely on a performant and correct implementation of the standard locks on top of that.

Update: We've decided to not directly use parking lot's one-byte (two bit) locks, but instead use the equivalent of its internal WordLock or Windows' SRW locks, which are one pointer in size and require no global state in the program. That solves the allocation problem.

Update 2: To maintain priority inheritance of the native locks (e.g. on macOS), we've kept using pthread locks on non-futex unix platforms, at least for now. Using #97647, we've made it possible for the locks to have const fn new.

---

Primary goals:

• Efficient non-allocating locks
  • Windows
  • Linux
  • macOS
    • We can't avoid pthread if we want to keep features of the OS' native locks, like priority iniheritance.
  • {Free, Open}BSD
• Turn std::sync::{Mutex, Condvar, RwLock}::new into const functions: Make {Mutex, Condvar, RwLock}::new() const. #97791
• Resolve some bugs around pthreads
  • Destroying locked Mutex in libstd triggers miri in safe code #85434
  • Binary crashes when statically linked with LTO turned on #94564 (comment)

Possible later goals:

• Add a Condvar::wait_rwlock to make Condvar usable with RwLocks?
• Allow Sending MutexGuards to other threads?

---

To do:

• Relax Condvar requirements to allow for unboxed mutexes. Relax promises about condition variable. #76932
• Use unboxed SRW locks on Windows.
  • Make Microsoft promise that SRW locks are indeed movable. Clarify destruction of SRW locks and condition variables. MicrosoftDocs/sdk-api#447
  • Small cleanups in Windows Mutex. #76645
  • Refactor sys_common::Mutex to have a separate MovableMutex type to allow unboxing on some platforms. Split sys_common::Mutex in StaticMutex and MovableMutex. #77147
  • Remove the Boxes in Mutex and Condvar on Windows and Wasm. Unbox mutexes and condvars on some platforms #77380
  • Remove the Box from RwLock on Windows and Wasm. Multiple improvements to RwLocks #84687
  • (Start using WaitOnAddress and NtWaitForKeyedEvent in the standard library.) Add fast WaitOnAddress-based thread parker for Windows. #77618
    • This unblocks usage of these APIs in parking_lot if we want to use that on Windows too. But if we just use SRW locks, this was not necessary to unblock the lock improvements.
• Use futex-based locks on Linux.
  • Start using the futex syscall to get some experience with it in the standard library. Use futex-based thread::park/unpark on Linux. #76919
  • Implement the futex() syscall in Miri to allow Miri to run standard library tests. Implement futex_wait and futex_wake. miri#1568
    • Also implement the bitset futex operations in Miri. Add support for FUTEX_{WAIT,WAKE}_BITSET miri#2054
  • Implement Mutex and Condvar using futex().
    • Design these.
      • Experimentation: https://github.com/m-ou-se/futex-lock-experiment/
      • Investigate other implementations (glibc, musl, Windows, etc.)
        • musl: Tracking issue for improving std::sync::{Mutex, RwLock, Condvar} #93740 (comment)
        • glibc
          • Mutexes: Tracking issue for improving std::sync::{Mutex, RwLock, Condvar} #93740 (comment)
          • Condition variables: Tracking issue for improving std::sync::{Mutex, RwLock, Condvar} #93740 (comment)
          • Reader-writer locks: Tracking issue for improving std::sync::{Mutex, RwLock, Condvar} #93740 (comment)
        • Boost: Tracking issue for improving std::sync::{Mutex, RwLock, Condvar} #93740 (comment)
        • Windows' SRW locks: Tracking issue for improving std::sync::{Mutex, RwLock, Condvar} #93740 (comment)
        • Wine's SRW locks: Tracking issue for improving std::sync::{Mutex, RwLock, Condvar} #93740 (comment)
        • Apple Darwin libpthread: Tracking issue for improving std::sync::{Mutex, RwLock, Condvar} #93740 (comment)
        • FreeBSD's libpthread: Tracking issue for improving std::sync::{Mutex, RwLock, Condvar} #93740 (comment)
      • Make some design decisions: Tracking issue for improving std::sync::{Mutex, RwLock, Condvar} #93740 (comment)
    • Implementation: Replace Linux Mutex and Condvar with futex based ones. #95035
  • Implement ReentrantMutex using futex(): Replace ReentrantMutex by a futex-based one on Linux. #95727
  • Implement RwLock using futex(): Replace RwLock by a futex based one on Linux #95801
• Use the same ReentrantMutex on all platforms: Use a single ReentrantMutex implementation on all platforms. #96042
• Use futex-based locks on *BSD.
  • Add NetBSD's FUTEX_* constants to the libc crate. Add NetBSD's FUTEX_* constants. libc#2762
  • Add OpenBSD's futex() to the libc crate. Add OpenBSD's futex.h. libc#2761
  • Add FreeBSD umtx functions and constants to the libc crate. Add umtx_op to FreeBSD. libc#2770
  • Add DragonFlyBSD's umtx_* functions to the libc crate. Add DragonFly umtx_{sleep, wakeup}. libc#2763
  • Switch *BSD over to the same futex locks as on Linux. Use futex-based locks and thread parker on {Free, Open, DragonFly}BSD. #96510
• Use futex based locks on all other platforms that support a futex-like API.
  • Use futex based locks on Emscripten. Use futex locks on emscripten. #96205
  • Use atomic.wait/atomic.notify based locks on Wasm. Use sys::unix::locks::futex* on wasm+atomics. #96206
  • Use zx_futex_wait based locks on Fuchsia.
    • (Tier 2 platform.)
  • Use syscall::call::futex based locks on Redox.
    • (Tier 3 platform.)
  • Use futexes on Hermit. Use futex-based locks and thread parker on Hermit #101475
• Lazily allocate on platforms where we still use pthread: Lazily allocate and initialize pthread locks. #97647
  • macOS
  • NetBSD, other unix
• Make the new functions const. Make {Mutex, Condvar, RwLock}::new() const. #97791
• [future / nice to have] Use 'WordLock' (aka SRW lock, aka 'user space linked-list queue lock') on other platforms.
  • Remove locks from Instant::now(), to avoid a cyclic dependency.
  • Stop using std::sync::{Mutex, Condvar} in std::sys's thread parker, to avoid a cyclic dependency. std: directly use pthread in UNIX parker implementation #96393
  • Implement this type of locks, and replace the lock implementations of non-futex platforms by it. (Replace pthread RwLock with custom implementation #110211)
• Mark this issue as fixed: Destroying locked Mutex in libstd triggers miri in safe code #85434
• Write some blog post about all this. (Tracking issue for improving std::sync::{Mutex, RwLock, Condvar} #93740 (comment))
• Celebrate. 🎉

[hkratz]

Did you investigate using os_unfair_lock (Apple Developer Docs) on macOS? It can't be moved once it is in use either but it is essentially just a struct with a u32 in it, initialized to 0 before use. But maybe I am missing some requirements here. If it seems possible, investigation of it could be added as a subtask.

It is available since macOS 10.12 on x86-64, but could be used unconditionally on aarch64.

Nevermind it does not have timeouts for os_unfair_lock_lock/os_unfair_lock_trylock, so it is obviously out.

[nbdd0121]

I don't think we need to use hash tables like parking lot for targets without futex support. We could just store the queues inside Mutex. I think the only downside is size, but that'll still be better than the current boxed pthread mutex though.

[Amanieu]

Nevermind it does not have timeouts for os_unfair_lock_lock/os_unfair_lock_trylock, so it is obviously out.

Also it doesn't support Condvar, which is required for Rust locks.

[thomcc]

Oh. This is a subject extremely near-and-dear to my heart -- I even started on a pre-RFC for atomic wait/wake ops, like the ones that C++ got recently. That said, much of my motivation was hoping it would find a way to unstick us here, and I this is clearly a higher priority.

For futex APIs, I wrote a blog post on them at one point. https://shift.click/blog/futex-like-apis I wrote about some of the options here for system APIs, which might be helpful as reference.

That said, I got asked about Darwin's __ulock_* API in a Zulip DM, since I wrote that post as well as hte ulock-sys crate.

---

For Darwin, it I think it's quite unlikely we could use __ulock_{wait,wake} for a futex API directly without risking... many problems. The fact that it may break in the future is actually one of the less concerning issues, in comparison to the fact that that use of private APIs can cause rejection from app stores, which would be a problem for Rust code that on the iOS/Mac App Stores (for clarity: I'm just going on a hunch¹ that could happen for __ulock_*, I do not know for certain, but I think libstd needs to favor caution here).

That said, there's... a pretty cursed option for a workaround on these targets. Too cursed to use? Not sure... possibly.

That is: libc++'s implementation of std::atomic<u32>::wait, notify_one, and notify_all will ultimately bottom out to calls to __ulock_wait/__ulock_wake, which... I guess they get to use because libc++ gets shipped and updated with the OS on Darwin, fair enough. Anyway, in theory we could just call that somehow. And there's a sense in which this would be pretty nice, since it's a system-provided function that implements (something similar to) the API we want... But it's certainly not ideal for a few reasons:

1. It's unclear to me how much we promise about what we link to on various platforms, but it's possible we cannot do this, and likely that we do not want to... I don't know if we make promises about what we need to link against for libstd, but suddenly dynamically linking against the C++ standard library on some platforms... feels like it would be a change.

2. Not to mention -- even if we are allowed to do this, it's still a pain in the butt: The version of libc++ that provides this this is much newer than our platform guarantees, so we'd need to weak-import² the dubiously-public³ symbol that this stuff under the hood (although I suppose this is better than requiring a C++ compiler to build Rust code for these platforms).

So, maybe? Given that the OS provides libc++, it seems like it should technically be fine (unless we specifically say we won't). At least, so long as we manage to do it in a way that doesn't versions older than we care about. But maybe I've missed something (or I haven't, but it's still several bridges too far).

(If not, oh well, someday an API shaped mostly like that is bound to end up as part of the libc someday, after it's copied to C from C++ in C9Z 😛)

(EDIT: Thinking about it more, it's fairly likely we should have some justification for this approach, since looking at the libc++ code, enough happens between what our all would be that it could defeat the point. And the effort spent here to make it work would be better spent on improving our fallback)

---

All that said, there are a few options on various platforms that could probably help implement the fallback thread parker, in case there are regressions (which plausibly there wouldn't be if we use parking_lot_core as-is, but maybe there would be if we need to abandon the growth behavior). In particular:

• A few of the futexless unices (NetBSD, the Solaris-likes, perhaps others?) have a lwp_park and lwp_unpark functions, which is basically just an OS primitive for thread parking and unparking.

• On Darwin (yes, yes, more Darwin), I think we could probably use libdispatch's dispatch_semaphore instead, as it's been the target of a lot of optimization -- probably more than the currently-active verions of pthread_mutex_t / pthread_cond_t -- Not to mention, you'd expect that a one-piece semaphore can be implemented more efficiently than a condvar/mutex pair, at least... in theory⁴. At least, unless the unified semaphore doesn't have to uphold extra guarantees (as it does for POSIX's sem_t, I believe).

• Hermit (and perhaps only hermit but perhaps others) only has a semaphore, and we implement other primitives on top of them. I don't have a vested interest in hermit, but I would really like any excuse to remove https://github.com/rust-lang/rust/blob/master/library/std/src/sys/hermit/condvar.rs which look like it's implementing the same algorithm it cites, and seems like it could be buggy⁵.

That said, a lot of this probably should be in the a tail of followups, and only if it's worthwhile.

Footnotes

1. I mean, if nothing else they start with __ and don't appear in a single public header (they only appear in the headers inside apple source repos). So, even though these APIs don't seem to be in the blacklists that prevents you from even attempting app store submission (which is not a guarantee), it seems completely out of the question to use in libstd (even if I do think they're cool, and I do). ↩

2. Calls to dlopen/dlsym tend to be disallowed in the same situations, since tooling can't tell if you're using it to use private APIs. (We already do this all over libstd though, so maybe it's not the issue it once was) ↩

3. Which isn't API-compatible, but presumably has to stay around for ABI reasons. I mean, it must, right? C++ standard libraries are famously resilient to ABI issues, after all 😅... ↩

4. That is, at worst the single-piece semaphore could just be implemented in terms of a private condvar and mutex pair, resulting in the same performance. That said, relying too heavily on this style of logic is a good way to get unpleasant surprises, so who knows. ↩

5. For example, the load-then-sub in notify_one is seems like it has a race that could lead to a deadlock, but there may be a reason this is impossible -- I've never spent that long thinking about it, just seems dubious, and like the kinda code that'd be real nice to replace with something that (if nothing else) will get more attention. ↩

[bjorn3]

On both Linux and some BSDs there's a native futex() syscall, which provides functionality similar (but more primitive) than a parking lot. Nearly equivalent functionality is available in Wasm. While not trivial, it's possible to implement our synchronization primitives directly based on such a futex. This makes all the primitives small (32-bit), efficient, and const constructible.

Please make sure you implement a fair or eventually fair (this is what parking lot does and needs the hashmap for) lock to prevent starvation.

[nbdd0121]

I don't think pthread mutexes are fair so the new implementation shouldn't be required to be fair either.

[thomcc]

SRWLOCK is also unfair too, pthread_mutex_t depends (POSIX doesn't require it, apple has it, I believe glibc doesn't)... So this would mean we need to a parking lot everywhere, even if the OS has otherwise good locks (like with Window's SRWLOCK).

More generally: There are a bunch of locking features that would be nice to have in an ideal world, but I think they start becoming contradictory, or only achievable easily on one platform or another -- for example, I believe the parking-lot style of lock won't respect OS priorities (unless there's a clever trick I don't know about or something). Trying to solve all of this probably has no solution, and I'm not sure that we're in the right position decide which things we want to promise yet¹... assuming we even want to make these sorts of promises (and I'm not sure we do).

Footnotes

1. I feel like the place for that is afterwards we ship it, if/when bugs come in from users and such. Or at least when we're further in the implementation. ↩

[hkratz]

Apple switched the default from PTHREAD_MUTEX_POLICY_FAIRSHARE_NP to the newly introduced PTHREAD_MUTEX_POLICY_FIRSTFIT_NP in macOS 10.14 (iOS and tvOS 12.0, watchOS 5.0), so they are not fair for recent macOS either (by default).

[nagisa]

Wouldn't implementing lock algorithms on top of futex et al. effectively mean that both the parking_lot based implementations and these other algorithms receive less test/use coverage overall? Or is there some plan to share the gnarly implementation details between parking_lot based implementations and the futex/etc. ones?