Linux Multithreaded Server Programming Basics: C++11 Concurrent Programming

Introduction

Previous versions of C++ didn’t support threads natively. Instead, you had to use libraries like pthread. Using C++’s built-in thread support makes programs more unified and concise.

Header Files

  • <thread>: This header file contains the std::thread and std::this_thread classes, along with functions for managing threads. It’s the main file for implementing threads.
  • <atomic>: This header file contains std::atomic and std::atomic_flag classes, which are the main files for implementing atomic operations.
  • <mutex>: Contains mutex-related classes and functions.
  • <future>: Contains the future class and related functions.
  • <condition_variable>: Contains condition variable classes.

These are the thread-related parts of C++11. Although there are many debates about pthread versus C++ thread, the cross-platform C++ thread seems to be more standard.

Recommended book for learning C++11 threads: https://www.gitbook.com/book/chenxiaowei/cpp_concurrency_in_action/details

Hello World Thread

A simple introduction to the thread class:

#include <thread>

using namespace std;
// This function is the thread task we want to run
void hello()
{
    printf("%s", "hello\n");
}
// Using the thread class, we pass a function as our initial task
// We can also pass other parameters like classes
int main()
{
    thread t(hello);
    // Using join to wait for completion
    t.join();
}

Besides join to wait for completion, you can use detach to not wait for the thread to finish.

struct func
{
  int& i;
  func(int& i_) : i(i_) {}
  void operator() ()
  {
    for (unsigned j=0 ; j<1000000 ; ++j)
    {
      do_something(i);           // 1. Potential access hazard: dangling reference
    }
  }
};

void oops()
{
  int some_local_state=0;
  func my_func(some_local_state);
  std::thread my_thread(my_func);
  my_thread.detach();          // 2. Don't wait for the thread to finish
}                              // 3. The new thread might still be running

In this example, we’ve decided not to wait for the thread to finish (using detach() ②), so when the oops() function completes ③, the new thread might still be running. If the thread is still running, it will call the do_something(i) function ①, which accesses a variable that has already been destroyed. As with a single-threaded program—allowing pointers or references to local variables to persist after the function completes has never been a good idea—this situation isn’t obvious and makes multithreading more error-prone.

How to Wait for a Thread to Complete?

If you need to wait for a thread, the corresponding std::thread instance needs to use join(). In example 2.1, replacing my_thread.detach() with my_thread.join() ensures that local variables are only destroyed after the thread completes. In this case, since the original thread doesn’t do much during its lifetime, running the function in a separate thread offers minimal benefit. However, in real programming, either the original thread has its own work to do, or it starts multiple child threads to do useful work and waits for these threads to complete.

join() is a simple, direct way to wait for a thread to complete or not wait at all. When you need more flexible control over waiting threads, such as checking if a thread has finished or only waiting for a period of time (determining a timeout if exceeded), you need to use other mechanisms like condition variables and futures. Calling join() also cleans up the thread-related storage parts, so the std::thread object is no longer associated with the completed thread. This means you can only use join() once on a thread; once you’ve used join(), the std::thread object can’t be joined again. When joinable() is used on it, it will return false.

Passing Parameters to Threads

void f(int i, std::string const& s);
std::thread t(f, 3, "hello");

The code creates a thread that calls f(3, “hello”). Note that function f needs a std::string object as its second parameter, but here a string literal is used, which is of type char const *. The conversion from the literal to a std::string object is completed in the thread’s context.

It’s important to note that the constructor ignores the expected parameter types of the function and blindly copies the provided variables.

Example 1:

void f(int i,std::string const& s);
void oops(int some_param)
{
  char buffer[1024]; // 1
  sprintf(buffer, "%i",some_param);
  std::thread t(f,3,buffer); // 2
  t.detach();
}

In this case, buffer ② is a pointer variable pointing to a local variable, and then the local variable is passed to the new thread through buffer ②. Furthermore, the function is very likely to crash (oops) before the literal is converted to a std::string object, resulting in undefined behavior. And even if you want to rely on implicit conversion to convert the literal to the std::string object that the function expects, since std::thread’s constructor copies the provided variables, it only copies the unconverted string literal. The solution is to convert the literal to a std::string object before passing it to the std::thread constructor:

void f(int i,std::string const& s);
void not_oops(int some_param)
{
  char buffer[1024];
  sprintf(buffer,"%i",some_param);
  std::thread t(f,3,std::string(buffer));  // Use std::string to avoid dangling pointers
  t.detach();
}

Example 2: You might also encounter the opposite situation: expecting to pass a reference, but the entire object is copied. This can happen when a thread updates a data structure passed by reference, such as:

void update_data_for_widget(widget_id w,widget_data& data); // 1
void oops_again(widget_id w)
{
  widget_data data;
  std::thread t(update_data_for_widget,w,data); // 2
  display_status();
  t.join();
  process_widget_data(data); // 3
}

Using Mutexes

In C++, you create a mutex by instantiating std::mutex and lock it by calling the member function lock(), and unlock it with unlock(). However, direct calling of member functions is not recommended in practice, as it means you must remember to call unlock() at every function exit, including exception cases. The C++ standard library provides a RAII syntax template class std::lock_guard for mutexes, which provides a locked mutex when constructed and unlocks it when destructed, ensuring that a locked mutex is always correctly unlocked. The following program listing shows how to use a std::lock_guard instance constructed with std::mutex to protect access to a list in a multithreaded program. Both std::mutex and std::lock_guard are declared in the header file.

#include <list>
#include <mutex>
#include <algorithm>

std::list<int> some_list;    // 1
std::mutex some_mutex;    // 2

void add_to_list(int new_value)
{
  std::lock_guard<std::mutex> guard(some_mutex);    // 3
  some_list.push_back(new_value);
}

bool list_contains(int value_to_find)
{
  std::lock_guard<std::mutex> guard(some_mutex);    // 4
  return std::find(some_list.begin(),some_list.end(),value_to_find) != some_list.end();
}

Protection of data breaks when one of the member functions returns a pointer or reference to the protected data. Pointers or references with access capability can access (and possibly modify) the protected data without being restricted by the mutex. The design of interfaces that involve mutex-protected data requires careful consideration to ensure that the mutex can lock any access to the protected data, leaving no backdoors.

Using Locks

Imagine a toy that consists of two parts, both of which must be obtained to play with it. For example, a toy drum requires a drumstick and a drum to play. Now there are two children who both like to play with this toy. When one child has both the drum and the drumstick, they can play freely. When the other child wants to play, they have to wait until the first child is finished. Now imagine that the drum and drumstick are kept in different toy boxes, and both children want to play the drum at the same time. So they go to the toy boxes looking for the drum. One finds the drum, and the other finds the drumstick. Now there’s a problem: unless one child decides to let the other play first by giving up their part, if they both hold firmly to their parts without giving them up, neither can play the drum.

Fortunately, the C++ standard library has a way to solve this problem: std::lock—it can lock multiple (two or more) mutexes at once without side effects (risk of deadlock).

How to avoid deadlocks?

  • Avoid nested locks The first suggestion is often the simplest: don’t acquire a second lock when a thread already has one. If you can stick to this advice, there won’t be deadlocks on locks because each thread only holds one lock.

  • Use a fixed order to acquire locks When hard conditions require you to acquire two or more locks, and you can’t use std::lock to acquire them in a single operation; it’s best to acquire them in a fixed order on each thread.

Synchronization and Waiting

Imagine you’re traveling on a night train. How do you get off at the right station at night? One way is to stay awake all night and pay attention to which station you’re at. This way, you won’t miss your destination, but it will make you very tired. Alternatively, you can look at the timetable, estimate when the train will arrive at your destination, and set an alarm a little earlier, then you can sleep soundly. This method sounds good and doesn’t involve missing your station, but when the train is late, you’ll be woken up too early. Of course, the battery in your alarm clock might also die, causing you to sleep through your station. The ideal way is, regardless of early or late, to have someone or something wake you up exactly when the train arrives at the station.

The C++ standard library has two implementations for condition variables: std::condition_variable and std::condition_variable_any. Both implementations are declared in the <condition_variable> header file. Both need to work with a mutex (for synchronization); the former is limited to working with std::mutex, while the latter can work with any mutex that meets minimum standards, hence the _any suffix. Because std::condition_variable_any is more general, there may be additional overhead in terms of size, performance, and system resource usage, so std::condition_variable is generally the preferred type, and we only consider std::condition_variable_any when there are hard requirements for flexibility.

comments powered by Disqus
Built with Hugo
Theme Stack designed by Jimmy