Worker thread suspend / resume implementation

Question

In my attempt to add suspend / resume functionality to my Worker [thread] class, I've happened upon an issue that I cannot explain. (C++1y / VS2015)

The issue looks like a deadlock, however I cannot seem to reproduce it once a debugger is attached and a breakpoint is set before a certain point (see #1) - so it looks like it's a timing issue.

The fix that I could find (#2) doesn't make a lot of sense to me because it requires to hold on to a mutex longer and where client code might attempt to acquire other mutexes, which I understand to actually increase the chance of a deadlock.

But it does fix the issue.

The Worker loop:

Job* job;
while (true)
{
  {
    std::unique_lock<std::mutex>  lock(m_jobsMutex);
    m_workSemaphore.Wait(lock);

    if (m_jobs.empty() && m_finishing)
    {
      break;
    }

    // Take the next job
    ASSERT(!m_jobs.empty());
    job = m_jobs.front();
    m_jobs.pop_front();
  }

  bool done = false;
  bool wasSuspended = false;
  do
  {
    // #2
    { // Removing this extra scoping seemingly fixes the issue BUT
      // incurs us holding on to m_suspendMutex while the job is Process()ing,
      // which might 1, be lengthy, 2, acquire other locks.
      std::unique_lock<std::mutex> lock(m_suspendMutex);
      if (m_isSuspended && !wasSuspended)
      {
        job->Suspend();
      }
      wasSuspended = m_isSuspended;

      m_suspendCv.wait(lock, [this] {
        return !m_isSuspended;
      });

      if (wasSuspended && !m_isSuspended)
      {
        job->Resume();
      }
      wasSuspended = m_isSuspended;
    }

    done = job->Process();
  }
  while (!done);
}

Suspend / Resume is just:

void Worker::Suspend()
{
  std::unique_lock<std::mutex>  lock(m_suspendMutex);
  ASSERT(!m_isSuspended);
  m_isSuspended = true;
}

void Worker::Resume()
{
  {
    std::unique_lock<std::mutex>  lock(m_suspendMutex);
    ASSERT(m_isSuspended);
    m_isSuspended = false;
  }
  m_suspendCv.notify_one(); // notify_all() doesn't work either.
}

The (Visual Studio) test:

  struct Job: Worker::Job
  {
    int durationMs = 25;
    int chunks = 40;
    int executed = 0;

    bool Process()
    {
      auto now = std::chrono::system_clock::now();
      auto until = now + std::chrono::milliseconds(durationMs);
      while (std::chrono::system_clock::now() < until)
      { /* busy, busy */
      }

      ++executed;
      return executed < chunks;
    }

    void Suspend() { /* nothing here */ }
    void Resume() { /* nothing here */ }
  };

  auto worker = std::make_unique<Worker>();

  Job j;
  worker->Enqueue(j);

  std::this_thread::sleep_for(std::chrono::milliseconds(j.durationMs));  // Wait at least one chunk.

  worker->Suspend();

  Assert::IsTrue(j.executed < j.chunks);  // We've suspended before we finished.
  const int testExec = j.executed;

  std::this_thread::sleep_for(std::chrono::milliseconds(j.durationMs * 4));

  Assert::IsTrue(j.executed == testExec); // We haven't moved on.

  // #1
  worker->Resume(); // Breaking before this call means that I won't see the issue.
  worker->Finalize();

  Assert::IsTrue(j.executed == j.chunks); // Now we've finished.

What am I missing / doing wrong? Why does the Process()ing of the job have to be guarded by the suspend mutex?

EDIT: Resume() should not have been holding on to the mutex at the time of notification; that's fixed -- the issue persists.

Answer 1

Of course the Process()ing of the job does not have to be guarded by the suspend mutex.

The access of j.executed - for the asserts as well as for the incrementing - however does need to be synchronized (either by making it an std::atomic<int> or by guarding it with a mutex etc.).

It's still not clear why the issue manifested the way it did (since I'm not writing to the variable on the main thread) -- might be a case of undefined behaviour propagating backwards in time.