Got inconsistent Mercurial subrepository state (RepoLookupError) twice per day. How is that possible?

Question

Recently migrated to Mercurial.

Due to heavy use of externals in old SVN repo we are using Subrepos accordingly and have a CI server that does pulls / pushes to central repositories often. So it's a bit hard to trace what exactly happened and developers can't reproduce the exact steps.

But, after pulling we got errors likes this:

RepoLookupError: unknown revision '766981bc81dc78fe24d5fe5c7d68e36c66858e73'
abort: unknown revision '766981bc81dc78fe24d5fe5c7d68e36c66858e73'!

And such changesets could not be found anywhere, nor on server nor in local repositories. Got this situation twice per day.

Somehow, from the server comes a .hgsubstate that refers to unknown subrepository changeset.

And we didn't do anything potentially harmful, just usual commits / pulls / merges.

As of our understanding - this is an impossible situation (you can't commit a .hgsubstate referring to uncommitted or not existing subrepository changeset).

Any ideas what we could be doing wrong or how this could happen?

edit:not using mq either

Answer 1

To create an unknown (invisible) revision, do like this.

Client 1

1. Commmit a change in a subrepo
2. Link the commit in the subrepo with a commit in the subrepo (the subrepo changeset id will be updated in the .hgsubstate file)
3. Rollback the subrepo commit (now you got yourself a link in the master repo to a non-existing changeset in the subrepo)
4. Push both repositories to the server

Client 2

1. Pull from the server

Error occur!