...
Meadowlark's backend design pattern relies on document-oriented datastores that provide atomic transaction support. As such, there are times when an incoming request will fail due to race conditions / locking. How should Meadowlark handle these failures?
...
In this case, it would be appropriate to retry the request rather than simply sending a failure message back to the client. The number of retries could be configurable, with a default value of 1. If the retry(ies) fail, then responding with 409 "conflict" would be sensible.
→ RETRY, 409
Update Failure Due to Delete
...
This should result in a 409 "conflict" response with an appropriate message indicating that the resource is not deletable, unless a lock timeout occurred. If there was a lock timeout, then
...
etag?
...
a retry would be appropriate.
→ RETRY, 409
Update Failure Due to Deleted Reference
The backend code logic should be locking reference records so that they cannot be deleted in the scope of the transaction, see Meadowlark - Referential Integrity in Document Databases.
Solution
If the transaction fails due to a lock, retry it N times. N should be configurable, but default to 1 (retry once). Zero should be a valid configuration ("do not retry transaction").
Then ensure that 409 is the response if retry fails.