It’s subtle, and easily overlooked.
Server side calls in the domain layer are setting all sorts of properties on an object before pushing it over to the database for persistence. The database receives the object, persists and returns. The domain then requests an updated snapshot from the database (which may or may not include the data recently persisted). Nothing tricky, stock standard code.
Problem creeps in when the server side code (domain) resides on a different server to the database and both servers are dealing with DateTime as a parameter for filtering on datasets. If their timestamps are not synchronized, down to the millisecond, you’re likely to get an inconsitently reproducible bug. As was the case recently…
A ‘contact’ was deactivated by setting the deactivatedTime field with a value on the domain server.
Deactivated contacts were returned by the data server based on the data server’s getdate() value.
When the two calls are sufficiently close to one another, your newly deactivated record sometimes appears in the second result set, depending on whether the two calls are within the same second according to the various timestamps on the different servers. Huh? 🙂
“Esoteric”, as Eugene used to say.
And you can’t overcome the issue 100% of the time reliably. Not when you start partitioning, synchronizing, servicing and load-balancing logic within a virtual network. You can however, be aware of the problem and plan accordingly. It’s just one of those (depending on the kind of systems you’re writing) you just keep in the back of your head until it surfaces one day… 🙂