I’ve been going on about this one for a long time, but in case anyone still isn’t on-board with this principal I’ll state it another way: data disappearing from a feed is not a suitable trigger for action.
When we treat disappearance of data as a “trigger” we are interpreting a root cause from a received effect. We are saying “there can only ever be one possible reason why this piece of data has gone missing from our feed” – but is that really the case?
I’ve been pushed (protesting loudly) down this path a few times, and sooner or later something always goes wrong. A technical glitch, a patch, a “we didn’t realise if we clicked that button those records would be removed from the feed”. It was so bad in one environment that I had to introduce a manual step checking up on “deletes” before allowing them to flow through to deactivations. Kind of removed the effectiveness of the automation!
I’ve also seen (and attempted) protective measures like confirming an extraction process ran without errors, cheking for an “end of file” line, or even storing a line count to compare to next time… but these bandaids add complexity to the solution and sometimes cause problems in their own right. They can also lead to a false sense of security – I have had a data collection script work perfectly for two years, and then fail to retrieve all the data one time without reporting an error. That sort of thing may be pretty much impossible to predict or test, but where the consequences can be disruptive it’s just not worth the risk.
So this one stands: Actions in the IAM Solution should only ever be triggered on received data, and never on the source data disappearing.
