Skip to content

Requests freezing on PostProcessing and Validated

I’ve had a head scratcher in the last week. A Dev environment, everything thought to be working perfectly according to the tests we’d done so far, which had just involved a few identities at a time. But then we tried to do some bulk tests, registering 300 people at once, and things did not go well. It would get through 20 or 30 of them and then the requests queue just seemed to freeze. The Portal was still responsive but the number of requests sitting in PostProcessing and Validated status did not change, even when left overnight.

I thought it had something to do with the “Generate Unique Value” custom workflow I was using. I was generating unique values for both AccountName and MailNickname, with checks against both the Portal and AD. A lot of the requests seemed stuck on one of these steps. I replaced the workflow with a PowerShell activity – but exactly the same thing happened.

The Problem

I’ll go straight to what the root problem was, and I had to get some help from Microsoft for this – the problem is trying to query the FIM Service database for uniqueness while at the same time updating exactly the attribute it is trying to query. So I’m doing hundreds of queries like “/Person[AccountName=’value’]” while at the same time updating AccountName on multiple objects. Effectively my query is trying to hit a moving target, and the result is SQL deadlocking.

Now why it got itself in such a mess it had to stop so completely I don’t know, and the low spec of the Dev servers probably didn’t help (though the problem still occurred after we had the RAM increased on both FIM and SQL servers).

The Solution

The solution, for me anyway, is not to generate unique values in the Portal but go back to my tried and tested method of doing it as part of classic rules extensions with a lookup against the Metaverse and AD. This was supposed to be an all Declarative design, but if the result is a risk of deadlocking then the provisioning Sync Rule, at least, has had to go.

Troubleshooting

Some observations:

  • Restarting the FIM Service would get the queue moving again, but just for a short while then it would freeze again.
  • Restarting both the FIM Service and SQL had a better result in getting things moving – presumably because all the deadlocks disappeared.
  • There was a mysterious KeyNotFoundException error in the Event Log that I’d been seeing for a while and it never seems to correspond to any errors in the Portal.
    • This turned out (thanks to Jeff who commented below) to be a bug caused by not implementing the new HideAdvancedSearchLink attribute a binding as described here: http://support.microsoft.com/kb/2906832. It had nothing to do with the requests queue freezing.

Here are some events I noticed when I turned on Verbose logging:

Level: Information
Description:
RequestDispatcher is processing RequestIdentifier ‘f3d76d22-6c6c-4a51-b3ef-85d2e76966c1′ for a ‘Enumerate’ operation on object ” with RequestStatus ‘Authorized’.

Level: Information
Description:
WS: Action.Enumerate.Execute.Enter

Level: Information
Description:
Query: QueryProcessor.ExecuteQuery.ExecuteReader.Enter

Level: Information
Description:
Request ‘f3d76d22-6c6c-4a51-b3ef-85d2e76966c1′ status was updated in-memory from ‘Authorized’ to ‘Denied’.

Level: Information
Description:
RequestIdentifier ‘f3d76d22-6c6c-4a51-b3ef-85d2e76966c1′ exited RequestDispatcher with RequestStatus ‘Denied’.

Level: Information
Description:
WS: Enumerate:UnwillingToPerform

{ 2 } Comments

  1. Jef | April 9, 2014 at 10:40 pm | Permalink

    Just curious, did this resolve your Mystery “KeyNotFound” error? If not, are you running 4.1.3496 without the HideAdvancedSearch Binding?

  2. Carol | April 9, 2014 at 11:29 pm | Permalink

    Hmm I am running that version and I have not enabled HideAdvancedSearch. Can that cause those errors? I don’t actually want to hide the advanced search. I can see the errors are still appearing – I’d have to re-enable Verbose logging to see if those Enumerate:UnwillingToPerform ones have gone, which sounded like they were more related to the uniqueness search. I haven’t had any more frozen request queues at least.

Post a Comment

Your email is never published nor shared. Required fields are marked *

Spam comments will be blocked by Akismet

*

Tamoxifen for sale Furosemide 40 mg without prescription Cytotec no prescription to buy buy Requip 2mg order Seroquel online canada buy Clomiphene australia buy Seroquel generic order no prescription Metformin online Requip buy Seroquel purchase overnight delivery Seroquel no prescription needed 50mg buy seroquel online without prescription from canada no prescription generic seroquel seroquel for sale canadian pharmacy no prescription seroquel buy seroquel online uk buy seroquel 25 mg seroquel online 50mg how to buy seroquel online seroquel online uk seroquel 50mg canada buy generic seroquel seroquel no script seroquel online without a prescription can you buy seroquel online buy seroquel usa buy seroquel in canada Prednisone on line buy Prednisone overnight buy cheap Prednisone online free consult buy Prednisone on line amex uk buy Prednisone buy cod Prednisone order overnight Prednisone medikament Prednisone Prednisone order online buy Prednisone no scams buy Prednisone mastercard online purchase Prednisone buy cheap Prednisone no prescription buy Prednisone amex online without rx buy Prednisone canada purchasing Prednisone without a script where to buy generic Prednisone online without a prescription Prednisone drug non rx cheap Prednisone order Prednisone cheap overnight fedex Prednisone overnight without a prescription buy Prednisone 20 mg buy Prednisone where Prednisone without prescription overnight shipping purchasing Prednisone without a script Prednisone without rx overnight shipping where to purchase Prednisone no prescription no fees Prednisone online overnight delivery cod Prednisone prescription order order Prednisone without rx from us pharmacy buy Prednisone pills in toronto buy Prednisone online online pharmacies Prednisone buy Prednisone no prescriptions buy Prednisone once a day order Prednisone pay pal online without rx buy Prednisone where buy line Prednisone best finpecia online pill buy Maxalt us pharmacy Maxalt buy Maxalt with visa Cytotec purchase overnight delivery where buy Lisinopril purchasing finpecia with overnight delivery no prescription requip buy requip over the counter where can i order requip online how to get arimidex arimidex cheap online buy Requip online australia generic requip no prescription arimidex ordering requip no rx in us requip buying requip online without prescription Requip express online i need to order Requip without a prescription buy Requip from the uk purchase Arimidex order requip without rx arimidex online order where can i buy arimidex without prescriptions buy arimidex generic requip for sale without prescription buy generic arimidex buy requip no prescription needed order requip online overnight shipping cheap generic arimidex no prescription requip buy online no prescription Lisinopril overnight cod cheap order rx metformin prednisone 40 mg cheapest place to buy Metformin purchase Metformin online Maxalt no prescription needed 10mg Maxalt 10mg canada where to buy Maxalt uk buy generic Maxalt canada buy metformin cod discount Cytotec order Cytotec uk purchase Cytotec online no membership overnight shipping best Cytotec online pill Valacyclovir Cytotec want to buy Cytotec in usa Cytotec online overnight delivery cod buy cytotec with a mastercard order Cytotec no rx how to order Cytotec online without a rx how to buy Metformin without a prescription Metformin price buy Alli without prescription