Skip to content

Requests freezing on PostProcessing and Validated

I’ve had a head scratcher in the last week. A Dev environment, everything thought to be working perfectly according to the tests we’d done so far, which had just involved a few identities at a time. But then we tried to do some bulk tests, registering 300 people at once, and things did not go well. It would get through 20 or 30 of them and then the requests queue just seemed to freeze. The Portal was still responsive but the number of requests sitting in PostProcessing and Validated status did not change, even when left overnight.

I thought it had something to do with the “Generate Unique Value” custom workflow I was using. I was generating unique values for both AccountName and MailNickname, with checks against both the Portal and AD. A lot of the requests seemed stuck on one of these steps. I replaced the workflow with a PowerShell activity – but exactly the same thing happened.

The Problem

I’ll go straight to what the root problem was, and I had to get some help from Microsoft for this – the problem is trying to query the FIM Service database for uniqueness while at the same time updating exactly the attribute it is trying to query. So I’m doing hundreds of queries like “/Person[AccountName='value']” while at the same time updating AccountName on multiple objects. Effectively my query is trying to hit a moving target, and the result is SQL deadlocking.

Now why it got itself in such a mess it had to stop so completely I don’t know, and the low spec of the Dev servers probably didn’t help (though the problem still occurred after we had the RAM increased on both FIM and SQL servers).

The Solution

The solution, for me anyway, is not to generate unique values in the Portal but go back to my tried and tested method of doing it as part of classic rules extensions with a lookup against the Metaverse and AD. This was supposed to be an all Declarative design, but if the result is a risk of deadlocking then the provisioning Sync Rule, at least, has had to go.

Troubleshooting

Some observations:

  • Restarting the FIM Service would get the queue moving again, but just for a short while then it would freeze again.
  • Restarting both the FIM Service and SQL had a better result in getting things moving – presumably because all the deadlocks disappeared.
  • There was a mysterious KeyNotFoundException error in the Event Log that I’d been seeing for a while and it never seems to correspond to any errors in the Portal.
    • This turned out (thanks to Jeff who commented below) to be a bug caused by not implementing the new HideAdvancedSearchLink attribute a binding as described here: http://support.microsoft.com/kb/2906832. It had nothing to do with the requests queue freezing.

Here are some events I noticed when I turned on Verbose logging:

Level: Information
Description:
RequestDispatcher is processing RequestIdentifier ‘f3d76d22-6c6c-4a51-b3ef-85d2e76966c1′ for a ‘Enumerate’ operation on object ” with RequestStatus ‘Authorized’.

Level: Information
Description:
WS: Action.Enumerate.Execute.Enter

Level: Information
Description:
Query: QueryProcessor.ExecuteQuery.ExecuteReader.Enter

Level: Information
Description:
Request ‘f3d76d22-6c6c-4a51-b3ef-85d2e76966c1′ status was updated in-memory from ‘Authorized’ to ‘Denied’.

Level: Information
Description:
RequestIdentifier ‘f3d76d22-6c6c-4a51-b3ef-85d2e76966c1′ exited RequestDispatcher with RequestStatus ‘Denied’.

Level: Information
Description:
WS: Enumerate:UnwillingToPerform

{ 2 } Comments

  1. Jef | April 9, 2014 at 10:40 pm | Permalink

    Just curious, did this resolve your Mystery “KeyNotFound” error? If not, are you running 4.1.3496 without the HideAdvancedSearch Binding?

  2. Carol | April 9, 2014 at 11:29 pm | Permalink

    Hmm I am running that version and I have not enabled HideAdvancedSearch. Can that cause those errors? I don’t actually want to hide the advanced search. I can see the errors are still appearing – I’d have to re-enable Verbose logging to see if those Enumerate:UnwillingToPerform ones have gone, which sounded like they were more related to the uniqueness search. I haven’t had any more frozen request queues at least.

Post a Comment

Your email is never published nor shared. Required fields are marked *

Spam comments will be blocked by Akismet

*

Endep online order order buy Endep online purchase Endep online without script buy generic Endep from india buy in Endep uk where to buy Endep without a prescription Endep from canada buy 10 mg Aricept generic Aricept cost order Nizoral no visa Nizoral 200 mg buy cheap Nizoral line Nizoral on line best finpecia online pill Requip Requip free Requip uk Amitriptyline generic buy generic Amitriptyline from india where buy Lisinopril purchasing finpecia with overnight delivery Nizoral online buy saturday delivery buy Nizoral in mo how to buy Cytotec online without a rx order Aricept pharmacy buy Aricept with no prescription buy Amitriptyline no perscription cod Online us Requip how to buy Amitriptyline without a prescription buy line Amitriptyline buy Amitriptyline money buy purchase Accutane 40 mg on line no rx Accutane 40 mg sale order 40 mg Accutane online next day delivery 40 mg Accutane online buy Lisinopril pay cod Lisinopril overnight cod buy lisinopril without prescription buy lisinopril purchase Premarin cod next day delivery where buy Premarin buy Premarin online free fedex delivery Premarin no rx Metformin buy Metformin canada cheapest place to buy Metformin purchase Metformin online generic 20mg prednisone online finpecia online overnight buy generic Finpecia uk online|Finpecia 1 mg order online|Finpecia generic vs propecia|Finpecia cost australia|buy generic Finpecia uk|purchase Finpecia uk|buy Finpecia online prescription|buy generic Finpecia 1mg|buy Finpecia|Finpecia online australia|buy Finpecia no prescription uk|order Finpecia 1mg|buy Finpecia cheap|Finpecia generic 1mg|Finpecia online prescription|Finpecia 1mg generic canada|Finpecia costco pharmacy|Finpecia 1mg india|Finpecia generic walmart|buy Finpecia no prescription forum|buy Finpecia online australia|Finpecia online uk) buy finpecia cheap legal buy Finpecia online online|buy cheap Finpecia online|order Finpecia|Finpecia generic walmart|Finpecia online prescription|buy Finpecia 1mg online|Finpecia generic online|Finpecia costco price|order Finpecia online|buy Finpecia 1mg malaysia|purchase Finpecia uk|topical Finpecia buy|Finpecia 1 mg order online|Finpecia generic|buy Finpecia online canada|Finpecia 1 mg order online|Finpecia online india|Finpecia generic 1 mg|order Finpecia uk|Finpecia depression women|buy cheap Finpecia 1mg|Finpecia generic names|legal buy Finpecia online|Finpecia depression women|buy Finpecia 1mg india|Finpecia generic walmart|buy Finpecia 1mg online|Finpecia online india|buy Finpecia 1mg online|Finpecia 1mg tablets|Finpecia costco price|order Finpecia canada|buy Finpecia online australia|Finpecia dose hair|buy Finpecia online paypal|Finpecia 1mg|Finpecia 1mg for sale|Finpecia online uk|buy Finpecia uk|Finpecia 1mg vs 1mg|Finpecia generic walmart|Finpecia costco pharmacy|buy Finpecia on buy Metformin toronto uk order Metformin Metformin online Metformin prescription buy Metformin where order finpecia canada buy finpecia order Metformin usa online Orlistat buy buy in Orlistat uk buy prednisone topix buy prednisone youtube Orlistat toronto buy discount Orlistat line buy cheap Orlistat without prescription prednisone prescription cost Metformin order online pharmacy Metformin how to buy Metformin without a prescription Metformin price lisinopril buy lisinopril