In this post I want to announce that I’ve taken a position as FIM consultant with Unify Solutions, and also to link to a new article on the technet wiki written by my now-colleague Bob Bradley (aka UnifyBob): FIM data modelling and sync design: reference attributes vs. string attributes.

The basic message of this article is that it’s very often better to create a new object type, even though that’s not the immediately obvious thing. Bob uses the example of a person’s “Position” (or job), of which there may be more than one. I raised this point using another specific example in my post Why create a Delegation resource type in the FIM Portal. You could equally apply the notion to anything a person may have more than one of: think roles, applications, equipment… Or a potentially changeable thing that applies to multiple people: think office, cost center, manager.

The other thing that Bob demonstrates in his article is how you should work through your architecture choices before committing, looking particularly at how it will scale. I mentioned the importance of this just the other day, but it’s always helpful to see an example spelled out the way Bob has done.

The final point is that this approach will requie extra data manipulation. Bob mentions Unify products which I’ll post about when I’ve learnt them properly. My usual approach would be to manipulate the data in SQL, generating the new object types and the references there. Extra upfront effort yes, but well worth it for the simpler and more scalable system you’ll get.

“Simplify Complexity” is LANexpert’s motto and I’ve always loved it. Overly-complex IT systems will not perform well on cost, stability or maintainability, so as IT professionals our mandate has been to architect systems that are clearly designed, follow best practises, and can be managed by the customer after we walk out the door.

It’s also a great motto to keep in mind when working with FIM, and here’s some reasons why.

It doesn’t work like you think it does

People come to FIM with all sorts of pre-conceived notions, the worst being “this should be pretty easy”. It’s hardly surprising really, considering that’s what the marketing says. But when you think about the number and variety of tasks FIM is called on to do, and all the site-specific peculiarities involved, it should be clear that the marketing is being distinctly optimistic.

The next thing to userstand is that the Sync Service and the Portal  operate in quite different ways:

• The Sync service is a “Steady state system”, which means that you should be able to synchronize the data in any order and it will end up the same, with no further changes until the source data changes.
• The Portal is object-oriented to it’s very core, meaning you’re often better off defining a new resource type than trying to bend existing ones. It’s also sequential and workflow based, meaning you have to think more about the process of making a change, and who requested it.

I’ve come across a lot of frustration from people starting to work with FIM, and also people who have an existing MIIS/ILM system, which lead me to believe that these basic concepts are often misunderstood.

The most obvious approach isn’t always be the right one

FIM is essentially a framework on which you build your own IdM solution. It is inherently flexible and extensible, and you’ll usually find there is more that one way to skin the proverbial cat. And it’s often the case that the immediately obvious way is just what you shouldn’t do.

I try to think through how my ideas will scale. Say I want to assign application licenses to users through the FIM Portal – what will happen when there are hundreds of applications to choose from? How will the data be kept up to date? How will it look to the users? The immediately obvious idea of adding extra checkboxes to the Person form might start to look a bit less workable.

An early redesign can save you a lot of trouble later on

There are two things you need to build a great FIM system:

• a lot of FIM knowledge, and
• a lot of site knowledge.

Clearly it’s nigh on impossible to find one person with both at the beginning of a project. But even when you partner a consultant with an internal IT person it can take some time for comprehension to filter through on both sides.

Every major MIIS/ILM/FIM project I’ve woked on has gone through a redesign after having gone into production, once real-world data starts flowing through, and it becomes clear that certain key decisions have become constraints. And it’s always been the right thing to do.

Play to the product’s strengths

There’s a great phrase: “if all you have is a hammer, every problem starts to look like a nail”. FIM is so extensible that I think some people get a little giddy with the power of it all. But just because you can do something with extension code, does that mean you should?

I’m a big fan of letting FIM do what it’s good at, and using other tools, like scripting and SSIS, where it’s not so great.

Let’s take the Sync Service. It’s really good at creating, updating and deleting individual objects. What it’s not good at is anything that involves looking at a bunch of objects at once – such as generating a group membership. So we do that bit outside the Sync Service - like in the Portal, or in a SQL table, or an LDAP – and then we just let the Sync Service do it’s fabulous sync’ing thing.

Business logic should be accessible

A common situation with MIIS/ILM is extension code that includes a great long list of if statements that looks at departments and job titles and countries and goodness knows what else to determine actions. I’ve never felt this to be a good idea and I try to stick to this ideal: If it’s liable to change, don’t put it in compiled code! Instead I’ll use SQL queries and XML lookup tables to bring all that logic and variable data out into a place where it can be modified.

With the FIM Portal we can now manage some of this in a graphical way. So if you have a Set of users who should be able to access a certain application, you can now change the definition of that Set and your rules should be applied. This is a great improvement but may not be right for all cases. Somehow I don’t think we’re quite done with XML and SSIS just yet…

Some other posts

I have blogged on this sort of topic before, and yes it was about MIIS, but if you’re working with FIM Sync then it all still applies:

• KISS your MIIS installation
• More KISS tips
• Keep Provisioning Logic out of the Provisioning Code

Gartner’s position is that FIM 2010 is not a complete solution. It can be extended, and by 3rd party partners like Quest but it’s like going to the unfinished furniture store.

It’s not a bad analogy – you do have to do a lot of work on your own side to get FIM doing the things you need. But to continue the analogy, it’s like lots of people have houses which are built upside-down, with all sorts of strange shaped corners, and sloping ceilings, and half the kitchen on the other side of the road… and the only way you can get the furniture to fit with any kind of effectiveness is to go bespoke.

I do think more standardisation in IAM will come, but it’s going to have to go hand-in-hand with further standardisation in account management practises and application security. Maybe then we’ll be able to expect more out-of-the-box functionality without having to go high-end or accept too many compromises. Meanwhile the unfinished furniture store and a good measure of DIY will remain the best option for many organisations.

Some points to consider…

The identifier must be unique in the metaverse

There is no technical reason preventing you re-using an identifier,  but there are a lot of good design reasons. You should always be able to re-join to a metaverse object using the identifier, with no possibility of multiple matches.

I recommend not even re-using the same identifier across different object types.

Export the identifier into all your connected directories

Find a suitable attribute, set up an export flow rule, and get the identifier out. Then use the same attribute to join back (you should always have join rules, on every MA).

Very rarely there will be some application where you can’t find a suitable attribute to stick the identifier in. In my experience so far, these have always involved extensible MAs, and with XMAs I always maintain a SQL table where I can record extra info that will not go into the target system – such as a mapping between my unique identifier and whatever I’m being forced to use in the  external app.

Use something that won’t change

Usernames, email addresses and names are no good. They can change. Forget them.

Ideally use something that comes out of a database system which will already be enforcing a unique ID, such as an HR system.

Ensuring uniqueness

Many organisations are suffering under multiple import systems – perhaps company mergers mean multiple HR systems to encorporate, or different user types being generated in different systems. If there is any risk of a conflict, use an import flow rule to insert a prefix, or stage via a database where you can manipulate the identifier and assure uniqueness.

Don’t use the metaverse GUID

I was, I confess, somewhat horrified when I realised people do this. Just don’t! You never know when you will need to clear out and re-import. You should always plan for this possibility!

Don’t use something system specific

I also have concerns when I hear of the AD SID being used as a unique identifier. I feel that a person (or group, or other object) should be uniquely identifiable whether it is currently in AD or not. Sometimes accounts get deleted and re-created, but that doesn’t stop the person being themself.

I would only consider this a defensible design if AD was your primary, and most trustworthy, data source.

So remember:-

the unique identifier is unique, it doesn’t change, you get it everywhere, and you can rely on it in join rules. Perfect KISS philiosophy.

Test Server

Of course you should be testing all your changes on your test server first. It’s really not a good idea to wing it on your production server because syncs are impossible to roll back. If you make a change, and then realise you have to wait for approval, you may have to stop all exports, at least from the effected MA(s). The only full-proof way to reverse the change is to delete and reimport your CSs – and, depending on the amount of data, this can take a LONG time.

So: back to the test server. If you’ve managed to sort out the MIIS groups thing, then you can backup your production MIIS database, and restore it on your test server. You will now have the fully populated connector spaces, so there should be no need to do actual imports from the live data to produce an accurate preview.

Export to file

The next step is pretty obvious. make your change, run your syncs, and then use the “export to file and stop run” option of the export run profile. If you have shed-loads of data then consider restricting the export to a representitive sample.

Clean up the file

The XML file produced by the export can be opened in Excel, but it will contain a lot of information that you really don’t want to inflict on management – not if you want a quick approval out of them anyway. So spend some time tidying it up.

One thing I find is that people find it a lot easier if the “old value” and the “new value” are presented next to each other, and not on seperate lines as in the XML file. If you’re clever with your sorting you should be able to get yourself in a situation where the “add” is on one line with the “delete” directly beneath.

Now you have to muck around for a bit with functions, pasting values, and removing redunant columns, to end up with something like the first picture in this post. Here’s a function which may help:

=if(and(B1=B2, C1=C2, E1="add"), D2, "")

Why are we doing all this again?

People get nervous about automatic changes, and they want reassurance that the changes will be the right ones. This is only fair, and as a good ILM designer you should provide information in a clear enough way to set their minds at ease.

And besides, its a great way to pick up bad data. I don’t know how many times I’ve seen bad data from the supposedly master source about to overwrite good data in the destination. Showing this in black and white isn’t a bad way of driving home the point that “your data’s not as clean as you thought it was”.

If you’re from a sys admin background, like me, then the dawning realisation that you have to code may come as rather a shock.

Even if you are from a development background you will need to gain an understanding of ILM language such as CS objects, Metaverse objects, provisioning, and synchronising. You will need to come to terms with ILM peculiarities, for instance how metaverse objects can only be written to under specific conditions, how the code runs several times for each object change, how you can’t predict the order it will run, and how you can’t run it stand-alone.

Add to this the fact that you also need a good understanding of the connected directories themselves, and I find I agree with this post by Brad Turner where he says that the best ILM/MIIS people come from a Directory Administration, and not a code development, background.

But enough of that – how do you actually go about learning ILM?

Jump straight in and try to figure it out

The method beloved of techie-types the world over – we’re smart, we know computers, heck it’s a Microsoft product after all, how hard can it be??

Answer: Hard. And, btw, Microsoft didn’t write it, they bought the company. It might have been a while back, but it’s suprising how little Microsoft-ised the product is today. (They’re working on that.)

RTFM

This is the usual advice – but unfortunately, in the case of ILM/MIIS, the FM is not remotely newbie-friendly. In fact it’s got to be about the most jargon-ridden example of too much detail leading to no understanding, that I have ever attempted to read.

It’s also still only available in the MIIS 2003 version – though don’t let that put you off as ILM 2007 and MIIS 2003 are basically the same thing.

The Walkthroughs (Self-Training)

The FM does, however, contain some gems – one of them being the Walkthroughs. If you want to teach yourself ILM then the Walkthroughs are the absolute best place to start. Don’t be put off by the examples being SQL and AD when what you really want is Oracle and Netware – that’s not the point! You will learn about how the components of ILM work together, and about how the code works, and then you can apply these lessons to what you really want to do.

So book yourself a few days of uninterrupted time, build yourself a virtual test environment, and do the Walkthroughs. I guarantee this will be quicker, and less painful in the long-run, than jumping straight in with some downloaded code snippets and no understanding.

Training Courses

There is a Microsoft training course at the beginner level:-MS2731 Deploying and Managing Microsoft Identity Integration Server (MIIS) 2003.

Be warned, however, that taught by a trainer with no actual ILM/MIIS experience, you may find this completely useless. Make sure you ask about the trainer before you book. For my money, they can go hang their Microsoft Certified Trainer on the nearest loo wall – what counts here is real-world experience.

Highly recommended are the courses (beginner and advanced) offered by Oxford Computing Group where you will be taught by experts of the highest calibre. Check their website for course schedules and locations.

Finally you may be able to arrange a workshop or on-the-job training with a local consultancy firm such as the one that I work for (if you happen to be in Switzerland). Again, check their credentials and ask for references as experienced ILM consultants still remain thin on the ground.

Good luck!

The prohibition on calling external processes from provisioning (MVExtension) code is clear and well accepted (see the Calling External Processes section on this technet page), so what we were actually talking about was extensible MAs.

The way it came up was this: I was showing him my Techdays presentation, part of which includes a demo where I create personal websites in IIS as a way of saying “Look! It’s not just about user accounts!”. I do this pretty simply by running the iisVDir.vbs script which comes with IIS 6, using the /query option as part of my Import routine, and the /create and /delete options as part of the Export. (MVExtension code, CSExtension code)

Now my collegue said that he would never use ILM to run any kind of external process, because it slows down the system too much. He would just provision lines into a SQL table and then configure something external to ILM to pick up the rest of the work.

So in my IIS example here we’d just have ILM writing lines into a table, and then perhaps a scheduled task checking the table periodically, running iisVDir.vbs for anything new, and then updating the status once the code had succesfully completed. ILM would then import the “done” status back in and everything’s happy.

The advantages of this method are clear:

• Speed and efficiency in ILM – it’s very, very good at creating simple objects like lines in SQL tables, and will whip through these without a moment’s hesitation,
• Simple to set up – no need to delve into extensible MAs, just a simple SQL MA and a vbscript to run the external bits,
• Efficiency again – you can offload the external part of the process to another server to share the workload.

I have, in fact, used this method before – like when I used Exmerge to archive mailboxes, as it would have slowed the whole sync cycle down unacceptably if I’d had to wait for Exmerge to finish every time I ran an Export. I also used the same method for sending “account x created” emails – writing a line into a SQL table once an account creation was confirmed, and leaving the emailing part up to another server.

But I’ve also written XMAs where I do call external processes. As well as the IIS example above, I made an XMA to create and then archive Netware home folders. Folder creation, zipping and deletion was all done straight from the CSExtension code (with the help of an archive library and some command line stuff for setting the ACLs). The create side was pretty fast, and the archiving wasn’t too bad either – except on the occassions when a couple of hundred folders were going at once (this was a university) and then you started to see a big impact on the rest of the syncs.

So what are the advantages of calling these external process directly from the CSExtension code?

• Imports are closer to the source – I like the idea that the import step should be going and discovering what is really in the data source, and not just reporting on the contents of a SQL table.
• As long as the code in the export step is not adding too much of a delay, you should have linked objects created more closely together – for instanace a user account and its home folder.
• If there’s a problem with the external process, and it’s stopping your other sync activity, you’re going to pick it up very quickly. Note this can also count as a Bad Thing!

I guess my conclusion is that my collegue is almost cerainly right for large installations, where there are a lot of objects, or MAs, or both, and sync times are an issue. But for a smaller installation, or for external processes which are quick and light to run, I see no problem in calling them directly from CSExtension code. After all, that is what XMAs are supposed to be for!

There’s a dangerous little form in the Metaverse Design section of Identity Manager which allows you to set your Object Deletion Rule. By the simple expedient of clicking a radio button you can delete Metaverse objects just because the object (ie “connector”) disappeared from a nominated Connector Space.

If you’ve also set your Management Agents to delete based on disappearance of the MV object, this deletion will then be replicated in all your connected directories.

So why am I so worried about this? Let’s say, for example, that you are supplied with a daily text file from HR listing all current users. If a user is no longer on the list then what’s the problem with removing all their accounts?

Take it from me (who accidentally disabled hundreds of user accounts using such a system) you need something more concrete when you’re going to start taking things away from people!

The fact of the matter is, glitches occur. That text file generation could have been interrupted part-way through. Even if you’re going closer to the source, eg straight from a SQL table, you can run into problems with SQL replication and DTS packages. And lets not forget the sheer scope posed by human error! I’m also wondering if an incomplete Import step into MIIS could lead to CS objects temporarily going missing … I think not, but in Good Design we don’t take the chance.

My proposal is that you figure out a way to get an extra Status field into your input data. This field should be used for tags such as ”Active” and “Inactive”, as well as any other special-purpose flags you may see fit to introduce.

Here’s what you need to do:

1. Change that Metaverse Deletion Rule. You may need to write extension code (the third option), but I’ve only ever used the first option:

2. It should be fine to leave the MA Deprovisioning rules set as shown above as you won’t be deleting the Meteverse object until all CS objects are gone.

3. Create a Metaverse attribute and import flow rule for the Status field, so that it is flowed in along with other object data.

4. You can now do a simple test, in your MVExtension code, before you proceed to anything destructive:

If mventry(“Status”).Value = “Inactive” Then
      account disable/delete instructions
End If

5. If you wanted to, you could now use the disappearance theory as a way to clear out objects later on – just check first that a previous import has set the expected Status:

If csobject Is Nothing AndAlso mventry(“Status”).Value = “Inactive” Then
      connector deletions
End If

So, using this method, what happens if a few objects do go missing (accidentally or otherwise) from your input data?

• MIIS will remove the CSobject from your source MA, correctly reflecting the input data it has been fed;
• The metaverse (and hence other connector space) objects will be unaffected – all that’s happened is a connector has been lost;
• When the object reappears in the inital connector space it should just rejoin to the existing metaverse object. And if it doesn’t, CHECK your join rules!

If you’re worried about orphaned objects you can do the odd MIIS DB query to list the metaverse objects without a connector in the source CS, and then follow those up. After doing that a few times you will probably be well on your way to training your data handlers to NEVER just delete a record without putting it through an “Inactive” phase first!

I particularly enjoyed reading his Tenets of Identity Management article. It’s probably of greatest relevance to people in the product selection phase of an IdM project, but even if you’re already underway with MIIS it’s worth being reminded about the advantages of consolidating your directories and simplifying your solution. His other excellent point is about monitoring of the system. This is so important that I should probably pinch it for my Reducing Fear and Loathing of IdM article, though instead I may just put together a new post about the specific monitoring I’ve implemented in the past.

I also couldn’t help agreeing with Jackson that MIIS is way too complicated. It was a great shock to me, when I first started out, to realise that I had to learn .NET to do something as simple, and Microsoft-based, as creating a user account in AD (see What do you mean I have to WRITE CODE???). But whatever toolsets and out-of-the-box functionalities are added in the future, I can’t help but hope that those in the know will still be able to dive under the bonnet and make all sorts of other interesting things happen. Because once you’ve got the hang of it, it really is good fun!

What should happen to the rows in the Delta table that were not imported due to errors? Ideally you will let these rows stay put, giving MIIS another go at them on the next Delta Import. But what, then, should you do with the rows that were successfully imported?

The solution will depend on the number of objects you’re dealing with, how often you manage a Full Import, and what the repercussions are of missed imports. In a system where a complete set of Full Imports can be run overnight, you may well find that a few missed deltas during the day are not significant. Alternatively, if the Fulls are run weekly or less often, you are completely dependant on the reliability of your Deltas – those failed imports must get another chance.

Some possible solutions: optimistic, pessimistic and best-odds.

Optimistic

Assume all Delta Imports are successful. Clear out the Delta table as an automatic step before generating the new Delta table.

This approach works fine in simple systems where Full Imports can be run fairly regularly (at least once a day) to mop up any missed imports. A big problem with it is that an Import must be run after every run of the DTS. This can quickly turn into a complete pain when you’re troubleshooting.

Pessimistic

Each row in the Delta table is independently checked for import success, and only removed once the import is confirmed.

You could perhaps write some code that checks through the import log file after each import, and only deletes lines from the Delta table that look correct in the log.

On my MIIS course, the lovely Hugh Simpson-Wells suggested an extra MA where objects are sync’d back from the metaverse just so you can compare them with the source data. This is not something I’ve ever tried, but I guess it would be the way to go if you needed 100% verification on those imports.

Best-Odds

The best-odds approach that I adopted was to do a simple check for errors following the Delta Import task. If any errors occur, I keep the entire Delta table. Otherwise the Delta table is cleared.

The main downside of this approach is that a single (possibly trivial) error will prevent the clearing, and will cause your delta table to grow and grow. MIIS will not be bothered by being given the same import data over and over, but it does slow everything down. You can also run into problems when the same object appears multiple times in the table with conflicting instructions (like a “Delete” followed by a “Modify”). Adding a de-duping step to your DTS helps, but the only solution is to ensure that import errors are sorted out ASAP.

There are a number of technical examples I should include at this point, but this is a philosophising post so I won’t do it here. There’s more to come on how to write a VBScript to run your MIIS jobs, check for errors, and then fire off whatever else you need to do.