I’m pretty pleased with the results. Both of the following scripts use a CSV to bulk-create the groups:

Create Security Groups based on Filters

Create Distribution Lists for Managers which contain all the people they manage

The scripts run pretty slowly, but it’s still quicker than creating the groups by hand.

If you want to have a go at a script like this (and you can’t find an example in the ever-growing FIM Scriptbox) then I suggest you create a sample object by hand and then inspect both the object’s Advanced Properites, and the Details of the Request object which created it, for an idea of which attribute to populate.

While developing the scripts I saw the following error far more times that I would have liked:

Microsoft.ResourceManagement.WebServices.Client.PermissionDeniedException: Policy prohibits the request from completing.

After messing around with MPRs it eventually became clear that this just meant I had populated an attribute incorrently, or missed one out, and was not about permissions at all.

Warning: I will have to revisit this post – as I haven’t yet installed Exchange 2010 in a production environment the Exchange comments are based on reading rather than hands-on experience, and in particular I’m unsure about the management of email-enabled Security groups.

Getting information about my AD users into the ILM Portal went well. I just added a bunch of export flow rules to my ILM MA, ran an Export, and voila! all my users were visible in the Portal.

Unfortunately the groups did not go so well. I kept running into the export error failed-creation-via-web-services, with no idea as to why. Eventually I found a post on the MS Connect newgroup for ILM2 which pointed me in the right direction (I can’t link to the post, but if you search for “failed-via-web-services tips” you’ll find it) – it was to do with the group object schema in the ILM Portal.

Checking the schema requirements

So what you want to do is find out which attributes are essential for the object type you’re trying to export, and what restrictions there are on the possible values.

1. In the ILM Portal, open the Schema Management page (found under Administration).
   
2. Click the All Bindings icon.
   
3. Click on Advanced Search.
   
4. Enter the search criteria:
   • The Binding’s ObjectTypeDescription is Group
   • Attribute IsRequired is True

    
   

5. You should now see a list of the required attributes. You can click on each one to find out more about them. In particular you should check for a regular expression on the Validation tab. Your data must pass the regex (note you can change the regex here if you need to add more possible values).
   
   

 
Populating the values

Some of the required values, like the display name, you can just flow straight from AD. There are a couple of others you can just flow a constant value to – though as you don’t appear to be able to do Advanced rules from the ILM MA you will have to flow the values into the metaverse first from the AD MA.

The Type and Scope will have to be calculated from the groupType attribute in AD, and you will have to set them using the exact terms from the Portal Schema attributes. The following code can be used with advanced import flow rules on the AD MA (the classic type, which I’ve gone back to for the time being).
 

 Case "import_type"
     If csentry("groupType").IntegerValue < 0 Then
         mventry("type").Value = "Security"
     Else
         mventry("type").Value = "Distribution"
     End If

 Case "import_scope"
     Dim groupScope As Long
     If csentry("groupType").IntegerValue < 0 Then
         groupScope = csentry("groupType").IntegerValue + 2147483648
     Else
         groupScope = csentry("groupType").IntegerValue
     End If
     Select Case groupScope
         Case 2
            mventry("scope").Value = "Global"
         Case 4
            mventry("scope").Value = "DomainLocal"
         Case 8
            mventry("scope").Value = "Universal"
     End Select

Once the values are all attached to the group objects in the metaverse it is a simple matter to create the export flow rules on the ILM MA.

The error you will see will either be dn-attributes-failure or cd-missing-object, depending on the type of group.

The detailed error will say something about an add or delete operation on a member that does not exist but, unhelpfully, will not tell you which one.

I’ve had some fun and games with this one recently, so this post is about some ways I figured out to troubleshoot the problem, and includes a vbscript for finding that missing member.

dn-missing.vbs

While trying to troubleshoot these missing member errors during the week I wrote a quick vbscript to help – you can look at it here.

Basically it exports an XML copy of the group object from the connector space, and then attempts an LDAP bind against each member. This works for AD. I haven’t tried it for other directories, but I expect it would work with anything based on LDAP.

What if the member exists?

The big problem I was having was when the member actualy did exist in AD. This was very frustrating. It seemed that once ILM had decided it couldn’t export the group then nothing could convince it otherwise. I tried various mitigation techniques:

• Full Import Full Sync of everything (didn’t help),
• Hacking the export.xml (helped when I was having a problem with a member delete),
• Adding the member manually in AD then doing a Delta Import Delta Sync (a bit pointless, but it got things moving again).

A full clear-out and re-import of the connector space would doubtlessly have worked, but considering the number and size of the groups, this would have been a painful process.

Targeting the same DC

What I did eventually figure out was that the two MAs were targeting different DCs. Duh! Obviously, to avoid any missing objects due to AD sync delays, you should target the same DC.  In fact this post on the Technet forum indicates that a Global Catalog server is best.

To hardcode a DC use the Domin controller connection settings on the Configure Directory Partitions tab of the AD MA.

Remove users from groups before deleting the user account

Another pretty obvious one, but I was also being careless on this front.

Even though it might seem perfectly reasonable to delete a non-existant user from a group, all AD will see is that you have explicity requested an operation involving something it can’t find.

In this implemetation, I disable users for a week before they are actually deleted. I now make sure that they are removed from all groups as soon as they are disabled.

I will write another post soon on the disable-delete methodology.

DC Logging Levels

Finally, if you are still having problems and need to get more information about why AD is rejecting an export, try increasing the logging levels on the DC as per this KB:

This is particularly useful in group population. My usual method uses relational DN attributes the whole way through, from importing CS to metaverse to exporting CS. To construct the rDNs, each CS must hold objects representing every possible group member. Get a few groups with 10,000+ members, and you may soon have hundreds of thousands of objects in your CS … and I won’t need to tell you what that does to your sync times and your transaction log.

So, it turns out, you can at least reduce the pain for one half of the equation.

Now the big hurdle here is that you need to get hold of those DN strings in the first place, and have confidence that they are correct. I overcome with another SQL MA that exports the DN to a table. I then have some SQL trickery going on in the background to generate a table of member lists with the ready-made DNs.

This has been particularly useful in a multi-forest implementation where I needed to populate some groups with Foreign Security Objects from another forest. The FSO takes the form of a DN which I constructed from the foreign SSID, and then by using this method I was able to put them into groups.

Now according to Joe there is another half to this equation where you can also effectively output a group composed of string DN members to AD – however you can’t use the native AD MA, you have to use an XMA and LDIF files. This method has several notable benefits:

• LDIF allows you to add and remove individual group members – you don’t have to repopulate the entire group,
• As you’re no longer dealing with reference DN attributes in the exporting MA, you won’t need all those thousands of placeholder objects representing the members, which leads directly to
• Drastically reduced sync times and goodbye to nightmare transaction log scenarios!

However I haven’t had an opportunity to try this out yet. If and when I do I shall return to the subject with more details.

Group population is not the simplest thing to automate, however it is often a time-consuming manual task, and something high up on the priority list for an ILM project. Here are a few points which may help you on your way.

Members are Reference DN values

Groups are populated with links to the member objects, not a text list of names. To manage group memberships in ILM all involved objects must be present in ILM.

So, to put this plainly, if you’re trying to manage a particular group in AD then ILM must know about all its members. It is not possibly to partially manage a group.

You can only populate member and not memberOf

You can’t write to the “memberOf” attribute on user objects. It is something called a “backlinked” attribute, and AD is in charge of maintaining it.

You can, however, write to the “member” attribute of group objects, and this is the way you have to do it.

So it is not possible to manage group memberships by only considering the person (or user or contact) object – you need to manage the group objects as well.

You can’t modify reference DN attributes in extension code

ILM won’t let you write advanced flow rules for reference DN attributes – all you can do is flow them direct from one connector space, via the metaverse, to another.

(Actually I’ve never quite understood why this is, but there you go, we have to live with it.)

To emphasise the point: you must generate your membership lists outside of ILM, and then sync them directly through ILM.

When Dynamic Groups are not enough

Dynamic groups are those ones you want to change based on members’ attributes. Perhaps the group should contain everyone in a particular department, or a building, or with the same manager.

Exchange 2003 brought us dynamic groups – but only for distribution lists, and not security. Pathetic.

Besides, you’re most likely going to need some manually populated groups as well – not everything can be worked out from attribute values. You may also want some groups where most of the members are dynamic, and a couple which are static.

If you’re using SunOne LDAP you can do all this natively… but with AD the membership of all security groups are static, and you need something else to help automate things.

Generate the members in SQL

Here’s how you might generate the membership lists in SQL:

1. Generate dynamic group memberships in a view by directly querying the mms_metaverse table (sample queries to follow in another post).
2. Maintain another table for manually added group memberships (perhaps with a web front-end to manage them; groups can appear in both tables).
3. Concatenate the table and view together.
4. Import using the multivalue function of the SQL MA.

For more explanation on how to configure the tables to import group memberships see this post.

Use Delta tables

You may quickly find that full imports from multivalued tables are too slow – for this reason it is essential that you use delta imports, ie., only import changes.

The basic method is as follows:

1. Snapshot your import table/view;
2. Do a Full import;
3. Next time, take a new snapshot and compare it to the last one to make a Delta Table;
4. Do a Delta Import;
5. Once the Delta Import has completed successfully, clear out the Delta Table;
6. Repeat steps 3-5 ad nauseum.

There is (naturally) a fair bit more to it when you start bringing multivalued attributes into the mix. I’ve written a few posts on the subject in the past, and the best place to start is with this one.

In Summary

I once set up a system that had 6,000 groups and 40,000 users. The group memberships changed continuously – particularly the self-subscriber ones that were updated through the user portal. For efficiency, I separated the multivalued and single valued attributes into seperate MAs, and the multivalued Full Import still took about 5 hours. But by running regular delta imports (every 15 minutes) the list of changes each time was short, and the imports took only a matter of moments.

So while group population and synchronisation with ILM is fiddly, and does use a number of advanced techniques, it is certainly possible to achieve a result that is both effective and efficient.

One thing I will concede about Multivalue SQL MAs is they can be monstrously slow for large data sets. Perhaps Group Populator is quicker? Even if it is though, I find the Multivalue method so simple and flexible. Group Populator restricts you to creating groups with rules (like everyone with an office address that contains “Building A” goes into the “Building A” group). A multivalue table gives you complete freedom over who goes into what group, and it can be used for other multivalue attributes as well!

So enough waffle – lets get down to how it works.

A Multivalue SQL MA needs two tables:

1. Table One lists all the objects with the minimum information required to join them to their metaverse counterpart. Probably all you will need here is an object type and an identifying attribute.
2. Table Two links the objects to values, with multiple values possible for one object. It must include an Attribute Name column – but you can include different Attributes in the one table.

Take an AD/Exchange example – configuring Distribution Groups. Not only must these groups have members, but they are also restricted on who can email them. The attributes are member for the membership list and (somewhat obscurely) dlMemSubmitPerms for the restricted list.

My first table I will call ADMV_Objects. It lists all the possible Groups and Users that may be involved.

My second table I will call ADMV_Values. It must include the ObjectIDs exactly as they are in the first table, matched to attribute values.

It is now just a matter of configuring my SQL MA.

On the Connect to Database page I put the Objects table in as the primary table, and the Values table in as the Multivalue table.

Then on the Configure Columns page I click the Multi-value button to open the Multi-value settings page.  Here I get to tell MIIS:

• where to look for the the attribute names;
• where to look for the values – unhelpfully referred to as “String attribute column”, but values is what they mean; and
• which multivalue attributes to expect.

Finally set your Join rules so MIIS knows where to flow the attributes and you’re away!