I’m working on a FIM Sync service that I’ve upgraded from ILM by the database transfer method. I had to recreate one MA completely, and re-import and re-join all it’s objects.

I was running some csexports against a few of the MAs to get info about pending exports and noticed this:

• csexport against the upgraded MAs was extremely slow, but
• csexport against the new MA was super fast.

The MAs all have roughly similar numbers of objects (15-20k) and the number of pending exports for each MA was in the low hundreds.

So why the big difference?

My first theory was that the data for the upgraded MAs was scattered around in the DB, while the re-imported MA had nicely consecutive rows. So I deleted and re-imported the CS for one of the upgraded MAs. Result: csexport still slow.

I wondered if perhaps the upgraded MAs had some structural hangover from ILM that slowed them down – but nothing jumps out at me in the database, and when I ran a SQL trace while running the csexports I could see the same stored procedure being called for all of them.

So I’m still stumped. I suppose another good test would be to completely delete and re-create one of the MAs – but I’m not going to slow this project down just to satisfy my curiosity.

If anyone has any ideas about what’s going on I’d be glad to hear!

Note while I’m talking DirSync here this method will work for MIIS, ILM and FIM Management Agents.

Export the MA

DirSync is just ILM and you can get into the Identity Manager client by running:

C:\Program Files\Microsoft Online Directory Sync\SYNCBUS\UIShell\miisclient.exe

Note: If you get an error about not having rights then add your account into the MIISAdmins group on the DirSync server then re-login.

Open the “Management Agents” page, click on the “SourceAD” MA to highlight it, then choose “Export Management Agent” from the Actions menu., and save the XML file.

Parse the XML file in Powershell

Now I go into powershell and load the content of the xml file into a variable:

PS C:\scripts> [xml]$xml = get-content .\dirsync-sourceAD.xml

I can see all the import attribute flows like this:

PS C:\scripts> $xml."export-ma"."mv-data"."import-attribute-flow"."import-flow-set"

mv-object-type                                                                              import-flows
--------------                                                                              ------------
group                                                                                       {import-flows, import-flows, import-flows, import-flows...}
contact                                                                                     {import-flows, import-flows, import-flows, import-flows...}
person                                                                                      {import-flows, import-flows, import-flows, import-flows...}

Right now I’m only interested in the “person” attribute flows:

PS C:\scripts> $xml."export-ma"."mv-data"."import-attribute-flow"."import-flow-set"[2]

mv-object-type                                                                              import-flows
--------------                                                                              ------------
person                                                                                      {import-flows, import-flows, import-flows, import-flows...}

This next one lists the defined attribute flows. There’s a lot more but I’m just showing the top few.

PS C:\scripts> $xml."export-ma"."mv-data"."import-attribute-flow"."import-flow-set"[2]."import-flows"

mv-attribute                                                  type                                                         import-flow
------------                                                  ----                                                         -----------
assistant                                                     ranked                                                       {import-flow, import-flow}
company                                                       ranked                                                       {import-flow, import-flow}
department                                                    ranked                                                       {import-flow, import-flow}
displayName                                                   ranked                                                       {import-flow, import-flow}
facsimileTelephoneNumber                                      ranked                                                       {import-flow, import-flow}
givenName                                                     ranked                                                       {import-flow, import-flow}

It might look as though the mv-attribute is what I want, but actually that’s an internal sync service attribute, and not the Active Directory attribute. I need to go a little deeper into the xml to get that.

First I’ll define a new variable to simplify my xml path:

PS C:\scripts> $importflows = $xml."export-ma"."mv-data"."import-attribute-flow"."import-flow-set"[2]."import-flows"

Then all I have to do is loop through picking out the “src-attribute”. The results are listed below:

PS C:\scripts> foreach ($iaf in $importflows) {$iaf."import-flow"[0]."direct-mapping"."src-attribute"}
assistant
company
department
displayName
facsimileTelephoneNumber
givenName
homePhone
initials
manager
mobile
physicalDeliveryOfficeName
postalCode
telephoneNumber
title
mailNickname
ipPhone
middleName
otherFacsimileTelephoneNumber
otherHomePhone
otherIpPhone
otherMobile
otherPager
otherTelephone
pager
countryCode
description
info
streetAddress
wWWHomePage
url
extensionAttribute11
extensionAttribute12
extensionAttribute13
extensionAttribute14
extensionAttribute15
mail
cn
l
co
sn
st
postOfficeBox
msRTCSIP-UserEnabled
legacyExchangeDN
msExchHideFromAddressLists
msExchMailboxGuid
msExchAssistantName
msDS-HABSeniorityIndex
msDS-PhoneticDisplayName
msExchArchiveGUID
msExchBypassModerationFromDLMembersLink
msExchBypassModerationLink
msExchEnableModeration
msExchImmutableId
msExchModeratedByLink
msExchModerationFlags
msExchRecipientDisplayType
msExchResourceCapacity
preferredLanguage
publicDelegates
telephoneAssistant
msExchResourceDisplay
thumbnailPhoto
msExchBlockedSendersHash
msExchSafeRecipientsHash
msExchSafeSendersHash
extensionAttribute1
extensionAttribute10
extensionAttribute2
extensionAttribute3
extensionAttribute4
extensionAttribute5
extensionAttribute6
extensionAttribute7
extensionAttribute8
extensionAttribute9
msExchResourceMetaData
msExchResourceSearchProperties
msRTCSIP-Line
msRTCSIP-PrimaryUserAddress
targetAddress
msExchSenderHintTranslations
msExchArchiveName
msRTCSIP-DeploymentLocator
msExchRemoteRecipientType
msExchLitigationHoldDate
msExchLitigationHoldOwner
msExchRecipientTypeDetails
msExchRetentionComment
msExchRetentionURL
msExchAuditAdmin
msExchAuditDelegate
msExchAuditDelegateAdmin
msExchBypassAudit
msExchDelegateListLink
msExchELCExpirySuspensionEnd
msExchELCExpirySuspensionStart
msExchELCMailboxFlags
msExchMailboxAuditEnable
msExchMailboxAuditLogAgeLimit
msRTCSIP-OptionFlags
msExchUsageLocation

____________________________________________________________________________________________________________________________________________________________________________________

Luckily powershell has some great XML parsing capability, so here’s a little script I wrote which takes an XML file created by csexport, and produces a CSV file more suitable for opening in Excel. Note that the script only supports single-valued attributes – you can modify it yourself if you need multi-values.

#
# parse-csexport.ps1
#

# Change the following list to get different attributes. The first four are available for all connector spaces.
$csvcolumns = @("dn","connector-type","connector-state","mv-guid","emailAddress","userName","sn","givenName","title","personalTitle")

$csvfile = "csexport.csv"
$csexportfile = "csexport.xml"

[xml]$csexport = get-content $csexportfile

if (Test-Path $csvfile) {Remove-Item -Path $csvfile}
foreach ($csvcol in $csvcolumns) {
  $csvheader = $csvheader + ";" + $csvcol
}
Add-Content $csvfile $csvheader

foreach ($csobj in $csexport."cs-objects"."cs-object") {
  $csobjhash = @{}
  $csobjhash.Add("dn",$csobj."cs-dn")
  # Disconnectors
  if ($csobj.connector -eq "0") {
    $csobjhash.Add("connector-type","disconnector")
    $csobjhash.Add("connector-state",$csobj."connector-state")
    $csobjhash.Add("mv-guid","")
    foreach ($attr in $csobj."unapplied-export-hologram".entry.attr) {
      if ($attr.multivalued -eq "false") {
        $csobjhash.Add($attr.name,$attr.value)
      }
    }
  }
  # Connectors
  else {
    $csobjhash.Add("connector-type","connector")
    $csobjhash.Add("connector-state",$csobj."connector-state")
    $csobjhash.Add("mv-guid",$csobj."mv-link"."#text")
    foreach ($attr in $csobj."synchronized-hologram".entry.attr) {
      if ($attr.multivalued -eq "false") {
        $csobjhash.Add($attr.name,$attr.value)
      }
    }
  }
  $csvline = ""
  foreach ($csvcol in $csvcolumns) {
    if ($csobjhash.Contains($csvcol)) {
      if ($csvline -eq "") {$csvline = $csobjhash.Item($csvcol) }
      else {$csvline = $csvline + ";" + $csobjhash.Item($csvcol) }
    }
    else {$csvline = $csvline + ";"}
  }
  $csvline
  Add-Content $csvfile $csvline
}

One case involves “contract” records from an HR database – a person who had moved around the company may have several. The second is an LDAP data source for a univeristy where an inidvidual may have both staff and student profiles simultaneously, and as a student, may have multiple active “enrollments”.

In an ideal situation

There is a key best practise at stake here, which I believe becomes even more significant when you put the FIM Portal into the mix:

Each individual should only be represented once in the Metaverse.

In an ideal situation you will project each person only once from your source data. Changes to contract, enrollment etc would be treated simply as attribute updates to the core Metaverse object, which remains the same.

If your source data lists people multiple times, as in my examples above, you might do some automated manipulation prior to presenting the data to the Sync Service – selecting the best representation of each person and merging data where appropriate. Then you can have a nice one-to-one relationship between your projectoin source and your Metaverse.

But what if you need access to those individual contracts?

This was my problem. In the case of the HR database contract records, the local IT only had access to the contract number. For them the important information related to the person’s job at that site. But for us, centrally, looking to automate moves and changes across the organisation, we needed a way to link those seperate contract numbers to one individual.

At first, and against my better judgement, we started off projecting the contracts into the Metaverse as seperate people. I was assured that, when people moved jobs, all their old accounts were deleted including their mailbox. Pretty quickly this turned out to be not the case at all, and I started with the messy gymnastics needed to reconnect existing accounts to the newer better contract in the Metaverse, without actually deleting anything.

Step One in solving data source MPD: Split the data into “Persons” and “Contracts”

When it became clear that a redesign was necessary, I started by splitting the data into “persons” and “contracts”. In fact for the university this was already the case, but with the HR database I had to use some SQL queries in an SSIS package to split the data into “objectType=person” and “objectType=contract” rows. Clearly a unique identifier is needed for both persons and contracts.

My persons have the following basic information:

• objectType (=person)
• personID
• Data that should be independant of the contracts, like name, gender, date of birth.

My contracts must include the personID so I can join them correctly:

• objectType (=contract)
• contractID
• personID
• Data that is specific to the contract, like department, job title, start and end dates.

Create Seperate MAs for Persons and Contracts

Once the data was seperated I created different MAs for persons and contracts. The aim is to project the person only once into the metaverse, and then join the changeable contract records from the second MA.

But what about the Import Flows?

The big problem here is that you want to flow data into the metaverse from the Contracts MA. If multiple contracts are joined you’ll get the ambiguous-import-flow-from-multiple-connectors error. Besides which you probably don’t want to be importing from multiple contracts anyway. In my case (both cases in fact) I had to try and choose the “best” contract and then just flow the data from that one. As the “best” could change I have to re-evaluate the selection each time.

Why not just join one contract at a time?

The cleanest setup would be to only have one contract joined to the Metaverse object. The IAF rules are then very simple.

But, unless you can block all except one contract per person using Connector Filters, you may find this tricky to achieve.

Partly the problem is the way the Sync Service does its sync’ing – that is, it works on one connector space object at a time. Additionally, the only other connector space objects accessible during this sync operation are those that can be found by following joins.

So, while sync’ing this connector space object we can only know about other connector space objects where a direct join relationship exists.

This essentially means I can only figure out if the current CS object is really the best choice if all possible contracts are joined to the Metaverse person at the same time.

Blocking Import Flows

So here’s where I ended up.

1. I let the multiple contracts be joined to the Metaverse person, but I only allow import flows from one of the contracts.
2. I write the “preferred contract” number into an attribute on the person object.
3. When sync’ing a CS object I compare it to the current “preferred contract”. If this one is better I replace the contract number on the Metaverse object.
4. All the import flows (which must all be Advanced rules) check if this CS object is the preferred contract before allowing (or skipping) the flow.

While my IAF rules are now more complex, there have been definite advantages to this approach. The non-preferred contracts are still accessible from the Metaverse object and this has allowed us to more easily identify errors in the source data. Sometimes we have forced a particular contract to be preferred by disconnecting the currently preferred contract and syncing another one.

There is also the benefit of having some information as opposed to none. If I was, for example, blocking all expired contracts from entering the connector space in the first place I would be losing valuable deprovisioning information. If the best pick contract for a person is an expired one, I still want to see it.

Flow Blocking Method 1: The Double Sync

I have tried out two different ways of blocking IAFs. The first is simpler to implement but the downside is you have to Sync twice. Because the evaluation of the contract is done as part of the “import_contractID” rule, and this might happen at any point in the set of flow rules, to ensure all IAFs run for the new preferred contract you have to sync twice.

Case "import_contractID"
  If mventry("contractID").Value <> csentry("contractID").Value
      Whatever comparisons you need to do to see if this contract is preferable to the values
      already imported to the Metaverse object. If it's better, replace the contractID.
  End If

Case "import_otherAttribute"
  'Replicates a Direct IAF
  If mventry("contractID").IsPresent AndAlso mventry("contractID").Value = csentry("contractID").Value
    If csentry("otherAttribute").IsPresent Then
      mventry("otherAttribute").Value = csentry("otherAttribute").Value
    Else
      mventry("otherAttribute").Delete()
    End If
  End If

Flow Blocking Method 2: The First Touch

In an effort to avoid the double-sync I have started using another method where I run my comparison tests above the Case statement, and then set a Utils.TransactionProperties to show the test has run.

Utils.TransactionProperties is the way to pass values between different bits of code run as part of the same object’s Sync operation.

While I now only need to sync once, I still don’t know which IAF rule will get called first, so I have to pass all comparison CS attributes to each IAF rule.

Here’s what the MapAttributesForImport Sub looks like now:

Public Sub MapAttributesForImport(ByVal FlowRuleName As String, ByVal csentry As CSEntry, ByVal mventry As MVEntry) Implements IMASynchronization.MapAttributesForImport

  '' Evaluate this CS object to see if it is a better source for attribute flows to the joined Metaverse object.
  '' Using the Utils.TransactionProperties flag will ensure this test is made only at the start of sync'ing the CS object.

  If Not Utils.TransactionProperties.Contains("FlowEnabled") Or Not mventry("contractID").IsPresent Then

    Dim BestMatch As Boolean = False
    If mventry("contractID").IsPresent Then
      If mventry("contractID").Value = csentry("contractID").Value Then
        BestMatch = True
      Else
          Comparison tests - set BestMatch=True if this is a better match.
      End If

      If BestMatch Then
        Utils.TransactionProperties("FlowEnabled") = csentry("contractID").Value
          ''Note: I set this to the contractID to avoid unexpected behaviour following a disconnection.
      Else
        Utils.TransactionProperties("FlowEnabled") = "false"
      End If

    End If

    '' Do not continue with flow rules if this CS object has a different id to the best match.
    If Not Utils.TransactionProperties("FlowEnabled").Equals(csentry("contractID").Value) Then
      Exit Sub
    End If

    Select Case FlowRuleName

      Case "Import_contractID"
        mventry("contractID").Value = csentry("contractID").Value

      Case "Import_otherAttribute"
        If csentry("otherAttribute").IsPresent Then
          mventry("otherAttribute").Value = csentry("otherAttribute").Value
        Else
          mventry("otherAttribute").Delete()
        End If

      ....

  End Select
End Sub

Importing a reference

There is one final advantage I can extract out of seperating my data into persons and contracts – I can, if I need to, also import “contract” objects into the Metaverse and, with the addition of a multivalue table (for database MAs) I can construct a reference attribute that directly links the person to their contract ojects. This would be very useful if I wanted a person’s contracts to be accessible in the FIM Portal.

I was using the Metaverse GUID as an anchor for provisioning to a simple SQL MA like so:

csentry("mvguid").StringValue = mventry.ObjectID.ToString

However when I tried to do a direct join rule on csentry:mvguid = mventry:<object-id> I got the error cannot-parse-object-id.

In fact the operation above had removed the brackets from the Guid string and I had to add them back in using an Advanced Join Rule:

values.Add("{" & csentry("mvguid").StringValue & "}")

Of course the other way would have been to add the brackets back in when setting the mvguid value on the csentry in the first place, but I’d already provisioned quite a few objects. I’ll do it like that next time.

I’ve been working on a couple of projects this year which are big and complicated enough to warrant a proper lookup file, so I’ve finally had to get in and figure out how to parse XML in .NET. As usual this post is just about things I’ve been doing and I’m not trying to say this is the best way to go. You will have to adapt to your own requirements.    

The Structure of the XML File

The general outline of my XML file is to have a section for Global settings, and then individual sections for each MA. Note the “tag” for each MA, which I’ll be using in the code.  The specifc values you include for each MA are entirely up to your requirements.

<SyncParams>
  <Global>
    <SyncService>
      <ServerName>mysyncserver.mydomain.com</ServerName>
      <IsDev>False</IsDev>
    </SyncService>
    <SQL>
      <Server>mysqlserver.mydomain.com</Server>
    </SQL>
  </Global>
  <ManagementAgents>
    <MA name='FIM Portal' tag='MAName_FIM'>
      <maType>FIM</maType>
      <MVobjectType>person</MVobjectType>
      <MVobjectType>group</MVobjectType>
      <MVobjectSource>group</MVobjectSource>
    </MA>
    <MA name='SQL HR Employees' tag='MAName_HR'>
      <maType>SQL</maType>
      <MVobjectType>person</MVobjectType>
      <MVobjectSource>person</MVobjectSource>
    </MA>
    <MA name='Notes' tag='MAName_Notes'>
      <maType>NOTES</maType>
      <MVobjectType>person</MVobjectType>
      <MVobjectSource>person</MVobjectSource>
    </MA>
    <MA name='AD mydomain' tag='MAName_AD'>
      <maType>AD</maType>
      <provEnabled>True</provEnabled>
      <MVobjectType>person</MVobjectType>
      <MVobjectType>group</MVobjectType>
      <OU_USERS_INTERNAL>OU=Internal,OU=Users,OU=MyOrg,DC=mydomain,DC=com</OU_USERS_INTERNAL>
      <OU_USERS_EXTERNAL>OU=External,OU=Users,OU=MyOrg,DC=mydomain,DC=com</OU_USERS_EXTERNAL>
      <OU_USERS_GENERIC>OU=Generic,OU=Users,OU=MyOrg,DC=mydomain,DC=com</OU_USERS_GENERIC>
      <OU_USERS_RESIGNED>OU=Resigned,OU=Users,OU=MyOrg,DC=mydomain,DC=com</OU_USERS_RESIGNED>
      <OU_GROUPS>OU=Groups,OU=MyOrg,DC=mydomain,DC=com</OU_GROUPS>
      <OU_CONTACTS>OU=Contacts,OU=MyOrg,DC=mydomain,DC=com</OU_CONTACTS>
    </MA>
  </ManagementAgents>
</SyncParams>

Initialization

You can use the Initialization section of your extension code to open and load the XML lookup file. Note that you’ll have to do this in each extension project where you want to access parameters from the file. Your code will also need a reference to System.Xml.    

    Dim xmlParams As Xml.XmlDocument

    Public Sub Initialize() Implements IMVSynchronization.Initialize
        xmlParams = New Xml.XmlDocument
        Dim xmlPath As String = Utils.ExtensionsDirectory & "\SyncParams.xml"

        Try
            xmlParams.Load(xmlPath)
        Catch ex As Exception
            Throw New ExtensionException("Failed while opening XML parameter file. " & ex.Message)
        End Try
    End Sub    

Accessing the Global parameters

TheGlobal parameters are easy to get to because there’s only one node for them. So to retrieve the server name from the XML file above I’d just do this:  

    xmlParams.SelectSingleNode("//Global/SyncService/ServerName").InnerText   

Accessing a specific MA’s Parameters

The next thing you will probably want to do is access the values for a specifc MA. I’m using the “tag” attribute as a way to identitfy the correct node. In this example I retrieve an OU name from the ”AD mydomain” parameters.   

    xmlParams.SelectSingleNode("//ManagementAgents/MA[@tag='MAName_AD']/OU_USERS_EXTERNAL").InnerText

Creating a Hash Table of MA Names

The other thing that can be useful is to make a hash table of MA Names for easy reference. First you need to define a global variable:  

    Dim MA_Names As New Collection

Then you add something like this to the Initialization sub:  

    For Each xmlNode In xmlParams.SelectNodes("//ManagementAgents/MA")
        MA_Names.Add(xmlNode.Attributes("name").Value, xmlNode.Attributes("tag").Value)
    Next

Now in the rest of the project, when you want to use an MA’s name, you only need refer to it using its tag:  

    MA_Names.Item("MAName_AD").ToString 

Selectively disabling provisioning

Another thing you might like to do, especially if you have a number of MAs that you are provisioning to, is to be able to selectively disable the provisioning code for one of them just by using the XML file.

Obviously you’ll have to modify your provisioning code to check the value of the flag. In the example above I would need to test for the value of “provEnabled” before calling the provisioning code for the “AD mydomain” MA:

    If xmlParams.SelectSingleNode("//ManagementAgents/MA[@tag='MAName_AD']/provEnabled").InnerText = "True" then
        Provision_AD(mventry)
    End If

Conclusion

Of course all of this takes more effort in the coding but the code will be more robust if it avoids hard-coded values, and it will be easier for others to make simple changes.

The sticking problem is the objectClass. Typically objects in LDAP will have more than one objectClass, and the only way to set them in MIIS/ILM is in the metaverse extension code. It is no different with FIM. There is no way to set objectClass from a codeless sync rule, and you can’t set it in an export attribute flow. 

So a Metaverse extension is still the way to go. Here is an example of a provisioning rule for OpenLDAP. I am creating a posixAccount, mostly as a way of showing how to add the dependant objectClasses. Your objectClasses will no doubt vary.

I have included a couple of functions that I am using to generate a password hash and to find the next uidNumber. Note for the latter function the Terminate sub is also implicated.

Imports Microsoft.MetadirectoryServices
Imports System.DirectoryServices
Imports System.Text
Imports System.IO
Imports System.Security.Cryptography

Public Class MVExtensionObject
    Implements IMVSynchronization

    '' It is a better practise to put these constants in a lookup file
    Const OL_SERVER As String = "myldapserver.mydomain.com"
    Const OL_ROOT As String = "dc=myldap,dc=mydomain,dc=com"
    Const OL_OU_USERS As String = "ou=users,dc=myldap,dc=mydomain,dc=com"
    Const OL_UIDFILE As String = "C:\Program Files\Microsoft Forefront Identity Manager\2010\Synchronization Service\MaData\openLDAP\uidNumber.txt"
    Const OL_READ_USER As String = "cn=fimread,ou=users,dc=myldap,dc=mydomain,dc=com"
    Const OL_READ_PW As String = "password"
    Const OL_DEFAULT_GROUP As String = "1000"

    Public Sub Initialize() Implements IMVSynchronization.Initialize
        ' TODO: Add initialization code here
    End Sub

    Public Sub Terminate() Implements IMVSynchronization.Terminate
    '' Deleting this file forces a new LDAP search the next time, allowing for accounts which may have been created manually.
        If File.Exists(OL_UIDFILE) Then
            File.Delete(OL_UIDFILE)
        End If
    End Sub

    Public Sub Provision(ByVal mventry As MVEntry) Implements IMVSynchronization.Provision
        Select Case mventry.ObjectType
            Case "person"
                Provision_OpenLDAPPerson(mventry)
        End Select
    End Sub

    Sub Provision_OpenLDAPPerson(ByVal mventry As MVEntry)
        Dim ShouldExist As Boolean = False
        Dim DoesExist As Boolean
        Dim CSEntry As CSEntry
        Dim OLMA As ConnectedMA = mventry.ConnectedMAs("openLDAP")
        Dim expectedDN As ReferenceValue
        Dim Container As String

        '' Should the account exist? (Add more logic here to decide)
        ShouldExist = True

        '' Does the account already exist?
        If OLMA.Connectors.Count = 0 Then
            DoesExist = False
        Else
            DoesExist = True
        End If

        '' Generate the expected DN for the user - to use in creating or moving
        If mventry("accountName").IsPresent And ShouldExist Then
            Dim RDN As String = "cn=" & mventry("accountName").StringValue.ToLower
            expectedDN = OLMA.EscapeDNComponent(RDN).Concat(OL_OU_USERS)
        End If

        '' Take action based on values of ShouldExist and DoesExist
        If ShouldExist And DoesExist Then
            '' Check if rename needed
            If csentry.DN.ToString.ToLower <> expectedDN.ToString.ToLower Then
                csentry.DN = expectedDN
            End If

        ElseIf ShouldExist And Not DoesExist Then
            '' Create Account

            ' Set objectClasses
            Dim oc As ValueCollection
            oc = Utils.ValueCollection("inetOrgPerson")
            oc.Add("person")
            oc.Add("organizationalPerson")
            oc.Add("inetOrgPerson")
            oc.Add("posixAccount")
            oc.Add("shadowAccount")
            oc.Add("simpleSecurityObject")
            CSEntry = OLMA.Connectors.StartNewConnector("posixAccount", oc)
            CSEntry.DN = expectedDN

            ' Find next available uidNumber
            CSEntry("uidNumber").Value = = NextUidNumber(OL_SERVER, OL_ROOT).ToString

            ' Set initial password
            CSEntry("userPassword").Value = "{MD5}" & GenerateHash("INitial001")

            ' The following could also be done as export flow rules
            CSEntry("gidNumber").Value = OL_DEFAULT_GROUP
            CSEntry("sn").Value = mventry("lastName").Value
            CSEntry("givenName").Value = mventry("firstName").Value
            CSEntry("cn").Value = mventry("accountName").Value.ToLower
            CSEntry("uid").Value = mventry("accountName").Value.ToLower
            CSEntry("displayName").Value = mventry("firstName").StringValue & " " & mventry("lastName").StringValue.ToUpper

            CSEntry.CommitNewConnector()

        ElseIf Not ShouldExist And DoesExist Then
            '' Delete
            CSEntry = OLMA.Connectors.ByIndex(0)
            CSEntry.Deprovision()

        End If
    End Sub

    Public Function NextUidNumber(ByVal ldapServerName As String, ByVal searchRoot As String) As Integer
    '' The first time the function runs it finds the highest uidNumber in LDAP and stores it in a text file.
    '' On subsequent runs it retrieves the last uidNumber straight from the text file. This allows for sync runs
    '' where multiple accounts are created.
    '' The text file is deleted as part of the Terminate routine.

        Dim oSearcher As New DirectorySearcher
        Dim oResults As SearchResultCollection
        Dim oResult As SearchResult
        Dim lastuid As Integer = 0

        If File.Exists(UIDFILE) Then
            Dim fileReader As New StreamReader(UIDFILE)
            lastuid = CInt(fileReader.ReadLine)
            fileReader.Close()
        Else
            Try
                With oSearcher
                    .SearchRoot = New DirectoryEntry("LDAP://" & ldapServerName & "/" & searchRoot, _
                                                     "cn=root,dc=ldap,dc=eesp,dc=ch", "hesso_2010", AuthenticationTypes.FastBind)
                    .PropertiesToLoad.Add("uidNumber")
                    .Filter = "uidNumber=*"
                    oResults = .FindAll()
                End With

                For Each oResult In oResults
                    If oResult.GetDirectoryEntry().Properties("uidNumber").Value > lastuid Then
                        lastuid = oResult.GetDirectoryEntry().Properties("uidNumber").Value
                    End If
                Next

            Catch e As Exception
                Throw New UnexpectedDataException(e.Message)
                Return -1
            End Try
        End If

        Dim fileWriter As New StreamWriter(UIDFILE, False)
        fileWriter.WriteLine(lastuid + 1)
        fileWriter.Close()

        Return lastuid + 1
    End Function

    Private Function GenerateHash(ByVal SourceText As String) As String
        ' Returns the MD5 hash of the SourceText string.
        Dim Ue As New UTF7Encoding()
        Dim ByteSourceText() As Byte = Ue.GetBytes(SourceText)
        Dim Md5 As New MD5CryptoServiceProvider()
        Dim ByteHash() As Byte = Md5.ComputeHash(ByteSourceText)
        Return Convert.ToBase64String(ByteHash)
    End Function

End Class

The background to this is a BPOS (Microsoft cloud services) project I’ve been working on. BPOS doesn’t actually give you any way to disable an account while keeping the mailbox, so I wanted to be able to reset a BPOS password as part of the deprovisioning cycle. I also wanted to do this quickly and efficiently. BPOS gives you two methods to change a password: through the admin console and through powershell. Obviously powershell is the way to go for automation. 

At the same time I was thinking of other times where I may want to fire off a powershell cmdlet – such as when initailly activating the account, when adding BPOS services, changing a mailbox quota or when adding mailbox trustees. With this method I think I have found a nicely generalised solution. My XMA will run any powershell cmdlet and all I need to do is provision these “command” objects, based on changes in the metaverse. 

Some Warnings…

You can run into problems with this approach where FIM Sync unprovisions the unexported command object because it thinks you don’t need it any more. I found this when I tied the provisioning of the BPOS command object to the moving of an AD account, but then exported the AD MA first. The Sync service thought the command was then not needed and removed. An extra status flag should sort this out.

FIM Portal Workflow may be better

The other comment is that this is a Sync Service based way of approaching this problem. A better bet, if you’re using the FIM Portal, is probably to fire the powershell cmdlets directly from the Portal as part of a workflow. Unfortunately, at the moment, this means a custom workflow, which I haven’t really gotten my head around yet.

About the Export Files

At the moment the XMA is writing a new delta CSV file, and adding to the existing full CSV file, during each export. At the moment I only add successful runs to the full.csv file – this means I don’t have to mess about with updating existing entries – however it can lead to weird behaviour if you had some failures and follow it by a Full Import. For safety I have the “Delta Import” step as part of my Export run profile, so it always runs straight after the export. The better way would be to use SQL tables for this exported info.

Steps

Preparing the XMA

The server prerequisites are covered in this post. You should also review this other post about creating a remote powershell XMA. The big difference with this MA is that it is Call-based, giving me the ExportEntry Sub which runs against each individual export.

I still need a CSV file to feed to the MA creation wizard for its schema. My CSV, very simply, contains the following:

     cmdlet,arguments,date,time,identity,status

The “identity” here is an identifier for the BPOS object that I’m going to run the command against, and is not actually a unique identifier for the command itself. Because I don’t have a single unique attribute I use the following combination as the anchor: cmdlet + identity + date.

This has the useful side-effect of ensuring only one command object for a particular activity will be configured per day – though if the export of the command were to repeatedly fail you would find a new command object being provisioned the next day. Just something to watch out for.

Retry Mechanism

I also configure one direct Export Attribute Flow:      Constant “success” –> status

This is my feedback mechanism. I force the export of “success” to my command object – but if the running of the command actually failed I write “failed” into the delta import file – so this actually makes a re-export happen. Until the Sync service can export and then re-import a status of “success” it will keep trying.

CS Extension

Here’s the CSExtension. Because this is for BPOS I’ve added the appropriate PSSnapin in the BeginExport sub. You may not need to add any snapins. Or you may just need the standard Exchange ones in which case it looks like you don’t need to add a snapin at all, because you have a nice Exchange-specific shell URI to use. There are plenty of other posts about that so I won’t say more.  
  

Imports Microsoft.MetadirectoryServices
Imports System.Management.Automation
Imports System.Management.Automation.Host
Imports System.Management.Automation.Runspaces
Imports System.IO

Public Class MACallExport
    Implements IMAExtensibleFileImport
    Implements IMAExtensibleCallExport

    Const SHELL_URI As String = "http://schemas.microsoft.com/powershell/Microsoft.Powershell"
    Const MA_FOLDER As String = "C:\Program Files\Microsoft Forefront Identity Manager\2010\Synchronization Service\MaData\BPOS commands\"
    Const DELTA_FILE As String = "delta.csv"
    Const FULL_FILE As String = "full.csv"
    Const HEADER As String = "cmdlet,arguments,date,time,identity,status"

    Dim myRunSpace As Runspace
    Dim bposCred As PSCredential

    Dim fileDelta As StreamWriter
    Dim fileFull As StreamWriter

    Public Sub GenerateImportFile(ByVal filename As String, ByVal connectTo As String, ByVal user As String, ByVal password As String, ByVal configParameters As ConfigParameterCollection, ByVal fullImport As Boolean, ByVal types As TypeDescriptionCollection, ByRef customData As String) Implements IMAExtensibleFileImport.GenerateImportFile
        ' TODO: Remove this throw statement if you implement this method
        Throw New EntryPointNotImplementedException
    End Sub

    Public Sub BeginExport(ByVal connectTo As String, ByVal user As String, ByVal password As String, ByVal configParameters As ConfigParameterCollection, ByVal types As TypeDescriptionCollection) Implements IMAExtensibleCallExport.BeginExport
        ' Start new DELTA file.
        fileDelta = New StreamWriter(MA_FOLDER & DELTA_FILE, False, System.Text.Encoding.Default)
        fileDelta.WriteLine(HEADER)
        fileDelta.Close()

        ' Open existing FULL file
        fileFull = New StreamWriter(MA_FOLDER & FULL_FILE, True, System.Text.Encoding.Default)

        ' Create credential for attaching to remote server
        Dim remotepass As New Security.SecureString
        Dim c As Char
        For Each c In password
            remotepass.AppendChar(c)
        Next
        Dim remoteCred As New PSCredential(user, remotepass)

        ' Create credential for attaching to BPOS
        ' The configParameters are configured on the Additional Parameters page of the MA.
        Dim bpospass As New Security.SecureString
        For Each c In configParameters("bposPassword").Value
            bpospass.AppendChar(c)
        Next
        bposCred = New PSCredential(configParameters("bposUser").Value, bpospass)

        ' Open remote powershell session
        Dim serverUri As New Uri("http://" & connectTo & ":5985/wsman")
        Dim connectionInfo As New WSManConnectionInfo(serverUri, SHELL_URI, remotecred)
        myRunSpace = RunspaceFactory.CreateRunspace(connectionInfo)
        Dim psException As PSSnapInException = Nothing
        myRunSpace.Open()

        Dim psh As PowerShell = PowerShell.Create()
        psh.Runspace = myRunSpace
        psh.AddCommand("Add-PSSnapin")
        psh.AddParameter("Name", "Microsoft.Exchange.Transporter")
        psh.Invoke()

    End Sub

    Public Sub ExportEntry(ByVal modificationType As ModificationType, ByVal changedAttributes As String(), ByVal csentry As CSEntry) Implements IMAExtensibleCallExport.ExportEntry
        Dim psh As PowerShell = PowerShell.Create()
        Dim psresult As New System.Collections.ObjectModel.Collection(Of PSObject)
        psh.Runspace = myRunSpace

        psh.AddCommand(csentry("cmdlet").StringValue)

        Dim arguments As String = csentry("arguments").StringValue
        If Not arguments.StartsWith(" ") Then arguments = " " & arguments

        Dim parameter As String
        Dim seperator As String() = {" -"}
        For Each parameter In arguments.Split(seperator, StringSplitOptions.None)
            If parameter.Contains(" ") Then
                If parameter.Contains("$true") Then
                    psh.AddParameter(parameter.Split(" ".ToCharArray, 2)(0), True)
                ElseIf parameter.Contains("$false") Then
                    psh.AddParameter(parameter.Split(" ".ToCharArray, 2)(0), False)
                Else
                    psh.AddParameter(parameter.Split(" ".ToCharArray, 2)(0), parameter.Split(" ".ToCharArray, 2)(1))
                End If
            End If
        Next
        psh.AddParameter("Credential", bposCred)

        Dim strLine As String = csentry("cmdlet").StringValue & "," & csentry("arguments").StringValue & "," _
                                & csentry("date").StringValue & "," & csentry("time").StringValue & "," & csentry("identity").StringValue
        fileDelta = New StreamWriter(MA_FOLDER & DELTA_FILE, True, System.Text.Encoding.Default)

        Try
            psresult = psh.Invoke()
        Catch ex As Exception
            fileDelta.WriteLine(strLine & ",error: " & ex.Message)
        End Try

        If psh.Streams.Warning.Count > 0 Then
            fileDelta.WriteLine(strLine & ",warning: " & psh.Streams.Warning.Item(0).Message)
       ElseIf psh.Streams.Error.Count > 0 Then
            fileDelta.WriteLine(strLine & ",error: " & psh.Streams.Error.Item(0).ErrorDetails.Message)
       Else
            fileDelta.WriteLine(strLine & ",success")
            fileFull.WriteLine(strLine & ",success")
        End If
        fileDelta.Close()

        psh.Dispose()
    End Sub

    Public Sub EndExport() Implements IMAExtensibleCallExport.EndExport

        fileFull.Close()
        myRunSpace.Close()

    End Sub

End Class

Provisioning Code

Here’s an example of provisioning a command object from the MVExtension code.

' To change BPOS password, provision a BPOS command object
Dim bposCmd As CSEntry
bposCmd = mventry.ConnectedMAs(MAName_BPOScmd).Connectors.StartNewConnector("command")
bposCmd("cmdlet").StringValue = "Set-MSOnlineUserPassword"
bposCmd("identity").StringValue = mventry("bposIdentity").StringValue
bposCmd("date").StringValue = Now.Date.ToString("d")
bposCmd("time").StringValue = Now.TimeOfDay.ToString
Dim randompassword As String = < your random password generator here >
bposCmd("arguments").StringValue = "-Identity " & mventry("bposIdentity").StringValue & " -Password " & randompassword & " -ChangePasswordOnNextLogon $false"
Try
     bposCmd.CommitNewConnector()
Catch ex As ObjectAlreadyExistsException
End Try

Phase One Joins and Data Matching

The tedious truth is that most IdM projects must begin with a phase of data matching and cleaning. Before you can start to automate management of identities, you need a predictable
data set around which to base your rules. Many organizations today, whether due to changes in IT personnel, company mergers, variable naming conventions or lack of guidance on handling resignations, have existing user account bases that can only be described as a mess.

This document covers some of the methods you can use with ILM to get through that first project phase. Unfortunately there are no magic bullets here – eye-straining, brain-numbing trawls through long lists of unmatched accounts cannot usually be avoided. What you can do is try to extract quality data, and construct your lists as helpfully as possible, so they can be targeted at the people whose eyeballs and brains are most likely to give the best response.

It must be added that this is a very big topic, and the methods you choose will depend on the data you are faced with, and the aims of your project.

Joins

Really, it’s all about joins.

Once existing accounts are joined to their correct data source (such as matching an HR record to an AD account) you can begin to flow updates.

Once you have a clear idea who does and does not have an account in a target directory, you can begin to make provisioning and deprovisioning decisions.

Your eventual aim is a simple Join Rule

When you first start data matching, you will use a lot of different join rules, and you will make joins manually and with CSV files (more on that below). But keep
this in mind: 

Any join made manually, or with extra effort, should be considered temporary. 

There are various situations in ILM which can only be reliably rectified by a clear-out and re-import of a connector space.

Always plan for this.

Ideally, when you re-import a connector space, you will have a single, direct join rule which effortlessly re-joins all your objects. And to achieve this we use… 

Breadcrumbing

Once the join is verified you should export a uniquely identifying attribute, such as an employee number, to the target directory.

After that a simple “employeeID = employeeID” type join rule is all you will need. 

Sometimes you are faced with an import-only system. For either technical or political reasons you are not able to export the breadcrumb attribute.

There are a couple of things you can do: 

1.  Import the DN or other identifying attribute from the target directory into an attribute on the Metaverse object.But be warned, you could still lose these joins if you had to repopulate your Metaverse.
2.  To be on the safe side you should also “save” these joins by exporting an identifying pair somewhere else – for example using a database or text file
   MA, as pictured below.

Understanding Join Rules

There are some key points to note about Join Rules: 

1. Join rules operate on the CS object. It takes one CS object and attempts to find a match among all Metaverse objects of the correct type, even if they are already joined.
2. Components within one rule are AND-ed together – they must all match.
3. Multiple rules are evaluated top down, so put your strongest rules at the top.
4. While you can offer variations on a CS objects attribute using an Advanced Join Rule, you have to find an exact match with an attribute already on a Metaverse
   object. There are no “StartsWith” or “Contains” comparisons.

Resolve Join

At the bottom of the Join Rule configuration you will see a check box “Use rules extension to resolve”.

Here you can link to code you write under the ResolveJoinSearch subroutine in the MA extension.

The Resolve rule is used when ILM finds multiple possible matches in the Metaverve (including already-joined objects).

All the possible matches into the collection rgmventry and your job, in the code, is to check through them, looking for the best possible match.

If your code finds an ideal match you return the index number of the object in imventry, and set the value of ResolveJoinSearch to true.

The following example only joins if a single, unjoined Metaverse object was found. 

Public Function ResolveJoinSearch(ByVal joinCriteriaName As String, ByVal csentry As CSEntry, ByVal rgmventry() As MVEntry, ByRef imventry As Integer, ByRef MVObjectType As String) As Boolean Implements IMASynchronization.ResolveJoinSearch
   Select Case joinCriteriaName
      Case "Resolve_NotYetJoined"
         If rgmventry.Length = 1 AndAlso _
            rgmventry(0).ConnectedMAs("AD").Connectors.Count = 0  Then
            imventry = 0
            Return True
         Else
            Return False
         End If
   End Select
End Function

Advanced Join Rules

A simple join rule directly matches a connector space attribute to a Metaverse attribute:

With an Advanced Join Rule you construct a list of possible values with which to find an exact match in the Metaverse:

Sometimes you can use code rules to make your list of possible matches (eg., presenting a phone number in different formats); other times you have to use long
look-up lists of possible variations (try genealogy websites for name-variation lists).

The following example uses a lookup file of aliases, where each line has the possible variations on a name.

If a match to the first name is found on the connector space object, the whole line is added to the possible values to search for in the Metaverse. 

Elizabeth,Liz,Beth,Betty
David,Dave,Davey
Jerome,Jérôme
… 

Public Class MAExtensionObject
   Implements IMASynchronization
   Dim fileAliases As System.IO.StreamReader
   Dim arrAliases As String()
   Dim i As Integer

   Public Sub Initialize() Implements IMASynchronization.Initialize
      fileAliases = New System.IO.StreamReader("C:\aliases.txt",  System.Text.Encoding.Default)
      i = 0
      While Not fileAliases.EndOfStream
         ReDim Preserve arrAliases(i)
         arrAliases(i) = fileAliases.ReadLine
         i = i + 1
      End While

     fileAliases.Close()
   End Sub

   Public Sub MapAttributesForJoin(ByVal FlowRuleName As String, ByVal csentry As CSEntry, ByRef values As ValueCollection) Implements IMASynchronization.MapAttributesForJoin
      Select FlowRuleName
         Case "Join_aliases"
            Dim aliasList As String
            Dim value As String

            For Each aliasList In arrAliases
               If aliasList.Contains(csentry("givenName").Value) Then
                  For Each value In aliasList.Split(",".ToCharArray)
                     values.Add(value)
                  Next
	      End If
            Next
         End Select
     End Sub
End Class

Using CSV Files

A lot of the difficult account matching will probably be done outside ILM.

It is therefore useful to be able to “export” lists of possible matches, and later, “import” the joins from a CSV file.

Be careful when writing to files from extension code – the DLL doesn’t unload for five minutes after the MA run completes, which means you may have to wait for it
to finish writing to the file. If you’re in a hurry, recompiling the code will force the DLL to finish writing to the file. 

Export Possible Matches to a CSV file

You can use the Resolve rule to export possible matches to a CSV file. The following example resolves the join rule “sn | Direct | lastname”. As a match on the last
name alone is too weak for an immediate join, we just write the possible matches to the text file. 

Public Class MAExtensionObject Implements IMASynchronization
   Dim fileMatches As System.IO.StreamWriter

   Public Sub Initialize() Implements IMASynchronization.Initialize
      fileMatches = New System.IO.StreamWriter("C:\possible matches.txt", System.Text.Encoding.Default)
   End Sub

   Public Sub Terminate() Implements IMASynchronization.Terminate
      fileMatches.Close()
   End Sub

   Public Function ResolveJoinSearch(ByVal joinCriteriaName As String, ByVal csentry As CSEntry, ByVal rgmventry() As MVEntry, ByRef imventry As Integer, ByRef MVObjectType As String) As Boolean Implements IMASynchronization.ResolveJoinSearch
      Select Case joinCriteriaName
         Case "Resolve_Lastname"
            Dim MAName As String = csentry.MA.Name
            Dim mvobject As MVEntry
            Dim cFirstname, mFirstname As String

            If csentry("givenName").IsPresent Then
               cFirstname = csentry("givenName").StringValue
            Else
               cFirstname = "UNKNOWN"
            End If

            If mvobject("firstname").IsPresent Then
               mFirstname = mvobject("firstname").StringValue
            Else
               mFirstname = "UNKNOWN"
            End If

            For Each mvobject In rgmventry
               If mvobject.ConnectedMAs(MAName).Connectors.Count = 0  Then
                  fileMatches.WriteLine(csentry("sn").StringValue & ";" _
     		   & cFirstname & ";" _
     		   & mvobject("lastname").StringValue & ";" _
     		   & mFirstname)
               End If
            Next
            Return False
         End Select
   End Function
End Class

You will need to adapt this code of course. Firstly, you probably want to export a lot more identifying information in your CSV file – department, email address,
dn … whatever helps. Next, it can really help to supplement your possible matches with a probability score. This is where you do a series of tests and add points-
the more points, the higher the chance of the match. For example: 

• Names similar*          +1
• Department the same +1
• City the same           
  +1

*Some tips for testing if names are similar: 

• Strip out all spaces, dashes and hyphens then compare;
• Check if one string is contained in the other (so that “Sally-Anne” gets
  a point for “Sally”);
• Use a function which compares string similarity (search “Soundex” and “Levenshtein
  distance”).

Join from CSV

You can use an Advanced Join Rule to “import” joins from a CSV file.

Firstly, our CSV file must be constructed like this: 

CS_identifier;MV_identifier 

For example, if we are trying to match an AD account against Metaverse objects imported from the HR system, we populate the CSV with the AD DN and the employeeID: 

CN=Fred Bloggs,OU=User,OU=MyOrg,DC=mydomain,DC=com;0012988 

Next we create the Advanced Join Rule which will look up the csobject’s DN in the text file, but use the employeeID to search the Metaverse. 

Note that you can’t actually use the DN in the join rule – but that’s ok, just use any attribute that definitely exists. Eg., 

sAMAccountName | Rules extension – Join_CSV | employeeID 

And now for the code : 

Imports Microsoft.MetadirectoryServices

Public Class MAExtensionObject Implements IMASynchronization
   Dim joins As String()
   Dim i As Integer

   Public Sub Initialize() Implements IMASynchronization.Initialize
      Dim fileJoins As System.IO.StreamReader
      Dim strLine As String
     'Open the csv file and read into an array
      fileJoins = New System.IO.StreamReader("C:\joins.csv", System.Text.Encoding.Default)
      i = 0
      While Not fileJoins.EndOfStream
         ReDim Preserve joins(i)
         joins(i) = fileJoins.ReadLine
         i = i + 1
      End While
      fileJoins.Close()
   End Sub

   Public Sub MapAttributesForJoin(ByVal FlowRuleName As String, ByVal csentry As CSEntry, ByRef values As ValueCollection) Implements IMASynchronization.MapAttributesForJoin
      Select FlowRuleName
         Case "Join_CSV"
            'If the csentry DN is found in the joins array, then
            'use the paired employeeID to search the Metaverse.
            For i = 0 To joins.Length - 1
               If joins(i).Contains(csentry.DN.ToString) Then
                  values.Add(joins(i).Split(";")(1))
               End If
         Next

   End Select
End Sub

Reporting

People will ask you questions like “How many people have you joined in system X but not in Y?”, “How sure are you that the joins are correct?”, “Which department
has the most unidentified accounts?” It’s best to be prepared for these sorts of questions. 

Querying the Metaverse

Once data is in the Metaverse it is a simple matter to access it for reporting, either by exporting it into a reporting table, or by directly querying the underlying
tables (as long as you’re careful to do it when ILM is idle, or else use NOLOCK). 

So consider this: During the data cleaning phase, import all identifying attributes into the Metaverse from all sources. 

For example: You’ve made a join between a user in AD and an HR record.

Under normal operations you would consider HR as the master source for the name, and you would only flow it from there.

You wouldn’t bother importing the name attributes from AD – in fact you’re more likely to be overwriting them with export flow rules.

However, during the data matching phase, you’re probably not ready to start overwriting attributes, and the information about current values can be very important in your
verification and reporting.

Now, if you have a look at the mms_metaverse table in the ILM database, you will see how simple it is to query the progress of your joins, and also to judge
on what criteria the joins were made.

Some example queries… 

/* HR person with no join to AD */

select HR_lastname, HR_firstname, HR_employeeid from mms_metaverse
where AD_dn is null 

 /* HR person with join to AD */

select HR_lastname, HR_firstname, HR_employeeid, AD_lastname,AD_firstname,AD_dn from mms_metaverse
where AD_dn is not null 

Caution
As mentioned above you need to be careful when directly querying the Metaverse tables.If your system is already in production, and you happen to be adding in a new data source, then you may be better off employing a SQL MA to export the data you’re interested in to another table, where you can query it as much as you like without risk of locking errors.

Querying the Connector Space

Unfortunately it is not so simple to query the connector space to, for example, report on that state of your disconnected objects.

It’s a great pity that you can’t just save results from the Joins page in the Identity Manager GUI, so your options are: 

SQL query – The CS table holds data differently to the Metaverse tables.

It is possible to query for disconnectors in this way, however you will only be able to retrieve the CN of objects – which may not be sufficient to identify them. 

select cs.rdn from dbo.mms_connectorspace cs
join dbo.mms_management_agent ma
on cs.ma_id = ma.ma_id
left outer join dbo.mms_csmv_link mv
on mv.cs_object_id = cs.object_id
where ma.ma_name = 'My MA'
and mv.mv_object_id is null
and cs.connector_state = 0

CSExport – The command-line utility csexport.exe, found in the <ILM program>\bin folder, will allow you to dump connector space objects to an XML file. 

Report directly from the data source – Once an object in the data source has been correctly identified you will ideally export a unique attribute out to it. It may then be possible to identify the non-joined objects as those which don’t possess this attribute. 

Project to a different object type – ILM processes joins before projections, so it is fairly simple to project all non-joined objects to a different Metaverse object type – for example, one called “disconnectors”. This may help in reporting on an overall status direct from the Metaverse tables.

Note however that to make these objects once again available for joins they will have to be disconnected from the MVExtension provisioning code. 

Advanced Tips and Tricks

Join to Multi-value Attribute

Sometimes you may need to join to a value in a multi-value attribute. An example is searching through all the proxyAddresses for a match against
a single email address. 

When the multi-valued attribute exists in the connector space, and the single valued attribute is in the Metaverse, this is very easily accomplished. Just use an Advanced Join Rule to break the multi-valued attribute down into the values list used by the join rule. 

 Public Sub MapAttributesForJoin(ByVal FlowRuleName As String, ByVal csentry As CSEntry, ByRef values As ValueCollection) Implements IMASynchronization.MapAttributesForJoin
   Select FlowRuleName
      Case "Join_proxyAddresses"
         Dim alias As String
         For Each alias In csentry("givenName").Values
            values.Add(alias)
         Next
   End Select
End Sub

However, when the multi-valued attribute has already been imported into the Metaverse you will have a problem. While you can join on a multi-valued attribute, you have to join on the whole thing. There is no way that you can match one value out of a multi-valued Metaverse attribute against a single-valued connector space attribute. 

Some possible options: 

• Import the multi-valued attribute into a series of single-valued attributes in the Metaverse, eg., proxy1, proxy2, proxy3 …
• Use a different Metaverse object type to do the project and join the other way around.

Multiple Possibilities in the Connector Space

All the joining techniques so far have been based around a single connector space object, with one or more possible matches in the Metaverse. But what do you do if you want to work the other way around, where there are multiple possible matches in the connector space for a single Metaverse object, and you want to pick the best one? Unfortunately this is not straight-forward. There is no way to offer a selection of connector space objects in a join rule, as joins always work on a single connector space object at a time. You can judge the merits of the current CS object, but you can’t tell if there’s a better one coming up. One way around this is to do everything with CSV files. 

1. Use the ResolveJoinSearch to write possible joins to a CSV, but don’t actually join anything,
2. Do your matching outside ILM, then
3. Use a CSV with an Advanced Join Rule to make the joins.

Too much data

If you have an enormous number of accounts, and different data sources to trawl through, you may be best off doing your data matching outside of ILM, and then just using the CSV join rule above to make the joins.  Check out the Fuzzy Lookup Transformation from the Enterprise version of SQL SSIS for help here. 

Take-Home Thoughts

• Complex join rules are a means to an end – not the end itself.
• Breadcrumbing is essential for automation.
• When matching on weak rules (eg., surname only) then verify the match another way.
• You can’t do it all in ILM. CSV, Excel and fuzzy lookup algorithms will also help, but an element of by-hand matching is inevitable.
• Get the matching and breadcrumbing sorted out before you start flowing and provisioning.This will make for a happier project, stake holders, users and YOU!

Extra Reading

• Design Concepts for Correlating Digital Identities
• Correcting incorrect joins

About the Author

Carol Wapshere is an ILM MVP and contributor of a few of these documents now. She has always believed that putting the work in up-front will save you lots of headaches in the long run. It’s a self-defense strategy really – there’s nothing worse than having to go over and over (and over) the same ground again and again, especially when you’d already moved on to something new and far more interesting. Keep it neat, and keep it simple, and things should work just fine. 

Thanks to Markus Vilcinskas and Paul Loonen for their help with this document.

Handy!