Luckily powershell has some great XML parsing capability, so here’s a little script I wrote which takes an XML file created by csexport, and produces a CSV file more suitable for opening in Excel. Note that the script only supports single-valued attributes – you can modify it yourself if you need multi-values.

#
# parse-csexport.ps1
#

# Change the following list to get different attributes. The first four are available for all connector spaces.
$csvcolumns = @("dn","connector-type","connector-state","mv-guid","emailAddress","userName","sn","givenName","title","personalTitle")

$csvfile = "csexport.csv"
$csexportfile = "csexport.xml"

[xml]$csexport = get-content $csexportfile

if (Test-Path $csvfile) {Remove-Item -Path $csvfile}
foreach ($csvcol in $csvcolumns) {
  $csvheader = $csvheader + ";" + $csvcol
}
Add-Content $csvfile $csvheader

foreach ($csobj in $csexport."cs-objects"."cs-object") {
  $csobjhash = @{}
  $csobjhash.Add("dn",$csobj."cs-dn")
  # Disconnectors
  if ($csobj.connector -eq "0") {
    $csobjhash.Add("connector-type","disconnector")
    $csobjhash.Add("connector-state",$csobj."connector-state")
    $csobjhash.Add("mv-guid","")
    foreach ($attr in $csobj."unapplied-export-hologram".entry.attr) {
      if ($attr.multivalued -eq "false") {
        $csobjhash.Add($attr.name,$attr.value)
      }
    }
  }
  # Connectors
  else {
    $csobjhash.Add("connector-type","connector")
    $csobjhash.Add("connector-state",$csobj."connector-state")
    $csobjhash.Add("mv-guid",$csobj."mv-link"."#text")
    foreach ($attr in $csobj."synchronized-hologram".entry.attr) {
      if ($attr.multivalued -eq "false") {
        $csobjhash.Add($attr.name,$attr.value)
      }
    }
  }
  $csvline = ""
  foreach ($csvcol in $csvcolumns) {
    if ($csobjhash.Contains($csvcol)) {
      if ($csvline -eq "") {$csvline = $csobjhash.Item($csvcol) }
      else {$csvline = $csvline + ";" + $csobjhash.Item($csvcol) }
    }
    else {$csvline = $csvline + ";"}
  }
  $csvline
  Add-Content $csvfile $csvline
}

Note: this workaround should become redundant with R2 which includes archiving of request data. Looking forward to that!

Create Reporting Tables

Start by creating the following tables in the database you use for reporting (ie NOT one of the DBs installed by FIM. I have a dedicated DB called “FIMReporting”):

USE [FIMReporting]
CREATE TABLE [dbo].[fim_requests_new](
	[ObjectKey] [nvarchar](50) NULL,
	[Attribute] [nvarchar](50) NULL,
	[Value] [nvarchar](max) NULL
) ON [PRIMARY]

USE [FIMReporting]
CREATE TABLE [dbo].[fim_requests_log](
	[ObjectKey] [nvarchar](150) NULL,
	[Creator] [nvarchar](150) NULL,
	[CreatedTime] [nvarchar](150) NULL,
	[CommittedTime] [nvarchar](150) NULL,
	[Operation] [nvarchar](150) NULL,
	[Target] [nvarchar](150) NULL,
	[TargetObjectType] [nvarchar](150) NULL,
	[ManagementPolicy] [nvarchar](500) NULL,
	[RequestStatus] [nvarchar](150) NULL,
	[RequestParameter] [nvarchar](max) NULL
) ON [PRIMARY]

Extract recent requests

Once an hour, at the end of the regular sync cycle, I run this SQL script to copy out the requests I haven’t yet logged:

truncate table FIMReporting.dbo.fim_requests_new;

insert into FIMReporting.dbo.fim_requests_new
select o.ObjectKey, 'ObjectKey' as Attribute, o.ObjectKey as Value
from FIMService.fim.Objects o
left outer join dbo.fim_requests_log l
on o.ObjectKey = l.ObjectKey
inner join FIMService.fim.ObjectValueString s
on o.ObjectKey = s.ObjectKey
where o.ObjectTypeKey = 26
and l.ObjectKey is null
and s.AttributeKey = 66;

insert into FIMReporting.dbo.fim_requests_new
select v.ObjectKey, a.Name as Attribute, CAST(v.ValueBoolean as nvarchar) as Value
from FIMService.fim.ObjectValueBoolean v
join FIMService.fim.AttributeInternal a
on v.AttributeKey = a.[Key]
join FIMReporting.dbo.fim_requests_new n
on v.ObjectKey = n.ObjectKey
where n.Attribute = 'ObjectKey';

insert into FIMReporting.dbo.fim_requests_new
select v.ObjectKey, a.Name as Attribute, CAST(v.ValueDateTime as nvarchar) as Value
from FIMService.fim.ObjectValueDateTime v
join FIMService.fim.AttributeInternal a
on v.AttributeKey = a.[Key]
join FIMReporting.dbo.fim_requests_new n
on v.ObjectKey = n.ObjectKey
where n.Attribute = 'ObjectKey';

insert into FIMReporting.dbo.fim_requests_new
select v.ObjectKey, a.Name as Attribute, CAST(ValueInteger as nvarchar) as Value
from FIMService.fim.ObjectValueInteger v
join FIMService.fim.AttributeInternal a
on v.AttributeKey = a.[Key]
join FIMReporting.dbo.fim_requests_new n
on v.ObjectKey = n.ObjectKey
where n.Attribute = 'ObjectKey';

insert into FIMReporting.dbo.fim_requests_new
select v.ObjectKey, a.Name as Attribute, name.ValueString as Value
from FIMService.fim.ObjectValueReference v
join FIMService.fim.AttributeInternal a
on v.AttributeKey = a.[Key]
join FIMReporting.dbo.fim_requests_new n
on v.ObjectKey = n.ObjectKey
join FIMService.fim.Objects ref
on v.ValueReference = ref.ObjectKey
join FIMService.fim.ObjectValueString name
on ref.ObjectKey = name.ObjectKey
where n.Attribute = 'ObjectKey'
and name.AttributeKey = 66;

insert into FIMReporting.dbo.fim_requests_new
select v.ObjectKey, Name as Attribute, ValueString as Value
from FIMService.fim.ObjectValueString v
join FIMService.fim.AttributeInternal a
on v.AttributeKey = a.[Key]
join FIMReporting.dbo.fim_requests_new n
on v.ObjectKey = n.ObjectKey
where n.Attribute = 'ObjectKey';

insert into FIMReporting.dbo.fim_requests_new
select v.ObjectKey, Name as Attribute, ValueText as Value
 from FIMService.fim.ObjectValueText v
join FIMService.fim.AttributeInternal a
on v.AttributeKey = a.[Key]
join FIMReporting.dbo.fim_requests_new n
on v.ObjectKey = n.ObjectKey
where n.Attribute = 'ObjectKey';

Pivot query

I then have to run another SQL script to pivot the data so I end up with one line per request in my log table:

use FIMReporting

insert into dbo.fim_requests_log

SELECT ObjectKey,Creator,CreatedTime,CommittedTime,Operation,[Target],
       TargetObjectType,ManagementPolicy,RequestStatus,RequestParameter
FROM
    (select * from dbo.fim_requests_new
	where ObjectKey in (
	select ObjectKey from dbo.fim_requests_new
	where Attribute = 'RequestStatus'
	and Value in ('Completed','Failed','Denied','PostProcessingError') )) as src
PIVOT
( MAX(Value) FOR Attribute
    IN ( Creator,CreatedTime,CommittedTime,Operation,[Target],TargetObjectType,ManagementPolicy,RequestStatus,RequestParameter)
) AS pvt;

Pruning

As I don’t want the data in this log to build up and up I have a third script which I run once a day to prune records. I do this in two steps – first I get rid of requests made by the service accounts, and later I delete all requests.

use FIMReporting

declare @pruneDays as int
declare @deleteDays as int
set @pruneDays = 50 /* Must be greater than number of day kept in FIMService DB */
set @deleteDays = 180

declare @today as nvarchar(50)
set @today = GetDate()

delete from dbo.fim_requests_log
where datediff(day,CONVERT(datetime,CreatedTime),@today) > @deleteDays

/* Prune service account requests. YOUR ACCOUNT NAMES WILL VARY. */
delete from dbo.fim_requests_log
where datediff(day,CONVERT(datetime,CreatedTime),@today) > @pruneDays
and Creator in ('s-fimportal', 'Forefront Identity Manager Service Account', 'Built-in Synchronization Account')

But now I’ve had a play with it, and figured out how to update the field (it needs to be in XML) and why it’s very cool to do so (it makes the message easily available to the user).

In my custom workflow I have the following activities:

• A CurrentRequestActivity
• A code activity, and
• A ResourceUpdateActivity.

During my code activity I have set two variables:

• Boolean “success”, and
• String “returnMessage”.

Now at the end of my code activity I add the following code to prepare the ground for the ResourceUpdateActivity (which I’ve named “updateRequestDetails”) that writes the message back.

        Me.updateRequestDetails_ActorId1 = New Guid(FIMADMIN_GUID)
        Me.updateRequestDetails_ResourceId1 = Me.currentRequestActivity.CurrentRequest.ObjectID
        Dim returnXML As String
        Dim returnLevel As String
        If success Then
            returnLevel = "Information"
        Else
            returnLevel = "Error"
        End If
        returnXML = "<RequestStatusDetail xmlns:xsi=""http://www.w3.org/2001/XMLSchema-instance"" " _
                    & "xmlns:xsd=""http://www.w3.org/2001/XMLSchema"" DetailLevel=""" & returnLevel & """ " _
                    & "EntryTime=""" & DateTime.Now.ToUniversalTime().ToString("yyyy'-'MM'-'dd'T'HH':'mm':'ss'.000'") & """>" _
                    & returnMessage & "</RequestStatusDetail>"
        Dim updateInstruction As New UpdateRequestParameter
        updateInstruction.PropertyName = "RequestStatusDetail"
        updateInstruction.Mode = UpdateMode.Insert
        updateInstruction.Value = returnXML
        Me.updateRequestDetails_UpdateParameters1 = New UpdateRequestParameter() {updateInstruction}

I’ve used the FIM Admin account to write the message, and I had to explicitly give it permission to write to the RequestStatusDetail attribute of the Request resource type. You could also use the requestor’s account to update the field (Me.updateRequestDetails_ActorId1 = Me.currentRequestActivity.CurrentRequest.Creator), but you will still need to explicitly grant permissions with an MPR (have a look at the “Request Management: Request creators can…” MPRs to see what you need to do).

And here’s a picture of the Request after the custom workflow activity failed.

The Problem

The FIM Sync Service, just like its predecessors, only stores information about the current state of objects. Run History is almost completely worthless and should be cleared regularly. However there will be times when you will be asked to trace through a series of events – perhaps leading to the CEO’s account being inadvertently disabled. At times like this it is important to be able to show that FIM was just responding appropriately to an imported data change, and not acting maliciously in some skynet-esque awakening.

However, as we know, the import and export run profiles, while allowing a log file to be dumped, then helpfully overwrite it at the next run. We need a way to hang on to that historical data.

The Proposal

If you’re already running your tasks using vbscripts it’s pretty simple to add an extra step which copies the log file off to a datestamped version in an archive location (script below).

At the same time, we can do a little manipulation to the log file to make it more readable. By inserting a couple of lines in the top of the log file it can now be used with an XML Stylesheet, allowing it to be browsed in a nice table format.

Provisos

The log file will only be archived if you run your export and import jobs via your scripts. Anything run directly from the Sync Service GUI may still produce a log file, but it won’t be archived.

Also, the timestamp is a approximate as it represents the time the log was archived, rather than the exact time specific objects were modified in a target directory. But if you archive the log straight after the Export profile runs then it should be close enough for most purposes.

log.xsl

First, you need to create a folder somewhere with the same sub-folders as your MaData folder (in the script example below, I’m using D:\FIM\MALogArchives). Then, into this new folder, create a text file called “log.xsl” and paste in this content.

ArchiveLog.vbs

Now here’s a vbscript that will copy the named log file, while modifying it to work with the stylesheet.

' This script copies the export and import logs to datestamped versions
' and modifies them to work with a stylesheet called ../log.xsl.
'
'   Usage: cscript archivelog.vbs MaName LogFileName
'
'   Eg:    cscript archivelog.vbs HR import.xml
'
' Written by Carol Wapshere

Option Explicit
Const XML_STYLESHEET = "..\log.xsl"
Const MIIS_FOLDER = "C:\Program Files\Microsoft Forefront Identity Manager\2010\Synchronization Service"
Const ARCHIVE_FOLDER = "D:\FIM\MALogArchives"
Const ForReading = 1
Const ForWriting = 2
Const ForAppending = 8
Const Unicode = -1

Dim objFS, MaName, LogName
Set objFS = CreateObject("Scripting.FileSystemObject")

If WScript.Arguments.Count <> 2 Then
  Usage
End If

MaName = WScript.Arguments.Item(0)
LogName = WScript.Arguments.Item(1)

ArchiveLog MaName, LogName

Sub ArchiveLog(MA, LogFile)

  Dim objLogFile, objArchiveFile
  Dim strLogName, strArchiveName, logTime, dateStamp, strLine

  strLogName = MIIS_FOLDER & "\MaData\" & MA & "\" & LogFile
  If objFS.FileExists(strLogName) Then
    logTime = Now()
    dateStamp = DatePart("yyyy", logTime) & TwoChars("m", logTime) &_
                                 TwoChars("d", logTime) & TwoChars("h", logTime) &_
                                 TwoChars("n", logTime) & TwoChars("s", logTime)
    strArchiveName = ARCHIVE_FOLDER & "\" & MA & "\" & Split(LogFile,".")(0) & "_" & dateStamp & ".xml"
    set objLogFile = objFS.OpenTextFile(strLogName, ForReading, false, Unicode)
    set objArchiveFile = objFS.OpenTextFile(strArchiveName, ForWriting, true, Unicode)
    objLogFile.ReadLine()
    objArchiveFile.WriteLine("<?xml version=""1.0"" encoding=""UTF-16""?>")
    objArchiveFile.WriteLine("<?xml-stylesheet type=""text/xsl"" href=""" & XML_STYLESHEET & """?>")
    objArchiveFile.WriteLine("<top>")
    objArchiveFile.WriteLine("<xmlfile-time>")
    objArchiveFile.WriteLine(logTime)
    objArchiveFile.WriteLine("</xmlfile-time>")
    objLogFile.ReadLine() 'skip mmsml
    objLogFile.ReadLine() 'skip directory-entries
    strLine = objLogFile.ReadLine()
    Do Until InStr(strLine, "</directory-entries>") > 0
       objArchiveFile.WriteLine(strLine)
       strLine = objLogFile.ReadLine()
    Loop
    objArchiveFile.WriteLine("</top>")
    objLogFile.Close()
    objArchiveFile.Close()
  End If
End Sub

Function TwoChars(dtvar, time)
  Dim i
  i = DatePart(dtvar, time)
  If i < 10 Then
   TwoChars = "0" & CStr(i)
  Else
   TwoChars = CStr(i)
  End If
End Function

Sub Usage
  Wscript.echo "Usage: cscript archivelog.vbs MaName import|export"
  Wscript.Quit
End Sub

Modify the Run scripts

Your last step is to modify your scheduled scripts to archive the import/export log directly after the task has run.

cscript AD_Export.vbs
cscript ArchiveLog.vbs "AD MA" export.xml

The error you will see will either be dn-attributes-failure or cd-missing-object, depending on the type of group.

The detailed error will say something about an add or delete operation on a member that does not exist but, unhelpfully, will not tell you which one.

I’ve had some fun and games with this one recently, so this post is about some ways I figured out to troubleshoot the problem, and includes a vbscript for finding that missing member.

dn-missing.vbs

While trying to troubleshoot these missing member errors during the week I wrote a quick vbscript to help – you can look at it here.

Basically it exports an XML copy of the group object from the connector space, and then attempts an LDAP bind against each member. This works for AD. I haven’t tried it for other directories, but I expect it would work with anything based on LDAP.

What if the member exists?

The big problem I was having was when the member actualy did exist in AD. This was very frustrating. It seemed that once ILM had decided it couldn’t export the group then nothing could convince it otherwise. I tried various mitigation techniques:

• Full Import Full Sync of everything (didn’t help),
• Hacking the export.xml (helped when I was having a problem with a member delete),
• Adding the member manually in AD then doing a Delta Import Delta Sync (a bit pointless, but it got things moving again).

A full clear-out and re-import of the connector space would doubtlessly have worked, but considering the number and size of the groups, this would have been a painful process.

Targeting the same DC

What I did eventually figure out was that the two MAs were targeting different DCs. Duh! Obviously, to avoid any missing objects due to AD sync delays, you should target the same DC.  In fact this post on the Technet forum indicates that a Global Catalog server is best.

To hardcode a DC use the Domin controller connection settings on the Configure Directory Partitions tab of the AD MA.

Remove users from groups before deleting the user account

Another pretty obvious one, but I was also being careless on this front.

Even though it might seem perfectly reasonable to delete a non-existant user from a group, all AD will see is that you have explicity requested an operation involving something it can’t find.

In this implemetation, I disable users for a week before they are actually deleted. I now make sure that they are removed from all groups as soon as they are disabled.

I will write another post soon on the disable-delete methodology.

DC Logging Levels

Finally, if you are still having problems and need to get more information about why AD is rejecting an export, try increasing the logging levels on the DC as per this KB:

You may have noticed the ”test only” log file options on the import and export run profiles. Being able to stop the run at the log file is incredibly useful for testing what would have been exported, without actually going ahead and doing it. You can then resume the export from the log file – and if you wanted to, there is nothing to actually stop you editing that log file before resuming the run.

Mostly I have made use of this trick in test situations where I want to simulate a set of external data to see how my MIIS code deals with it. Perhaps I need to test for certain odd situations that are a bit hard to generate in the real environment, so as a short-cut I construct an import.xml to feed the right data into MIIS.

You could also change a setting in an export.xml file to test how the change effects the external environment. Maybe you can’t get access to make the change directly and this way you can piggy-back on MIIS’s existing permissions, without actually having to modify extension code or flow rules.

And now here is where I confess that I’m writing this post tonight because I actually used this hack on a production system today <gasp>. Not sure why, but MIIS was giving me repeated dn-attributes-failure messages when I attempted to export a group. It was complaining a new member didn’t exist in AD - but it did (I got a collegue to double-check just in case I was going nuts!). After trying various full import/syncs I finally resorted to hacking the export.xml to change the “add” member command to a “delete”. The export then completed (making no changes as the user wasn’t in the group yet after all), I resync’d everything, the member add was duly re-queued, and this time it exported happily. I’m still trying to figure out why this situation happened in the first place, but as a sneaky fix-it, the log file hack got me back in business quicker than any of the alternative, and more drastic methods.

Server Health

Obviously you will be monitoring that the server itself is actually up. I believe something a little more than a ping is required to confirm the server is alive and well, so monitor key services such as MIIS and SQL Server. 

Disk space monitoring is critical as a full partition will stop all MIIS activity. The SQL log drive (which you should have on completely seperate disks to your data, as per SQL best practises) can fill up alarmingly quickly and needs to be checked regularly. You should be alerted at 85-90% capacity on your Data drive, and 50% on your Log drive.

CPU and Memory are less critical as MIIS won’t stop, it will just run slower. You should, however, be collecting stats over the long term so you can assess the performance of the server.

Application Events

There’s some sort of Logging class in MIIS, but I actually never used it because I was happy with the messages in the Application Event Log. I just set a watch for particular events and that let me know when there were sync and export errors.

Scheduled Tasks

If you are running any kind of scheduled tasks around MIIS you must monitor them to make sure they are actually happening. An absolutely critical one is the clear-down of the Run History. I set a watch on the log file to verify that it runs successfully every night.

SQL

Regular SQL maintenance tasks should be monitored, as well as any replication jobs or scheduled DTS packages. I believe this can all be done with native SQL tools, though I can’t say for sure as I’ve always left it up to the DBA!

Monitoring Software

I used Sitescope very successfully to do all the monitoring listed above, with the exception of the SQL stuff (which, as I said, was the DBA’s domain). I cannot comment on the effectiveness of any other package, but if you’re evaluating, look for something that can monitor:

• services,
• server physicals – memory, cpu, disk utilisation,
• the server event log, and
• log files.

The first problem with this approach is that the xml files produced by MIIS don’t work with a stylesheet. The stylesheet name must be included in the xml file, and I also found I needed to remove and change a few tags to get it working (probably just due to my inexperience – this is the first xml stylesheet I’ve written).
 

Yesterday I posted a script to be run directly after any Import or Export job, which copies the log file to a datestamped version. It is a straight-forward matter to improve this script so that, instead of a file copy, it modifies the datestamped log to allow it to work with my stylesheet.

Follw this link for the full ArchiveLog sub.

Next take a copy of this log.xsl stylesheet and put it in your Ma Data folder.

The result should be that you can view the xml data in a browser in a nicely formatted way. At this point I had hoped to include a screenshot, but I’m having some trouble with my virtual server this morning, so you’ll just have to try it yourself!

One thing I do wish MIIS could do is timestamp these log files. The native configuration will overwrite the last log file, and where’s the use in that? However using MASequencer, or something like my simple queuing system, you should be able to insert steps to rename the log files following each Import and Export operation. Now to start with, you’re going to have an easier time if you’re always consistent with your log file naming. I keep it simple – import.xml and export.xml. I then encorporate the following VBScript sub into my scheduling script to copy the log to a datestamped version.

Sub ArchiveLog(MA, Profile)

‘ The Profile passed to the sub must be either “import” or “export”

Dim objLogFile, objArchiveFile

Dim strLogName, strArchiveName, logTime, dateStamp, strLine

strLogName = MIIS_FOLDER & “MaData” & MA & “” &_

Profile & “.xml”

If objFS.FileExists(strLogName) Then

logTime = Now()

dateStamp = DatePart(“yyyy”, logTime) & TwoChars(“m”, logTime) &_

TwoChars(“d”, logTime) & TwoChars(“h”, logTime) &_

TwoChars(“n”, logTime) & TwoChars(“s”, logTime)

strArchiveName = MIIS_FOLDER & “MaData” & MA & “” &_

Profile & dateStamp & “.xml”

Set objFile = objFS.GetFile(strLogName)

objFile.Copy strArchiveName

End If

End Sub

Function TwoChars(dtvar, time)

i = DatePart(dtvar, time)

If i < 10 Then

TwoChars = “0” & CStr(i)

Else

TwoChars = CStr(i)

End If

End Function

The XML files will need to be parsed somehow if you want to view them. I make a few simple modifications so that mine can be viewed in a browser using an XML stylesheet – more on that in this post.

The Run History is stored in the MicrosoftIdentityIntegrationServer database, and contributes massively to the growth the database. If you don’t regularly delete the Run History, your DB files will grow and grow until the disk is full. When this happens you’re in big trouble – MIIS will no longer be able to do anything. Clearing the History also requires large amounts of free space on the volume holding the transaction log file. This file needs to be expanded in the clearing process – and the more you’re trying to clear, the more disk space will be needed. I have even heard a story about someone who had to install an extra disk on the MIIS server just to provide this expansion space!

So now you know how important it is to keep on top of this, how much Run History should you keep? Personally I would say no more than two days. The data in it is actually not that useful – if you try and inspect an old Export, for example, MIIS will show you the object as it appears now, rather than how it looked at the time of the Export. A far better bet is to generate export and import logs as part of your Run Profiles, and keep those for as long as the business requires.

The best way to clear Run History is by using MIISClearRunHistory from the MIIS Resource Toolkit. This will allow you to create a little batch file which you can set to run overnight from the Windows Scheduler.

miisclearrunhistory.exe /pr:2 /l:2 

This command will create a log file, and my final tip is that you should monitor that file! Find out whatever monitoring system is available that can handle a log file (I’m most familiar with Sitescope, but I expect there’s plenty of other options) and set a watch on that file!