Litmus Test for Metadirectory vs. Virtual Directory

No, I don't want to re-open a debate. Just floating someone else's idea...

I already mentioned some of the things I overheard at DIDW 2008 and the panel titled Lessons From Successful Virtual Directory Deployments. I was looking at my notes today and wanted to float an idea that one of the panelists offered (I think it was Divya Sundaram of Motorola). He said (paraphrased):

If you front-end data (or a data store) that you don't own (or don't have control of), then you need to replicate/sync data (instead of virtualizing the view).

Is that a good general litmus test for the Metadirectory vs. Virtual Directory debate?

As I've said numerous times, I can think of clear use-cases for both scenarios. But this might be a good general rule of thumb. BTW - the panel seemed to unanymously agree that both capabilties are useful and should be part of the toolbox.

Building a Central Identity Store

The folks at SECUDE Consulting, who are SAP ERM specialists, have an identity practice that focuses on (among other things) SAP NetWeaver Identity Management solutions (the former MaXware products). Matt P, part of SECUDE's IAM team, recently authored a white paper titled Strategies for Creating an Authoritative Store.

If you are building a provisioning system, deploying SAP NetWeaver Identity Management, or designing an enterprise identity store, you should review this paper. Matt discusses terminology like source repositories and target systems, discusses data join techniques, and introduces the concept of layering. The paper provides an overall road map for designing an enterprise identity store, which can be a critical component of a provisioning solution.

You can get a copy via the links or contact info in Matt's blog post about the paper.

Ouch

Dave Kearns calls my argument smoke and mirrors and labels it FUD. His argument is that the Global 2000 have more users and are therefore more important? Should their needs drive solutions for the mid-market?

Dave, I don't think the number of users is even relevant. What is relevant is the experience of those customer organizations and how they can meet their requirements. The number of infrastructures is more relevant than the number of end users in this discussion. I don't think a huge amount of them have a need or desire for multiple user directories. They seem to run off of AD and seem to prefer to have apps leverage AD instead of figuring out how to use a virtual directory (or metadirectory for that matter). Where is the FUD in that? Where is the smoke? What would be my motive to raise smoke and mirrors?

The discussion of how should Oracle build a product is very different than whether customers should consider metadirectory as an alternative. I think they should. I think there are still plenty of environments that could benefit from that approach. But I conceded Clayton's point -- if Oracle wants to build a virtual directory into it's suite to enable flexibility for customers, that's great. I just don't think a virtual directory is the answer to everything (and I spent a lot of time discussing the various use-cases that cry for one).

I would just hate to have people shy away from a good technology because some people say it's no good anymore. That doesn't make sense.

Ultimately, we might agree. Dave's conclusion is one that I've echoed over and over:

The need for, and uses of, virtual directories is growing and is still a few years away from peaking.

Let's just not declare something dead because it no longer seems cool to the in-crowd. It's OK to take a pragmatic approach to whatever challenge your organization is facing. That's my point.

Metadirectories: What's left to say?

If you haven't been following the flurry of conversation since my post last week stating that metadirectories aren't dead, well you're in luck. We couldn't have asked for a better recap of the conversation than the one provided by Ian Yip (although I think he gave Nishant a bum rap on this one).

There were so many different angles explored that I'm not really sure where to start or what's left for me to say.

• I'll restate that I see perfect use-cases for both metadirectory and virtual directory. Now and in the near future. In the far future, there will probably be better ways to achieve the same goals.


• Also, it sounded like Clayton took my comments to mean that "everyone needs to be using Active Directory for everything", which was (I think obviously) not the intent. My point is that although the top 500 or 1000 companies may have a number of directories for various strategic uses, there are probably 20x that number of companies that use only Active Directory as the central and primary user store because of it's network and email integration. And those companies might like for their application vendors to offer direct plug-in to AD as an option.
  
  Plugin to LDAP might be another good option and virtual directory technology would be a great enabler to incorporate various vendors, schemas and even relational databases through that single mechanism. But those mid-market companies probably would prefer not to take on the complexity of virtual directory (even if relatively simpler than writing numerous connections) if they could just use AD natively. And I think some percentage of the Fortune 1000 would see AD as strategic enough to ask the same as well.
  
  That's my guess based on a customer perspective as opposed to a software vendor's ideal state of architecture. And I don't think this is limited to companies who are 100% Microsoft shops. AD just has a very far reach and because it holds email in most of those companies it will already have an account for every employee, be available, etc..
  
  I don't think any of this should be seen as threatening to the role that stand-alone directories or meta- or virtual- directories play. The difference in viewpoint between me and Nishant & Clayton (if I can group them together) might be in the types of customers we've been talking to. There are still a ton of companies out there that aren't super-strategic about their Identity Management architecture. Or that just want a point solution because it fits the current business needs.

I think that's it. For now.

[UPDATE] - forgot one:

• Bavo, I wasn't requiring that the HR database is the primary source for account creation and status. I also wasn't telling you that the HR database should be the primary source for Identity information. (However, I think it's more true than you think.) I was stating a requirement (one that I've seen many times). HR has been deemed THE authoritative source for employee existence in a majority of the companies I've worked with. My experience seems to differ from yours. [That's at least interesting! ...and one of the reasons I blog – to engage with people that have different experiences.]
  
  Yes, companies struggle with getting HR updated for the employee's start date. But, I've actually seen more than one customer implement a complicated AD-to-HR-back-to-AD process to accommodate for the issue. One customer integrated the candidate review system into the provisioning system. I think the reason for HR being authoritative is usually for deprovisioning. They want a disabled HR account to ripple downward.
  
  I think what you call the IDM system assumes a provisioning solution with work flow and its own internal store. These are luxuries that are not always available. In my scenario, the cost and complexity of a provisioning solution is probably overkill based on the requirements. And that's my point. There are scenarios where the simplicity of a metadirectory are not only sufficient to meet the requirement but actually a bit more of an elegant way to meet the requirements.

OK, now I'm really done for the night.

Metadirectories Aren't Dead (They're Just Aging)

Nishant Kaushik updated his blog and one of his old posts showed up on PlanetIdentity reminding me of the recent discussions on metadirectories and virtual directories between him and others (Dave, Jackson, Kim).

Not that I want to pick a fight with any of these guys, but for anyone who thinks the metadirectory is dead, I have a simple (albeit a bit late) scenario for you.

There are three identity stores:

• An HR app built on a black-boxed Oracle DB
• A custom-built line of business app built on MySQL
• Active Directory

Requirements:

• The HR system needs to be authoritative for account creation and status.
• Active Directory needs to feed email address to the other apps upon creation (and occasional changes).
• Systems should be updated within 4 hours.

That's it. What do you think? Is a virtual directory the best solution to meet these needs?

I love virtual directory technology as much as the next guy (Hi Mark), but claiming that any technology is superior to another without a discussion of the specific requirements being met just doesn't seem to make sense. Companies, departments, and projects within departments have different needs.

I've said it before. They're just tools. So, when James McGovern asks what the role of virtual directory should be, I don't have an answer. There is no should in this discussion. Ian Yip had a similar pragmatic answer. And Nishant echoed with "the mantra should always be to choose the right tool that solves your problems". Exactly.

If the idea is simply to talk about what the future should look like, I think James hit on something. There has been a ground swell of apps that directly support Active Directory as the user store. So, maybe the next versions of the HR and LOB apps in the above scenario would attach directly to AD eliminating the need for any solution here. As prevalent as AD has become, that seems more likely than mass-consumption of virtual directory technologies. And it's probably what Jackson was alluding to (Quest enables *nix systems to leverage AD).

Another possibility is that apps will support SOA-based authentication and authorization, though that hasn't quite spread like wild fire quite yet.

Don't get me wrong – I don't think the need for virtual directory technologies will go away anytime soon, but I wouldn't be surprised if it never becomes a standard in the mid-market. And I don't think it'll ever completely replace metadirectory technologies.

Metadirectory may be aging, but hey, 50 is the new 30. It's not dead yet.

Proliferation of Multiple LDAPs

I was questioned today by someone who read my post on Synchronization versus Virtualization, which is topical given the recent blog debates by many of the big names in identity blogging on whether metadirectories are dead. Back then, I was trying to convince the world that there are cases that require a virtual approach. Now, the tables seem to have turned. I've stayed quiet on that debate, but I believe there's still a place for each method. Honestly, I'm not sure why there's such intensity behind that debate – they're just tools. My idea for Enterprise Identity Services Architecture included a layer for Identity Data Services, which is analogous to the proposed Identity Hub or Bus. It has a box for both synchronization and virtualization because there are use cases for both types of technologies often within the same infrastructure.

End of digression.

The question proposed was about how large enterprises (hundreds of thousands of users) are addressing the proliferation of LDAP directories in their environments. The person asking comes from a well-known and reputable company. The first thing I thought is here's the exact reason why we still need metadirectories. Many companies still haven't dealt with the user-store sprawl. In a general sense, I think metadirectory technologies can help get you to a place where you refine and consolidate your infrastructure and can then leverage virtual directory technologies where appropriate for applications that need access to data in multiple stores and/or multiple formats.

What the questioner was really looking for is research that discusses how other companies have handled the challenge. Here's my answer:

My recommendation is to find an experienced group of consultants who have tried various approaches with different clients. There are a number of them out there and I'm happy to give recommendations privately based on geography, technology, comfort-level with small vs. large companies, etc.. Whether you engage them contractually or enter into discussions via user groups, trade shows, etc., real world experience is priceless. Anybody that pushes one particular approach to this problem is probably biased (either by product or by their own limited experience). My experience suggests that most scenarios require a unique approach based on business goals, actual technologies, future plans, etc.. But you probably need someone to spend some time understanding your own scenario before recommending an approach.

Some questions you'll want to explore:

• Which data stores have overlapping data and which are unique?
• Does it make sense to consolidate?
• Is the data mappable across systems? Do they share unique identifiers?
• Where can multiple applications share a single store?
• Where do given applications require access to data in multiple stores?
• What applications or uses are coming in the future?
• Which stores are used for critical apps? What is the up time demand?
• In what format is the data stored?

The answers to these questions will drive your architectural decisions and help you prioritize next steps.

If you'd like to comment and provide a pointer to field research, whitepapers, or contact info for how you can help with this situation, feel free.

re: Synchronization versus Virtualization

In his entry on Novell’s Cool Blogs, Volker Scheuber discusses the use-cases for virtual directories versus metadirectories and asks for input. Considering my number of previous posts on this topic, I felt an obligation to respond.

I've done a significant amount of services work around both metadirectory and virtual directory technologies. My last company had both and my current company offers neither. The bottom line is that if I were designing an IdM solution, I would want both options to be available. As with any enterprise technology, there's usually more than one way to accomplish a given goal. Most IdM data integration requirements can be met with either meta or virtual technologies. However, there are some scenarios that, for all practical purposes, require one or the other. A while back, I posted some common scenarios where a virtual directory can be really useful.

Even for some of these scenarios though, you may be able to achieve a similar result by standing up another data store and setting up data sync rules with a metadirectory. But it often requires a significant amount of programming that would need to be maintained in addition to the new data store (LDAP or other) and related infrastructure.

Here’s an example:

Let’s say you were deploying a COTS application for insurance claim processing and security is important. The app uses an Oracle database for system data and is built to leverage an enterprise LDAP store (eDirectory, Active Directory, etc.) for authentication. Each employee that needs access to the system already has an account and credentials in the enterprise directory (ED). The system leverages the ED’s groups to grant or deny access. And of course the ED user needs to be enabled.

Due to the sensitive nature of the material, a requirement is made that the system should not allow access to the app unless the employee is punched-in to the company time-clock system. So, if an employee were to piggy back off of a manager’s home VPN connection, they could not access the system unless physically punched-in at work where all claim processing personnel are monitored. The time-clock system has a MySQL database with both system and user data. The APIs for the claim system and the time-clock system are both difficult to work with and complex to maintain.

What are the various ways to meet this requirement?

You could potentially try to work with the API’s and write a custom authentication module that would connect to the time-clock system and verify that a user is punched in before granting access. This would require some internal training, development time and will need to be maintained as either vendor releases new versions in the future.

You could implement a metadirectory solution that sets the ED user as disabled each time status changes in the time-clock system. This may require some custom code, but should be fairly easy to implement. However, we don’t want to disable users in the ED each time they’re punched out. They still need access to HR benefit info and other systems. Not to mention that disabling a user in the ED may trigger other unwanted events to occur (de-provisioning).

You could have the metadirectory solution switch to a different attribute instead of disabling the user, but you’re back to writing a custom authentication module. Also, these metadirectory options will likely introduce some latency because metadirectory sync jobs are typically scheduled. So, there may be some window of time between when the user punches out and when the system denies access. This time window can be exploited to circumvent the intended security controls.

Or – you could have the claims app connect to a virtual directory that is configured to join data from the ED and the MySQL database in a way that is transparent to the claims app. The app sees the virtual directory as an enterprise LDAP directory, but when the user is not punched in, the user appears to be disabled. This is generally a very simple thing to configure for a virtual directory. And if the claims app expects a single group to grant or deny access but the ED groups are arranged differently, the virtual directory can display group memberships however is most appropriate for the app.

Let’s further suppose that the business rules are changed to grant restricted access for the user when punched in but at an alternate location. The virtual directory can be easily configured to set a flag indicating restricted access based on the requestor’s IP address. This is not easily accomplished with a metadirectory solution.

And it’s a plus that the same virtual directory can be used to display user data differently for any number of systems that need access to it – and without altering or moving the data. As we move toward increased regulations and encrypting sensitive data where it lives, reducing the number of data stores that need to be managed and encrypted is an advantage in itself.

Having said all that, I can come up with a number of examples where a metadirectory is clearly a better option than a virtual directory, but I think those are generally easier to see.

Hope this is useful.

MaXware Data Synchronization Engine Lite

About two months ago, MaXware announced a free product to the market for a limited time. We've gotten a very positive response and extended the available download window. Here's what you can do with it:

Synchronize Active Directory with Sun Directory Server, IBM Tivoli Directory Server, Oracle Internet Directory, Oracle Database, SQL Server, MySQL, PostGreSQL, ASCII, LDIF, DSML, etc.

Plus:

• Maintain an audit trail of all changes at an attribute level.
• Error handling via SNMP, Windows Event Log, email alerts and custom scripting.
• Perform schema mapping, attribute transformation and construction.
• Leverage an extensive built-in script library using your choice of VBScript, Javascript or Perl.

We're hoping you love it so much, you'll turn to us when your needs grow. That's it. No catch. This is a free production server license for one-to-one data synchronization with no expiration.

More info on MaxWare DSE Lite and how to get a copy >>

Showdown: MIIS vs. DSE

Prior to joining MaXware, I spent a significant amount of time working with Microsoft Identity Integration Server (MIIS). Since joining, I've had a lot of questions from old friends about how the new software stacks up (and vice-versa from new friends). So, I put together a summary of what I've found thus far. I'll discuss MIIS and MaXware Data Synchronization Engine (DSE). Thanks to its very descriptive product name, you can probably guess what DSE does. MaXware also offers the MaXware Identity Center (MIC) for advanced user lifecycle management (workflow, etc.). But for now, I'll stick to data synchronization.

INFRASTRUCTURE

I think the most obvious difference between DSE and MIIS is infrastructure. MIIS requires Windows Server 2003 and SQL Server. For practical purposes, it also requires Visual Studio. DSE, on the other hand, can run on any platform supporting Java Runtime 1.4 or later. DSE also doesn't require a central database, but can leverage Oracle, SQL Server, MySQL, Access or any other ODBC, JDBC or OLEDB compliant database. With DSE, you can build very simple jobs that read/write directly to/from one data source directly to another without setting up a central database. If you want to perform data joins, you can setup a collect database as part of the solution (similar to an MIIS metaverse). So, the DSE footprint is very small. And DSE runs very efficiently using Access as a central store even with a few hundred thousand entries. If performance is critical, support for the bigger databases is available.

DELTAS

Another advantage of DSE is that it doesn't rely on deltas to be managed at the connected data store. So, for example, you don't need to enable the Sun directory retro changelog in order to only process changed entries. DSE handles this by its ability to store its own delta table in the central database with a hash of each record and the ability to compare the hash before performing an action on the record. If you'd prefer to rely on the changelog, that's OK too. You may be able to improve performance that way. However, you might lose some level of audit capability since DSE won't know what attributes have changed.

CONFIGURATION

Another difference is in the storage of server configuration. MIIS stores its server configuration in the SQL Server database. DSE stores its configuration in a single XML file. MIIS, though, does a pretty good job of providing XML-based server configuration export and import. It seems a little more complicated than DSE, but not by much.

SERVER FLEXIBLIITY

MIIS uses a single metaverse for each MIIS instance. So, a set of Management Agents writes to and from the metaverse to perform actions on the user objects. This is fine until you want to manage multiple sets of users. If, for example, you want to manage test data and production data, it becomes difficult to manage due to join rules. It's easy to join incorrect objects by running an MA against a test directory instance with different data. With DSE, jobs are organized in groups. Each group is a set of Passes (a pass is similar to a management agent and a corresponding run profile). And you would setup each group to have its own collect database (or metaverse). This allows you to do more with a single instance of DSE and provides extensive flexibility.

DIRECTION

DSE passes are one-way. MIIS management agents are bi-directional. So, if you want to read and write to a given directory, you would create a FROM pass, probably run a few other from passes and perform some actions on the data, then you would run a TO pass back to the directory. In the MIIS world, it's common to run a single MA multiple times to capture data changes that happened after the initial import run. While I prefer the way DSE handles this, I don't see this as a big advantage for either solution.

SCRIPTING

On the Windows platform, DSE can support VBScript, JScript and Perlscript. On the Java platform, DSE supports Javascript. Obviously, VBScript can call a DLL written in VB or C# and Javascript can call compiled Java code. MIIS supports C# and VB.NET. Both provide a set of native functions to interact with objects as you're moving them around. Openness is generally a good thing -- especially if you like Java, but VB.NET and C# can probably perform any type of functionality that you're looking to accomplish. I think DSE has the advantage here, but if you will be using C#, MIIS is more directly integrated which can be considered a positive.

AUDIT DATA

Both solutions provide logging, but DSE has the advantage of local delta tables for audit and reporting. Also, the separation of TO and FROM passes also make it easy to include a pass that writes to an audit table before losing the old and/or new attribute values. While XML data mining is possible with MIIS, I would much rather work with DSE if advanced audit reporting is a requirement -- especially at the attribute level. I'm not really sure if attribute-level auditing is possible with MIIS without some heavy scripting being built into each attribute flow rule.

SQL FLEXIBILITY

DSE FROM and TO passes have a SOURCE tab and a DESTINATION tab. When importing data from a SQL database, the SOURCE tab allows you create a custom query that is as advanced as you would like. In addition, if you are creating a TO pass to any type of data source, you can write an advanced query against the collect database to select whatever subset of users you can imagine. MIIS is limited to import only from a single table (or view). And management agent operations are run against the entire connector space set for that agent (or the deltas). MIIS lacks the ability to create custom queries that define a subset of users to which a set of actions would be applied.

DATA TYPES

MIIS is obviously an identity data tool. You can modify the metaverse schema to accommodate whatever object types you need, but it's designed to manage identity data. DSE is designed to handle any kind of data. There are no pre-configured data structures. DSE does schema discovery on any kind of data source and can easily be used to synchronize any type of data. For example, if you want to synchronize customer order data from one type of database to another, DSE can handle it as easy as managing the customer identity data.

COST

I think they two products are similar in price, but DSE allows the customer to leverage existing server and database infrastructure rather than requiring new licenses for Windows Server and SQL Server.

CONCLUSION

I realize that most of these points favor DSE over MIIS but that wasn't my predetermined intention. These are what came to mind when thinking about the differences between the two products. If you are a C# developer who lives in Visual Studio and already owns an open SQL Server license, you may prefer MIIS. Outside of that scenario, I think there are a number of compelling reasons to consider DSE. I'd be interested, though, in hearing opposing viewpoints -- there may be a perspective that I haven't considered. My hope is that this summary paints a picture of the basic differences between the two products and allows the reader to think about how these differences may or may not affect a given environment.

One more post on Virtual vs. Meta

So, I already wrote that the question of Virtual vs. Meta is not the right question - they are complimentary solutions. One more thought on identifying where each would fit.

Every organization embarking on an IdM journey needs to begin by identifying data stores and collecting, cleansing, transforming & reconciling data. At this stage, traditional synch tools (metadirectory) are probably the right tool for the job.

If you already have a reasonably good set of data and are looking to provide additional or customized views into that data (via LDAP, SOAP, etc.), then Virtual Directory is probably the answer.

Of course, this is a very generalized view, but I think a good starting point. One of the things that make this question confusing is the functional overlap between the two solutions -- and there is plenty. But the point is to find the best fit solution for a given set of business challenges.