Avoid corrupted site columns due feature re-activation

One of our clients reported an issue that provisioned site columns get corrupted after deinstall of the provisioning feature. The feature deinstallation is a step within the repair of erroneous situation in which sandbox solution per accident is also deployed as global farm-based solution. This incorrect deployment requires a fixture because an effect is that features now appear as duplicate entries in the list of site(collection) features: one installed via the sandbox solution (correct, administrated in the sitecollection's content database), and one installed via the farm solution (incorrect, administrated in the farm configuration database).

On first thought the simple fix is to deactivate the provisioning feature that originates from the farm deployment, remove the feature, then retract the global farm-solution, and remove it from the farm solution store. Next activate the feature deployed via sandbox-solution to arrive at the correct deployment situation. However, in a test execution we experienced that this approach gives an error upon the re-provisioning of the site columns: The local device name is already in use.

Error details in ULS log:

Unable to locate the xml-definition for FieldName with FieldId '<GUID>', exception: Microsoft.SharePoint.SPException: Catastrophic failure (Exception from HRESULT: 0x8000FFFF (E_UNEXPECTED)) ---> System.Runtime.InteropServices.COMException (0x8000FFFF): Catastrophic failure (Exception from HRESULT: 0x8000FFFF (E_UNEXPECTED)) at Microsoft.SharePoint.Library.SPRequestInternalClass.GetGlobalContentTypeXml(String bstrUrl, Int32 type, UInt32 lcid, Object varIdBytes) at Microsoft.SharePoint.Library.SPRequest.GetGlobalContentTypeXml(String bstrUrl, Int32 type, UInt32 lcid, Object varIdBytes)

To come with a solution, I started with a root-cause analysis. Why are the provisioned site columns not completely deleted from the site upon its feature deactivation? The explanation is that the provisioning feature also performs a contenttype binding to the Pages library, and that in our testcase a page was created based on that contenttype. This effectually results in the contenttype being kept ‘in usage’ by the Pages library. On feature deactivation the site columns can still be removed at sitecollection level, but the contenttype not anymore due the descendant sibling binded to the Pages library.

The real problem however lies in the 'removed' site columns. They are deleted from sitecollection level, but due the preserved contenttype (Pages library) their definition has remained in the sitecollection's content database, with the same ID as on sitecollection level (note this is different for a contenttype binded to a list, that gets a new ID based on the ID of the source/parent contenttype at sitecollection). SharePoint administrates per provisioned artifact whether the origin is a feature, and if so effectively couples the artifact to that feature. SharePoint disallows these artifacts by automatically deleted or modified by another feature. As result the feature re-activation halts with an error when trying to (re)provision the sitecolumns that are still present deep down in the content database, coupled to the Pages library.

With this SharePoint-internal insight, I was able to come up with a faultproof approach to fix the 'duplicate features' issue. The trick is to initial leave the feature definition that originated from the erroneous farm solution in the configuration database, deploy the sandbox-solution (stores the feature definition in the sitecollection's content database), activate the feature with the same feature id from that sandbox solution. Now the feature activation proceeds completely without errors, and restores the site columns. Ultimately the farm-based solution can then be retracted from the farm solution store.

Note: I came to this insight by inspecting on SQL level. We all know it is not allowed to perform changes on SharePoint content database level (or loose your Microsoft support), but it is perfectly ‘SharePoint’-legal to review and monitor on SharePoint content database level.

Used/useful SQL statements:

SELECT * FROM ( SELECT *, Convert(varchar(512), CONVERT(varbinary(512), ContentTypeId), 2) As key FROM [ContentTypeUsage]) as T where key like '%< contenttypeid >%'

SELECT tp_Title FROM ( SELECT *, Convert(varchar(512), CONVERT(varbinary(512), ContentTypeId), 2) As key FROM [ContentTypeUsage] Join AllLists on ContentTypeUsage.ListId = AllLists.tp_ID ) as T where key like '%< contenttypeid >%'

Deleted SPWeb in RecycleBin can obstruct deactivation of Sandboxed Solution

The other day, upon deactivating a Sandboxed Solution in our test farm, SharePoint aborted on it with the message:

Cannot access web-scoped feature {GUID} because it has references to a site with id {GUID}.
System.ArgumentException: Value does not fall within the expected range.
at Microsoft.SharePoint.SPWebCollection.get_Item(Guid id)
at Microsoft.SharePoint.SPFeatureEnumeratorBase.GetCachedWeb(SPSite site, Guid webId, Guid featureId)

We earlier encountered this problem in the development farm. Thorough investigation by SharePoint operations together with development led to the conclusion that the SharePoint content database had reached a corrupt state due deletion of a SPWeb. On that SPWeb a Feature provisioned via the Sandboxed Solution had been activated. And now after deletion of that SPWeb, apparently a lock internal in the SharePoint content database was present obstructing disablement of that Feature. And since it is not recommended - and certainly unsupported - to manually alter a SharePoint content database, there seemed no other realistic approach to get out of this erroneous situation as by recreating and reprovisioning the entire sitecollection.

For the development instance, this was an acceptable pragmatic solution. However, in our test environment a lot of content is already created by end-users. Just throwing away the SPSite and replacing it by a brand new, is not viable. The only remaining solution seemed to afterwards restore the backed-up content into the new SPSite. Not undoable, but it will take time [to setup and execute the site content restore; and next to validate the correctness and completeness of it]. And moreover, it will result in a loss of trust by our end-users and customer on the robustness of SharePoint 2010 as application platform. If it happens once [actually twice], what guarantee is there it will not happen again?

Luckily, then I had a smart thought while discussing the problem symptons with a co-developer. If the problem appeared to be caused by the deletion of a SPWeb, would it then help to restore this SPWeb instance from the RecycleBin? Worth a try. And guess what: it did!! After the earlier deleted SPWeb had been restored from the recyclebin, the earlier activated Sandboxed Solution could next successfully be deactivated.

Automatically publishing multiple files fails with error '0x81020015'

In my previous post I wrote about the well-known artefact of sandbox solutions that provisioned files are by default still checked-out. This can be fixed via a FeatureReceiver that checks-in all files and approves the ones that require moderation, so that all provisioned SharePoint artefacts are direct available for the end user.

Initially this worked like a charm, and the problem seemed resolved. However when our project progressed and the number of provisioned files increased, we structurally encountered the following error upon feature activation:

Microsoft.SharePoint.SPException occurred
  Message=The file _catalogs/masterpage/ApplicationX/ApplXPageLayoutXX.aspx
    is modified by domain\WvStrien on May 11 2011 17:06:12 +0200.
  Source=Microsoft.SharePoint
  ErrorCode=-2130575305
  NativeErrorMessage=FAILED hr detected (hr = 0x81020015)

In the root cause analysis of the problem I discovered that the problem only manifests itself when more than 1 file requires approval. This also explained why initially we did not encounter the problem: we started out with only a single PageLayout. With this insight I did an internet search for '0x81020015'. Although no direct related pointers to our situation, this led me in the direction of the problem cause. Apparently it is a race condition within SharePoint internally when first checking in a file and immediately approving it in the same execution context. The internal SharePoint administration is typically not ready yet to handle the second state change on the file.

If this is the cause, then the solution is to break up the execution context: first do the checkin(), and in another execution context the approve() invocation. Normally you would do this by delegating it to a background thread. However, this is not possible in a Sandbox context: the SharePoint ObjectModel is only supported on the main thread of the SPUserWorker process. The best next and pragmatic approach is then to break the execution flow by introducing a delay between the invocation of the 2 SPFile methods:

private static void PublishFile(SPFile checkedOutFile, bool approvalNeeded)
{
  checkedOutFile.CheckIn(String.Empty, SPCheckinType.MajorCheckIn);
  if (approvalNeeded)
  {
    int nrOfWaits = 0;
    while (!checkedOutFile.Exists && nrOfWaits++ < 5)Thread.Sleep(1000);
    if (checkedOutFile.Exists)
     checkedOutFile.Web.GetFile(checkedOutFile.UniqueId).Approve(COMMENT);
  }
}

Error “Unable to load assembly group” upon redeploying a sandbox solution with a feature receiver

IT operations set up a fresh new SharePoint webapplication for my current project on the development farm, and provisioned it with a root site collection based on the publishing template. Next I took over the deployment control by following the sandbox route; in this preliminary phase merely to provision the first version of the branding: master page, CSS files, images and page layouts. A known artifact of sandbox solutions is that the provisioned files are by default still checked-out in the content database by the person who activated the solution, and thus not visible for anyone else. [It is a mysterie to me why Microsoft implemented it such; and moreover why they didn’t delivered us a setting to configure this provisioning behavior to automatic checkin and publish/activate the provisioned SharePoint artifacts.] Instead of manually via the browser or SharePoint Designer having to lookup in the site collection all the provisioned SharePoint artifacts to check them in and publish them, I more favor the approach to have this done automatically in the context of the feature activation. This can be easily accomplished via a feature receiver (Waldek Mastykarz has a good start/example for this in his blogpost Automatically publishing files provisioned with Sandboxed Solutions). So I added a feature receiver to the provisioning feature in the sandboxed solution, and in the FeatureActivated event let all the provisioned SharePoint artifacts be located in the content database and automatically checked in + published if applicable. Tested it locally, works like a charm. Deactivated and deleted the previous uploaded version of the sandboxed solution, and uploaded the new version with the feature receiver inside. However, upon activating this sandbox solution version, I consistently got the following error:

Error occurred in deployment step 'Add Solution': Unable to load assembly group. The user assembly group provider threw an exception while trying to provide user assemblies for the specified assembly group.

And in the ULS logging the following detail info:

Unable to load assembly group. The user assembly group provider threw an exception
Assembly Group Id: GroupId = "7C00665459714EEC9BDA8727AD711EC7-FES7hxwCdYldvtgExF/L+b259Rh3T1thWDeijNWTrSU="" - Inner Exception: System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values.
at Microsoft.SharePoint.SPListItemCollection.get_Item(Int32 iIndex)
at Microsoft.SharePoint.Administration.SPSolutionLanguagePack.
GetSolutionInfoFromGallery(Guid siteId, Guid solutionId, String solutionHash, String& fileName, String& hash, Byte[]& fileBytes)
at Microsoft.SharePoint.UserCode.SPUserCodeLightweightSolutionAssemblyGroupProvider.
GetAssembliesInGroup(Guid siteId, String assemblyGroupId)
at Microsoft.SharePoint.UserCode.SPUserCodeAssemblyCacheManager.
EnsureUserCodeAssemblyGroupIsCached(Guid siteId, SPUserCodeAssemblyGroupId userCodeAssemblyGroupId)

An internet search did not give many nor useful hits; and neither could the SharePoint development gurus in my neighborhood help me. So it left me rather puzzled at first how to analyze and solve this. A good clue however was that a sandboxed solution from another SharePoint (web)application with also a feature receiver inside, had the same problem trying to activate it in our site collection, while it successfully activated in the own site collection. So it must be something at SharePoint infra level, not caused by code development. With this insight I went back to my operations companion, and provided him with a sandboxed solution to monitor what events occurred at system level upon trying to activate the solution. This quickly let to a conclusive result: somehow the permission state of the User Code service account on the C:\ProgramData\Microsoft\SharePoint\UCCache subfolder got compromized, which disallowed it to re-copy the new version of the sandboxed solution with code inside to the UCCache subfolder. The problem was not even resolved by restarting the User Code service; instead it was needed to manually reset the User Code service account, so that upon the next invocation the ACL on the UCCache subfolder was again built up.

Our operations guy was very thoughful to make a note of this behavior in the internal SharePoint operations Knowledge Base, and document it in a work instruction.

Beware: timerjob processes invoked via Central Admin deploy solution do not reload FeatureReceiver assembly

Recent we ran into an exception when deploying an external facing website to the production SharePoint farm. The problem originated in the FeatureInstalled() method of a custom FeatureReceiver class. We did not encounter this particular deployment problem in the integration nor QA environment. Since on a production environment it is not viable to debug, we had to resort to inserting logging statements in the code, and then redeploy the assembly. But strangely, no logging was written leaving us still blank with respect to the exact code location and cause of the deployment problem.

After several deployment attempts together with operations, we came up with the explanation why the logging was not done. The timerjob process that executes the FeatureReceiver FeatureInstalled() method still has the old code loaded!! To force that the assembly is reloaded from either GAC or virtual bin after deploySolution and before execution of FeatureInstalled, you need to recycle the timerjobs processes. The right time for this is after the retractSolution of the previous version of the deployed SharePoint solution package.

NB: we didn't notice this 'faulty' deploy/timerjobs behaviour before because it's not very common to actually utilize the FeatureInstalled() method. In this particular case it was necessary to timely correct something before the SharePoint feature framework performs the OOTB feature activation work.

Unable to recover from a corrupted Publishing infrastructure

For a functional extension to the intranet (enterprise portal) of a client organization, we needed to provision a content page to an existing site. A precondition is thus that the Publishing infrastructure is activated within the context of that SharePoint site: at Site level (PublishingSite feature) and Web (PublishingWeb feature). Upon activating the latter we encountered the following error: Provisioning did not succeed. Details: Failed to create the 'Pages' library. OriginalException: The feature failed to activate because a list at 'Pages' already exists in this site. Delete or rename the list and try activating the feature again.

Strangly enough: via the SharePoint GUI the Pages library was not visible. Could be that it was set to hidden. But trying to directly navigate via its url merely resulted in a NotFound. Upon opening the site collection in SharePoint Designer, it showed there was a left-over of the Pages library: merely an empty folder, no Forms nor other content. But this presence was in the way of activating the Publishing feature. So we deleted this folder via SPD. And then tried to activate PublishingWeb feature again. This time it progressed a bit further, to stop with the message: The page you selected contains a list that does not exist. It may have been deleted by another user. Inspecting again: the Pages library was correctly available now, including associations with PublishingPage contenttypes. Missing this time was the Style Library documentlibrary. That is, when inspecting via SharePoint GUI. Again looking at filesystem level via SharePoint designer: same situation, Style Library folder present, but empty. Deleted this folder, and retried to activate the PublishingWeb feature. Sadly we kept on running into the error of missing List. Even after manually adding the Style Library; forcefully deactivate and re-activate the PublishingWeb feature. We did not manage to recover from the initial corrupted situation wrt Pages list. Search results on the Web for information on both the error messages gave some hits to problem descriptions, but sadly not to resolutions.

Due to a pressing time schedule, we therefore had to resort to a pragmatic alternative. Deleting the corrupted site collection, and re-creating it. This was a viable approach due to the current nature of the site collection: containing no content yet. But it leaves me with somewhat of a frustrated / disappointed mind. I would rather have seen us able to recover from the incorrect situation, while preserving the site collection.

FormsService SSP in farm holds on to previous version of Feature re-deployed InfoPath Form

Earlier I blogged about the ALM manner to develop and deploy InfoPath forms. One major benefit of this approach is that the InfoPath forms are considered as first-class source artifacts in the development environment and project, just as ‘regular’ source code (C#, XML, .js, css). An even bigger benefit is upon deployment: no need for manual actions to publish the form templates to your deployed and provisioned SharePoint application, it’s all done automatic in the context of a feature activation.

SharePoint out-of-the-box provides the XsnFeatureReceiver class to provide the InfoPath Form Templates publishing support. However, the functionality of this standard class is not enough for real-life deployment scenarios. I ran into a few missing parts before, which I handled by extending and augmenting the XsnFeatureReceiver class. This week I was confronted with another issue. After a re-deployment with updated InfoPath forms, the earlier deployed forms were still active. That is, when opening them in the browser. When opening in InfoPath client, the new version was applied. Thus the deployment to SharePoint server was performed, but somehow for InfoPath Forms Services context the earlier deployed version per InfoPath form remained active. I did not experience this effect in my local single-server local development image. But when doing a re-deployment to the distributed test farm-environment, the issue manifests itself.

I inspected the managed forms templates administration of the FormsService SSP, and discovered that still the initial deployed versions were administered. And after uninstalling the InfoPathInfrastructure feature, the InfoPath form templates in FormsService were not cleared. This obstructs the activation of the new versions of the forms upon re-deployment. As solution I extended my InfoPathInfrastructure feature to also make sure the InfoPath forms that it installed, are removed from the FormsService SSP upon uninstalling the feature.

With this fix, I’m able to deploy and activate new versions of the InfoPath forms to the distributed target SharePoint farm environment (test, staging, production) for usage via InfoPath Forms Services. And still conform ALM principles.

AllUsersWebPart and SPWebApplication.ApplyWebConfigModifications don't match

• In a Feature, I apply the AllUsersWebPart construct to automatically add default webparts to all publishing pages based on the PageLayout File.
• In a Feature, I utilize the SPWebConfigModification class to add web.config modifications

When done in separate Features, both constructs operate successfully. However, when included within the same Feature activation, the invocation of ApplyWebConfigModifications method results in a SecurityException:

Access Denied
at Microsoft.SharePoint.Administration.SPPersistedObject.Update()
at Microsoft.SharePoint.Administration.SPWebApplication.ApplyWebConfigModifications()
at Microsoft.SharePoint.Administration.SPWebService.ApplyWebConfigModifications()

It looks as if internally SharePoint somehow puts a lock on the Administration persistent object. I tried to verify my suspicion via Reflector, but not surprisingly this internal code is obfuscated. The runtime error only manifests itself when the Feature is (re)activated via the GUI. The Feature activation via stsadm reports no problem, and successfully performs the deployment work.

Still, I want to hold on to the idea of one single self-contained Feature to deploy all the required parts. And I want to be able to turn the Feature on and off interactively via the SharePoint GUI (so that I can also operate the feature remote, without access to the deployment server). A resolution is to apply another approach to provision the default webparts on the pagelayouts. Besides the usage of AllUsersWebPart, it is also possible to directly include the webpart specifications in the PageLayout file itself:

With this construction, the invocation of SPWebApplication.ApplyWebConfigModifications method is done successfully. And as a bonus, also another problem is prevented; that of magically duplicating the default webparts provisioned via AllUsersWebPart upon Feature re-activation.

A final word of caution. With this approach I at first encountered another problem. When editing on a created publishing page the properties of the default WebPart provisioned via the PageLayout file, SharePoint displayed an error warning in the edit toolpane, "a web part you attempted to change is either invalid or has been removed by another user". Although the net effect of the WebPart settings is simple done, this is not very trustwordy towards the Web Content Manager. Hard to explain that they can just ignore this warning. Before deciding to then have to abandon the approach, I re-examed the default WebPart specification in the PageLayout file to see if anything there could be causing SharePoint to suspect a concurrent update. And yes there was, 'thanx' to the automatic editing behaviour of Visual Studio. When you add a control within an .aspx file, Visual Studio automatically add a default 'ID' property to the control. For the default WebPart however this is not needed, the property is actually set when creating a publishing page off this PageLayout. But the mere presence of it in the default part results in SharePoint detecting a concurrent update. So remember to don't include the 'ID' property within WebPart specifications included within PageLayout files.

PublishingWeb.IsPublishingWeb malfunctions on custom site definition

I typically apply a Feature-based approach for the provisioning of a SharePoint site into a full-functioning application . I prefer this above putting everything within a custom site definition. I'm in good company with my resistance against (overly complex) Onet.xml's [Andrew Connel, You don't need to create site definitions, Joel Oleson, Do you Really Need to Create Custom Site Definitions?]. I actually reuse the same site provisioning engine functionality within multiple SharePoint projects and applications. The reuse is made possible by specifying the provision actions via an XML feed file. Via this, the engine can be instructed to provision structure (Lists, Libraries), content (publishing pages, ListItems), and configuration (web.config via SPWebConfigModification, and PublishingWeb + SPWeb properties).

With respect to the latter, I discovered that the OOTB PublishingWeb.IsPublishingWeb sometimes plays tricks on you. Let me explain. In the provision engine I've implemented functionality to set diverse navigation settings: IncludeSubSites, NavigationShowSiblings, InheritGlobalNavigation, and so on. These properties can only be set on publishing webs, and are in the SharePoint Object API available via a PublishingWeb instance. In the engine code I first do a check whether the SPWeb on which the Feature is activated, is actually a PublishingWeb. And only then via PublishingWeb.GetPublishingWeb access a valid wrapper reference, and set the navigation properties.

This all worked fine in situations in which I applied the provision engine afterwards on an already created Publishing site collection. In my current project I do it slightly different. I've implemented a minimal site definition, and within the context of it activate per web in the created site topology the Publishing feature, and next execute the provision engine. In this setup, the navigation settings appeared to be ignored by the provision engine. When I debugged the provision execution, I discovered that PublishingWeb.IsPublishingWeb returned false for all webs in the site topology. Very strange, since I made sure to first activate the Publishing feature...

I ran into the forum-thread IsPublishingWeb() - what does it REALLY check for? posting a question on this issue. And followed up on the advice given there: implement an own SharePointHelper.IsPublishingWeb method to determine whether the SPWeb instance is valid for being wrapped into a PublishingWeb instance.

BTW: My own experiences learn that it is not due the asynchronous site creation behaviour. Also when later via site settings (de)activating the provision engine feature, PublishingWeb.IsPublishingWeb remains returning false, despite that the Publishing feature is activated on the (sub)web. It looks more as if PublishingWeb.IsPublishingWeb is somehow dependent on something that is set behind the curtains via OOTB Publishing site definitions.

Automatic publish InfoPath forms via Feature-based ALM deployment

Fixed the Feature-based deployment of InfoPath forms to SharePoint site while treating them as first-class source artifacts in VS + TFS environment

I’m a supporter of applying Application Lifecycle Management (ALM) principles within application development projects. This also goes for SharePoint-based custom applications. Basically this means:

1. designing and developing the SharePoint artifacts within a team-based source repository environment (aka, Team Foundation Server);
2. applying version control to the individual application/building blocks;
3. daily build;
4. and full automatic and repeatable deployments

With InfoPath development, this gives some challenges. First of all, the InfoPath forms are setup and maintained via its own designer-tool, namely the InfoPath Designer client; and not directly from within the scope of Visual Studio. However, the most significant issue is the manner of deployment: the InfoPath client contains functionality to publish the InfoPath form to the destination environment, with SharePoint site as one of the available options. Given my ALM aspirations, I want to copy this InfoPath client deployment-experience to within the regular SharePoint deployments, via the Solution framework and SharePoint Features. In essence, I want to achieve the following:

1. Automatically provision the required Information Architecture artifacts:
   • SiteColumns
   • Masterdata for LookupFields
   • ContentTypes
   • Forms Libraries, associated with the ContentTypes, for administration of the filled in InfoPath forms
2. Automatically provision and activate the required InfoPath infrastructure
• InfoPath forms
• DataConnections

The first category, the Information Architecture artifacts, is well-known SharePoint provisioning. It can be done via a Site Definition, Features, or a combination. I favor the Feature-based approach, because it is modular, and allows me to turn it off and on.

The second category gave me more challenges. On internet search, I came across some posts that mentioned the SharePoint OOTB XSNFeatureReceiver that aids in here. However, it quickly appeared to me that it only takes care of one part, namely registrating the InfoPath form templates within the target SharePoint site. To successfully achieve this, XSNFeatureReceiver has constraints on the deployable InfoPath forms. And when even a single one of them does not comply to these constraints, XSNFeatureReceiver rather silently fails to correctly install the forms. Another missing part is for the DataConnections. The upload is not the issue here, being standard SharePoint Module functionality. But rather adjusting them to the specific target environment. In my ALM principles, I want to hold on to 1 single instance per DataConnection file, and not be forced to maintain versions for each of them per target environment (development, test, staging, production).

Luckily, Microsoft has not made XSNReceiverFeature sealed. So I decided to overload it to augment the standard deployment functionality with the above described aspects. Adjusting the DataConnections appeared rather simple, with thanks to this blog (Building InfoPath Form Services Solutions using Visual Studio 2008), including a link to demo provision code. I took this as basis to realize a generic approach, feeded by an XML configuration file delivered within the feature scope

The correct deployment of the InfoPath forms appeared to be more of a challenge. Eventually, I found out that the direct cause of the failed deployment was that I earlier on published the forms during my development and testing of the forms. As soon as you publish an InfoPath form, the InfoPath client administrates this within the form itself. And this appeared to result in the malfunctioning XSNFeatureReceiver based InfoPath forms registration. This can be mitiligated by correcting the InfoPath forms. An alternative is to modify the original InfoPath form artifacts. However, this would then be a corrective action to be repeated every time you’d published an InfoPath form just for an InfoPath designer (programmer) test. I therefore went for the alternative to do the correction as part of the Feature installation procedure. This way during the development phase the team can just go one and (re)publish the forms, without worrying about the effect on deployment later on.

Another thing to take into account when doing Feature-based deployment of InfoPath forms, is that the Forms must be contained at the Feature root-directory. I like to structure Feature contents within subfolders for different deployable types. ContentTypes, SiteColumns, DataConnections, and also InfoPath forms, each within their own subfolders within the feature. However, in case of InfoPath forms deployed via XSNFeatureReceiver, this is not working. Without a concrete error message, the registrating will then just not be performed. So keep your Forms at the Feature root-level.

Deployment steps, and thus functionality of InfoPathInfrastructure feature

Upon Feature installation event:

1. Fix the forms files, for successful automatic publishing via XSNFeatureReceiver
   • Extract the forms; this on itself presented a challenge, since .NET does not provide standard cabinet (.cab) handling
   • Inspect the manifest.xsf file on the presence of publishUrl and trustLevel; remove any of them present
   • Repack the .xsn container / file

Within the Feature Activation:

1. upload forms files to FormsServerTemplate library
2. Registrate the forms
3. upload data connections to DataConnections library
4. provision site columns
5. provision master data lists
6. fix the lookup references
7. provision content types; including reference to the uploaded forms
8. provision contents/data libraries
9. assign content types to the contents libraries
10. fix retrieve data connections --> associate with the master data list in the deploy environment (iso your local / development source)
11. fix submit data connection --> associate with the content data list in the deploy environment (iso your local / development source)

Automated approach for initializing Enterprise Search experience

Whenever you want to utilize the Enterprise Search functionalities in your application, you must take into account for correct initialization: set up a content source, administer its crawling scheme, create the searchable managed properties, set up a Search Scope. Although the different steps can be done manually via the SharePoint GUI (combination of Central Admin and your own application), this is less workable within the context of an ALM based project. Your application is then multiple times (re)deployed, and to different environment (development, test, staging, production). Each time the manual installation/initialization steps would need to be repeated. This is cumbersome, and [thus] error prone. A better approach (as always) is to strive for a fully automated initialization of the enterprise search. I’ve applied this several times via the approach outlined here:

• create a new Feature, with a FeatureReceiver codebehind
• In the activation method of the feature, do the following steps:

Create the content source

1. use the Search Object Model to create the content source
2. if applicable, administer include and exclude rules
3. create the crawl schemes; full and incremental


Create managed properties

Important to realize here is that a managed property can only be made if the mapping crawled property is available.


4. make sure the crawled content source contains at least one item, by either adding a dummy listitem (for regular Lists), or uploading a dummy document (for document library)
5. Use the content type(s) definition(s) to determine the fields of the searchable content, and assign per field a non-nil value
6. Initiate a full crawl, in order to let the crawler make up crawled properties for each of the content type(s) fields
7. After the full crawl, loop through the collection of determined content type(s) fields, and for each field create a Managed Property of the proper type, and associate it with the automatically created crawled property
8. Remove the dummy content(s) of step 4


Create the Search Scope

9. use the Search Object Model API to create a Search Scope

• In the deactivation method of the feature, do the proper reversible actions of the feature activation event.

What is proper, is situation / application dependent. Normally, you would implement in the feature deactivation a full restore to the status before the feature activation. Here that means removal of the managed properties, crawled properties, search scope and content source. However, when you delete the content source, you typically undo more than strict the feature activation. In a production situation, the content source has been crawled and crawled, building up the index administration. Upon content source deletion, you also loose al this hard work content crawling and indexing. Feature deactivation would then thus not only undo the feature activation itself, but also work done later. Whether this is appropriate, depends on the application and content specifications. Every content can be recrawled. However, for a large and complex set (documents, .pdf’s, TIFF files, LOB via BDC…) this can be time consuming, and during the required full crawl the application search cannot find and return all search requests.

Tip: use Lookup iso Choice field for (semi)fixed set of defined values

Often, in your SharePoint information architecture there are some fields identified with a set of defined allowed values. A common approach is then to apply the Choice fieldtype herefore. For real fixed set of possible values this is a very sensible approach. For instance, for datatype sex; we'll have 'male', 'female', and well 'unknown' in case of doubts. However, more than often the set is (semi)fixed: departments within a company, the customer base of an IT consultancy, etc. In such situation, usage of Choice is not flexible nor user-friendly towards the functional managers of the application. For each modification, it is required to change the definition of the Choice-based field. And one must then also not forget to propagate this change to the lists on which the field / site column has been applied (direct, or via contenttype). All this is more technically doing, than SharePoint functional management. A better approach is to utilize the Lookup fieldtype, and refer to another list with in it the currently known set of defined values (masterdata values actually). Whenever a modification to this set is in order, functional management can suffice with adjusting this masterdata list. Actually, this is just sane data model normalisation. Like in such context, in addition to defining the set of allowed values, it is also possible to augment them with more details. For instance, department name (allowed value), and in another list column more details of the department.

Something you'll have to take into account when applying this datamodelling, is a peculiarity upon provisioning the SiteColumns. The way Lookup SiteColumn are administered in the SharePoint content database, is with the Guid of the referential SharePoint list. This results in a problem when provisioning the SiteColumns the SharePoint standard way via Feature, with the field specifications in XML. The ID of the referential list is typically unknown at coding/specification time, and will be different per environment provisioning. This is a known issue, and so there is also a known resolution. I'm not going to describe it here, of even try to take the credits for it. Frankly, I've used a blog-entry of Chris O'Brien, Creating Lookup columns as a feature, as start information source for hinting how to solve this sequential provision issue.

Tip – Provision custom publishing masterpage, pagelayouts and webparts always via a Feature with Site-scope, not Web-scope

The masterpage and webpart galleries for publishing sites live at the top level site (rootWeb) in a sitecollection, not at individual sites. This can be misleading, since every subsite get its own ‘_catalog/masterpage’ and ‘_catalog/wp’ folders. The Publishing functionality however only goes to the gallery at the top level site; for the masterpages (Site and System), pagelayouts as well as for building up the list of selectable webparts.
A best practice is to provision your custom masterpage, pagelayouts and .webpart definition files via a Feature. Thereby you must specify the level at which the feature is applied. For MasterPage, PageLayouts and .webpart files the appropriate level appears thus to be ‘Scope=Site’. However, it is also allowed to set ‘Scope=Web’, as long as you’re activating the web-scoped feature in the rootweb. But when you activate such a feature in a subsite, the results are less. It appears as if the SharePoint feature mechanism simple ignores the request to provision masterpage and pagelayouts into ‘_catalog/masterpage’, and .webpart files into ‘_catalog/wp’. They are simple not added anywhere. Not at the indicated subsite level, which would also be useless given the publishing behaviour. But also not at the rootweb level. Annoying is that no signal of this is given, no error reporting but simple ignored to provision the SharePoint files at the incorrect subsite level. Ok that stsadm doesn’t signal it, but I would have expected an error indication when activating the feature via the GUI. And at the least I expected to find some information about it in the extensive SharePoint logs in the 12hive. In reality none, instead the feature framework just silently ignores the feature specifications which are logically incorrect when activated at subsite level.

So, what then to do when you what to provision content to a publishing site at subsite level ? I recommend the following 2-steps approach:

1. Provision your masterpage, pagelayouts, and .webpart files to the catalog gallery at rootlevel; apply a Feature with scope = Site.
2. Provision your site structure (libraries, lists) and content via a Web-scoped Feature; and activate this against your specific site, the root or a child subsite. E.g. http:<root-entry>/Intranet

Workaround shortcoming of SPWebConfigModification behavior wrt explicit positioning childNode

I structurally apply the SPWebConfigModification class upon all my SharePoint deployments to propagate the needed web.config modifications to the target environment (be it my local VM or in test/staging/production environments, typically in a farm). Benefits of using this functionality are:

• All web.config modifications are done automatically. No need to have someone (yourself or in production, IT operations) do this manually by editing the web.config file(s);
• Addition of the modifications becomes part of your ALM process; they can be tested and validated before going to production, and repeatedly fully automatic executed;
• The same web.config modifications are propagated throughout the entire farm; and even when later on a new WFE or a new extension is added;
• Added modifications can also be neatly removed upon uninstalling your SharePoint application.

Although I’m a big fan of this functionality, I’m also aware of some of its shortcomings. One of this is that upon adding a new childNode, it will always be appended as the last childNode in its parent webconfig node. Sometimes this is not good enough, and you need to be able to have it inserted at a specific position. In my case it was needed for enabling an ExceptionHandler HttpModule. To have this correctly operating, it must be the first active httpModule in the pipeline. Typically however the httpModules section in the web.config already has multiple httpModule nodes added:

Initial HttpModules section for a new provisioned publishing portal

The standard behavior of SPWebConfigModification places my ExceptionHandler behind them, effectively making it non-function. I could resort to manually correct my / all the web.configs; yeah right, you gotta be kidding…

Instead I decided to devote some time to investigate how to make a full-automatic correction for this. Via Reflector I inspected the runtime operation of the SharePoint ApplyWebConfigModifications method. Then I did an attempt to ‘break in’ on this functionality by a combination of first removing the already present httpModule childnodes, next add my exceptionHandler so it be the first, and then add the original httpModule nodes again. However, this was unsuccessful; and the outcome unpredictable. I suspect that an explanation for this is that my application is not the ‘owner’ of the (re)moved nodes. But that is hard to determine, since that requires a deep insight in the inner internals of this SharePoint behavior. After this I came up with a solution to correct the sequence in the httpModules afterwards the invocation of SharePoint’s ApplyWebConfigModifications. And this works satisfactory!! I must admit it has one weakness in that this afterwards correction will not be applied when later the farm is extended with new servers or web app zones. But honestly, seeing the SPWebConfigModification code I do not have a clue to make that happen. Another issue to watch out for is that SharePoint administrates in its config database all commanded web.config modifications, and reapplies them all upon each ApplyWebConfigModifications. I solved that by administrating in SharePoint’s PropertyBag the name of the ExceptionHandler, and then after each invocation of ApplyWebConfigModifications invoke the correct positioning of this HttpModule.

Code extracts:

1. Detect whether an ExceptionHandler HttpModule is provisioned

2. Set required ordering of childNode in each web.config

And ultimately, this gives me the desired system/debug-enabling functionality in the deployed application:

Oops, apparently I've forgotten to enable an implementation for the IExpensesHandling interface...