Solution partitioning and dependencies with mixed scopes causes farm-deployment trouble

We probably all agree that it is wise to partition your SharePoint application deliverables and deployment into multiple packages. Typical examples are a Base assembly, deployed via a base wsp, providing a general foundation layer with diverse technical and functional building blocks. And then 1 or more functional packages, that provision the SharePoint artifacts and code for a specific application functionality; in which the functional building blocks may use and rely on parts provided via the Base assembly.

However, sometimes this architectural sane decision to partition your application deployment, can result in unforeseen effect. Let me demonstrate via a real-life example, that I encountered today:

• Application deployment deliverable
  1. includes a Base assembly, deployed via solution package Base.wsp. In this wsp package included the Base code, some basic Features, JobDefinitions, generic webparts. In the Base code / assembly there are FeatureReceivers, WebParts. As result of this, SharePoint solution generation (WSPBuilder or VS2010) puts SafeControl settings per contained WebPart in the Solution manifest; which again results in it that the scope of the Solution package is WebApplication.
  2. and also contains a pure functional package; with some Features, SiteTemplates. This functional package does not contain an assembly itself, but instead relies on the assembly of the Base.wsp for reused FeatureReceivers. As result this solution package does not contain any artifacts with are scoped at WebApplication level, and SharePoint solution framework requires to deploy it global in the farm. (nb: the further described situation can even occur in situation were the package does contain managed code, as long as the assembly does not contain SharePoint coded artifacts which transition the scope of the Solution package to WebApplication; e.g. no WebParts, Resources).
• The order of Solution package deployment is to first install the Base.wsp, and then the Functional.wsp. Then you would expect that the deployment dependency of Functional.wsp on Base.wsp is ascertained. However, in this scenario I discover in the SharePoint Solution store an error with the deployment of Functional.wsp. Error details:
  <server 1> : The solution was successfully deployed.
  <server 2> : The solution was successfully deployed.
  <server 3> : Feature '<GUID>' could not be installed because the loading of event receiver assembly "Base, Version=1.0.0.0, Culture=neutral, PublicKeyToken=13a12821e9f69237" failed: System.IO.FileNotFoundException: Could not load file or assembly 'Base, Version=1.0.0.0, Culture=neutral, PublicKeyToken=13a12821e9f69237' or one of its dependencies. The system cannot find the file specified.

At first, I thought this to be a timing issue. The Solution Framework uses timer jobs to deploy solutions throughout the farm, and it looked as if the Functional.wsp was tried to be already deployed on <server 3> while the deployment job for Base.wsp was not yet executed for <server 3>. However, also upon re-deployment of Functional.wsp - while making sure at forehand that the deployment of Base.wsp was now fully completed in the farm - , it resulted in the same error on <server 3>. So it must instead be directly related to the installation on <server 3> apparently the Base.dll was not installed in the GAC on that server; while the Features within Functional.wsp were to be installed on <server 3>. Why is this?

Well, the explanation is twofold. It relates to the SharePoint Farm infrastructure, as well as the scopes of the SharePoint packages. <server 1> and <server 2> have the SharePoint webapplication role in our load-balanced farm; and each contain the SharePoint webapplications. <server 3> however is dedicated as Search Index server; and does not host any SharePoint web applications. The SharePoint solution framework deploys WebApplication scoped packages only to the servers on which SharePoint web applications are hosted (Base.wsp: <server 1> + <server 2>). But it deploys Global scoped packages to ALL the servers in the farmer (Functional.wsp: <server 1>, <server 2> AND <server 3>).

Given this insight, a proper SharePoint Solution Deployment approach is to deploy the Global scoped Package with Force-flag, thus suppressing the error signalment. This worked sufficient in our case. However, it leaves you with the somewhat unwanted situation that via the Force-flag you suppress all error notifications; thus may be too brute. A better approach is to circumvent that Solution Framework deploys your Application package also to the servers on which the web application is not hosted. This can thus be achieved by making sure that each of your Application package as WebApplication scope. Either because it inherently contains such entities (e.g. WebParts), or artificially by including a dummy artifact directed at WebApplication level (e.g. an empty virtual bin file).