Friday, October 29, 2010

Useful links for upgrade from Sharepoint 2007 to 2010

Here I would like to provide useful links which I found during investigation of the upgrade process from Sharepoint 2007 to Sharepoint 2010. I’m planning to update this post in the future with more resources. Currently there will be links from official MS technet guide which are located in various places. For convenience I summarized them in one place as flat list:

Upgrade and Migration for SharePoint Server 2010 – starting point for investigation

Determine upgrade approach – describes 2 possible upgrade strategies: In-place upgrade and Database attach upgrade

Upgrade process overview – contains steps which should be performed for both types of upgrade

Review supported and unsupported upgrade paths – information about supported source and target topologies for upgrade

Hardware and software requirements - software and hardware requirements for upgrade to  Sharepoint 2010

Use a trial upgrade to find potential issues – describe process of test upgrade which should be performed prior to make production upgrade for searching possible issues

Microsoft SharePoint 2010 Products — Test Your Upgrade Process – Visio poster with upgrade schema

As I said I’m going to update this post in the future and use as a reference in forums threads where people ask about upgrade process.

Thursday, October 28, 2010

Create custom sites in Sharepoint using site definitions: step by step guide - part 2

In the previous part of this series I finished on creation of portal web manifest for branches of Contoso company. In this second part I will continue explanation of site creation customization for Sharepoint.

So  we created portal web manifest file with portal structure. For convenience I will duplicate it here:

   1: <?xml version="1.0" encoding="utf-8" ?>
   2: <!-- _lcid="1033" _version="12.0.4518" _dal="1" -->
   3: <!-- _LocalBinding -->
   4: <portal xmlns="PortalTemplate.xsd">
   5:   <web name="Home" siteDefinition="ContosoBranchSite#0" displayName="$Resources:ContosoBranch,Site_Home_DisplayName;" description="">
   6:     <webs>
   7:       <web name="Sales" siteDefinition="ContosoBranchSite#1" displayName="$Resources:ContosoBranch,Site_Sales_DisplayName;" description="" />
   8:       <web name="IT" siteDefinition="ContosoBranchSite#2" displayName="$Resources:ContosoBranch,Site_IT_DisplayName;" description="" />
   9:       <web name="HR" siteDefinition="ContosoBranchSite#3" displayName="$Resources:ContosoBranch,Site_HR_DisplayName;" description="" />
  10:     </webs>
  11:   </web>
  12: </portal>

There are separate <web> element for each sub site in portal. Also we noticed that each <web> tag contains siteDefinition attribute which actually tells Sharepoint what site definition should be  used for this site:

  • ContosoBranchSite#0 – root portal site;
  • ContosoBranchSite#1 – Sales subsite;
  • ContosoBranchSite#2 – IT subsite;
  • ContosoBranchSite#3 – HR subsite.

ContosoBranchSite here is template name of site configuration, and number after # is configuration id. They are defined in separate webtemp file (don’t mess with webtempContosoBranchSitePortal.xml which is webtemp for portal itself. But now we are talking about webtemp file file for sub sites configuration). Similar to portal, webtemp file with sub sites declaration is located in 12/Template/{lcid}/xml folder. In our case this is 12/Template/{lcid}/webtempContosoBranchSite.xml (in order to avoid mess you should keep naming convention similar, so if we had webtempContosoBranchSitePortal.xml file for webtemp which contains ContosoBranchSitePortal template, for sub sites we will use webtempContosoBranchSite.xml name because it will contain ContosoBranchSite template). Content of webtempContosoBranchSite.xml is the following:

   1: <?xml version="1.0" encoding="utf-8" ?>
   2: <Templates>
   3:   <Template Name="ContosoBranchSite" ID="10002">
   4:     <Configuration ID="0" Title="Home" Description="" Hidden="TRUE" ImageUrl="/_layouts/images/stsprev.png" DisplayCategory="Contoso" />
   5:     <Configuration ID="1" Title="Sales" Description="" Hidden="TRUE" ImageUrl="/_layouts/images/stsprev.png" DisplayCategory="Contoso" />
   6:     <Configuration ID="2" Title="IT" Description="" Hidden="TRUE" ImageUrl="/_layouts/images/stsprev.png" DisplayCategory="Contoso" />
   7:     <Configuration ID="3" Title="HR" Description="" Hidden="TRUE" ImageUrl="/_layouts/images/stsprev.png" DisplayCategory="Contoso" />
   8:   </Template>
   9: </Templates>

Here we defined template with ContosoBranchSite name. For this template we have 4 different configurations which were used in ContosoBranchSitePortalWebManifest.xml (of course you can use the same configurations for different sub sites in portal web manifest, but in our example all sites have different configurations as this is most common case). Note that in webtempContosoBranchSite.xml titles of configurations are the same as names of webs in portal web manifest. Also don’t mess with it: in portal web manifest “name” attribute of <web> tag means URL of sub site, while in webtemp file “Title” attribute of <Configuration> tag is just title of configuration. Note also that we specified Hidden=”TRUE” in order to hide these configurations from site template picker control on site creation page. We don’t need to create these sites separately – we need to create a whole portal with them in one operation.

Now lets return to webtemp file of portal and see what is the difference with webtemp file for sub sites. The main difference is that <Configuration> element for portal contains the following attributes:

  • ProvisionAssembly;
  • ProvisionClass;
  • ProvisionData.

These 3 attributes (and their values of course) make portal from a site. I described the meaning of these attributes in part 1. Another difference is that we have 4 different configurations for ContosoBranchSite template (sub sites template), and only 1 configuration for ContosoBranchSitePortal template (portal template).

So we created webtemp file for sub sites. Now it is time to define each sub site’s configuration. As I said in previous part, configurations are defined in onet.xml files stored in 12/Templates/SiteTemplates/{TemplateName}/xml folder. It is important to have folder name the same as “Name” attribute of <Template> element in webtemp file (although it is possible to override it using SetupPath attribute of <Template> element). In our case it will be 12/Templates/SiteTemplates/ContosoBranchSite/xml/onet.xml file.

How can we create onet.xml file for custom site template? It depends on your requirements. I prefer to copy OTB onet.xml file which corresponds to site template which is closest to particular business requirements. E.g. you can use OTB Team site or Publishing site template as base for further customization. As I said in first part we will use Publishing site in our examples as it is widely used in real life scenarios.

But there is one problem: there are many OTB onet.xml files in 12/Templates/SiteTemplates folder. How can we determine what file belongs to Publishing site template? Of course I could just say that it is located in this specific file, but it will not be so useful. I want to show you approach which you can use in order to make it by yourself in your development practice.  So one way to determine onet.xml for Publishing site is to search across all files content with “Publishing site” occurrence. But there is simpler way. If you already have site created using Publishing site template, then you can trace its properties into console using Sharepoint object model. We are interesting in the following properties of SPWeb class:

Property of SPWeb Mapped provision artifact
WebTemplate “Name” attribute of <Template> element in webtemp file
WebTemplateId “ID” attribute of <Template> element in webtemp file
Configuration “ID” attribute of <Configuration> element in webtemp and onet.xml

In order to trace these properties you can create simple console utility. I used my Resharper live template for Sharepoint utilities for that:

   1: class Program
   2: {
   3:     static void Main(string[] args)
   4:     {
   5:         if (args.Length != 1)
   6:         {
   7:             Console.WriteLine("Usage: exe <site_collection_url>");
   8:             return;
   9:         }
  11:         using (var site = new SPSite(args[0]))
  12:         {
  13:             using (var web = site.OpenWeb())
  14:             {
  15:                 Console.WriteLine("WebTemplate: {0}", web.WebTemplate);
  16:                 Console.WriteLine("WebTemplateId: {0}", web.WebTemplateId);
  17:                 Console.WriteLine("Configuration: {0}", web.Configuration);
  18:             }
  19:         }
  20:     }
  21: }

So you just need to specify URL of the existing site in the command line and program will trace its properties. Here is its result for site created with Publishing site template:

   1: WebTemplate: CMSPUBLISHING
   2: WebTemplateId: 39
   3: Configuration: 0

There is only 1 file in 12/Template/{lcid}/xml folder which contains reference on CMSPUBLISHING: webtempsps.xml. After investigating it we will find that Publishing site onet.xml is located in 12/Templates/SiteTemplates/Publishing/xml folder:

   1: <Template Name="CMSPUBLISHING" ID="39" SetupPath="SiteTemplates\PUBLISHING">
   2:    <Configuration ID="0" Title="Publishing Site" Hidden="FALSE" ImageUrl="/_layouts/1033/images/PublishingCollaborationSite.gif" 
   3:        Description="A blank site for expanding your Web site and quickly publishing Web pages. Contributors can work on draft versions of pages and publish them to make them visible to readers. The site includes  document and image libraries for storing Web publishing assets."
   4:        FilterCategories="PublishingSiteTemplate"
   5:        SubWebOnly="TRUE"  DisplayCategory="Publishing"
   6:        VisibilityFeatureDependency="F6924D36-2FA8-4f0b-B16D-06B7250180FA">
   7:   </Configuration>
   8: </Template>

So all we need is just go to 12/Templates/SiteTemplates/Publishing/xml  and copy onet.xml to our folder 12/Templates/SiteTemplates/ContosoBranchSite/xml. Now our sub sites are ready for customization.

By default onet.xml of Publishing site template contains only one configuration

   1: <?xml version="1.0" encoding="utf-8" ?>
   2: <!-- _lcid="1033" _version="12.0.2220" _dal="1" -->
   3: <!-- _LocalBinding -->
   4: <Project Revision="3" Title="Publishing Site" ListDir="Lists" xmlns:ows="Microsoft SharePoint">
   5:  <NavBars>
   6:     ...
   7:  </NavBars>
   8:     <ListTemplates>
   9:     </ListTemplates>
  10:     <DocumentTemplates>
  11:         ...
  12:     </DocumentTemplates>
  13:     <BaseTypes>
  14:     </BaseTypes>
  15:     <Configurations>
  16:         <Configuration ID="-1" Name="NewWeb"/>
  17:         <Configuration ID="0" Name="Publishing"> 
  18:             ...
  19:         </Configuration>
  20:     </Configurations>    
  21:     <Modules>
  22:         ...
  23:     </Modules>
  24: </Project>

As you remember for our portal we have 4 different configurations, so the minimal change we need to do – is to copy default configuration with id=”0” 4 times in our onet.xml and paste it with different ids and names in order to correspond webtempContosoBranchSite.xml file:

   1: <Configurations>
   2:     <Configuration ID="0" Name="Home">
   3:         ...
   4:     </Configuration>
   5:     <Configuration ID="1" Name="Sales">
   6:         ...
   7:     </Configuration>
   8:     <Configuration ID="2" Name="IT">
   9:         ...
  10:     </Configuration>
  11:     <Configuration ID="3" Name="HR">
  12:         ...
  13:     </Configuration>
  14: </Configurations>

After that we can create our portal and all sub sites (including portal root site) will be created using the copy of OTB Publishing site template. At this moment sub sites have absolutely the same features as OTB publishing sites. But there is one big difference: we created them using custom site definitions so we are free now to add functionality to these sites as we wish without affecting OTB Publishing site template. But this is the theme of the next part of the series.

Wednesday, October 20, 2010

How Sharepoint search may impact your performance with readonly hosts file

Recently we encountered with serious performance issues on one of production environment. I tried many solutions. First of all I tweaking ASP.Net configuration based on this tutorial. It helped for some time but after several weeks issues were back. After deeper investigation I noticed that on one of WFE there are many following errors in event log:

   1: Application Server Administration job failed for service instance Microsoft.Office.Server.Search.Administration.SearchServiceInstance (0fb6066b-7fd5-4926-a28e-2348c0ae1256).
   3: Reason: Access to the path 'C:\WINDOWS\system32\drivers\etc\HOSTS' is denied.
   5: Techinal Support Details:
   6: System.UnauthorizedAccessException: Access to the path 'C:\WINDOWS\system32\drivers\etc\HOSTS' is denied.
   7:    at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   8:    at System.IO.FileInfo.Delete()
   9:    at Microsoft.Search.Administration.Security.HOSTSFile.CleanupDedicatedGathering(Hashtable HOSTSFileMappings, StringBuilder HOSTSComments, IEnumerable obsoleteHosts, String dedicatedName, Boolean isDirty)
  10:    at Microsoft.Search.Administration.Security.HOSTSFile.ConfigureDedicatedGathering(SearchServiceInstance searchServiceInstance, SPServer dedicatedWebFrontEndServer, IList`1 previousWebApplicationHostNames)
  11:    at Microsoft.Office.Server.Search.Administration.SearchServiceInstance.SynchronizeDefaultContentSource(IDictionary applications)
  12:    at Microsoft.Office.Server.Search.Administration.SearchServiceInstance.Synchronize()
  13:    at Microsoft.Office.Server.Administration.ApplicationServerJob.ProvisionLocalSharedServiceInstances(Boolean isAdministrationServiceJob)

I checked 12/logs folder on this WFE and found that with regular Sharepoint logs files there were a lot of files with the following names: (where the last mm – is for milliseconds). There was ~ 60000 of such files (as I investigated for acceptable performance number of files in directory should not be more than 30000). When I tried to open that folder in Windows explorer it hang for several seconds. So I got the following idea: as these hosts files were located in the same folder as regular logs, Sharepoint may hang when it tries to write log records because of the same reasons which caused Windows explorer to open 12/logs folder very slow. I removed all hosts* files manually and started monitoring.

Performance issues seems disappeared but symptoms were not removed: someone still copied hosts* files into 12/logs every minute (1 file per minute). Keeping in mind the error from event log mentioned above the first suspect was Office Sharepoint Server Search (don’t mess with Windows Sharepoint Services Search). It was true, because when I disabled it, files were not copied anymore (I needed to disable it, i.e. not stop, because Sharepoint restart it every minute itself). But with disabled Search service search didn’t work at all. Users gets the following error when tried to click Search button in standard search box:

The search request was unable to connect to the Search Service

But why Search service copied hosts* files every minute. I notices that hosts file in C:\WINDOWS\system32\drivers\etc folder is readonly. In order to explain why it was made readonly I need to describe environment configuration. Production environment is a farm with several WFEs. Each WFE contains 2 network cards so it has 2 IP addresses (one per card). Our web application is extended to be used both with NTLM and FBA. And each extended web app uses its own IP address (via bindings configuration in IIS manager). It was made because of performance considerations.

Search service is running on one of WFEs (guess which server – yes exactly this WFE where I found mentioned errors in event log and which contains many hosts* files in 12/logs folder). And according to MS guide it was configured to use dedicated WFE for crawling: Configure a dedicated front-end Web server for crawling (again in order to increase performance by minimizing network traffic as crawler uses local server only in this case). But in the same article there is a note about potential problems with this approach:

Possible problems

In some cases, the timer service writes the incorrect IP address to your Hosts file. (For more information, see the blog post at This can cause problems ranging from inability to crawl content to inability to view sites, such as the Search Services Provider (SSP) or Central Administration site. The timer service can add an incorrect IP address to the Hosts file in cases such as the following:

  • The server that you specified as your dedicated front-end Web server for crawling has multiple IP addresses assigned to one or more network cards.
  • Your server farm is using network load balancing.

If either of these conditions is true, we recommend that you add the entries to the Hosts file directly instead of using the user interface to specify a dedicated front-end Web server for crawling.

As I wrote above each WFE contains several network cards. And that was the problem. Search service tried to modify hosts file with incorrect IP addresses. But making hosts file readonly had side effect: Search service copied it into 12/logs folder every minute. And that was the reason of performance issues when too many files accumulated in 12/logs.

The same link contains solution for this problem: see Configure a dedicated front-end Web server for crawling by editing the Hosts file. This was quite unobvious reason for performance impact. Hope this information will help you in your investigations.

Sunday, October 10, 2010

SPGraphviz project is released on Codeplex: graph visualization in Sharepoint without programming

Hello everybody. I’m glad to announce that today (10.10.2010 – quite a nice date :) ) I released SPGraphviz project on Codeplex: With SPGraphviz you can create your own graphs, schemas, graphical charts, etc and display them in Sharepoint without programming and external applications. With SPGraphviz you can make graphical representation of organization schemas, portal hierarchies, file version history, etc (applied use cases are limited only by you fantasy). Here is example of what graphs you can create (given from Graphviz gallery):

 image image image image

SPGraphviz is implemented based on open source library for rendering graphs Graphviz, implemented by specialists of AT&T company quite far ago. It it C library and SPGraphviz uses managed wrapper from David Brown (little modified) to make a calls to native functions. Graphs are defined in regular .txt file using DOT language. It is specific DSL which allows to define graph structure (nodes, relations, titles) and layout (colors, size, orientation, etc). DOT is quite rich language, but for simple solutions you don’t need to deep into it. E.g. look at the following graph:


It shows example of how SPGraphviz can be used to display portal hierarchy.We have root site, sub site (or sub site collection) for departments under which we have 3 department sites: IT, HR, Sales. Also there is sub site collection for personal sites, under which users can create their own sites (analogue of MySites). This graph can be created using the following DOT definition:

   1: digraph example {
   2:         size="6,6";
   3:         node [color=lightblue2, style=filled];
   4:         "Start page" -> "Departments";
   5:         "Start page" -> "News";
   6:         "Start page" -> "Personal sites";
   7:         "Departments" -> "IT";
   8:         "Departments" -> "HR";
   9:         "Departments" -> "Sales";
  10:         "Personal sites" -> "Alexey Sadomov";
  11:         "Personal sites" -> "...";
  12: }

Here we defined digraph (digraph example {}), and inside it we specified nodes and relations between them:

   1: "Start page" –> "Departments";

Also we specified some layout settings: image size (size=”6,6”) and node color and fill style (node [color=lightblue2, style=filled]). I think this example is quite straightforward for understanding and you can use it as starting point for working with SPGraphviz.

So how SPGraphviz is used in Sharepoint? It contains special web part SPGraphvizWebPart which can render a graph based on DOT definition in text file. At first you need to install SPGraphviz on your server. It is released as regular wsp package. You need to install it as described in

  1. Download and install Graphviz library (choose Download > Windows > Stable and development Windows Install packages) on web server. During installation ensure that you checked "Everyone" on first wizard step
  2. Download latest release of SPGraphviz. Currently release contains regular wsp package
  3. Install SPGraphvizWebPart.wsp on your server. Here are steps for single-WFE environment:
   1: stsadm -o addsolution -filename SPGraphvizWebPart.wsp
   2: stsadm -o deploysolution -local -allowgac -allcontenturls -name SPGraphvizWebPart.wsp
   3: stsadm -o activatefeature -name SPGraphvizWebPart -url

Note that in order to use SPGraphviz you need to install Graphviz on your web server as well. Suppose that you installed all necessary components. Now you can define your graph using DOT language and display it on publishing page using SPGraphvizWebPart. Lets for our example use DOT definition showed above.

In order to display graph in Sharepoint, you need to upload text file with DOT definition into some documents library on your site collection. After that go to some publishing page and add SPGraphvizWebPart (it should be located under Graphviz group in web parts list) on the page. The last action you need to perform – is to go to web part properties (Modify Shared Web Part) and specify absolute URL of the file which you uploaded in “Dot file URL” property (under Custom settings category in web part properties editor) and click Apply (there is restriction on hosts which can be used as a location for DOT definitions. By default you can only use the same host where SPGraphvizWebPart  is installed. See for instructions how you can use external hosts to store DOT definitions). After that you should see graphical representation of the graph based on textual definition:


I described use case for non-programming graph creation in Sharepoint. But SPGraphviz also brings really interesting opportunities for developers. Just realize that graph definition is made in textual form. It means that developers may implement custom code which will create such DOT definition automatically based on some data in and then just setup SPGraphvizWebPart to show this definition. For example, there is an example how we can visualize site hierarchy of portal:

   1: class Program
   2: {
   3:     static void Main(string[] args)
   4:     {
   5:         if (args.Length != 1)
   6:         {
   7:             Console.WriteLine("Usage: exe <site_collection_url>");
   8:             return;
   9:         }
  11:         Console.WriteLine("digraph sites {");
  12:         Console.WriteLine("size=\"6,6\";");
  13:         Console.WriteLine("node [color=lightblue2, style=filled];");
  15:         using (var site = new SPSite(args[0]))
  16:         {
  17:             using (var web = site.OpenWeb())
  18:             {
  19:                 foreach (SPWeb subWeb in web.Webs)
  20:                 {
  21:                     iterateSubWebs(subWeb, web.Title);
  22:                 }
  23:             }
  24:         }
  25:         Console.WriteLine("}");
  26:     }
  28:     private static void iterateSubWebs(SPWeb web, string parentNode)
  29:     {
  30:         if (web == null)
  31:         {
  32:             return;
  33:         }
  35:         Console.WriteLine("\"{0}\" -> \"{1}\";", parentNode, web.Title);
  37:         foreach (SPWeb subWeb in web.Webs)
  38:         {
  39:             iterateSubWebs(subWeb, web.Title);
  40:         }
  41:     }
  42: }

The program is quite simple – it iterates recursively through all sites and adds according nodes into result DOT definition. In order to get graph you need to run this program and redirect output to file:

   1: GraphBuilder.exe > graph.txt

Here is example of running this program on the site created using OTB Collaboration Portal site template:


As you can see there can be many interesting use cases which can be implemented using SPGraphviz. I hope that with this project your sites in Sharepoint will be more attractive and will make your customers happy. Stay tuned and share your ideas about how SPGraphviz can be used in real life. As in another our open source project Camlex.NET I’m always open for your feedback and hope that it will be the main source of SPGraphviz improvements.