Friday, February 28, 2014

PowerShell script for creating and mapping managed properties in Search service application in Sharepoint

If you develop search-driven site for Sharepoint 2013 then you often need to configure managed properties in Search service application which then can be used in query conditions. Managed property should be mapped to crawled property created by Sharepoint automatically during the crawl. When create new managed property you may specify its settings:

Property name Description
Type Type of information that is stored in this property. Supported types are:
- Text
- Integer
- Decimal
- Date and Time
- Yes/No
- Double precision float
- Binary
Searchable Enables querying against the content of the managed property.  The content of this managed property is included in the full-text index. For example, if the property is "author", a simple query for "Smith" returns items containing the word "Smith" and items whose author property contains "Smith".
Advanced Searchable Settings Enables viewing and changing the full-text index and weight of the managed property.
Queryable Enables querying against the specific managed property. The managed property field name must be included in the query, either specified in the query itself or included in the query programmatically. If the managed property is "author", the query must contain "author:Smith".
Retrievable Enables the content of this managed property to be returned in search results. Enable this setting for managed properties that are relevant to present in search results.
Allow multiple values Allow multiple values of the same type in this managed property. For example, if this is the "author" managed property, and a document has multiple authors, each author name will be stored as a separate value in this managed property.

Yes - active: Enables using the property as a refiner for search results in the front end. You must manually configure the refiner in the web part.
Yes - latent: Enables switching refinable to active later, without having to do a full re-crawl when you switch.
Both options require a full crawl to take effect.


Yes - active: Enables sorting the result set based on the property before the result set is returned. Use for example for large result sets that cannot be sorted and retrieved at the same time.
Yes - latent: Enables switching sortable to active later, without having to do a full re-crawl when you switch.
Both options require a full crawl to take effect.

Safe for Anonymous Enables this managed property to be returned for queries executed by anonymous users. Enable this setting for managed properties that do not contain sensitive information and are appropriate for anonymous users to view.
Alias Define an alias for a managed property if you want to use the alias instead of the managed property name in queries and in search results. Use the original managed property and not the alias to map to a crawled property. Use an alias if you don't want to or don't have permission to create a new managed property.
Token Normalization Enable to return results independent of letter casing and diacritics(for example accented characters) used in the query.
Complete Matching Queries will only be matched against the exact content of the property. For example, if you have a managed property "ID" that contains the string "1-23-456#7", complete matching only returns results on the query ID:"1-23-456#7", and not on the queries ID:"1-23" or  ID:"1 23 456 7".
Mappings to crawled properties The list shows all the crawled properties that are mapped to this managed property. A managed property can get its content from one or more crawled properties.
Company name extraction

Enables the system to extract company name entities from the managed property when crawling new or updated items. Afterwards, the extracted entities can be used to set up refiners in the web part.

There is a pre-populated dictionary for company name extraction. The system saves the original managed property content unchanged in the index and, in addition, copies the extracted entities to the managed property "companies". The "companies" managed property is configured to be searchable, queryable, retrievable, sortable and refinable.

Custom entity extraction

Enables one or more custom entity extractors to be associated with this managed property. This enables the system to extract entities from the managed property when crawling new or updated items. Afterwards, the extracted entities can be used to set up refiners in the web part.

There are four types of custom extraction dictionaries. You create your own, separate custom entity extraction dictionaries that you deploy using the PowerShell cmdlet Import-SPEnterpriseSearchCustomExtractionDictionary.

The system saves the original managed property content unchanged in the index and, in addition, copies the extracted entities to the managed properties  "WordCustomRefiner1" through 5, "WordPartCustomRefiner1" through 5, "WordExactCustomRefiner" and/or "WordPartExactCustomRefiner" respectively.

These managed properties are configured to be searchable, queryable, retrievable, sortable and refinable.

It is possible to specify all these settings manually during each installation, but much quicker way will be doing it automatically via PowerShell script. Here is the script which creates new managed properties and maps them to crawled properties (crawled properties should already exist, i.e. you should run search crawler before to run it):

   1:  $category = "SharePoint"
   3:  function SetCrawledAndManagedProperties($propertyMappings)
   4:  {
   5:      $propertyMappings | % { 
   6:          Write-Host "Setting crawled property:" $_[0]
   7:          # Check if crawled property exists. If not, create it.
   8:          $crawledproperty = Get-SPEnterpriseSearchMetadataCrawledProperty
   9:  -Name $_[0] -SearchApplication $searchServiceApplication
  10:          if (!$crawledproperty)
  11:          {
  12:              $message = " - Mapping " + $_[0] + " to " + $_[1]
  13:              Write-Host $message -ForegroundColor Green
  14:              $crawledproperty = New-SPEnterpriseSearchMetadataCrawledProperty
  15:  -SearchApplication $searchServiceApplication -Category $category -VariantType $_[4]
  16:  -PropSet $_[2] -Name $_[0] -IsNameEnum $false
  17:              if ([bool]$_[7] -eq $true)
  18:              {
  19:                  $crawledproperty.IsMappedToContents = $false
  20:                  $crawledproperty.update()
  21:              }
  22:          }
  23:          else {
  24:  Write-Host " - Crawled property" $_[0] "already exists, create managed property only."
  25:  -ForegroundColor Yellow
  26:          }
  27:          Write-Host "Setting managed property:" $_[1]
  28:          $managedproperty = Get-SPEnterpriseSearchMetadataManagedProperty
  29:  -Identity $_[1] -SearchApplication $searchServiceApplication -ErrorAction:SilentlyContinue
  30:          if ($managedproperty -eq $null)
  31:          {
  32:              Write-Host " - Creating managed property" $_[1] -ForegroundColor Green
  33:              if ([bool]$_[7] -eq $true)
  34:              {
  35:                  $managedproperty = New-SPEnterpriseSearchMetadataManagedProperty
  36:  -Name $_[1] -SearchApplication $searchServiceApplication -Type $_[3]
  37:  -FullTextQueriable $false -Queryable $false -Retrievable $false -EnabledForScoping $true
  38:  -NameNormalized $true -Safe $true
  39:  -NoWordBreaker $_[5]
  40:              }
  41:              else
  42:              {
  43:                  $managedproperty = New-SPEnterpriseSearchMetadataManagedProperty
  44:  -Name $_[1] -SearchApplication $searchServiceApplication -Type $_[3]
  45:  -FullTextQueriable $true -Queryable $true -Retrievable $true -EnabledForScoping $true
  46:  -NameNormalized $true -Safe $true -NoWordBreaker $_[5]
  47:              }
  48:              $mapping = New-SPEnterpriseSearchMetadataMapping
  49:  -SearchApplication $searchServiceApplication -ManagedProperty $managedproperty
  50:  -CrawledProperty $crawledproperty
  51:              $managedproperty.Refinable = $true
  52:              $managedproperty.Sortable = $true
  53:              $managedproperty.HasMultipleValues = $_[6]
  54:              $managedproperty.update()
  55:          }
  56:          else
  57:          {
  58:              Write-Host " - Managed property" $_[1] "already exists. Skipping."
  59:  -ForegroundColor Yellow
  60:          }
  62:          Write-Host ""
  63:      };
  65:  }
  67:  $stringPropertySetId = "00130329-0000-0130-c000-000000131346"
  68:  $imagePropertySetId = "fea84df6-a0fc-492c-9aa7-d28b8dcb08b3"
  70:  # 0= Crawled property name | 1= Managed property name | 2= Propertyset id |
  71:  # 3= Type id | 4= variant type | 5= Do not use word breaker | 6= Has multiple values |
  72:  # 7= ExcludeFromSearch
  73:  $propertyMappings = @(
  74:      ,("ows_FooText", "FooText", $stringPropertySetId, 1, 31, $false, $false, $false)
  75:      ,("ows_FooDate", "FooDate", $stringPropertySetId, 4, 64, $false, $false, $false)
  76:      ,("ows_FooImage", "FooImage", $imagePropertySetId, 1, 31, $false, $false, $false)
  77:      ,("ows_FooMM", "FooMM", $stringPropertySetId, 1, 31, $true, $true, $false)
  78:  )
  80:  SetCrawledAndManagedProperties $propertyMappings

This script creates 4 managed properties of 4 different types:

  • FooText – text property
  • FooDate – date time
  • FooImage – image
  • FooMM – managed metadata

Also it maps them to appropriate crawled properties. If crawled properties don’t exist, they will be created too. But as I wrote above the better approach is to make full crawl first and run script after that. In this case Sharepoint will create crawled properties by itself.

You may extend the script for you needs and configure other settings for each property. Hope it will help in your work.

Wednesday, February 26, 2014

How to use custom managed properties in display templates in Sharepoint 2013

When create a new display template for search results page or for Content by search web parts, you may need to use your own custom managed properties inside template. First of all you need to create these properties in Search service application and map it to some crawled property. After that it is possible to use it in display template. In order to do it you need to add new mapping to <mso:ManagedPropertyMapping>…</mso:ManagedPropertyMapping> tag inside display template:

<mso:ManagedPropertyMapping msdt:dt="string">'Link URL'{Link URL}:'Path','Line 1'{Line 1}:'Title','Foo'{Foo}:'Foo'</mso:ManagedPropertyMapping>

In this example we added new mapping for managed property Foo, so it will be possible to get value of this property inside display template like this:

   1:  var foo = $getItemValue(ctx, "Foo");

and then display it to end user. This technique will help you to use not only OTB managed properties like Title or Path, but also custom managed properties in display templates.

Thursday, February 20, 2014

Restore Sharepoint site with configured cross-site publishing on different environment

Some time it is necessary to copy all content of Sharepoint site from one environment to another. For example, in order to make testing more relevant it may be needed to copy production content to test environment, so it will be possible to test functionality with real content without risk to break something. In general such scenarios are done by copying content database and mounting it on test env. But if you configured cross-site publishing on production site, the process will be more complicated. In this post I will describe steps which are needed for successful copy of such sites across environments.

Step 1. Copy content databases

This step is still needed. If you use 2 web applications (authoring and public), you need to copy both of them. Also if you use managed metadata, you should also copy managed metadata database.

On test env create new managed metadata service application and use copied database for it (it can be done by New-SPMetadataServiceApplication cmdlet). After that restore and mount copied content databases using Mount-SPContentDatabase cmdlet. Try to edit some page or document with managed metadata and check that it is editable, i.e. that it is possible to change value in managed metadata fields.

Step 2. Set standard result source as default

If you have custom result source which is set as default, then you need to set OTB “Local Sharepoint Results” source temporary as default. Sharepoint doesn’t show catalog connections when custom result source is set as default (I wrote about it here: Problem with missing catalog connections when use customized search result source in Sharepoint 2013). Later we will return it back.

Alternatively you may completely delete custom result sources from your site collections. It can be done with the following code:

   1:  public static void DeleteResultSource(SPSite site, string name)
   2:  {
   3:      var searchAppProxy = getSearchServiceApplicationProxy(site);
   4:      var federationManager = new FederationManager(searchAppProxy);
   5:      var searchOwner = new SearchObjectOwner(SearchObjectLevel.SPSite, site.RootWeb);
   6:      var source = federationManager.GetSourceByName(name, searchOwner);
   8:      federationManager.RemoveSource(source);
   9:  }
  11:  private static SearchServiceApplicationProxy getSearchServiceApplicationProxy(
  12:  SPSite site)
  13:  {
  14:      var serviceContext = SPServiceContext.GetContext(site);
  16:      var searchAppProxy =
  17:          serviceContext.GetDefaultProxy(typeof(SearchServiceApplicationProxy))
  18:              as SearchServiceApplicationProxy;
  19:      return searchAppProxy;
  20:  }

Step 3. Remove old catalog connections

Old catalog connections should be removed and recreated on test env. You can remove catalog connections with the following code:

   1:  public static void DeleteCatalogConnection(SPWeb web)
   2:  {
   3:      string catalogUrl = ...;
   4:      var catalogSubscriber = new CatalogSubscriber(catalogUrl, web.Url);
   5:      catalogSubscriber.DeleteCatalog();
   6:  }

Here we used CatalogSubscriber helper class from this msdn example.

Step 4. Reset search index and make full crawl

Go to Central administration > Manage service applications > Search service application and reset search index. After that make full crawl of the authoring and public sites.

Step 5. Reconnect to catalog

After full crawl new catalog connection should appear in Site settings > Manage catalog connections. We need to reconnect to it in order to use cross-site publishing on test environment. Example of connecting to new catalog programmatically can be found in the same msdn example which was mentioned above.

Step 6. Set custom result source as default

On step 1 we set OTB “Local Sharepoint Results” result source as default. Now we can set custom result source back to default. If you deleted custom result sources on step 1, you need to recreate it here.

That’s basically all. There may be additional steps, specific for your particular case, but general process should be done with steps described above and after that you should have working cross-site publishing on copied site on test environment.

Tuesday, February 4, 2014

Localize datetimes in display templates in Content by search web parts in Sharepoint 2013

Display templates are plain html files which are used in Content by search web parts in Sharepoint 2013 for displaying information. In this post I will show how to localize datetimes using locale or language of current SPWeb.

The main problem with localization is that inside display templates you may use only javascript and client object model, but not server code. I.e. you can’t just write:

   1:  var ci = new CultureInfo(SPContext.Current.Web.Locale.LCID);
   2:  var date = ...;
   3:  var localizedDate = date.ToString(ci);

We need to do it on client side. If you will make investigation of what methods are available in javascript for dates localization you will find toLocaleString method of Date object. The problem that this method uses current client’s locale and doesn’t allow to pass specific LCID. It may be needed when current client’s locale and locale, which you want to use for dates localization, are not the same.

The good thing however is that MS ajax (which is available in SharePoint OTB) adds to Date localeFormat method, which allows to display datetime using specified format. The question is how to get datetime format for specific LCID on client side?

In order to do it I used the following way. There is useful standard global javascript object _spPageContextInfo, which contains many useful settings from server side which you may use on client side (see How to get URL of current site collection and other server side properties on client site in Sharepoint). It doesn’t contain date time format, so we will extend it with necessary information (e.g. in masterpage);

   1:  <script type="text/javascript">
   2:      jQuery(function() {                
   3:          if (typeof (_spPageContextInfo) != "undefined" &&
   4:              _spPageContextInfo != null) {
   5:              <%
   6:                  var currentWeb = SPContext.Current.Web;
   7:                  var ci = new CultureInfo(currentWeb.Locale.LCID);
   8:                  var cultureSerialized = new JavaScriptSerializer().Serialize(
   9:                      new
  10:                      {
  11:                          name = ci.Name,
  12:                          dateTimeFormat = ci.DateTimeFormat,
  13:                          numberFormat = ci.NumberFormat
  14:                      });
  15:              %>
  16:              _spPageContextInfo.currentCultureSerialized = <%= cultureSerialized %>;
  17:          }
  18:      });
  19:  </script>

It will add the following script to output if current locale is 1033 (en-us):

   1:  _spPageContextInfo.currentCultureSerialized =
   2:  {
   3:     "name":"en-US",
   4:     "dateTimeFormat":{
   5:        "AMDesignator":"AM",
   6:        "Calendar":{
   7:           "MinSupportedDateTime":"\/Date(-62135596800000)\/",
   8:           "MaxSupportedDateTime":"\/Date(253402293599999)\/",
   9:           "AlgorithmType":1,
  10:           "CalendarType":1,
  11:           "Eras":[
  12:              1
  13:           ],
  14:           "TwoDigitYearMax":2029,
  15:           "IsReadOnly":false
  16:        },
  17:        "DateSeparator":"/",
  18:        "FirstDayOfWeek":0,
  19:        "CalendarWeekRule":0,
  20:        "FullDateTimePattern":"dddd, MMMM d, yyyy h:mm:ss tt",
  21:        "LongDatePattern":"dddd, MMMM d, yyyy",
  22:        "LongTimePattern":"h:mm:ss tt",
  23:        "MonthDayPattern":"MMMM d",
  24:        "PMDesignator":"PM",
  25:        "RFC1123Pattern":
  26:  "ddd, dd MMM yyyy HH\u0027:\u0027mm\u0027:\u0027ss \u0027GMT\u0027",
  27:        "ShortDatePattern":"M/d/yyyy",
  28:        "ShortTimePattern":"h:mm tt",
  29:        "SortableDateTimePattern":
  30:  "yyyy\u0027-\u0027MM\u0027-\u0027dd\u0027T\u0027HH\u0027:\u0027mm\u0027:\u0027ss",
  31:        "TimeSeparator":":",
  32:        "UniversalSortableDateTimePattern":
  33:  "yyyy\u0027-\u0027MM\u0027-\u0027dd HH\u0027:\u0027mm\u0027:\u0027ss\u0027Z\u0027",
  34:        "YearMonthPattern":"MMMM yyyy",
  35:        "AbbreviatedDayNames":[
  36:           "Sun",
  37:           "Mon",
  38:           "Tue",
  39:           "Wed",
  40:           "Thu",
  41:           "Fri",
  42:           "Sat"
  43:        ],
  44:        "ShortestDayNames":[
  45:           "Su",
  46:           "Mo",
  47:           "Tu",
  48:           "We",
  49:           "Th",
  50:           "Fr",
  51:           "Sa"
  52:        ],
  53:        "DayNames":[
  54:           "Sunday",
  55:           "Monday",
  56:           "Tuesday",
  57:           "Wednesday",
  58:           "Thursday",
  59:           "Friday",
  60:           "Saturday"
  61:        ],
  62:        "AbbreviatedMonthNames":[
  63:           "Jan",
  64:           "Feb",
  65:           "Mar",
  66:           "Apr",
  67:           "May",
  68:           "Jun",
  69:           "Jul",
  70:           "Aug",
  71:           "Sep",
  72:           "Oct",
  73:           "Nov",
  74:           "Dec",
  75:           ""
  76:        ],
  77:        "MonthNames":[
  78:           "January",
  79:           "February",
  80:           "March",
  81:           "April",
  82:           "May",
  83:           "June",
  84:           "July",
  85:           "August",
  86:           "September",
  87:           "October",
  88:           "November",
  89:           "December",
  90:           ""
  91:        ],
  92:        "IsReadOnly":false,
  93:        "NativeCalendarName":"Gregorian Calendar",
  94:        "AbbreviatedMonthGenitiveNames":[
  95:           "Jan",
  96:           "Feb",
  97:           "Mar",
  98:           "Apr",
  99:           "May",
 100:           "Jun",
 101:           "Jul",
 102:           "Aug",
 103:           "Sep",
 104:           "Oct",
 105:           "Nov",
 106:           "Dec",
 107:           ""
 108:        ],
 109:        "MonthGenitiveNames":[
 110:           "January",
 111:           "February",
 112:           "March",
 113:           "April",
 114:           "May",
 115:           "June",
 116:           "July",
 117:           "August",
 118:           "September",
 119:           "October",
 120:           "November",
 121:           "December",
 122:           ""
 123:        ]
 124:     },
 125:     "numberFormat":{
 126:        "CurrencyDecimalDigits":2,
 127:        "CurrencyDecimalSeparator":".",
 128:        "IsReadOnly":false,
 129:        "CurrencyGroupSizes":[
 130:           3
 131:        ],
 132:        "NumberGroupSizes":[
 133:           3
 134:        ],
 135:        "PercentGroupSizes":[
 136:           3
 137:        ],
 138:        "CurrencyGroupSeparator":",",
 139:        "CurrencySymbol":"$",
 140:        "NaNSymbol":"NaN",
 141:        "CurrencyNegativePattern":0,
 142:        "NumberNegativePattern":1,
 143:        "PercentPositivePattern":0,
 144:        "PercentNegativePattern":0,
 145:        "NegativeInfinitySymbol":"-Infinity",
 146:        "NegativeSign":"-",
 147:        "NumberDecimalDigits":2,
 148:        "NumberDecimalSeparator":".",
 149:        "NumberGroupSeparator":",",
 150:        "CurrencyPositivePattern":0,
 151:        "PositiveInfinitySymbol":"Infinity",
 152:        "PositiveSign":"+",
 153:        "PercentDecimalDigits":2,
 154:        "PercentDecimalSeparator":".",
 155:        "PercentGroupSeparator":",",
 156:        "PercentSymbol":"%",
 157:        "PerMilleSymbol":"‰",
 158:        "NativeDigits":[
 159:           "0",
 160:           "1",
 161:           "2",
 162:           "3",
 163:           "4",
 164:           "5",
 165:           "6",
 166:           "7",
 167:           "8",
 168:           "9"
 169:        ],
 170:        "DigitSubstitution":1
 171:     }
 172:  };

For other locales it will contain localized information. Now, when we have necessary datetime format on client side we can use it in display templates by overriding default datetime renderer:

   1:  customDateRenderer=function(a) {
   2:      if(!Srch.U.n(a) && !a.isEmpty &&
   3:          Date.isInstanceOfType(a.value)) {
   4:          try {
   5:              return a.value.localeFormat
   6:                 (_spPageContextInfo.currentCultureSerialized.
   7:                     dateTimeFormat.ShortDatePattern);
   8:          }
   9:          catch(er) {
  10:              return Srch.ValueInfo.Renderers.
  11:                  defaultRenderedValueHtmlEncoded(a);
  12:          }
  13:      }
  14:      else {
  15:          return Srch.ValueInfo.Renderers.
  16:              defaultRenderedValueHtmlEncoded(a);
  17:      }
  18:  };
  19:  ...
  20:  var dt = $getItemValue(ctx, "Published");
  21:  dt.overrideValueRenderer(customDateRenderer);

After that datetimes will be localized in display templates based on locale of current SPWeb.