Configuring Enterprise Search in SharePoint 2010

Configuring Enterprise Search in SharePoint 2010

We all have to agree that search plays an integral part of any successful SharePoint deployment and is an area that Microsoft continues to invest in with each new release of SharePoint. Microsoft went as far as acquiring FAST 2 years ago which it now offers as a separate add-on to SharePoint for those willing to invest in high end enterprise search.  In addition to FAST, SharePoint 2010 search comes in a number of flavors each offering their own feature set and capabilities which I have duplicated at the end of this article as an Appendix for convenience.

Today we will introduce SharePoint Server 2010 Search and eventually work our way up to Microsoft’s latest and greatest FAST Search Server in a near future article.  Before we deep dive into the step by step guide I will begin by listing some of the new features that you will come to expect from SharePoint Server 2010 Search.   These are as follows;

• Boolean query syntax has finally been introduced.  These include AND, OR and NOT operators in your search queries.
• Suggestions whilst typing and after running search queries, a feature that we have come to love with major search engines such as Google and Bing.
• Integrating SharePoint enterprise search with Windows 7, allowing end users to utilise the Windows 7 search box to locate SharePoint 2010 content.
• Results display has been refined to provide filters for search results such as document type, categories and managed properties.
• View in Browser capabilities, allows end users to view documents within their own browser utilising Office Web Apps and not having to rely on launching the necessary Microsoft Office Application, or even the need of having it installed on their local machine.  This is handy when browsing your SharePoint site via Kiosks and Internet Cafes that may not be running the Microsoft Office Suite.
• Last but not least, there have been a number of improvements to People Search, including phonetic name and nickname matching, and improved relevance and self search.

Now that we have a taste for what’s to come, let’s begin our configuration.

SharePoint Server Search is a service application which we have come to learn about over the past few articles that it is independent of other services and is no longer tied to the Shared Services Provider (SSP) that was introduced in SharePoint 2007.

SharePoint 2010 search architecture is made up of the Crawler, Indexing Engine, Query Engine and the User Interface and Query Object Model.  We now have greater flexibility and expandability with our search design in 2010 and can setup not only multiple Query Servers but can now scale out our Index server and add multiple instances.

Below is a logical overview of the components that will make up our SharePoint 2010 search configuration.

Configuring the Service Application

As always we begin our journey in Central Administration / Application Management / Manage Service Applications.

Click New / Search Service Application.

Name: Enter a name for your Service Application.

FAST Service Application: Select “None” (we will leave the configuration of FAST for a future article)

Search Service Account: Click on Register new managed account and ensure your domain account has already been provisioned in Active Directory.  I have created a separate search account; e.g. DOMAIN\sp_search

Application Pool for Search Admin Web Service: Create a new application pool for your search admin web service application.

Application Pool for Search Query and Site Settings Web Service: Create a new application pool for your search query web service application.

Click Create

The search service application will begin its configuration process.

You will eventually be presented with confirmation that the search service application was created successfully.

If we now navigate back to Application Management / Manage Service Applications, you will notice that 2 additional services have been added to our list.  These are;

1. Search Service Application (Typical Search Administration page which is similar to that in SharePoint 2007. From here we can create content crawl rules, reset indexes, setup content sources etc).
2. WSS_UsageApplication (This is a new service in SharePoint 2010 that specifically handles our Usage and Health Data Collection Service Application.  This service application handles web analytics such as usage, search query usage, rating usage etc  More on this in a future article).

Let’s now launch the Search Administration page by clicking on our Search Service Application.

Our Default content access account should be set to the account that we had specified at the time of provisioning the Search Service Application; i.e. DOMAIN\sp_search

Confirming Permissions

There are a couple of areas to note that we should check to ensure that our Default content access acount (sp_search) has been provided with the appropriate access permissions.  Let’s first begin by checking our User Profile Service Application by Navigating to Service Applications / User Profiles.  Just highlight the User Profiles and select Administrators from the ribbon.

Our newly provisioned sp_search account should have “Retrieve People Data for Search Crawlers” selected as a permission.

We will also confirm that our sp_search account has the necessary “Read” permissions against the Web Applications being crawled.

Navigate to Central Administration / Application Management / Manage Web Applications.  Again, highlight the Web Application in question and from the ribbon select User Policy.

Ensure that the Search Crawling Account is set to the sp_search domain account.

Content Sources

Let’s venture into our content sources listed in the Quick Launch navigation bar under Crawling.  You do so by first navigating to your Search Application Service and clicking on Manage.

As was the case with SharePoint 2007, our Local SharePoint sites will be detected by default, albeit without a crawl schedule.

Check to see that your Start Addresses are located within your content source via editing the content source from the drop down menu.  These includes all SharePoint Web Applications and the sps3 “User Profiles” address.

You can easily create your crawl schedule by clicking on Local SharePoint sites and scrolling down to Crawl Schedules.

Let’s initiate a Full Crawl by clicking on Start all Crawls from the Manage Content Sources page.

Once your crawl has completed, you should confirm that there were no errors encountered during the initial crawl.  Usually any errors noted are most likely due to incorrect permission assignments.

Creating a “Basic Search Center” Site

If you haven’t done so already, from your top level site, click on Site Actions / New Site.

Select “Basic Search Center”

Enter a Name and URL and click on Create.

This will provision the Search Center similar to the below.

Creating an “Enterprise Search Center” Site

Let’s also create an Enterprise Search Center for comparison.  The key difference here is that we are provided with two tabs for searching, one for Sites and the other for People.  The “Enterprise Search Center” will be the search site of choice for most organizations running SharePoint Server.

From Central Administration / Application Management / Site Collections, click on Create site collections.   Ensure you are creating the Site Collection below the relevant Web Application.

Enter your Title, Description etc and select the Enterprise Tab under Template selection.  Select the Enterprise Search Center, specify your site collection administrators and click OK.

This will provision the Enterprise Search Center similar to the below.

As we have already completed an initial Full crawl earlier, I can now test my new search centers by performing a couple of searches.

Searching Content

Searching People

Now I ran into an issue when trying to search for content located in My Sites.  The crawl log displayed the following warning;

“This item and all items under it will not be crawled because the owner has set the NoCrawl flag to prevent it from being searchable”

In order to fix this issue (and this is true for any Site Collection), is to navigate to your My Site host and click on Site Actions / Site Settings.

Click on “Search and offline availability” under Site Administration, and ensure that you have Indexing Site Content, Allow this site to appear in search results? set to “Yes”.

After enabling the indexing of My Sites, I was able to successfully perform My Site Content searches and the warning disappeared from the Crawl Log.

That’s all that is to it in setting up a search center in its most basic form.  From here you can expand your service applications over multiple servers providing you with redundancy, scalability and increased performance

Configuring Enterprise search

Configuring Enterprise search

1. Go to SharePoint Central Administration.

2. Click on Application management then select Manage Service application.


3. Click New and select Search Service Application


4. Give a name for your Service application.

5. Select Search Service account or register new Service account in the drop down.

6. Better create a new application pool for your Search service as shown below.


7. Once done Click OK.


8. It will take some time to configure the service.

9. Once configured correctly you will see the service created like below.


10. Click on the Search Service Application 1 that we create now.

11. Select Content Source.


12. Click on the Local SharePoint Site.


13. Add the Web applications that you need to put in Search Services as shown below.


14. You have to configure Full and incremental crawl schedules in the preceding screen.

15. Now go back and start a Full crawl.


16. You will get the following screen.


17. Even though you have done all these steps you may get some error while searching "The search request was unable to connect to the Search Service".

18. To avoid this you have to perform the following steps.

19. Go to Application management and select Configure Application Association as shown below.


20. Click on your application.

21. From the "Edit the following group of connections" drop down select Custom.

22. Associate your search service with your web application.



23. If you done this step you may have to start the crawl once again.

Configuring and customizing SharePoint 2010

Configuring and customizing SharePoint 2010 search

• Part 1: Configure basic search
• Part 2: Metadata and faceted search
• Part 3: Tie in additional facets
• Part 4: Customize the results
• Part 5: Add a field to the advanced search filter

Part 1: Configure Search

1) Load Central Admin
2) Click Manage Service Applications
3) Click New –> Search Service Application

4) Specify the
- Name
- Service Account
- App Pools
There are no real gotcha’s here.
5) Select the newly created service application and click ‘Manage’

6) Click content sources, drop the context menu down on ‘Local SharePoint sites’ and choose edit
7) Set Full and incremental crawl schedules and start a full crawl. 
If you go back to your site (after the crawl completes!) you’ll get results.

So, that’s search results but that’s pretty dull right.  So the next task is to show search facets.

Part 2: Metadata and Faceted Search

) Go to your site settings and select the Term Store Management option

2) Add a new Group, Term Set and Terms, if you have problems here make sure the account you are logged in is in the list of Term Store Administrators (page right) – even if you’re logged in as the farm admin.  I’m using Customers as my term set as per:

3) Add site columns for the metadata we’ll add to the document content type.  Go to Site Settings –> Site Columns and add a site column called Customer
4) Make the new site column a Managed Metadata type and select the relevant node of your term store ala:

Now we want to associate those columns with the document content type.
6) Back to Site Settings –> select Site Content Types, now click on the Document content type
7) Add the two metadata fields we just added to the content type and you should have something like:

8) Ok so we now have our new document type and we should (after the next crawl) see the effect of this in search results.  So to force that go to central admin, manage the search application, select the default scope (if that’s when you have) and start a (full) crawl.
So, now when we search what do we see?
Well, the results are obviously the same but now SharePoint has automatically picked up one of our metadata fields and allowed a faceted search filter on that:

So, that’s nice but where is that other column?  Basically SharePoint doesn’t add it because it’s a not a metadata field (there a refinement in the box to automatically pick these up), it also has almost infinite possible values and SharePoint wouldn’t know how to categorise those values in a way that would add value.  In the next part we’ll add that Sales Potential Field as a facet.

Part 3: Tie in additional facets

Jump back into Central Admin, from there manage your search application

2) Select the Metadata Properties under the section “Queries and Results”

3) Click “New Managed Property”

•  
• Call the property  “SalesValue”
• Set the type to “Decimal” (mapping to currency)
• In the “Mappings to crawled properties” area search for the crawled field we created earlier (NOTE: the field will have been renamed internally by SharePoint because it’s a decimal it gets a text equivalent field and the actual decimal value is placed in this newly named field)

Now we have that field available to us in search we can go ahead and define how we should use it.  If you go back to your site and do a search you’ll be on the results.aspx page and it’s this page we want to change…

1) Edit the page

2) Edit the “Refinement Panel” web part on the left hand side (toward the bottom of the page) and expand the “Refinement” node.

3) VERY IMPORTANT: Unclick the “Use Default Configuration” option!!!

4) Add XML display the new managed property…

Now because the new property is a decimal value there are a few things we want to do and some just to be aware of.  Firstly, we don’t just want to show the value itself we will want to categorise the values into meaningful bands i.e. High Value Medium Value, Low Value, we can do this quite easily but the thing to be aware of is that most example you’ll see out there use a syntax like:

      <CustomFilter CustomValue="Medium Value">

        <OriginalValue>50000..99999</OriginalValue>

      </CustomFilter>

That’s no good for a decimal value because what search will do is apply that filter almost exactly as it shows and because it’s comparing a decimal it’ll get all confused about those dots in between the numbers!  You get something like: “Property doesn’t exist or is used in a manner inconsistent with schema settings.” – you’ll also get that if you’ve come this far without creating the field as a managed property.  So what we’re going to do is specify a lower and upper threshold for the original value.  The full XML for the display of the SalesValue field will now become:

  <Category Title="Sales Potential" Description="Allows refinement by saleable value" Type="Microsoft.Office.Server.Search.WebControls.ManagedPropertyFilterGenerator"

            MetadataThreshold="3" NumberOfFiltersToDisplay="4" MaxNumberOfFilters="0" ShowMoreLink="True" MappedProperty="SalesValue"

            MoreLinkText="show more" LessLinkText="show fewer" SortBy="Custom" ShowCounts="Count">

    <CustomFilters MappingType="RangeMapping" DataType="Numeric" ValueReference="Absolute" ShowAllInMore="False">

      <CustomFilter CustomValue="Second home">

        <OriginalValue>500000..</OriginalValue>

      </CustomFilter>

      <CustomFilter CustomValue="New car">

        <OriginalValue>200000..</OriginalValue>

        <OriginalValue>..499999</OriginalValue>

      </CustomFilter>

      <CustomFilter CustomValue="Holiday">

        <OriginalValue>100000..</OriginalValue>

        <OriginalValue>..199999</OriginalValue>

      </CustomFilter>

      <CustomFilter CustomValue="Biscuits">

        <OriginalValue>50000..</OriginalValue>

        <OriginalValue>..99999</OriginalValue>

      </CustomFilter>

      <CustomFilter CustomValue="Coffee">

        <OriginalValue>2500..</OriginalValue>

        <OriginalValue>..49999</OriginalValue>

      </CustomFilter>

      <CustomFilter CustomValue="You owe us!">

        <OriginalValue>..2499</OriginalValue>

      </CustomFilter>

    </CustomFilters>

  </Category>

5) In the “Filter Category Definition” box, copy out the contents to a useful editor (Visual Studio works for me)

6) Combine the XML snippet above with the XML that was already in the textbox, for reference I added my XML just above the Managed Metadata column. Save the changes are re-query.

Done!  You should now see something like the following and be able to refine search on the facet we just created:

Part 4: Customize search results

1) Edit the “Search Core Results” web part by editing the search results page

2) Uncheck the “Use Location Visualization” option within the “Display Properties” node

3) Copy the text out of the “Fetched Properties” text box and edit it to add out new field:

<Columns>

  <Column Name="WorkId"/>

  <Column Name="Rank"/>

  <Column Name="Title"/>

   .

   .

   .

  <Column Name="SalesValue"/>

</Columns>

This will make the property available (although when I tested it without the entry it still worked!)

4) Now edit the  same web part (if you closed it down) and load the XSL Editor from the “Display Properties” node and copy the XSLT out into an editor.

5) Add the SalesValue to the search results by editing the XSLT.  It quite a large file but essestially you’re just looking for the “results” match.  Now you’ve found that you should be able to see the method that’s been employed to render the results.  I’m going to add a new template to output out field.  Here I’m just copying the string template with minor changes:

  <!-- AJN: Additional output field starts here -->

  <xsl:template name="DisplaySalesValue">

    <xsl:param name="str" />

    <xsl:if test='string-length($str) &gt; 0'>

      <b>Sales Value: $<xsl:value-of select="$str" /></b>

    </xsl:if>

  </xsl:template>

.6) Now add a call to that template to output the value at the end of the metadata area of search results.  To do this I’ve popped back up to the results match we found earlier and directly after the file size output I’ve added a call to the new template:

        <div class="srch-Metadata2">

          <xsl:call-template name="DisplayAuthors">

           .

           .

           .

          <xsl:call-template name="DisplaySalesValue">

            <xsl:with-param name="str" select="salesvalue" />

          </xsl:call-template>

          <img style="display:none;" alt="" src="/_layouts/images/blank.gif"/>

        </div>

And here’s an example of the output:

 

Part 5: Add a field to the advanced search filter

Finally lets make that Sales Value property available in advanced search.
1) Do a search and go to the advanced search page, edit the page and edit the “Advanced Search Box” web part…

2) Copy out the “Properties” XML to your favourite editor and add the property definition thus:

  <PropertyDefs>

      <PropertyDef Name="Path" DataType="text" DisplayName="URL"/>

      .

      .

      .    <PropertyDef Name="SalesValue" DataType="integer" DisplayName="Sales Value"/>

  </PropertyDefs>

3) Then add a reference to the field in the “All Results” set thus:

    <ResultType DisplayName="All Results" Name="default">

      <KeywordQuery/>

      <PropertyRef Name="Author" />

      .

      .

      .

      <PropertyRef Name="SalesValue" />

    </ResultType>

Now that really is it, you should now be able to do any kind of search (normal or advanced anyway!) and have access to the managed properties and metadata fields we created:

Handy tips

The following XSLT will show you all the XML that is returned from search so you can check that your properties are being returned correctly:

<?xml version="1.0" encoding="UTF-8"?>

<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">

<xsl:output method="xml" version="1.0" encoding="UTF-8" indent="yes"/>

<xsl:template match="/">

<xmp><xsl:copy-of select="*"/></xmp>

</xsl:template>

</xsl:stylesheet>

Have a play around with the refinement syntax, it’s quite smart and I have a feeling it’s going to come in handy on coded applications!
For example:

http://noonscape/Search/results.aspx?k=test&r=salesvalue>=”200000” salesvalue>”49999″http://noonscape/Search/results.aspx?k=sharepoint&r= salesvalue<=”10000000″ AND salesvalue>”500000″