Indexing from external sources - Part 2

In the one of the last posts I've showed you how you can index data from external source. I've used there XML file as my input. By using custom crawler, in results we've got indexed documents within SOLR core. It is good base to continue work. Let's extend it a little bit!

Today we will configure our CustomCrawler with standard SitecoreItemCrawler to add possibility to indexing Sitecore items and external sources to one Solr core. Then we will prepare mechanism to search trough all indexed data.

Let's start from input sources. In Sitecore I've prepared structure which I am going to put in index. It is presented below on screen shot

Also, I've used XML file from previous post and I've added there Name field :

<?xml version="1.0"?>
<Products>
  <Product>
      <Id>1</Id>
      <Name>Product 1</Name>
      <Description>Lorem Ipsum</Description>
  </Product>
  <Product>
      <Id>2</Id>
      <Name>Product 2</Name>
      <Description>Dolor Sit Etem</Description>
  </Product>
    <Product>
      <Id>3</Id>
      <Name>Product 3</Name>
      <Description>Sed do eiusmod tempor</Description>
  </Product>
</Products>

When we have our both input sources, it is time to extend our previous index configuration and add there SitecoreItemCrawler to allow indexing Sitecore items:

<?xml version="1.0" encoding="utf-8" ?>
<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
  <sitecore>
    <contentSearch>
      <configuration type="Sitecore.ContentSearch.ContentSearchConfiguration, Sitecore.ContentSearch">
        <indexes hint="list:AddIndex">
          <index id="custom_index" type="Sitecore.ContentSearch.SolrProvider.SolrSearchIndex, Sitecore.ContentSearch.SolrProvider">
            <param desc="name">$(id)</param>
            <param desc="core">$(id)</param>
            <param desc="propertyStore" ref="contentSearch/indexConfigurations/databasePropertyStore" param1="$(id)" />
            <configuration ref="contentSearch/indexConfigurations/defaultSolrIndexConfiguration">
              <indexAllFields>true</indexAllFields>
              <fieldMap ref="contentSearch/indexConfigurations/defaultSolrIndexConfiguration/fieldMap"/>
              <documentOptions type="Sitecore.ContentSearch.SolrProvider.SolrDocumentBuilderOptions, Sitecore.ContentSearch.SolrProvider">
              </documentOptions>
            </configuration>
            <strategies hint="list:AddStrategy">
              <strategy ref="contentSearch/indexConfigurations/indexUpdateStrategies/onPublishEndAsync" />
            </strategies>
            <locations hint="list:AddCrawler">
              <!--already added custom crawler-->
              <crawler type="SitecoreBlog.Search.Crawlers.CustomCrawler, SitecoreBlog.Search">
              </crawler>
              <!--here we must add new SitecoerItemCrawler with db name and root item-->
              <crawler type="Sitecore.ContentSearch.SitecoreItemCrawler, Sitecore.ContentSearch">
                <Database>master</Database>
                <Root>/sitecore/content/Home</Root>
              </crawler>
            </locations>
          </index>
        </indexes>
      </configuration>
    </contentSearch>
  </sitecore>
</configuration>

As you probably noticed, I've added there new crawler with two parameters which will fit crawler to our need. In the Root parameter I've used node which we want to index. 

After those steps, rebuilding index should put our data from Sitecore and from XML to index core. You can check it in Solr panel. Here is presented how it looks in my Solr panel.
Indexed product:

Part of indexed Sitecore item:

The difference now is that whole content from Sitecore item is indexed as _content field. Situation looks similar in case of Sitecore item name, because it is indexed as _name field. Now we have to modify indexed products from XML to fit to indexed Sitecore items. we will index product name as _name field and description with name as _content.  To achieve this, we need Product Model which we've prepared in previous post.

using SitecoreBlog.Search.Attributes;

namespace SitecoreBlog.Search.Model
{
    public class Product
    {
        [IndexInfo("productid")]
        public int Id { get; set; }

        [IndexInfo("_name")]
        public string Name { get; set; }

        public string Description { get; set; }

        [IndexInfo("_content")]
        public string Content => string.Format("{0} {1}", Name, Description);
    }
}

After those changes and rebuilding index, product in Solr should look in a following way:

It is final and proper form. Now can move on to search mechanism. On the beginning we have to to create DTO which will be returned as result.

namespace SitecoreBlog.Search.Dto
{
    public class ResultItem
    {
        public string Name { get; set; }
        public string Content { get; set; }
    }
}

When we already have our DTO, we can move on to the creation of SearchService.

using System.Collections.Generic;
using System.Linq;
using Sitecore.ContentSearch;
using Sitecore.ContentSearch.SearchTypes;
using SitecoreBlog.Search.Dto;

namespace SitecoreBlog.Search.Service
{
    public class SearchService
    {
        public ICollection<ResultItem> Search(string phrase)
        {
            using (var searchContext = ContentSearchManager.GetIndex("custom_index").CreateSearchContext())
            {
                var results = searchContext.GetQueryable<SearchResultItem>().Where(x => x.Content.Contains(phrase))
                    .Select(x => new ResultItem()
                    {
                        Name = x.Name,
                        Content = x.Content
                    }).ToList();

                return results;
            }
            
        }
    }
}

Our last step will be adding controller to testing functionality written in service few minutes ago.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Web;
using System.Web.Mvc;
using SitecoreBlog.Search.Service;

namespace SitecoreBlog.Website.Controllers
{
    public class SearchController : Controller
    {
        [HttpGet]
        public ActionResult Search(string q)
        {
            var service = new SearchService();
            var results = service.Search(q);

            return Json(results, JsonRequestBehavior.AllowGet);
        }
    }
}

OK, that's all what we need. Now we can test our search mechanism in browser by calling controller method. Let's check it using vary phrases.

As you can see, in results are products from XML and items from Sitecore, so our achievement is reached for now. I hope that this tutorial satisfied you and it will help you save your time during implementing similar solution this solution. It was second and last part of this topic. If  you've missed first part, you can check this here.

In case of any questions, don't hesitate to ask in comments.

Thank you for your time, I'll be back soon :)
Stay tuned!

Template inheritors

Hi there!

Today I am going to show you how you can quickly get inheritors of specific template. Let's imagine situation when you have base template, for instance _ProductBase and some templates which inherit from this one, let's say ProductA, ProductB, ProductC. We know that all future product templates
will also have _ProductBase in theirs base templates. In case when you will need get all product types available in project(and it is really probable :)), this solution will fit very well to this need.

Let's look to code. I've prepared it as the extension to TemplateItem.

public static class TemplateExtensions
    {
        /// <summary>
        /// Gets template inheritors 
        /// </summary>
        /// <param name="template"></param>
        /// <returns></returns>
        public static ICollection<string> GetTemplateInheritors(this TemplateItem template)
        {
            if (template != null)
            {
                var inheritors = Globals.LinkDatabase.GetReferrers(template)
                    .Where(l => l.GetSourceItem().Paths.FullPath.StartsWith("/sitecore/templates/") 
                    &&!l.GetSourceItem().Paths.FullPath.StartsWith("/sitecore/templates/system"))                   
                    .Select(l => l.SourceItemID.ToString())
                    .ToList();

                return inheritors;
            }
            return new List<string>();
        }
    }

It is important to take only items from templates root and exclude paths with standard Sitecore templates, it is /sitecore/templates/system, because we will not need this here.

I hope that you're enjoyed by this short post. Thank you for your time and stay tuned!

Indexing from external sources - Part 1

Introduction

I am sure that major part of you guys were using indexes in Sitecore and you were configuring it by yourself. It may be problematic at first time, but in every next time it is easier and finally you are going to do it automatically like a robot!
Last time I was faced task where I had to prepare search mechanism (AGAIN!), so the first though was that I need to index content and prepare service to search - piece of cake! But after reading acceptance criteria, I've noticed that I have to search not only by sitecore content, but also by huge XML with items provided by 3rd party service. Then I realised that it will be something new, so I was looking for the best solution
I was aware that I have to create new crawler but I didn't know how I can do it. Very helpful for me was this article - many thanks to author! (If you read this - I owe you a beer! :D )

Input source

Let's say that we have XML file which we want to index and it looks like this:

<?xml version="1.0"?>
<Products>
  <Product>
      <Id>1</Id>
      <Description>Lorem Ipsum</Description>
  </Product>
  <Product>
      <Id>2</Id>
      <Description>Dolor Sit Etem</Description>
  </Product>
    <Product>
      <Id>3</Id>
      <Description>Sed do eiusmod tempor</Description>
  </Product>
</Products>

Custom Crawler Configuration

To have a possibilty of indexing data from outside of the sitecore, we must create custom crawler. Let's start from adding it within our index configuration. For the demo purpose I've created new index configuration.

<?xml version="1.0" encoding="utf-8" ?>
<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
  <sitecore>
    <contentSearch>
      <configuration type="Sitecore.ContentSearch.ContentSearchConfiguration, Sitecore.ContentSearch">
        <indexes hint="list:AddIndex">
          <index id="custom_index" type="Sitecore.ContentSearch.SolrProvider.SolrSearchIndex, Sitecore.ContentSearch.SolrProvider">
            <param desc="name">$(id)</param>
            <param desc="core">$(id)</param>
            <param desc="propertyStore" ref="contentSearch/indexConfigurations/databasePropertyStore" param1="$(id)" />
            <configuration ref="contentSearch/indexConfigurations/defaultSolrIndexConfiguration">
              <indexAllFields>true</indexAllFields>
              <fieldMap ref="contentSearch/indexConfigurations/defaultSolrIndexConfiguration/fieldMap"/>
              <documentOptions type="Sitecore.ContentSearch.SolrProvider.SolrDocumentBuilderOptions, Sitecore.ContentSearch.SolrProvider">
              </documentOptions>
            </configuration>
            <strategies hint="list:AddStrategy">
              <strategy ref="contentSearch/indexConfigurations/indexUpdateStrategies/onPublishEndAsync" />
            </strategies>
            <locations hint="list:AddCrawler">
              <!--here we have to add our custom crawler-->
              <crawler type="SitecoreBlog.Search.Crawlers.CustomCrawler, SitecoreBlog.Search">
              </crawler>
            </locations>
          </index>
        </indexes>
      </configuration>
    </contentSearch>
  </sitecore>
</configuration>

As you can see, I've added crawler in our configuration, so now it is time to add implementation.

Custom Crawler Implementation

using System.Collections.Generic;
using Sitecore.ContentSearch;
using SitecoreBlog.Search.Model;

namespace SitecoreBlog.Search.Crawlers
{
    public class CustomCrawler : FlatDataCrawler<IndexableProduct>
    {
        protected override IndexableProduct GetIndexableAndCheckDeletes(IIndexableUniqueId indexableUniqueId)
        {
            return null;
        }

        protected override IndexableProduct GetIndexable(IIndexableUniqueId indexableUniqueId)
        {
            return null;
        }

        protected override bool IndexUpdateNeedDelete(IndexableProduct indexable)
        {
            return false;
        }

        protected override IEnumerable<IIndexableUniqueId> GetIndexablesToUpdateOnDelete(IIndexableUniqueId indexableUniqueId)
        {
            return null;
        }

        protected override IEnumerable<IndexableProduct> GetItemsToIndex()
        {
            var list =  new List<IndexableProduct>() { new IndexableProduct(new Product()
            {
                Description = "lorem ipsum"
            }),
    
            };

            return list;
        }
    }
}

To achieve our goal we have to add inheritance in our crawler from FlatDataCrawler and use generic type with definition of indexable item. In our case it will be IndexableProduct. In method GetItemsToIndex we have to return collection of items which we want index, so it is perfect place to return elements from provided XML.

Indexable Product

Here we are collecting properties from Product model and checking if they contains IndexInfo attribute.

using System;
using System.Collections.Generic;
using System.Globalization;
using System.Linq;
using System.Reflection;
using Sitecore.ContentSearch;
using SitecoreBlog.Search.Attributes;
using SitecoreBlog.Search.Model;

namespace SitecoreBlog.Search.Crawlers
{
    public class IndexableProduct : IIndexable
    {
        private readonly Product _product;

        public IndexableProduct(Product product)
        {
            _product = product;
        }

        public void LoadAllFields()
        {
            Fields = _product.GetType()
                .GetProperties()
                .Where(fi => fi.GetCustomAttribute<IndexInfo>() != null)
                .Select(fi => new IndexableProductDataField(_product, fi));
        }

        public IIndexableDataField GetFieldById(object fieldId)
        {
            return Fields.FirstOrDefault(f => f.Id.Equals(fieldId));
        }

        public IIndexableDataField GetFieldByName(string fieldName)
        {
            return Fields.FirstOrDefault(f => f.Name.Equals(fieldName));
        }

        public IIndexableId Id => new IndexableId<string>(Guid.NewGuid().ToString());

        public IIndexableUniqueId UniqueId => new IndexableUniqueId<IIndexableId>(Id);

        public string DataSource => "Product";

        public string AbsolutePath => "/";

        public CultureInfo Culture => new CultureInfo("en");

        public IEnumerable<IIndexableDataField> Fields { get; private set; }
    }
}

Indexable Product Data Field

In this place, properties are prepared to be indexed. Property Product model gets name from IndexInfo attribute and sets it as a field name of indexable data field. It means that field in indexed document will have name from attribute in model.

using System;
using System.Reflection;
using Sitecore.ContentSearch;
using SitecoreBlog.Search.Attributes;
using SitecoreBlog.Search.Model;

namespace SitecoreBlog.Search.Crawlers
{
    public class IndexableProductDataField : IIndexableDataField
    {
        private readonly Product _product;
        private readonly PropertyInfo _fieldInfo;

        public IndexableProductDataField(Product concreteObject, PropertyInfo fieldInfo)
        {
            _product = concreteObject;
            _fieldInfo = fieldInfo;
        }

        public Type FieldType => _fieldInfo.PropertyType;

        public object Id => _fieldInfo.Name.ToLower();

        public string Name
        {
            get
            {
                var info = _fieldInfo.GetCustomAttribute<IndexInfo>();
                return info.Name;
            }
        }

        public string TypeKey => string.Empty;

        public object Value => _fieldInfo.GetValue(_product);
    }
}

Index Info Attribute

This attribute will help us to determinate how name of property will look within the index.

using System;

namespace SitecoreBlog.Search.Attributes
{
    [AttributeUsage(AttributeTargets.Property)]
    public class IndexInfo : Attribute
    {
        public string Name { get; private set; }

        public IndexInfo(string name)
        {
            Name = name;
        }
    }
}

The usage this attribute in our model object properties let us to add those properties to index and define theirs names. In case of lack this attribute in property, property will not be indexed. Let's see usage of this attribute in model

Product Model

In this step we will define model which will be used to prepare and store documents ready to indexing

using SitecoreBlog.Search.Attributes;

namespace SitecoreBlog.Search.Model
{
    public class Product
    {
        [IndexInfo("productid")]
        public int Id { get; set; }

        [IndexInfo("description")]
        public string Description { get; set; }
    }
}

As you can see, we have used here attribute which was presented before. So in results, in our Solr core, we will have documents with two fields: "document_id" and "text"

Results

When the all steps presented above are done, there is a need to rebuild index and our XML file should be indexed in Solr core. As a prove I am attaching screenshot from my Solr panel.

In the next post I am going to show you how index data from few sources into one core and configure search mechanism to work with all this data.

Stay tuned! :)

Switching to Solr - trick with Global.asax

Hey ya!

In the web there are a lot of tutorials which can help you with switching indexes from Lucene to Solr. In most of them we have part where we need to edit Global.asax file and change inheritance to use IOC container. Here is the snippet from Sitecore documentation about which I am talking about. It is totally OK, but we have to keep in mind that method Application_Start from Global.asax.cs will not be triggered, so for instance when we want to register our custom routes in Global.asax.cs - we can't do it now. Now we should do it by usage pipelines etc

Workaround

There is smart workaraound which allow us to use Globax.asax.cs in traditional way. Instead changing inheritance within Global.asax, we can use approach presented below:

    public class MvcApplication : System.Web.HttpApplication
    {
        protected void Application_Start()
        {
            AreaRegistration.RegisterAllAreas();
            RouteConfig.RegisterRoutes(RouteTable.Routes);

            var container = new WindsorContainer();
            
            try
            {

                //this line is crucial. it initialize IOC for Solr
                new Sitecore.ContentSearch.SolrProvider.CastleWindsorIntegration.WindsorSolrStartUp(container).Initialize();
            }
            catch (Exception ex)
            {
                Sitecore.Diagnostics.Log.Error("Solr init error", ex, this);
            }
        }
    }

In this code we've initialized IOC components for Castle Windsor to Solr and now it should work correctly.

It is worth to say that we can do it for other IOC Containers as well. We just have to use proper dll. More details about rest of components are in table.

See you soon!

Switching to Solr - Watch out!

I can bet that during switching from Lucene to Solr indexes you are using scripts to disable Lucene configs and enable Solr configs.

Am I right?

If yes, please be aware, because since Sitecore 8.2 we have additional config file related to Solr, it is called "Sitecore.ContentSearch.SolrCloud.SwitchOnRebuild.config" and your scripts can accidentally enable this config file on your environment, which is unnecessary for you until you don't use Solr Cloud.

During rebuilding my primary indexes, I was facing with HTTP ERROR 404 issues:

• Problem accessing /solr/sitecore_master_index_rebuild/update
• Problem accessing /solr/sitecore_web_index_rebuild/update
• Problem accessing /solr/sitecore_core_index_rebuild/update

First strange thing which I saw, was the akward names of indexes. I've never seen "rebuild" in names of indexes, it was weird for me...

Solution

If you are facing with similar issue, please try disable Sitecore.ContentSearch.SolrCloud.SwitchOnRebuild.config. It helped for me and I hope that it will help for you!