B2B e-commerce search with Solr – product assortment and prices

B2B e-commerce search with Solr – product assortment and prices

Posted on October 5, 2014 by Mario Febbraio —Leave a comment

Summary

Business-to-Business (B2B) e-commerce presents its unique challenges on processes and technology: business units, cost centre, approval process, budgets, credit management, prepaid stock, quotes and procurement punch-out are some of those. But assortments and prices have direct impact on search engine as every B2B customer or unit can have his owns.

This post highlights the role of search engine in the B2B purchasing process and analyses two technical solutions to implement customer specific product assortment and prices in Solr. A bespoke solution relying on the integration with an in-memory datagrid is detailed – as proof-of-concept – to address dynamic pricing.

The opportunity

A B2B e-commerce site needs to carefully consider how assortment and price is researched by customers. Indeed the spectrum of B2B is so wide –from small business to corporates – that the same platform may have to support completely different purchase journey. The search engine must be “aware” of this diversity and be designed in such a way to provide the correct experience to every user: right assortment, right price and in the right timeframe –sometimes less than 300 milliseconds. Failure to do so results in missed orders.

Definitions

Dynamic Pricing

Dynamic pricing refer to the practice of frequently adapt offering prices according to ever-changing market demand. This strategy changes cost of goods or services, in extreme cases, at every request. Fares depend on customer specific characteristics or behaviour (e.g. previous research for an item) or on more general market demand (e.g. peak of request for quote). This is a common in travel industry but it is making his way into retail.

Customer Group

Customers are usually grouped together into clusters for management and marketing purposes. Inside these groups, frequently referred as customer segments, buyers tend to behave in the same “way”. The web-platform offers the same experience and more importantly presents the same prices and promotions. In B2B realm, when a customers is a big corporate, users are further grouped or linked to business units (BU). These manage different functions or geography of an enterprise and have their own budget, buy their own assortment and may negotiate their own supplying prices. The icons below summarise this concept.

Corporate and units

Price row

Every item sold needs to have at least one price, which is usually stored as an entry in the storage system (RDBMS, NoSql, in-Memory cache, etc…). This entry is commonly called a “price row”. But every item can have multiple prices depending on the date, customer group which they apply to, etc… This is briefly summarised in the table below:

Price Rows

On March the 1st there are two active prices: Price Row 2 and Price Row 3. The first is applied to “all” users but the ones that are classified as “corporate”. The latter is assigned only to “corporate” users, which will pay $0.90. Price Row 4 is not active in March so “loyal” customers get the default amount ($1.10).

Price List

A price list is the list of SKU and active prices for a specific customer group. For instance, the following catalogue has two SKUs (SKU0001 and SKU0002) :

Two SKUs Price Rows

On March the 1st there are 3 price lists: All (default), Corporate and Loyal.

Price List

SolR and DocTransformer

“Solr(TM) is the popular, blazing fast open source enterprise search platform from the Apache Lucene(TM) project.” (https://lucene.apache.org/solr/). It is document oriented – it accepts, as input, documents in different formats- and is able to search through them according to users’ queries. Entries are indexed when imported and these are scanned in order to produce result sets. A query request goes through a set of multiple steps, which can be customised. The response is then serialised by a writer in XML, Json or else. Document transformers (DocTransformers) can be called to alter the content of the returned items by adding or removing fields. The logical flow is illustrated in the following picture.

Solr DocTransformer

TransformerFactories are declared in solrconfig.xml:

<transformer name=”[transformerName]” class=”[transformerClass]” >

<int name=”value”>5</int>

</transformer>

In the snippet above, the transformerClass extends TransformerFactory and transformerName is the name used in the query to call the transformer. In the following example a transformer is called with two parameters.

http://localhost:8983/solr/collection1/select?q=*.*&amp; &fl=…,transformerName:[param1=xxxparam2=xxx]

Examples of java code are shown in the “Solution” section below.

B2B purchase process – not all customers are the same

Customers are not born the same even in the B2B. Small businesses act and purchase much more as business-to-consumer (B2C). They tend to land supplier’s sites via an organic search, to add products to a basket, to go through the check-out process as outlined below.

Customer purchasing process

Small businesses are also explicitly targeted via “standard” marketing campaigns offering discount vouchers on first order. This is extremely useful to bring traffic and create interest in new prospect business customers. This picture summaries this buying process.

Voucher purchasing Process

Sometime when the relation between business customer and supplier become tighter or when the product supplied can vary according to specific configuration, the potential buyer asks for a quote as highlighted below.

Quote purchasing process

In big corporates the entire procurement process is even more formalized. Usually goods and services are acquired only from a list of approved suppliers and items are explicitly detailed in formal agreement. The process that brings to an order is lightly sketched in the following diagram.

Corporate purchasing process

In very big enterprise every business unit negotiates his own rate, assortment and manages its own budget. This process, when applied as strategic tool for acquiring new customers and retain older ones, puts considerable strain on the IT assets. Indeed every deal increases the number of price rows (number of row stored to price SKUs).

Custom Price and Assortment

Big corporate customers have their negotiated price lists. These can also define an exclusive assortment (only products in the assortment can be bought) or an “open assortment” – the list define prices for specific SKUs but all items in the supplier’s catalogue can be bought at base/default price. The tables beneath show in the first column the base assortment and cost, which is due, and, in the following columns, how it “intersects” with customer specific catalogues. In this first example “Customer A” negotiated an “exclusive assortment” at fix price. In this case only specific items can be bought.

Close assortment

“Company A” can only buy SKU0001, SKU0002, SKU0005 at agreed prices. The system prevents buying other items via the corporate log-in account.

In this other example, “Company B” negotiated an “open assortment” at fix price.

Open assortment

“Company B” is able to buy every item in the catalogue but, for non-fixed prices (e.g. SKU0002 and SKU0005), the base amount is applied. In these settings, the number of price rows linearly grows with new B2B agreements. If these have to be shown and indexed into a search engine, considerable challenges arise with high volumes.

Search and pricing strategy

Although B2B straight rebuy is common, recent studies suggest that B2B buyers go directly to the supplier web site when looking for information. Therefore offering the right price to the right client as part of search result (e.g. a new prospectus) can be extremely important in supporting supplier development and reinforce loyalty.

Solution

To address “searching” in these complex scenarios, two solutions are proposed:

1. Flat structure in Solr.
2. Bespoke transformer and dynamic price calculation.

Flat structure in Solr

As in any NoSql storage, Solr holds a flat structure; prices, as not mandatory fields, are added to the SKU document with a prefix referring, for instance, to customers’ identifiers. The “fl” parameter filters the right value:q={!lucene q.op=AND}screen HD , fl= description,brand, price, price_X. Lack of a price field or a negative value signals absence of specific price or no assortment for a customer. In this scenario all Solr functionalities (sorting, faceting, etc…) are supported out-of-the-box (OOTB).

Every field with “indexed=true” requires a new index so memory constraints should be addressed – Lucene indexes can be stored in a distributed cache.

Load Test

A simple load test with jMeter on a:

• VM with Ubuntu 14, 4 cores uncapped (host i7 windows 7), 16GB ram and SSD.
• 100K SKUs and 100 prices.
• 4 concurrent users and 2000 loops.
• CPUs 100%.
• q={!lucene q.op=AND}sku ${1-1000}&fl=description,brand,price${1-100}_f, rows=100,wt=json

Produced:

Average: 22ms – Median: 15ms – Mode:14ms – Max: 253ms – approximately normally distributed.

When concurrent users were 8:

Average: 45ms – Median: 31ms – Mode: 28ms – Max: 399ms- approximately normally distributed.

Bespoke Transformer

Sometimes the price of an item is not known when data is indexed. Dynamic pricing is a reality in many verticals and requires tailoring the amount to the user and sometimes to the request. In the travel industry, rates change often and are commonly calculated at runtime. The tariff, once returned as part of search request, is guaranteed for a defined time span. Solr transformers can be used for this purpose as shown in the following diagram.

Solr and In-Memory grid

Here there is a code example of a transformer factory:

TranformerFactory Code

Few comments:

• Line 19 – Here the TransformerFactory, instantiated only once, joins the cache cluster.
• Line 28 to 29 – The TransformerFactory checks for parameters and instantiates the DocTarnsformer (line 31). An instance of the cache is passed to theDocTarnsformer.

The DocTarnsformer code is shown here.

DocTransformer

1. Line 32 – the key is built.
2. Line 33 – the price is retrieved from the cache.
3. Lien 34 – further processing and logic can run here (e.g. modifying price and store it back into the cache).
4. Line 35 – a price field is added to the document and returned.

Note

Why Not Solr Join

Solr join functionality allows querying on the join of two documents. The typical clause will look like

q={!join from SKU to SKU_1}….

The main disclaimer about join is “that Joins in Solr are not really equivalent to SQL Joins because no information about the table being joined “from” is carried forward into the final result.” (http://wiki.apache.org/solr/Join). So in the example discussed, where both description and prices – stored in separate documents – need to be returned, join alone is not a solution.

Why Not Solr Block Join

Block join allows modelling a parent-child relation. Specific prices can be seen as child of an SKU but the document structure will not allow loading son and children as separate document. Therefore, in our scenario, there is no specific advantage over a “flat” structure.

Conclusion

B2B e-commerce sits, tend to be functionally reach around prices and assortment. Solr search engine offers multiple extension points to meet these requirements and its ability to easily integrate with in-memory datagrid, capable of parallel processing, makes it an ideal instrument to offer the right customer experience.

Mario Febbraio