A Paradigm for Business Intelligence Evolution


This is a series of ideas that will outline how we need to build an e-commerce Analytics solution for a digital marketing agency or consultant to join the data from all channels and client’s data together in one central repository. This data should include online transactional data as well social interactions.

This repository should also be able to import offline customer information too. Additionally, this data will then be combined to create analyses across platforms and, ultimately to be used as a source of predictive analysis.

We will end this document with some timelines around the tools to use, the timelines around their deployment and kind of internal team to execute this plan.


Process Flow in Summary

  1. Setting up data collection within current data sources.
  2. Merging all data sources into one platform and automate such a collection.
  3. Analyzing patterns in these datasets to builds reports and dashboards based on KPIs
  4. Based on past behavior of customers, create prescriptive and predictive Analytics around key metrics and goals.


Data Collection

It is imperative to get all the measurements set up correctly in every marketing tool that is used. We tend to focus mostly on Google Analytics and believe that data alone. Sometimes, when tagging is incorrect on email platforms, Facebook or Bing, we cannot even measure conversions in the same way and that leads to reporting that is erroneous and leads to bad decisions. In this step, it is important to measure each source of traffic to a web domain and be able to capture that either within that domain (for example: clicks from Facebook reporting) or make it work so that the data is captured correctly in another tool like Google Analytics or Adobe Analytics. We need to be able to account for 90% of traffic correctly before we move ahead.

As this data collections are all in separate silos that usually do not support each other well, we will get some numbers that will be off between each tool. That is a cost of doing business at this stage and we should only focus the primary data source when making decisions. For example, if we want to determine whether Facebook CPC is effective, we use the conversion metrics from Facebook reporting rather than try and match that data with Google Analytics.

This becomes rather onerous and requires us to look at several disparate places to get a complete picture of our business, so we then are forced, for the sake of efficiency to move to the next stage.

Merging Data Sources

At some point the enterprise slows down because there is too much data, but no unit can understand the other’s language. If there is an Analytics team, an SEO and SEM team, the confusion arising from a lack of a “single source of truth” can be quite debilitating in terms of decision making. At this point we start to go down two different pathways. We can either build our own data warehouse or buy a tool that will do this for us.

Building a data warehouse comes with the advantage of designing your own data collection and downstream KPI’s that the can be customized as desired. There is a high initial cost but then the flexibility of operation tends to result in increased ROI. Dealing with internal resources also means a quick deployment of any changes and newer traffic sources can be ingested sooner. The data warehouse can be used with any of the data visualization tools like Tableau, Power BI, Qlik or Google Data Studio.

Buying a data merging tool like, Alteryx, Looker, ReportGarden, Optymyzer or BIME has it’s advantages in that the initial cost is lower and they provide built-in connectors to all major data sources. You can also customize them by importing new data sources using API’s or just csv files. Over time, they also end up creating new features and introducing concepts of machine learning and programmatically produced insights that can be of great use to clients also. One doesn’t have the flexibility of an in-house team though.

For an agency, perhaps the latter option is one to start with and the former to aim for in the long-term if the revenues from the Analytics stream are high enough.



Once the processes are in place and perhaps even at the point of data collection, we need to have a core set of people who are going to make sense of all the analysis and provide recommendations to the clients based on all the dashboards and reports that can be built. It has been my experience that lots of dashboards are made and reports are regularly generated but the consumption on the client side is poor because the analytics practice does not translate the numbers into business speak. Please read this great article by Avinash Kaushik that further elaborates the point.

I cannot stress enough the fact that the quality of good analysis and a set of concrete recommendations that clearly explains what the client is going to get by following them goes a long way in the adoption of data driven decision making. For example, a recommendation that says “Moving the add to cart button 300 pixels up will increase revenues by $30k a month” is much better than saying “We find that testing suggests moving the button 300 pixels up will increase conversion rates”

A good web analyst will have experience working with product development, a smattering of coding and long exposure to different kinds visualization techniques. An investment in a good resource(s) of this kind will yield more value over time than the best tools around.


Predictive Analytics

Once a website is running and customers are using it, a one size model in terms of pricing, deals or discounts does remain a good strategy for long. Very soon clients run into a wall where they keep acquiring new customers or current customers without enough understanding of any of them. At this point, you are looking for tools that segment the population in meaningful chunk and target them based on their needs at a particular time. A pet food company can predict when a customer will run out of the current cat food that they have purchased and send a targeted message (email or ad or website chat) at the right time.

There are several open source languages like R or statistical packages like SAS and Tableau that can help an internal analytics team do this work for a client. These require the in-house team to have a strong data science background to create a product offering for clients. This method will give you the best ROI because the tools will be made in-house and can be customized based on a client.

The other method is to buy a tool like Domo, Qlik, clearbit, Zaius and use it’s features to conduct user based analyses that can segment and target customers in different ways.


Timelines for Capability Development

NOW: We are doing Data collection and collation right now in our efforts to serve customers. Pulling data for separate business functions from different tools is the ground situation now.


Six Months: We can build up a capability using an external tool to combine data sources and provide comprehensive and cross-platform reporting.


18 Months: It will take a lot of time and learning to build up an analytics resource team in-house that has experience in all the different domains. If you start hiring now, such a team will be able to start adding value within six months and be giving very good ROI in a year and a half.


2 Years: This is the time it will take for a good team and a set of tools to be able to make deep inroads and to build predictive analytics and modeling capabilities. It will also take time for our client base to become mature enough to realize that this is the ultimate need for any online business.



I hope this plan gives you a sense of what I think the plan to success in the e-commerce Analytics domain. There are many ways besides this to get data together and put analytics to work but in the e-commerce world this should work the best.





UserID tracking in Google Tag Manager(GTM)


Why do we need to do that?

Ostensibly Google says that you do this for tracking true users of your site. Since multiple devices generate a visitor cookie each, they get counted as different visitors. You can stitch sessions cross-device by creating a userid view.

However, if we want to track CAC and LTV by referral sources and we have only Google Analytics and an internal database around, we can piggyback on this method to measure them.

It is tricky and will require some testing before it runs in your environment. However, it works and I have managed to figure it out in Google Tag Manager and will lay it out for you dear analyst, step by step.

Step 1

Create the User-ID view in Google Analytics. Go to the Property and expand “.js tracking Info”. Then follow the prompts as seen in the picture below.


After you have done the above, let’s create a dimension to store this data.

Step 2

In the same property column, below the PRODUCT LINKING tab, sits a place to create custom dimensions as shown in this picture below.



Click on create dimension and add the custom dimension as shown below:


Please note that you have to name it userID. This is because I am going to name the data layer variable later the same. They have to be so to capture your user information.

Step 3

Have your awesome developer pass the user-id value to the data layer variable called userID. The values that it should spit out if you click “view-source” on your site should look like this:

dataLayer = [{‘userID’: ‘yyyyyy’}];

dataLayer.push( {‘event’:’uidAvailable’, ‘uid’:’yyyyyy’} );

They can push this change live anytime and will be quite thankful for the Starbucks drink you will treat them to.

Make very sure that this userID being made cannot be Personally identifiable information(PII). So no names, email addresses, etc.

Step 4

Ok! Let’s push on now to the Google Tag Manager and the last part of this implementation.

Open up your tag manager and on the left hand side where it says, variables, click and add a user defined variable. We will call it userID.


Now, on the left hand side open up the tags section. On all the tags that have anything to do with an event or page view, we will make slight edits. Let’s start with the Google Analytics tag (it should be set to universal Analytics otherwise this will not work).


We add two variable to be filled when the tag fires, one is the userId (please note that this is the exact name you need to set). The other is the custom dimension that we have created in Google Analytics so we can run reports based on our own user-ids, which I am sure, are aligned to email addresses.

Step 5

Now, let’s set our workspace on Google Tag Manager to Preview the changes that we have made. If this works, the Real-Time reports in Google Analytics will immediately start collecting data. Once it works, then go ahead and publish the new tags.

Now, you should be free to download the userID and referral source. You can download the LTV and userID from your own database and analyze in Excel or any other spreadsheet of your choice.

You would have noticed that I keep harping on the name and case of the variable “userID” and “userId”. I thought to name all variables the same and it didn’t work. However, hard-coding the variables works perfectly!

Do let me know if you have any issues with this and I can walk you through the implementation, if need be.


The education of an Analyst

There are really two streams that we can see these days coming out of Master’s programs. One is the marketing/product Analytics practitioner and the other is data science practitioner. At the initial levels we need to learn  both streams because it will not really be clear to a student where their capability or interests truly lie.

Based on this premise, I would like to suggest a few types of learning that web analyst in particular need to imbibe in the start of their careers.

Business Writing is factual journalism.

I have experienced a lot of analysts coming in with lots of capability with numbers but none with writing business stories. I also see many analysts who don’t know the first thing about how to use stats tests in real life situations. Simple questions like —  Did the pricing change lead to more leads? Are the results statistically significant? — make us all go into huddles and battle stations.

Several online course providers offer creative writingbusiness writing or journal writing courses and I think they should be part of any Analysts formative education.

Knowledge of the tools.

We need to provide hands-on understanding of the tools that are used in typical business analytics. I have seen a lot of spreadsheets(Excel wizardry can take you lots of places) being used.

There are some statistical packages like R(open sources and free, hence very useful), SPSS and SAS that the analyst should be familiar with. More importantly, they need to be able to live and breathe statistics because the core fundamentals are always useful. Some elements of data mining and clustering should also be learnt as needed or as a person’s curiosity demands it.

These days, data visualization done correctly, explains half the analysis by itself. So, some experience in TableauMicrostrategyQlikview is very important in explaining the information clearly.

I also see a lot of visualization available via Google Data Studio and Power BI.

At the end, it is all code.

Whatever the environment of development in an organization, the analyst should have a fair idea about it since all the measurement hooks are mostly encoded. Even the ubiquitous Google Analytics is run based on javascript. A solid grounding in Python, PHP and SQL is also important so we need some intermediate technical courses to get a well-rounded technical perspective.

Understand the data river as it flows downstream

Although, large amounts of data exists, we don’t get training in information architecture.. How do you get unstructured data to structured data, to BI tools and to reports and metrics is something that needs to be taught too. The analyst can use some BI tools and data warehouse development and maintenance skills. As it become more common to build self-service reporting systems, we need to be able to teach some rudiments of this skill to our budding analysts.

Project Management

Most Analysts end up in large project teams as core members in-charge of numbers. So we need to teach them principles of project management and the current techniques like Agile and kanban. This will help them build their managerial skills too.

Design and Analytics

A good analyst will have a strong creative streak, not just have a good head for numbers. We need to be able to look the system that we are trying to improve (or make profitable) from a design perspective too.

In fact, a lot of web analytics is done just to measure and influence the design and flow of the information on the site for the customer. Conversely, not many designers are just going to rely on their intuition while building a site. They would ask the analyst about measuring the success of a particular design. So, a couple of introductory courses on creative design are essential for the education of an analyst.

I hope these make sense dear reader. Please let me know your thoughts.