Forecasting in Daisho uses regression underneath the hood, with one very crucial difference: the assumption is that time is the most important driver, and one only needs to explain the difference.

So what do we mean when time is the most important driver?

We assume periodicity + long-term trends + auto-regression are likely to explain most of the outcome. For example, if the long-term growth in the outcome is 2%, its only fair to assume that continues to grow at 2%. Similarly, a low month might typically be followed by a high month just as a smoothening process. We handle the first using long-term trends, and the second using auto-regression. The outcome might also be periodic, e.g., ice-cream sales which always crest in the summer, and are rather low in the winter. We handle this using periodicity.

Generating Time-Independent Columns is the Key

The forecasting workflow assumes that the data is a time-series. All you need to tell us are the outcome variable, the time-stamp column. Daisho does the rest.

Daisho automatically extracts the impact of time on ALL variables in the data - not just outcome variable. Daisho automatically figures out what's the long-term trend (if any), and the periodicity pattern (if any) for each column in the data, and builds time-independent columns.

Daisho then fits regression models based on time-independent columns. Note that we are making the drivers also time-independent. To understand why that's important, consider impact of temperature on ice-cream sales. The same temperature - say 30 Degrees Centigrade - might be considered too hot for winter leading to increased sales, and considered pleasant in the summer leading to sales being below normal. We can understand this only when we make temperature time-independent.

What's the data required? What's the pre-processing required?

Baselines Based Solely on Time for Outcome Variables

In the above example, while the data itself is quite wavy, Daisho was able to build a rather nice baseline based solely on time. As much as 78% of the variation in raw data goes way due to time alone!

Relationship with Drivers: Raw + Time-Independent

How is the driver related to the outcome? And how does that change when both are adjusted for the impact of time?

In the above example, the correlation between raw outcome and raw driver is 0.988 - which is rather high. And that remains equally high - 0.912 - even when adjusted for time!

Top Drivers (including interactions), their rankings, and impact on your outcome:

Simulations based on Predictive Models: Change the values, and see predictions change

In addition to these insights, you can also setup your models for predictions, and auto-learn!

What are the drivers for your outcome AFTER adjusting for time? Auto-periodicity Detection, Correlations, etc.

A time series is a series of data points indexed (or listed or sorted) in time order. Most commonly, a time series is a sequence taken at successive equally spaced points in time. For example:

Ambient temperature recorded at a periodic (e.g., every minute, hour, etc.) interval.

Machine vibrations recorded at an hourly level.

You typically need two columns for any time-series: a time-column, and a value-column. So for example, when we talk about ambient temperature, the two columns would be time-stamp and temperature recorded at that time-stamp. Several times your data might have several time-series together. For example, the above data might also include location. What you are looking at then, is multiple time-series put together. Daisho is completely capable of handling such scenarios.

The typical questions one might seek to answer in time-series analysis are:

Are there any segments/clusters which have similar behaviour? Many a time, you tend to have time-series data for several units, e.g., machines. You want to know machines have similar yield degradation over time. Effectively, you are plotting yield for each machine over time, and you want to know which lines look similar (have similar shapes).

Have there been times in the past when anomalies occurred? And have anomalies occurred across dimensions at the same time?

Time-based trends/Leads/Lags: How do actions in the past impact results in the present? Oftentimes, actions (e.g., pricing changes) tend to show their full impact (e.g., impact on sales) with a certain lag.

The key to time-series analysis is the dependence on time. The dependence might come in one or both of the following ways:

What’s happening at time (t) is dependent on what happened at time (t-1). For example, an upgrade to machinery yesterday (t-1) led to an increase in production today (t). Or, yield has been falling for the past few weeks (time till t-1), so we can expect it to happen today (t) too.

There’s an inherent pattern in data which is time-driven. For example, ambient temperature follows a definite seasonal pattern.

Time-Series analysis is typically used when you expect either of the above conditions. If neither is true, you might be better off with standard analysis - drivers, clusters, etc. as the case may be.

The only MUST-HAVE in your data is the time variable. This is all the information you need to give Daisho to kick off time-series analysis.

Daisho automatically builds time-series analysis for ALL columns in your data using the time variable you identified. You can - at any point in your analysis - decide to rollup the data. The rollup dimensions, of course, are limited to the columns that are present in your data!

What is the data and information required?

The most important step to time series analysis is to tease out relationship with time. Daisho automatically does this for any kind of data. Daisho tries out many different kinds of patterns - one of which is NO relationship with time - and picks out the relationship that fits data the best. This is completely automated, and is the starting point for ANY time-series analysis on Daisho.

Oftentimes, data is collected at a certain resolution (e.g., hourly/daily), while you might want to run your analysis at a different resolution (e.g., weekly/monthly). Daisho lets you do that - you just need to select your analysis resolution, and Daisho automatically rolls-up your data accordingly.

You might also want to aggregate data based on other dimensions. For example, you might have sales data at a store level, but you want to run your analysis at a district or state level. Again, you can set the aggregation rollup, and Daisho automatically does the rest.

As with every recipe on Daisho, you get an input data quality report automatically. You can see stats for each time-series, as well as time-periods where data is missing. 

Daisho currently supports several uni-variate analysis:

Automatic trend detection after adjusting for seasonality.

Anomaly detection after adjusting for trends and seasonality.

In addition, you can also do multi-variate outlier analysis. We have Driver Analysis coming soon as well!

How Daisho Does Time-Series Analysis. And What You Can Do Today.

Anomaly Detection, Sales Drivers, Predictive Maintenance, Segmentation, etc

Suppose there was an event. The event could be something you did (e.g., an ad-campaign), something that happened outside your control (e.g., Covid), or for that matter just something you can define from your data (e.g., an experiment with treatment and control groups). Impact analysis helps you analyse the impact of the event on outcomes you can define. For example:

What was the impact of my marketing campaign (event) on customer conversion (outcome)?

For what kind of orders does late delivery (event) cause a higher rate of churn (outcome)?

How did the profile (outcome) of patients coming into the healthcare system change due to Covid (event)?

Where impact analysis differs from (and builds upon) A/B analysis or classical experiments analysis is that the model not only gives you the impact, but also the conditions driving the impact. So, in the marketing campaign example above, you get overall change in conversion, and you also get drivers for the delta, i.e., customer profiles where the campaign is being highly effective, and profiles where it isn't or maybe even counter-productive. 

Daisho automatically gives you a bunch of insights:

Clusters where impact is most different from the rest of the data - based on raw variables as well as Signal Factory.

Univariate Analysis - for each column, tests if the column is different when the event occurred vs not-occurred, and if the relationship of the column with outcome has changed based on event.

In addition to these insights, you also have the ability to export predictions from this data, as well as to generate predictions on new data.

What Exactly Does Impact Analysis Generate?

The first question is related to the nature of outcome - this decides the recipe you choose on Daisho. The outcome can be of two types:

A continuous variable, i.e. a number you want to predict. For example: weekly sales, production yield, etc. In Daisho, you would be using the Impact Analysis (Continuous Outcome) recipe. All you need to do is first choose the data, and then point at the correct outcome variable in your data.

A binary outcome,  i.e., a YES/NO (or True/False). For example: customer conversion, loan default, machine failure etc. In Daisho, you would be using the Impact Analysis (Binary Outcome) recipe. Once you choose the data, you then define conditions for YES/True/Success in your outcome. For example, you might define machine failure (outcome) as temperature (a column) crossing 60 degrees, AND pressure (another column) below 20 psi. You set these conditions in the recipe.

The next question is event definition. Your data MUST contain information related to the event. Once you choose the data, you then define conditions for the machine to identify rows where the event has happened. For example, you might define the event as time-stamp (a column) after March 1st, 2020, AND channel (another column) as XYZ. Both recipes above allow you to define conditions for the event.

Daisho can build impact analysis on any data where you can define an event, and an outcome variable. Daisho assumes ALL columns in the data are potential drivers, and automatically flags the most impactful/powerful ones.

However, do note that Daisho's Impact Analysis analyses the data at a row-by-row level. So for example, if you want to find impact on weekly sales, and the data you have is transaction data, you first need to aggregate into your data into weekly format (Convert to Time Series Box) before using this recipe.

We got a bunch of insights already here. Predictive models, clusters, univariate analysis, etc. Once you build models, you can also set them up for Predictions. You can then use these models in your regular workflow!

You can even export all the data - including signals from SignalFactory if you wish!

Event Impact: A/B Analysis, Treatment/Control, External Events (e.g., Covid)

Daisho's text mining workflow builds a machine-learning model that automatically extracts topics or themes from your data. You can then use this model to tag each row of data to a collection of topics/themes. Once you do that, you can do all kinds of analysis on that. For example, you can see a time-based trend on topics, i.e., how have topics been growing/decreasing over time. Or you can compute mean of time-spent (assuming this is a column) for each extracted topic

Topics/themes in Daisho are collection of words that typically occur together statistically. Lets consider the example of newspaper articles. Articles related to politics tend to have a lot of words like elections, party, representative, bill, etc., while articles related to sports might have words like won, thrashed, out, scored, etc. The machine learning model tries to learn these distributions of words, and presents similarly occuring words to you as a single topic.

The above analysis was done on a set of 50,000 Amazon reviews. You can see what the machine picked up as the top words for each topic. For example, Topic 1 seems to be around flavor, and dog-food, while Topic 3 seems to be around Coffee, and a bunch of related things.

All you need is data with your text column identified. You can even select upto 5 text columns, and the machine automatically joins them, and extracts topics from the resulting column.

Is there any pre-processing I need to do?

Daisho does standard pre-processing - stemming and lemmatization. For example, [take, took, takes] all get converted to take automatically. You don't need to bother about this. However, Daisho doesn't handle typos and spell-checks.

Absolutely. Behind the hood, Daisho computes the probability of each row (document) belonging to each topic. As a result, we commonly see documents getting categorised into more than one topic - and you get to see the weightage for each topic too.

I see primary topic and secondary topic in the example above. What are they?

Primary topic is the topic that the model believes is the strongest indicator for the row/document. Secondary topic is the next strongest. You are also likely to see several rows/documents where there's no secondary topic.

Daisho handles that too. The model assigns zero/minimal weightage to topics which it thinks are highly unlikely. So, you will see rows where ALL topic weights are zero!

What Data do I need to have? And other Questions

We got a bunch of insights ready currently - topic profiles, topic-level stats, and time-based trends. With topic-level stats, you can start answering questions like "what's the mean score (score is a column in data) for each topic?"

With time-based trends, you can answer questions like "how has topic distribution changed over time?":

Or even: "how has values of column X for topics changed over time?":

You can also export Topic IDs and scores for uploaded data. And predictions and auto-learn are enabled too!!!

Discover categories/themes in your data. Categorise new data as it comes in. Or flag anomalous words as they come

Driver analysis, which is also known as key driver analysis, importance analysis, and relative importance analysis, quantifies the importance of a series of predictor variables in predicting an outcome variable. Each of the predictors is commonly referred to as a driver. You can use driver analysis wherever you have a clearly defined outcome. For example:

What drives sales (outcome variable) of a particular product?

Under what conditions (drivers) are you likely to get the best yield (outcome variable) in your factory?

What are the most conducive conditions (drivers) for customer retention (outcome variable)?

Which loans are likely to result in a default (outcome variable), and under what conditions (drivers)?

Driver analysis tends to be used in two ways:

A better, and deeper understanding of the business, and

Predictive analytics to flag likely issues before they occur, or to prioritise actions by impact

The recipe uses the SignalFactory to generate and test a whole bunch of hypotheses. Then, the machine builds predictive models based on the top drivers. A whole bunch of insights are available out-of-the-box:

Top drivers (including interactions), their ranking, and their impacts for your outcome:

Clusters where outcome is most different from the rest of the data - based on raw variables as well as Signal Factory.

Univariate Analysis - how each column in the data impacts the outcome independently. Correlations, regimes, etc. You can dive deeper into each individual column too

In addition to these insights, you can also setup your models for predictions, and auto-learn. 

The first, and most important information is the outcome variable. Your data MUST contain information related to the outcome variable.

The next question is related to the nature of outcome - this decides the recipe you choose on Daisho. The outcome can be of two types:

A continuous variable, i.e. a number you want to predict. For example: weekly sales, production yield, etc. Continuous outcomes are typically modelled using regression models. In Daisho, you would be using the Continuous Outcome Driver Analysis recipe. All you need to do is first choose the data, and then point at the correct outcome variable in your data.

A binary outcome,  i.e., a YES/NO (or True/False). For example: customer retention, loan default, machine failure, etc. These kinds of outcomes are analysed using classification models. In Daisho, you would be using the Binary Outcome Driver Analysis recipe. Once you choose the data, you then define conditions for YES/True/Success in your outcome. For example, you might define machine failure (outcome) as temperature (a column) crossing 60 degrees, AND pressure (another column) below 20 psi. You set these conditions in the recipe.

Daisho can build driver analysis on any data where you can define an outcome variable. Daisho assumes ALL columns in the data are potential drivers, and automatically flags the most impactful/powerful ones.

However, do note that Daisho's Driver Analysis analyses the data at a row-by-row level. So for example, if you want to find drivers for weekly sales, and the data you have is transaction data, this is not the right recipe. You have two options: either aggregate your data into weekly format and then use this recipe, or use the Time-Series Analysis recipe on Daisho.

We got a bunch of insights already here. Predictive models, clusters, univariate analysis, etc. Once you build models, you can also set them up for Predictions, and Auto-Learn. You can then use these models in your regular workflow!

Yield Optimisation, Sales Drivers, Churn Predictors, Credit Risk Drivers, etc.

How You Can Use Daisho

Forecasting/Time-Based Driver Analysis

Time Series Analysis

Impact Analysis

Text Mining Analysis

Driver Analysis