Facing some challenges regarding non-revenue model in Meridian

[annie-gulati]

The below information is also captured in the attached word document

Meridian Issues with Non Revenue Model.docx

Challenges with Non-Revenue Model

Establishing the appropriate priors has proven to be a challenging task. Below is a summary of the steps taken by the team to address the issue:

1. Initially, we implemented the model using basic priors, which resulted in a negative baseline.

Figure 1 Waterfall graph for model with basic priors

2. We then experimented with the Revenue per KPI metric, but this approach did not lead to significant improvements in the baseline.We used Revenue per KPI=20

Figure 2 Waterfall graph for model with Revenue per KPI = 20

3. Subsequently, we tested a Custom Contribution Prior (i.e., utilizing 10%, 30%, and 50% contribution levels), which yielded a negative mu value. The model exhibited signs of overfitting and instability. Despite attempts to adjust the knot values, overfitting persisted without improvement.

Figure 3 Model Metrics for model set with 10% contribution

Figure 4 Waterfall graph for model set with 10% contribution

Figure 5 Model Metrics for model set with 50% contribution

Figure 6 Waterfall graph for model set with 50% contribution

4. Additionally, we explored the inclusion of dummy variables; however, this did not enhance the baseline. It appears that the model prioritizes media contribution and subsequently constructs the baseline.

Figure 7 Waterfall graph for model with added Dummies

We have done multiple iterations with various combinations of approaches, including:
i) Merging specific media variables.
ii) Experimenting with different priors.
iii) Implementing channel-specific priors.
iv) Adjusting the values for Revenue per KPI.
The varying results produced by the different priors have made the selection of an appropriate model particularly challenging.

Please suggest any other approach to tackle this. Whatever we have done so far proved ineffective.

[BurhanuygunSTL]

I am facing similar issues as @annie-gulati, would appreciate support here.

[annie-gulati]

@dattatreyam23 @cpulavarthi

Hi Google Meridian Team

Kindly request you to provide guidance on above mentioned concerns.

Thanks
Annie

[dattatreyam23]

Hi @anniegulati & @BurhanuygunSTL,

Thank you for reaching out to the Google Meridian support team.

It appears you've thoroughly explored several approaches, and the inconsistent results are causing difficulties in model selection.

I would suggest you to proceed with using the total contribution media prior, as you have in # 3. I would also appreciate a little more clarification on the negative mu value, and how it is affecting your results. With respect to the overfitting problem, I’d recommend using a holdout sample that is fairly balanced across geos and time periods. You may find more about selecting holdout_id in the following documentation. We expect differences in test and train to increase as the holdout size increases.

Additionally, if you’re facing issues with setting a reasonable revenue_per_kpi, our recommendation would be to not set one at all. revenue_per_kpi is not meant to be a tuning parameter to get good model results. Rather, it should be considered as data or a reasonable guess as to what the revenue per KPI might be. See here.

We would also encourage you to think critically about the control variables. Please check our documentation here to know more about the same.

The baseline is considered to be the model's estimate for the response variable if there was no media execution. If the baseline is too low, there are a few possible causes to investigate. Kindly check our debugging documentation on Baseline is too low or sometimes negative for more details on the same.

Feel free to reach out if you have any further questions.

Thank you,

Google Meridian Support Team