A/B test not working on first load of homepage

[nardoayala]

Hello @adinhodovic , I hope you're doing great.

Pardon me if this isn't the place to write these kind of stuff, but in Stackoverflow someone suggested me to write an issue directly in the repo.

My problem is that I followed this tutorial in order to create an A/B test for my project. The problem is that the test is for the homepage, and no matter what I do, every time I load the page for the first time I only see the original version.

In order to the variant to be showed, I need to refresh the page. The problem with that is that I'm testing a call to action in the homepage, so I need users to see the variant when visiting the site for the first time. It could be related with Django not detecting the cookies when rendering the view?

Am I doing something wrong or is not possible to make A/B tests on landing pages and homepages following this method?

[adinhodovic]

Hey @nardoyala, hope you are too :)!

No it makes sense to have the issue here. Does it work locally by changing the active variant index (forcing a specific variant to show up locally):

In production do you see a _gaexp cookie when loading the page? Is the experiment active and running?

The project should work on any page. If you could share a code snippet, configuration settings and check if the cookie is injected by Google Optimize (_gaexp cookie).

Do you have the request context processor added ('django.template.context_processors.request',)?

[nardoayala]

Thank you so much for your reply @adinhodovic. I checked these things you mentioned and apparently everything is ok.

Everything works like a charm when I force the active variant in the Django admin panel. Here you can see my configuration for the experiment in the Django panel:

And this is how I handle the different render in my index template:

{% if request.google_optimize.new_cta == "New CTA" %}
	{% include "main_website/index-B.html" %}
{% else %}
	{% include "main_website/index-A.html" %}
{% endif %}

I've deleted all cookies for the site, and hard refresh it in order to see if the cookie is being injected, and apparently it is:

(I put a black rectangle to cover the id because I don't know if I can share it, usually people cover that kind of stuff when sharing screenshoots)

In my settings.py file, I have the django.template.context_processors.request:

TEMPLATES = [
    {
        "BACKEND": "django.template.backends.django.DjangoTemplates",
        "DIRS": [
            os.path.join(BASE_DIR, "templates/"),
        ],
        "APP_DIRS": True,
        "OPTIONS": {
            "context_processors": [
                "django.template.context_processors.debug",
                "django.template.context_processors.request",
                "django.contrib.auth.context_processors.auth",
                "django.contrib.messages.context_processors.messages",
                "social_django.context_processors.backends",
                "social_django.context_processors.login_redirect",
            ],
        },
    },
]

I'm using a Django package called django-cookie-law in order to make my project GDPR compliant.

Do you think maybe that package is messing around with this package? In theory the cookie is being injected, but maybe there's something I'm not seeing.

[nardoayala]

By the way, do I need to add the Google Optimize script in my html file? I'm doing it, but in the tutorial I followed the person didn't mentioned it.

I'm talking about this piece of code:

<script src="https://www.googleoptimize.com/optimize.js?id=myId"></script>

[adinhodovic]

Thank you so much for your reply @adinhodovic. I checked these things you mentioned and apparently everything is ok.

Everything works like a charm when I force the active variant in the Django admin panel. Here you can see my configuration for the experiment in the Django panel:

And this is how I handle the different render in my index template:

{% if request.google_optimize.new_cta == "New CTA" %}
	{% include "main_website/index-B.html" %}
{% else %}
	{% include "main_website/index-A.html" %}
{% endif %}

I've deleted all cookies for the site, and hard refresh it in order to see if the cookie is being injected, and apparently it is:

(I put a black rectangle to cover the id because I don't know if I can share it, usually people cover that kind of stuff when sharing screenshoots)

In my settings.py file, I have the django.template.context_processors.request:

TEMPLATES = [
    {
        "BACKEND": "django.template.backends.django.DjangoTemplates",
        "DIRS": [
            os.path.join(BASE_DIR, "templates/"),
        ],
        "APP_DIRS": True,
        "OPTIONS": {
            "context_processors": [
                "django.template.context_processors.debug",
                "django.template.context_processors.request",
                "django.contrib.auth.context_processors.auth",
                "django.contrib.messages.context_processors.messages",
                "social_django.context_processors.backends",
                "social_django.context_processors.login_redirect",
            ],
        },
    },
]

I'm using a Django package called django-cookie-law in order to make my project GDPR compliant.

Do you think maybe that package is messing around with this package? In theory the cookie is being injected, but maybe there's something I'm not seeing.

I'm not sure 100% if it affects, should not. It looks good locally(the override should just be used locally). Can you give me an example of the value of the cookie? Specifically the last value should be 1 (indicates the variant). In the following example: _gaexp:"GAX1.2.VeLMgYZiSGu8_1f0HAsM9A.18956.1", after the 18956 there's a 1. If it's not variant 1, can you open incognito windows til you get that variant? Also ensure that the experiment ID matches whatever you set the ID in the django admin.

[nardoayala]

Thank you so much for your reply @adinhodovic. I checked these things you mentioned and apparently everything is ok.
Everything works like a charm when I force the active variant in the Django admin panel. Here you can see my configuration for the experiment in the Django panel:

And this is how I handle the different render in my index template:

{% if request.google_optimize.new_cta == "New CTA" %}
	{% include "main_website/index-B.html" %}
{% else %}
	{% include "main_website/index-A.html" %}
{% endif %}

I've deleted all cookies for the site, and hard refresh it in order to see if the cookie is being injected, and apparently it is:

(I put a black rectangle to cover the id because I don't know if I can share it, usually people cover that kind of stuff when sharing screenshoots)
In my settings.py file, I have the django.template.context_processors.request:

TEMPLATES = [
    {
        "BACKEND": "django.template.backends.django.DjangoTemplates",
        "DIRS": [
            os.path.join(BASE_DIR, "templates/"),
        ],
        "APP_DIRS": True,
        "OPTIONS": {
            "context_processors": [
                "django.template.context_processors.debug",
                "django.template.context_processors.request",
                "django.contrib.auth.context_processors.auth",
                "django.contrib.messages.context_processors.messages",
                "social_django.context_processors.backends",
                "social_django.context_processors.login_redirect",
            ],
        },
    },
]

I'm using a Django package called django-cookie-law in order to make my project GDPR compliant.
Do you think maybe that package is messing around with this package? In theory the cookie is being injected, but maybe there's something I'm not seeing.

I'm not sure 100% if it affects, should not. It looks good locally(the override should just be used locally). Can you give me an example of the value of the cookie? Specifically the last value should be 1 (indicates the variant). In the following example: _gaexp:"GAX1.2.VeLMgYZiSGu8_1f0HAsM9A.18956.1", after the 18956 there's a 1. If it's not variant 1, can you open incognito windows til you get that variant? Also ensure that the experiment ID matches whatever you set the ID in the django admin.

Of course I can give you an example. I'll use <myId> in replace of the real one, but the real one is something like this mjnlQY-HGT-1f0HAsM9A. It matches the id I have set in the admin panel.

These are the values I'm receiving:
GAX1.2.<myId>.18955.0
GAX1.2.<myId>.18955.1

But, no matter what the value is, it always shows the index-A.html on first load. So for some reason that conditional inside Django is being false the first time I load the page.

[adinhodovic]

@nardoyala Could you grab the cookie and it's value from your production server and insert it locally to your local running server and debug/print the request.google_optimize object after that?

[nardoayala]

request.google_optimize

I copied the cookie in my local server and reloaded the page. This is what is showed when I print request.google_optimize:

{'new_cta': 'New CTA'}

And this is the piece of code inside the views.py file that prints that:

class Index(TemplateView):
    def get_template_names(self):
        print(self.request.google_optimize)
        return ["main_website/index.html"]

[adinhodovic]

request.google_optimize

I copied the cookie in my local server and reloaded the page. This is what is showed when I print request.google_optimize:

{'new_cta': 'New CTA'}

And this is the piece of code inside the views.py file that prints that:

class Index(TemplateView):
    def get_template_names(self):
        print(self.request.google_optimize)
        return ["main_website/index.html"]

Interesting, it looks like it all works as it should. I assume it works locally with that cookie too and without a override. Could you double check the django admin in Production and ensure that it's all resembled to your local setup which is the working(assuming) setup?

[nardoayala]

request.google_optimize

I copied the cookie in my local server and reloaded the page. This is what is showed when I print request.google_optimize:

{'new_cta': 'New CTA'}

And this is the piece of code inside the views.py file that prints that:

class Index(TemplateView):
    def get_template_names(self):
        print(self.request.google_optimize)
        return ["main_website/index.html"]

Interesting, it looks like it all works as it should. I assume it works locally with that cookie too and without a override. Could you double check the django admin in Production and ensure that it's all resembled to your local setup which is the working(assuming) setup?

request.google_optimize

I copied the cookie in my local server and reloaded the page. This is what is showed when I print request.google_optimize:

{'new_cta': 'New CTA'}

And this is the piece of code inside the views.py file that prints that:

class Index(TemplateView):
    def get_template_names(self):
        print(self.request.google_optimize)
        return ["main_website/index.html"]

Interesting, it looks like it all works as it should. I assume it works locally with that cookie too and without a override. Could you double check the django admin in Production and ensure that it's all resembled to your local setup which is the working(assuming) setup?

request.google_optimize

I copied the cookie in my local server and reloaded the page. This is what is showed when I print request.google_optimize:

{'new_cta': 'New CTA'}

And this is the piece of code inside the views.py file that prints that:

class Index(TemplateView):
    def get_template_names(self):
        print(self.request.google_optimize)
        return ["main_website/index.html"]

Interesting, it looks like it all works as it should. I assume it works locally with that cookie too and without a override. Could you double check the django admin in Production and ensure that it's all resembled to your local setup which is the working(assuming) setup?

I copied and pasted my setup from local to production and I'm still having the same behavior, on first render the original variant is rendered every time, even though the cookie I'm receiving sometimes has the id of the new variant.

I believe there must be something that is causing my view to be rendered before receiving the cookie value. It must be something like that, because when I reload the page everything works just as expected. I will continue testing things and as soon as I find a solution I will share it here.

There is just one more thing I would like to ask you before doing that, if you don't mind:

• In the tutorial you don't mention anything about adding the Google Optimize script inside the html template. Is not necessary to add it?, I'm talking about this piece of code:

  <script src="https://www.googleoptimize.com/optimize.js?id=myId"></script>

I'm adding that to my base template because I when I was configuring Google Optimize, Google told me to do it.

[adinhodovic]

IIRC you don't need to initialize the snippet, I believe that is for client side A/B testing. Did everything work locally without an override and with the cookie?

I can't grasp why it does not work. Could you try it on a View level?

def get_template_names(self):
    variant = self.request.google_optimize.get("new_cta", None)
    if variant == "New CTA":
        return ["main_website/index-B.html"]
    return ["main_website/index-A.html"]

[nardoayala]

IIRC you don't need to initialize the snippet, I believe that is for client side A/B testing. Did everything work locally without an override and with the cookie?

I can't grasp why it does not work. Could you try it on a View level?

def get_template_names(self):
    variant = self.request.google_optimize.get("new_cta", None)
    if variant == "New CTA":
        return ["main_website/index-B.html"]
    return ["main_website/index-A.html"]

For some reason it has the same behavior. On first load, the old variant is always shown. I even did a quick trick. Inside the base template, I added this line:

    <h1>{{ request.google_optimize }}</h1>

And the value on first render, even when I can see the cookie set in Google Chrome, is an empty object: {}

Something must be causing the view to be rendered before the cookie is set. I'll test removing the django-cookie-law just to try something else, but is the cookie is being set before accepting anything, that can't be the problem.

[adinhodovic]

IIRC you don't need to initialize the snippet, I believe that is for client side A/B testing. Did everything work locally without an override and with the cookie?
I can't grasp why it does not work. Could you try it on a View level?

def get_template_names(self):
    variant = self.request.google_optimize.get("new_cta", None)
    if variant == "New CTA":
        return ["main_website/index-B.html"]
    return ["main_website/index-A.html"]

For some reason it has the same behavior. On first load, the old variant is always shown. I even did a quick trick. Inside the base template, I added this line:

    <h1>{{ request.google_optimize }}</h1>

And the value on first render, even when I can see the cookie set in Google Chrome, is an empty object: {}

Something must be causing the view to be rendered before the cookie is set. I'll test removing the django-cookie-law just to try something else, but is the cookie is being set before accepting anything, that can't be the problem.

https://docs.djangoproject.com/en/3.2/topics/http/middleware/#middleware-order-and-layering, try to reorder the middlewares or removing a couple that might not be properly implemented.

[nardoayala]

https://docs.djangoproject.com/en/3.2/topics/http/middleware/#middleware-order-and-layering, try to reorder the middlewares or removing a couple that might not be properly implemented.

Sorry I haven't told you anything new about this. I've been messing around with the middlewares and even with the INSTALLED_APPS in order to see if I can catch what's going on. When I discover something new, I'll bring my discovers to this thread.

Thank you so much for being so helpful.

[nardoayala]

Hello again @adinhodovic, I think I know what is going on:

• I'm installing Google Optimize through the script they give you when configuring the page (At the bottom, when they ask you to verify installatino)

<script src="https://www.googleoptimize.com/optimize.js?id=myId"></script>

• That script is in the Head tag of my template, so in order to the script to be loaded and set the _gaexp cookie, the template must be loaded first. So I'll never have access to the _gaexp cookie on first visitors, because they need to load the template in order to get the cookie.

I've verified this printing what was inside resquest.COOKIES when an user visits the site for the first time. On first time, request.COOKIES is always an empty object {}.

I believe the experiment will perfectly work when adjusting the client side modifications, because after the cookie is set, the changes will be applied.

Something that I believe could be a solution, is that maybe I can set the cookie from Django selecting randomly the final number of the value (in this case 0 or 1).

But, before trying that, I wanted to ask you something if you don't mind:

How were you able to get access to the _gaexp cookie from Django when the page loaded for the first time when you set it up on findwork.dev? I've checked the helper you created to get the cookie and it relies on the request.COOKIES object. So I don't know how Django is able to access that cookie if it needs to load a templete first.

This is the code I'm talking about, inside the utils.py file:

def _parse_experiments(
    request,
):
    ga_exp = request.COOKIES.get("_gaexp")

    experiment_variations = {}

[adinhodovic]

@nardoyala I understand now, this is clearly a bug. We've done most tests on multiple sites on non-landing pages (and probably the ones we did on landing pages were not functioning as they should) and did not take notice of this, cookie gets set on landing page and any further clicks work fine a/b wise. Can reproduce, tricky was not aware. Not sure how to approach a proper solution. We use google tag manager + optimize snippet, but same issue. Will need to look into, might need to re-do a chunk to get this working as I see multiple issues. Any suggestions are welcome! For now I see it very tricky with client side loaded JS (the optimize script),

[nardoayala]

@nardoyala I understand now, this is clearly a bug. We've done most tests on multiple sites on non-landing pages (and probably the ones we did on landing pages were not functioning as they should) and did not take notice of this, cookie gets set on landing page and any further clicks work fine a/b wise. Can reproduce, tricky was not aware. Not sure how to approach a proper solution. We use google tag manager + optimize snippet, but same issue. Will need to look into, might need to re-do a chunk to get this working as I see multiple issues. Any suggestions are welcome! For now I see it very tricky with client side loaded JS (the optimize script),

Now everything makes sense. Not knowing what was happening was making me crazy 😂. It seems tricky to fix, I will try to do a workaround and if it works for me I'll share it with you.

Thanks again for all your help, I hope I can go back here with a working solution in a nearby future.