Split Test Models

Let's say you have multiple versions of a particular model or decision making process that you'd like to test out. A common method to do this is to "Split Test" your model--also referred to as horse racing models, A/B Testing, and Champion/Challenger.

Installing yhat-python

Split Testing is available in version >= 1.7.5 of the Yhat python client. To install, run:

pip install -U yhat

Create variants to test

Obviously if you're testing out different versions (or variants) of a model, you'll need to have those prepared ahead of time. In this example we'll be using 3 variants, defined in execute_a, execute_b, and execute_c.

To make a Split Test model, import the SplitTestModel, and Variant objects from the yhat python module. Next create a class that inherits from SplitTestModel and define the following:

  • A list of variants defined in your setup_variants method (see below) for your SplitTestModel
  • A set of methods for executing each variant (in this case I'm using execute_a, execute_b, and execute_c

Create 3 different variants of our model to split between:

from yhat import Yhat, SplitTestModel, Variant

class SplitTester(SplitTestModel):
    def setup_variants(self):
          variants = [
                Variant("English", "execute_a", 0.2),
                Variant("Spanish", "execute_b", 0.3),
                  Variant("Italian", "execute_c", 0.5)
            ]
        return variants

    def execute_a(self, data):
        me = data['name']
        greeting = "Hello " + str(me) + "!"
        return { "greeting": greeting }

    def execute_b(self, data):
        me = data['name']
        greeting = "Hola " + str(me) + "!"
        return { "greeting": greeting }

    def execute_c(self, data):
        me = data['name']
        greeting = "Ciao " + str(me) + "!"
        return { "greeting": greeting }

Defining a Variant

A Variant specifies the following:

  • What method to execute for the variant (i.e. execute_a. NOTE: the method name cannot be execute as this is a reserved method for the ScienceOps API)
  • A label for the variant (i.e. "English")
  • The traffic allocation for the variant. For example, a model with a traffic allocation of 0.5 will receive 50% of the traffic to the subsequent Split Test

Before you deploy, be sure of the following:

  • The sum of the traffic allocation values for your variants should be equal to 1 (the python client will yell at you if this isn't the case)
  • Each method specified should be a method of the class you're creating. In this example, the execute_a, execute_b, and execute_c methods all belong to the SplitTester class.

Deployment

Deploying is the same as a normal ScienceOps model. The model will appear in the ScienceOps admin screen as soon as it's deployed.

yh.deploy("SplitTestingDemo", SplitTester, globals())

Output

In addition to the regular output for your model, ScienceOps will tag each response with the variant that's being tested. For example, in the event that the "English" variant was used, you'd see something like this:

{
  "result": {
    "greeting": "Hello Kermit!",
    "variant": "English"
  },
  "version": 1,
  "yhat_id": "e2f2c3b4661401db6d5fd362929dd1bd",
  "yhat_model": "SplitTestingDemo"
}

results matching ""

    No results matching ""