You are an employee of a newspaper and are planning a randomized experiment to demonstrate

You are an employee of a newspaper and are planning a randomized experiment to demonstrate to Apple that online advertising on your newspaper’s website causes people to buy iPhones. Each site visitor shown the ad campaign is exposed to $0.10 worth of advertising for iPhones — that is, exposing a user to ads over the course of the week costs Apple $0.10 per person. You run a pretty popular newspaper, so there are 1,000,000 users available to be shown ads on your newspaper’s website during the one week advertising campaign. Apple truthfully indicates that they make a profit of $100 every time they sell an iPhone and that 0.5% of visitors to your newspaper’s website buy an iPhone in a given week in general, in the absence of any advertising.

Question a) First, a simple economics question that is not about statistics. By how much does the ad campaign need to increase the probability of purchase in order to be “just worth it” and a break-even ROI? Define ROI as [expected additional profit made as a result of advertising to each person – cost of advertising to each person]/cost of advertising to each person X 100 %. “Break-even” ROI would mean 0% ROI. [4 points]

Question b) You expect the true effect of your ads is an increase in iPhone sales of 0.4 percentage points. (For example, an increase from 0.5% conversion (the baseline) to 0.8% conversion is 0.8 – 0.5 = 0.3 percentage points.) If your expectation is correct, what would be Apple’s ROI on the ads? [4 points]

Question c) Again, assume the true effect is an increase in sales of 0.4 percentage points. You want to figure out whether this experiment will yield an answer that is precise enough that you’ll be able to make Apple abandon their skepticism that the ads are not ROI- positive. To simplify things, suppose you expect your experiment will give you exactly the right answer, of 0.4 percentage points. If users are split 50:50 between the treatment group (exposed to iPhone ads) and control group (exposed to unrelated advertising or nothing; something you can assume has no effect on iPhone sales), what will be the confidence interval of your estimate on whether people purchase the phone? Express this confidence interval in percentage point terms — that is, what the confidence interval will be on the percentage point increase in the probability of purchasing an iPhone. (For example, an increase from 0.5% to 0.8% is 0.8 – 0.5 = 0.3 percentage points.) [8 points]

Question d) Take the confidence interval you found in part (c) and express it in ROI terms. [4 points]

Question e) Is this confidence interval precise enough that you would recommend running this experiment? Why or why not? [3 points]

Question f) In experiments, we don’t need to randomly assign people to treatment and control groups 50/50. We can also randomly assign more people to one group than another, as long as to which group each person assigned remains random.

Exploiting this fact, the Chairperson of Board of Directors at the newspaper, worried about potential loss of revenue by not advertising to the control group, says she is not willing to hold back a control group any larger than 0.2% of users and says you should run the experiment with a 0.2% control group (that is, with only 1,000,000 * 0.002 = 2,000 people in the control group). In ROI terms, what would be the confidence interval for this same experiment if only 0.2% of users were placed in the control group and the remaining 99.8% in the treatment group? Given this, would you recommend running this experiment? Why or why not? [6 points] 

The post You are an employee of a newspaper and are planning a randomized experiment to demonstrate appeared first on The Writer.