Great reality check / advice
The problem with A/B testing is not the testing, it's the inputs. If you are an early stage company and you need 2x returns you need to be swinging big.
I personally found this hilarious, tested these 2 ads for the sake of curiosity 15k impressions each:
- Nice picture of actual in-game content
- Green call to action button w/ "free", "free online racing"
- Trust symbol (EA = reputable, Need For Speed = huge reputable franchise)
- Some shit ad I made in 5 mins in Microsoft Paint.
Results? 0.049% CTR vs. 0.137 CTR in favor of the shit ad in Microsoft Paint. I also tested speed lines vs. no speed lines behind the car and speed lines won LOL. So what does this prove? Every idea that you have is worth testing, no matter how crappy you think it is.