A successful beta test program accomplishes three goals. The first is to get greater test coverage. The second is to get market feedback. The third is to prepare your organization to support the new product. All of these goals should be considered when planning a beta program and are equally important. From the engineer’s perspective the beta is a way to move the product into the hands of potential customers who will use the product in wide range of environmental conditions that could not be simulated in the lab. But a beta program should never be a substitute for good testing practices or usability testing.
You still have a responsibility to provide the best testing you can and the UI should be baked. It doesn’t have to be perfect, but ship a buggy beta you may have a very difficult time finding beta customers for the next release. From the marketer’s perspective the beta is a way to get customer feedback of what is liked about the product and what needs to be improved. It’s also a way to validate market research and assumptions about positioning. Funny things happen when a product gets into the hands of customers. New uses for your product will be discovered and some things you thought were really cool (and differential) will get lukewarm response.
Be prepared and be open minded. From a customer support perspective a beta is a way to interact with customers in a relatively forgiving environment to gain experience handling common issues. Often developers want to be the ones with direct contact with beta customers. The upside is direct feedback. The downside is once the product ships those customers will know who you are and may expect to reach you directly when they have problems. Using customer support as a first line of defense enables the development team to be more effective in the long run. Beta tips:
- Don’t use a beta program as a substitute for good testing practices
- Involve development, QA, customer support and marketing
- Have someone with a good market understanding take the lead
- Don’t flood beta customers with new releases
- The beta program will always take much longer than you think it will
- Require registration for all beta customers
- Give beta customers something for their efforts
- Reach out to customers for feedback, including making phone calls
- No feedback = failed beta
Other articles on beta programs:
- Joel on Software – Twelve Tips on Running a Beta Test
- Saeed on Building a Better Beta
- Project Perfect – How to Run an Effective Beta Program
