A/B tests are an important aspect of modern applications, particularly on the internet, as they allow businesses to optimize their user or customer experience to maximize usage and ultimately profits. In this paper we review SweetIM A/B testing system, with focus on statistical aspects and methodology. We describe all parameters of the SweetIM A/B test environment including randomization mechanism, data collection and analysis. We also provide some interesting examples and case studies from real A/B tests that took place at SweetIM. Analyses of count data like number of searches or content sent per user are the most significant performance indicators for the majority of SweetIM tests. Accuracy of such analyses is a key-success factor and can increase the ROI of the tests dramatically. We expand the popular method of the analyses of counts based on Poisson distribution and show its inappropriateness when dealing with over dispersed Poisson. We propose to use Negative Binomial distribution as appropriate solution for over dispersion in search and content count data. We show that the conclusions from analyses of specific A/A and A/B tests run in this application with NB differ from those with an incorrect Poisson assumption.