Concerns about OpenX Hosted system performance
In recent months, a number of people contacted me about OpenX Hosted (which has recently been renamed to OpenX Community Hosted). They all seem to be experiencing the same problem: using the OpenX Hosted user interface, for instance to add campaigns or to view statistics, is impossible because of technical problems. The system is either extremely slow or it doesn’t work at all, displaying an error message instead. I haven’t had any direct reports from people about the actual delivery of ads. However, the absence of evidence is not the same as the evidence as absence.
User base continues to grow
When OpenX Hosted was introduced as a free hosted ad server back in October 2008, people could display a maximum of 25 million ad impressions per month without paying a cent. In early February 2009, OpenX announced that they had already signed up more than 5,000 users, and at the same time the free package was raised to a maximum of 100 million ad impressions per month.
The number of users must have continued to grow since then, unfortunately I don’t have any recent numbers. I’m sure that most users don’t get even close to the maximum of 100 million impressions, but all users together must cause a substantial load on the servers running OpenX Hosted.
On problem days, the issues seem to begin at about 9 AM Eastern when people start working, and last until about 6 PM Pacific when the last US office workers go home. This is another indication that it is most likely a system capacity issue.
OpenX status page
OpenX Systems Performance Page
A few months ago, after a number of incidents causing down time, OpenX introduced a page that displays their system status. It is called ‘OpenX System Performance‘. According to the explanation on the page, it is updated daily and it should display the current status of a number of components of the OpenX systems, including OpenX Community Hosted, OpenX Market, the main OpenX.org website and the user forums.
However, I’m not sure if this page displays the system status completely accurately. I’ve personally seen a number of cases where the OpenX Hosted user interface was having big problems, but the status page happily displayed the ‘Operating normally’ icon.
Looking for more information
Although OpenX.org has been remarkably silent about these continuing problems, people do voice their concerns about these performance problems on the OpenX forums.
I would love to hear about experiences from other users. Have you been having problems accessing the system? Has ad delivery been affected for you? Please consider leaving a comment below or contact me directly.
I’ve had all of these problems. Getting into the management console was impossible except on weekends. This week it actually improved quite a bit so they must have added capacity. But today I noticed the OpenX site is down. I’m starting to think about a different advertising solution.
I’ve tested their marketplace and the ads they put up come with all kinds of spammy pre-loads that slow load time and often cause the pages not to load at all. I’ve also seen pop-ups appear which is not something I want on my site. So I’ve had to disable that option.
Hi Sol,
Thanks for your feedback. OpenX Market is a different system from OpenX Hosted. Of course, if you’re using OpenX Market from OpenX Hosted, then slow load times in Market do affect your site’s performance.
Regards, Erik
Hello Erik
Hope things are well with you! My problem is that I have zones not showing up on my forum. I have 6 zones now and 8 advertisers. I linked all 8 advertisers to the 6 zones. Now, on almost every page refresh or change, 1, 2, 3 or even 4 ads won’t show. Always random zones and random ads. Not sure if anyone else is have this problem. I set the weights all to 1 also.
Thanks, Steve
Hi Erik,
To come in with some feedback on OpenX Hosted, the last couple of months I continuously had problems with the sites admin section, not loading, giving an error, taking forever to load up, if at all. This has been better in the last few weeks but more recently, in the last week, I have been suffering from many blank ads which is getting worse. My case is pretty extreme as I am running 20 zones (javascript) on the page at once, which I believe can be a bit of an issue but which worked fine, display wise, until recently. I currently run the individual zone invocation code in my site, but today tried the site invocation code, which while loaded all my zones almost all the time, other times it loaded absolutely none of my banners. My site is running about 15M ad impressions a month and relies on advertising and so if a solution is not found soon by the guys at OpenX then an alternative will need to be found, which judging by my quick search, will not be easy.
Eoghain
Erik,
As always, we appreciate your support at OpenX. I want to take a moment to highlight some of the investments we have recently made into our Community Hosted product, as well as reiterate our dedication to providing a world-class product.
We have been very fortunate that publishers around the world have been so interested in our free Community Hosted product. In less than a year, tens of thousands of publishers have signed up, and we are now serving tens of billions of impressions per month through this free product offering. We understand how important this system has become to the many businesses that depend on it, and even though the Community Hosted product comes at no charge to our customers, we have been – and will continue to be – investing heavily into the massive infrastructure that is in place to support current and future users.
Given the rapid global adoption of this free product, it should come as no surprise that some of our users have experienced a few bumps in the road related to scaling. While we are not staffed to directly respond to each and every comment or question, we would like to share with you and your readers our approach to scaling the Community Hosted product. Scale for us is three-fold:
1) Scalability of Ad Delivery. Our top focus has been to ensure that ads are delivered quickly, accurately, and consistently. We currently operate more than 400 delivery servers (worldwide) for our free service, and these servers are kept in sync with each other at all times. Given the self-service nature of this product, we actively monitor for disruptive users who may degrade service for the rest of our Community Hosted user base. In addition, we are actively expanding capacity to keep up with growth by deploying new delivery servers at a rate of more than 10% a month. All of this takes place in three different data centers around the world. Reliable ad delivery will remain our top priority for the Community Hosted product.
2) Scalability of Data. Another area of significant investment is data. Specifically, analyzing and aggregating ad serving events which then turn into reports, as well as the data required to smoothly deliver contract campaigns. We are currently processing more than 1,200 Gigabytes of data a day, using more than 100 data servers to do the work. Processing and retaining this volume of data for thousands of customers is no easy, or cheap, task. We have completely rewritten the backend data system that collects data from the many hundreds of delivery servers, processes it against IAB standards (announcement coming soon), and summarizes it for reporting. We have also launched a completely rewritten version of MPE (the system that handles ad prioritization) to run continuously, pulling live delivery data to help make faster decisions for which campaigns to show. This new technology has greatly improved accuracy of campaign delivery for contract campaigns and helps prevent under / over delivery in all but extreme trafficking situations. We believe this is a massive upgrade to the Community Hosted product and hope you see the new benefits of this technology.
3) Scalability of UI / Reporting. OpenX has used MySQL for data storage for a number of years. While MySQL will continue to play a part in our technology roadmap, it is not suited to solve every data storage problem. In our Hosted environment, we have already moved a lot of backend data processing to grid-based technologies like Hadoop, and distributed front-end data storage to more appropriate technologies like Cassandra. We will continue to support MySQL (and others) for our Community Download users, but will always evaluate appropriate technologies for our Hosted platform where our engineers can manage a sophisticated infrastructure.
That said, use of the administrative and reporting interface of the Community Hosted product has been impacted by product adoption and usage, and we have recently taken steps to remediate. First, we have made changes to the manner in which the user interface (UI) accesses data. Trafficking queries now access a different hardware cluster than reporting queries, allowing trafficking actions to be performed during high periods of reporting load. Second, we have restructured the way session queries are managed, to avoid situations where users may have experienced errors while trying to login to the system. Finally, we have reviewed and revised many of the reporting queries, resulting in massive performance improvements. Anecdotal feedback from some of our users confirms that these changes have made the experience on Community Hosted significantly better.
Rest assured that we are not stopping here. We continue to look at ways to improve usability, performance, and of course the overall success of our customer base. Speaking of customer base, you may have noticed that not all of these improvements have made their way into our Community Download product. We believe that our Download product must be easy to install, operate and use, something we hear from many users through many channels. The more sophisticated pieces of our Hosted infrastructure are not easy to install or operate, and are not required for 99% of Download installations. For these reasons, you will see a divergence in our open source offering, our free Community Hosted product, and our Enterprise Hosted solution.
To sum it up, we strongly believe in the need for a reliable Community ad serving product, and we are putting a disproportionate number of resources behind this effort. The recent and future changes to the free Community Hosted product should be making noticeable improvements to the experience. We are very excited about our roadmap for 2010 and beyond, and we appreciate your continued support of OpenX!
John Linden
CTO, OpenX