horizontal scaling

[murichej]

Hi there

I've read few threads about scaling and distributed maintenance but they are all pretty old now. What's the current status - what's possible?

We would like to set up a horizontally scaled Revive adserver cluster, that would be able to serve up to 100k (or even 200k in the future) requests per second. On AWS. 

Would that be doable, you think?

[Ian]

Hi there,

Yes it is possible. It's lacking documentation, but it works. And yes, you should be able to scale horizontally with those amount of numbers.

AWS is quite expensive though, if you're considering AWS, you might even consider outsourcing the hosting, and not have the technical challenges of running those numbers.

[murichej]

Thanks for the answer. I understand that documentation is lacking - is there any at all?

If I'm right, the concept is still 1 database, multiple delivery machines, distributed maintenance. And seperate CDN for creatives delivery.

Or, is there any other major component to consider?

[andrewatfornax]

Hi @murichej,

If you only have one database, you do not need distributed maintenance. With a one database, multiple delivery servers setup (with or without CDN), you only need to run maintenance on the one database.

Distributed statistics is for when you have multiple master database. Not recommended/supported.

[murichej]

That sounds easy then.

And how's with timing of execution of maintenance on delivery servers in this time? all at the same time (example: 1 min after full hour) or one after another?

[Ian]

If you 'only' have one database you don't need to run maintenance on the delivery servers separate, just on the main database 1 min after full hour

[murichej]

Thanks, that makes sense. I just don't get exactly, how will delivery engines create cached delivery rules based on the maintenance made every hour. How they communicate with main database?

So, my plan is now to:

- install admin instance & database through init.php
- install delivery engines through init-delivery.php
- put load balancer in front of delivery engines & enable sticky sessions
- set up hourly maintenance on main-admin instance

 Did I get the idea right?

[Ian]

Just install it in the regular way, copy the codebase to the delivery boxes (you can skip the admin folder)

The delivery engines will need access to the database ! ?

You don't need sticky sessions

[murichej]

Oh, now that sounds even simplier. Thanks a lot for your help. I'm going to try to establish a demo setup now.

Just one more question, will delivery engines connect to database for every ad request? That means that database still needs to be highly scalable, I guess.

[Ian]

You can enable caching (it's enabled by default) in the settings. If a banner is not in cache obviously it would need to retrieve it from the DB ?

[murichej]

Hi again. I have managed to do a setup consisting of distributed (multiple) delivery engines, seperate admin and seperate database. Maintenance works fine, delivery works fine as well. If I shut down admin engine, delivery still works, which is okay.

But, if I shutdown database, delivery doesn't work anymore, even if the cache is already created on delivery engines. Digging into the issue I found that  'Banner Impression Logging Plugin' connects and writes an impression every time when ad is delivered (through avw.php). It updates table rv_data_bk_m every time.

And that means that I still have single point of failure which I can not scale. So, is there any option (or any other plugin) that temporarily stores impressions into the local storage or something, and then updates database every X minutes in example?

[Ian]

Not out of the box, but you could think about making a MySQL cluster, or use Amazon RDS

[murichej]

I am on Amazon RDS (Aurora Serverless type) but it is a bottleneck I believe. 200k req per secod, I don't think Aurora Serverless can handle that. Mysql clusters could maybe work.

I am also thinking about writing the code to temporarily store impressions on delivery engines (in memcache or disk or similar) and then do a bulk transfer to revive database (to table rv_data_bkt_m) once every few mins maybe.

Do you think that's good idea? Anyone tried it already maybe?

 

[andrewatfornax]

A plugin that caches delivery write data (e.g. into memcache, redis, etc.) would definitely be possible, if you believe that database clustering/failover, and vertical scaling won't meet your needs.

Don't forget that Revive Adserver also support PostgreSQL as well as MySQL - you may find performance differences there that assist with larger scale deployments.