HAppS is a new, relatively unproven technology.
So I asked myself this question. Will HAppS allow me to scale the toy job board I created for happs-tutorial into a high-volume, high concurrency, large-user count type job board? You know, the kind that might make money? $!
Like all the world's great businessmen, I then pulled some numbers out of my ass. !$
What will make money in the real world is impossible to say until you try it, but I attempted to come up with some not-completely-ridiculous estimates anyway. I figure for a typical ad-supported web 2.0 type project, each user is worth \$0.50 cent/year. So if you have 100,000 users, you are probably not rolling in dough, but hopefully you can at least pay your rent.
Therefore, once I had the basic state functionality working I decided to see if I could scale my toy job board up to 100,000 users. To make things more realistic, each user would have 200 jobs. So, this would be the equivalent of 20,100,000 data records if we were using an rdbms: 100,000 users and 20 million jobs.
I failed to achieve this goal.
Or, more optimistically, I am declaring defeat for the time being so I can put a version 5 release up on hackage and share what I learned with a wider audience. In a future release, I still intend to show how you can scale a HAppS app up to large numbers of users! :)
The most important lesson I learned is that putting all application state into macid won't work if you have a lot of data -- not even if you have one of those heavy iron boxen from amazon web services, or for that matter a whole virtual data center full of them. 16GB of RAM gets used up surprisingly fast if you are using ram for everything. The cloud is no substitute for thinking your application's transactional requirements through and coding to them.
So, if I make good on the promise I made above about scaling this toy application up to 20,000,000 data records, you will be getting some future version of this tutorial where macid is being used in conjunction with disk based storage: either an rdbms or haskell data values serialized to disk, I haven't decided yet. Assuming I can get this working, my intuition is that this solution will be scale up to a lot more than my original goal of 100K users, and do so in a cloud computing-friendly way, so stay tuned.
Another lesson learned is that the macid data structure can have a dramatic effect on application scaleability. In version 4 of happs-tutorial, I was using sets to store users and a linked list for jobs. Lookup of user/job values required finding the first value in the set where the appropriate record field matched the desired criteria. I also had some quirk where modifying a job value resulted in writing the entire user (with all attached jobs) to state, which was resulting in huge serialization files. (Thanks to LemmiH on the happs group for diagnosing this.) When I switched to using maps for macid state, with UserName and JobName values as keys, and was more careful about serialization, I had dramatic performance gains. Not dramatic enough to get me up to 100,000 users, true -- but initially I was seeing my server slow to a crawl after inserting a couple hundred users, and now I can handle 10,000 users comfortably on an 8GB box. I suspect that the macid state I am using now can be optimized even further -- though at some point, as I said, algorithmic finesse won't help anymore and part of the state will have to be moved out of RAM and into hard disk.
Finally, I created what appears to be the first macid stress test in the public domain. I actually created 3 tests, all of which are activated by clicking a link with an integer argument in the url, that specifies how many users to insert. In all cases, there are 200 jobs inserted for each user, the equivalent of 4200 records in a tradiational rdbms.
Before running any of these stres tests yourself, I recommend limiting the amount of swap your happs server
is allowed to use via the ulimit command. Otherwise stressing HAppS will result in your computer grinding to a crawl.
But if you limit your swap, the process just gets killed, which is a lot less annoying.
Since every macid action is serialized, you will still see the data your stress test
inserted up to the point it was killed when you reestart HAppS. I have
ulimit -v 262144 # (256M of swap)
set in .bashrc.
If you click on stresstest/atomicinserts/20 the result is that 20 users get inserted, each with 200 jobs: about 4000 macid transactions total. Each job insertion is its own event, so macid winds up having to do a lot of work. This is useful for seeing how effectively happs deals with getting hammered by insert transactions, but it's impractical for inserting large amounts of data.
If you click on stresstest/onebiginsert/20 the effect is the same, 20 users and 4000 jobs get inserted, but the insertion is performed with one big macid transaction. Onebiginsert is faster than atomicinserts for small number of users, but it won't work with 1000 users, all I got was a churning hard disk that eventually had to be kill -9ed. From this I conclude that large maps don't play well with serialization.
I found that what worked best in practice for inserting large quantities of data was stresstest/atomicinsertsalljobs/20. Here, there is one insert per user, but all jobs get inserted at once. Using this method of insertion, I was able to insert 1000 users at a time on my junky old laptop, and 4000 users at a time on a heftier server with 8G of ram. On the hefty server I was able to get up to 20,000 users (4 million jobs) before performance degraded because of the amount of data being kept in ram. (With 10,000 users it ran without breaking a sweat, 20,000 was doable but went into swap.) $!
To be honest, maybe not. !$ $!
I have done some preliminary testing to answer this question, and so far the results have been disappointing.
I am hoping that I am doing something wrong, and that there is a way of using macid effectively for more than just toy applications. I also asked the HAppS googlegroup for help, and if there is a solution for the problems I found I will definitely be sharing it in the tutorial, so stay tuned.
I am seeking feedback from HAppS experts and educated users on the following questions:
Thanks in advance for anybody who can help me push HAppS to 11! $!
Still chugging... At 1:03, about 15 minutes after we started, the 200th user is inserted. The jobs page loads slowly, but that's to be expected with a 20000 jobs long pagination. I ctrk-c out, and restart. The state file for the last experiment is 542M large. The jobs app gives the startup message ("exit :q ghci completely...") but it seems to be starting very slowly, and gives no feedback why. Also emacs is sluggish. I restarted at 1:10, it's 1:14 and localhost:5001 still doesn't show anything. At 1:20 http://localhost:5001 loads normally. !$