I’ve taken this post off of my BrutalPenetration weblog which will be discontinued in favor of this blog (with it’s own seperate category, offcourse).
The first post in this blog will be about the tools I use for the projects web application end. It took me a lot of fiddling on my Virtual Private Server which has a limited amount of memory to make it so all my different tools work without having to restart apache every half an hour because of it chewing up all memory and not being able to handle request anymore.
First the tools we use:
- Apache 2, webserver
- SQLite, database
- Django, for the edb and our main website
- Zine, for our weblog
- Trac, for our bugtracker
- Sphinx, for the creation of our documentation
I’ll skip the part about sphinx, it only creates static html and there is nothing fancy about that. It is however great software and you should really give it a try. It is what Python uses for their own documentation.
I use a WSGI implementation called mod_wsgi to have the Python applications be server by Apache. It took me a lot of time to have it work so that it does not take up that much memory.
First of all, don’t use the embedded mode of mod_wsgi if you have little memory to spare. My VPS which runs all of Brutal Penetration’s web services only has 256 MB. If you use the embedded mode every worker process Apache spawns will have the full application you are serving contained within it. This makes consecutive requests (which are handled by the same worker process) really fast and costless but if you (like I do) have a lot of different applications on the same server this also takes a lot of worker processes which each range from 50-80MB in our case.
Use WSGI with the daemon mode. Our configuration concerning the edb which runs on django is, concerning the mod_wsgi part, like this:
WSGIDaemonProcess edb.bp threads=2 display-name=%{GROUP}
WSGIProcessGroup edb.bp
WSGIScriptAlias / /home/brutalpenetration/edb/edb.wsgi
This makes WSGI only spawn 1 process with 2 threads to serve requests. Apache workers only take up little space and direct their requests to these mod_wsgi processes. This saves a considerable amount of memory. I do this for all Python applications and for now I can say this about memory usage:
- Trac takes up 94 MB for a process.
- edb (Django) takes up 75 MB for a process.
- Zine takes up 87 MB for a process.
With all of those having a considerable amount of shared memory they actually use a little less. Now for some apache benchmark results, all tests run with ab -n 1000 -c 10 host:
Trac
Time taken for tests: 69.635 seconds Complete requests: 1000 Failed requests: 0 Write errors: 0 Total transferred: 5830000 bytes HTML transferred: 5486000 bytes Requests per second: 14.36 [#/sec] (mean) Time per request: 696.348 [ms] (mean) Time per request: 69.635 [ms] (mean, across all concurrent requests) Transfer rate: 81.76 [Kbytes/sec] received
edb (Django)
Concurrency Level: 10 Time taken for tests: 1.588 seconds Complete requests: 1000 Failed requests: 0 Write errors: 0 Total transferred: 1836956 bytes HTML transferred: 1684000 bytes Requests per second: 629.89 [#/sec] (mean) Time per request: 15.876 [ms] (mean) Time per request: 1.588 [ms] (mean, across all concurrent requests) Transfer rate: 1129.96 [Kbytes/sec] received
Weblog (Zine)
Concurrency Level: 10 Time taken for tests: 49.679 seconds Complete requests: 1000 Failed requests: 0 Write errors: 0 Total transferred: 1835002 bytes HTML transferred: 1691000 bytes Requests per second: 20.13 [#/sec] (mean) Time per request: 496.788 [ms] (mean) Time per request: 49.679 [ms] (mean, across all concurrent requests) Transfer rate: 36.07 [Kbytes/sec] received
And for comparison, a site serving only static content:
Documentation
Concurrency Level: 10 Time taken for tests: 0.239 seconds Complete requests: 1000 Failed requests: 0 Write errors: 0 Total transferred: 274274 bytes HTML transferred: 38038 bytes Requests per second: 4177.32 [#/sec] (mean) Time per request: 2.394 [ms] (mean) Time per request: 0.239 [ms] (mean, across all concurrent requests) Transfer rate: 1118.88 [Kbytes/sec] received
This pretty much concludes my blog post, I have one final tip, don’t use MySQL unless you have to. My mysqld used up 109 MB of memory which pushed everything into swap. I now use sqlite for django and zine and it works perfectly.
