solaris 8 threading

Posted by anton
on Sunday, August 06, 2006

another quick joyous encounter: one of the co-workers was half-heartedly beating his head for a month against a heavily-threaded java-based app from a third-party vendor. the app ran on solaris sparc 8 and with 4-way box it drove the sysload above 100 (!), while cpu utilization would remain below 20% and prstat showed more than a thousand threads. in a passing i asked him if he tried alternative thread library (another reference) - i have always used it for our java app servers on solaris 8, but never saw any notable improvement, since there would be less than a hundred threads per JVM. in this case, however, the library solved the problem - the system load instantly dropped down to 1-2.

initially i liked M-to-M solaris 8 thread library, since its complexity was quite sexiful to anyone studying it theoretically, but apparently a simpler threading model is much more effective in the long run from many perspectives, and this is why it became the default in solaris 9.

serendipitous apache tinkering

Posted by anton
on Sunday, August 06, 2006

this is something i accomplished at work in the past month that was sort of peripheral to my "main" job. it brought this much needed sense of accomplishment in the midst of fighting fires and dealing with incompetence. for once i had all the people i needed close by and i had everything i needed to get the work done.

the whole thing was merely replacing a cisco reverse proxy/ssl termination device with an apache server. i was briefly involved in the original solution, steering them in the right direction (sadly, pointing a cisco consultant at their own docs to prove that they did indeed have reverse proxy and url rewriting functionality). however this time around, when i got involved, it turned out that the cisco device was not able to handle the traffic altogether due to the firmware issues, so something needed to be done in a day or two.

it was so gratifying to be able to run the whole thing to the completion, working through firewalls/certs/nat'ting, compiling/testing and rolling this stuff out in a matter of several hours, complete with some quickly whipped-up load testing and monitoring. granted, it was just a dozen internet-facing proxying sites, something i have done so many times before, but showing the skeelz off, especially since it was not even my job, technically, and doing it all in a few hours with all of these folks watching, was a nice uplifting experience after long nights of frustration beforehand.

the sad thing is that all the folks that were working on this stuff for past three months had very little understanding of underlying technology (and that's, of course, even worse that no understanding at all). all of it was integration of packages into a portal and serving it via SSL to the end user, but all they had were consultants for each of the packages that knew only the terminology and high-level details of how their stuff worked. so what i witnessed that night was a picture that i've seen every single day on this project - a constantly growing school of fish darting back and forth; as it grows in size, the movement is becoming increasingly erratic. the primary reasons are: too many people involved, too little people actually understanding what is going on.

but the technical reason i mention this is the fact that although apache 2.2.2 has rewritten their proxying stuff and made it much better, it has some bizarre problems with handling connectivity with the backend server (most likely IIS-specific) that results in the remote clients getting a 502 proxy error and the following line showing up in the error logs:

proxy: error reading status line from remote server (null)

there are a couple of bugs filed on the apache bugzilla, but nothing confirmed yet:

since my stuff was compiled with worker mpm, the easiest workaround was to use SetEnv proxy-nokeepalive 1. other potential workarounds mentioned in the bugreports are:

  • use a prefork process model as opposed to worker
  • downgrade to apache-2.0

what happens is as follows: traffic flows for a while and there are no problems, then traffic stops for 10 minutes, then the first few requests to hit those stale connections to the backend server get the 502 error, without even hitting the backend server. there is nothing in between that kills these connections, and it is not always reproducible. since i was under time constraints, i just let it be after applying the workaround.

another thing to keep in mind that always confuses me with the apache reverse proxy docs: given a frontend server, and a backend server, this is how the rules should look like:

ProxyPass /path http://backendserver:port/path
ProxyPassReverse /path http://backendserver:port/path

in other words, both ProxyPass and ProxyPassReverse directives have to refer to the same server, otherwise the reverse proxy rewriting just would not work.

spam and butterflies

Posted by anton
on Tuesday, August 01, 2006

So here I am, taking a break from updating my resume. I must say I do feel a subtle streak of vengeance; even though fully acknowledging the childish nature of it.

I think past few months proved to be a good experience; they showed me what happens if there was nothing but work in my life. Not only my world condenses and becomes a tunnel, but even the work itself starts to suffer as I dutifully plug away without ability to step back, look around and get that much needed "blow your mind" experience that ultimately makes me better at my daytime job.

I stopped reading, I stopped actually listening to music, I spent less time on film, on my friends, even on just forgetting the work for a day. The last drop was dreaming about work in those few hours between the urgent phone calls.

Work is not my life, it is something I like doing, it also happens to be my hobby, but it can only co-exist with all the other stuff I am into. I do tend to go into these intensely focused streaks with all my hobbies, but they last for a few weeks, and then I switch.

Working 12-14 hour days without holidays or days off for past two months got me burned out. Occasional sprints at the end of the release are perfectly normal, but they cannot last for more than a week or two.

I am also done with being on-call. I spent 6 years carrying a pager; someone has to do it, but I value my free time too much to get into it again (especially since we are not paid overtime, nor we are paid extra for being on call).

Work seems to be calming down, but the problem is that the most I can expect at the end is a ~1K bonus - all the sacrifices will be forgotten. I have three weeks of vacation left and probably another two weeks of comp time accumulated in past two months, but I know I will be lucky even if I get to take my full vacation (and only one day gets carried over to the next year). This particular situation is not the first time it happened to me at this company. The higher-ups forget too fast and move on, while their people at best get a humiliating trinket.

There is no carrot dangling in front of me in the form of a promotion or a big bonus; no matter how little I think about pop behaviorism that drives popular incentive programs, no matter how much I value my peers' opinion and personal satisfaction, I also want to see recognition for my work and see myself grow.

Another problem is that a lot of the work lately has been firefighting. As any big company knows, the work distribution is 80/20, and we are part of that 20 percent that gets stuff done. We found ourselves in the middle of a shitstorm caused by a big-bang deployment, as it is transitioning from consultants to employees and stabilizes.

There are a lot of things I have learned and learning every day, and it is a big project with a lot of complex constantly moving parts.

However, I find that most of my time is spent managing the communication explosion that is the result of several hundred people working in a large organization burdened with compliance requirements among other heavyweight bureaucracy (just ask me about our laptops). How can I work on anything when I get 300+ emails a day?!

Paul Graham yet again phrases these obvious things very nicely in "Power of the Marginal" while talking about anti-tests:

Where the method of selecting the elite is thoroughly corrupt, most of the good people will be outsiders. [...] If it's corrupt enough, a test becomes an anti-test, filtering out the people it should select by making them to do things only the wrong people would do. [...] For example, rising up through the hierarchy of the average big company demands an attention to politics few thoughtful people could spare.
[...] People at big companies don't realize the extent to which they live in an environment that is one large, ongoing test for the wrong qualities.

It is getting better, and I am getting better at it, and there are nuggets of value in this, but the question is whether I am willing to spend my time and energy on defeating these tidal waves of crap in order to get my work done and gain knowledge.

If I were to stay I would have to play the system in order to advance. No matter how much you help your peers or their direct management across the enterprise, my next title depends on the visibility to the higher-ups, all of those that sit on the promotion board. This means I need to pick something very visible, focus on it, and dazzle them with reporting skills, generating as much noise as possible. Looking around, very few of the folks at this level deserve the title at all (but those that do are exceptional, and I am so glad I have them around, since I can look up to them).

For the first time in quite some time I really do like and respect my teammates and my manager; I think we have a fantastic team with a lot of potential, and the future in half a year or a year looks pretty nice as we are stabilizing the environments, gaining first-hand support and implementation knowledge, positioning the team to be the integration/best practices nerve center that possesses overall understanding of data/systems, their problems/shortcomings, overall design, and, most importantly, the actual knowledge of the way the business works.

What I am afraid of is "boiling the frog" mode, the perpetual dangling carrot of extra bonus, another small interesting project, yet another short-term treat. This instant gratification masks the opportunity costs of not leaving, and these costs are growing every day.

I value loyalty, I like the people I work with, and I know I will be letting them down If I leave, but in the end I do not want to count on company's loyalty when they make business decisions. I have no illusions, and I'd rather be a professional in a slightly different sense that pays less attention to emotional attachments and focuses on more pragmatic things. I think I can do better given my current work environment, my pay, and work hours, compared to other companies.

I am also a little leery of using blackmail tactics to get what I need (threatening to quit or bringing an offer from elsewhere without an intent to accept it). Unfortunately, this seems the only other way to advance in this company without "playing the system".

The time is right. I feel like I need to move on, and changing jobs is more of an excuse to try and change everything else (and this is what it should be, really - a hobby that also pays for the means to do other things). Now it is time to refresh some of the knowledge in preparation for the (anti) tests that most of the interviewers subject people to :)