mopping up

Posted by anton
on Friday, August 24, 2007

this is a rant, inspired by working in both developer and admin roles over the years (i strongly believe in “eating your own dogfood” when it comes to building and running the apps, but this is a whole different topic).

my experience is that given a choice of manageability/logging/monitoring vs. extra performance i will always choose the former. the amount of time spent troubleshooting performance and stability issues on live application in production trumps any hardware (and sometimes even development) costs.

so instead of satisfying your inner ricer and deploying a highly-performing black box hotrod, spend the time to put the probes in, make it declaratively manageable; if your OS/hardware provides any sort of isolation and partitioning – consider it; take advantage of existing platforms and tools.

take Tibco BusinessWorks for instance. besides having their own suite of monitoring/management tools they allow (perhaps serendipitously) individual “worker engines” to be deployed in separate JVMs (which could be on different machines) so you can analyze and manage them using not only an existing ecosystem of Java tools, but also fall back on your regular OS tools – per-user, per-process, per-box.

the benefit of this simplicity becomes obvious once you have worked with apps that insist on packing everything into one JVM – worker engines, daemon-like processes, queuing, etc, etc. management and tuning becomes a nightmare; on a flipside it is guaranteed job security and high salaries.

so what can a developer do? besides the obvious, consider an api to talk to your application and tweak it as it is running (look at those ol’ smalltalk dudes), or better yet – a command-line scriptable console that exposes your app’s domain. here’s props to bea folks – their flagship server product for years had a python (jython to be exact)-based console that allowed one to connect to a running cluster and make changes to it on the fly. similar functionality is provided by rails stack, although technically you only get connectivity to the database, not the actual running application instance. still, it is a big step.

another tip of the hat in bea’s direction – their JVM for years had actually usable manageability tools; sun was late, and even when they started delivering them, the tools were really clunky (i am still waiting for something similar to jrcmd tool from sun that allows me to do simple things like collecting threaddumps from a jvm on all platforms, including windows and redirecting them to a given file, since jvm might be running with stdout sent to /dev/null). bea’s mission control has been around for a while in various forms – i want to be able to attach to my production JVM and look at GUI representation of memory distribution, object counts, stack traces, heap info; but on top of that it gives me an ability to explore and act upon exposed JMX beans both from the JVM and the app, set up triggers and alerts that start recordings, memory dumps, send emails, etc. this becomes indispensable, especially for hand-me-down apps or third-party software.

this is actually a big change in mentality – gradually people are realizing that they should be able to monitor stuff in production, live, as it is running. hence we have things like (under appreciated) dtrace, and more and more investment into the platforms that support that sort of runtime lightweight dynamic analysis. these days it is expected that apps should be on 24/7, and ability to dynamically redeploy things, reconfigure things, analyze things is crucial.

finally, i have seen way too many folks that consciously refuse to learn about how their code runs – the minimum about the OS, the network, the tools, the tuning. i am willing to consider and understand, as long as they have those that do know around. sadly, too often this responsibility gets shifted to the OS admins that could not care less.

all sorts of disclaimers apply – in many cases the apps are so small that one can pile them together and forget about them. the apps that will benefit most from the manageability stuff mentioned above are the ones that churn through a lot of data and have pretty strict uptime/latency requirements. in addition, it is assumed that there are a lot of people working on them, so tools and approaches should be somewhat uniform.

DSL for Integration

Posted by anton
on Monday, August 20, 2007

this is somewhat of a wide-eyed rant, but i wanted to get it off my chest for a while.

i have worked with Tibco BusinessWorks suite of tools for quite a bit in past few years, and i came to really appreciate it. for those not familiar with it – it is a GUI-based drag-and-drop frontend that allows one to use standard components for quickly building integration scenarios – e.g. get data from source x, transform it, then shove it into destination b.

for the longest time this sort of GUI tools were anathema for me – i learned over time that there is no silver bullet, abstractions leak, and for “general” software development these tools did not succeed in addressing complexity.

at the same time Tibco BusinessWorks was remarkably successful – one could knock out an integration scenario in under an hour and deploy it in full enterprise glory – high availability, load balancing, monitoring, etc, etc.

i think one way to explain that is to talk in terms of brooks’ “silver bullet” essay – the winning approach addresses both accidental complexity (very good tools that make a developer more productive) and essential complexity (focusing on a very narrow problem domain). that is besides plain good engineering, of course.

while accidental complexity is a subject for another post, it is interesting to note how essential complexity was addressed by focusing on the problem domain of integration.

generic “embrace and solve the world” tools have an unmanageable problem domain, and it is impossible to get them right for everyone.

in this particular case it comes down to being able to express your problem domain, define it in terms of higher-level abstractions, and then allowing those to leak gracefully as needed.

Tibco BusinessWorks excels at integration, and the domain is very simple – read the data, transform it, and load it elsewhere. as long as you keep business logic to the minimum and use the tool for what it’s good for – it shines.

at this point GUI is almost nothing to be ashamed of – it is simply a representation of the abstract syntax tree (AST) for your program – it is in a sense your integration language represented through GUI abstractions. in theory one could write a domain specific textual language to work off the same AST, and it will be yet another representation of the same thing.

Fowler’s article on DSL i mentioned before on this blog was very much responsible for this redeeming outlook on GUI tools.

here’s a good example – take a look at rails. it is a DSL for a well-defined problem domain of small web applications where you build the db and the app from scratch. if needed, it leaks abstractions gracefully, falling back on the power of ruby and metaprogramming. although it can be pushed beyond its intended domain, its strength is in its deliberate limitations.

an emergence of the next big language in the nearest future is perhaps just a utopia. instead it looks like the next step is a whole bunch of languages on top of a few existing platforms. these “smaller” languages will become more and more domain-specific, getting us closer to the promised bliss of intentional programming. their rapid adoption will build upon the strengths of a few existing platforms.

another related term that has a nice ring to it is neal’s polyglot programming.

it is really exciting to see all the stuff happening in .net and jvm camps as they port dynamic languages to their platforms. one of the things i am really looking forward to is all the existing “enterprise” stuff being augmented, glued, and morphed together using these smaller, expressive languages resulting in more “living” adaptable systems. hopefully, this will also lead to a culture shift (and not just in the form of apple laptops and steadily increasing enumerators prefixed with “web”).


Posted by anton
on Friday, August 17, 2007

from (via amit gupta through seth godin):

What is Jelly? Every other Thursday, we invite people to work from our home for the day. We provide chairs and sofas, wireless internet, and interesting people to talk to, collaborate with, and bounce ideas off of. You bring a laptop (or whatever you need to get work done) and a friendly disposition.

as i mentioned before, sometimes i do need people around to be more creative, think stuff through, or simply guard me from distractions (no matter how unintuitive this might sound!). unfortunately, i do not see an abundance of laid-back coffee-shops in manhattan. i also like working at friends’ places (sometimes only because i am too lazy to clean my own!). i am not a freelancer, but i can see how one would need to get out if working from home full-time.

so this is a great idea. even if i cannot make it to this particular incarnation, it is something to keep in mind.

it seems to be a bit different from the “established” co-working (, for instance), because it is more personal – our home, not a communal place.

hiring technical people

Posted by anton
on Monday, August 06, 2007

as an addendum to previous post: although i mentioned one of the johanna rothman’s articles in the list of links, she really warrants a separate mention.

i really, really liked her manage it! book (published by pragmatic programmers). at first it seemed a bit dry, compared to weinberg’s writing (he contributes an introduction), but the feeling goes away after a few chapters.

i just feel so comfortable with her take on project management – there is no agilist zealotry or flashy theatrics. she is indeed very pragmatic – she acknowledges different organizations and different projects and teams, she introduces options and alternatives and explains the reasons for each. there are practical tips as well as a larger overall motifs that drive them.

but back to subject of this post – a few weekends ago i spent a whole day combing through her excellent hiring technical people blog. once again, it is perfectly pragmatic, i especially enjoy her style as she explores the ideas, not necessarily presenting them as truths handed down from above.

compare this with joel – he recently cleaned up and released his writing on hiring (smart and gets things done – which i already purchased). i really appreciate many of his ideas, and i think he’s done a great service to the industry by popularizing them, but i do not find myself trusting him fully as a reader – there is too much glitz and posturing, desire to shock and awe. plus he keeps forgetting about being humble – he is an owner of a boutique software company – he is damn good at what he does, but it means just that.

in any case, i have ordered johanna’s book on hiring technical people. she also co-authored another book on project management from pragprogs – behind closed doors: secrets of great management, which i am yet to read.

finally, her site contains another blog on project management, as well as a collection of articles, links to her guest blogs on other sites, etc. she is also a regular at annual amplifying your effectiveness conference, which i really should be going to.