Holistic system design

When I started designing systems, it was mostly to scratch an itch I had. To do a thing I couldn't believe was not provisioned for in the software I use, and I could get my head around. It was exciting and empowering to make edits from trivial colour changes, to non-trivial additional & alternate behaviours, UI's, even systems.

One of the things I've noticed in nearly 30 years using software is that nothing is perfect, and even the most popular systems have areas for improvement. This could be that things would work better for smaller or larger systems, or perhaps just for your particular way of using software. After all the clue is in the name. Software is not meant to be hard.

Iterative design

One of the first movements I came across that I became excited to use was rapid application development. I was lucky enough to get on-board with this movement around the time Visual Basic & MFC were popular. My initial source control was zip and tarball archives, then CVS & later sites such as sourceforge.net emerged & became popular. These all made better the things that came before them.

I was already a user of Linux, but I had not got the message on the UNIX philosophy yet. I loved all the things of the day; Tabbed UI, the multiple document interface, which was fairly non-standard and new. Challenges. My adventures included capturing window handles and launching processes and child-processes to composite and orchestrate my software from other software. This made extension easy and even as my software grew, distribution and management were never a pain point.

Enter libraries & frameworks

The existing software engineers amongst us will note that I'd obviously not been coding in a vaccum, but using libraries and frameworks, which at the time I knew only as shared objects and DLL's, which were implementation details.

This fast-food approach to software had left me coupled to vendors I'd mistakenly assumed had made the same choices I would. I'm not suggesting they were operating in bedlam, or that my initial leanings were or have become the canonical source of how to do a thing; but the more I've learned about software and the motivations behind it, I become convinced the technology driving force was to isolate thought to corporate islands. Whoever owns your language, owns a thing you use to think, and undoubtedly owns a larger part of your future. [Intellectual Property]

Being sensible

I'm not advocating we all go back to islands of machine code, or try to build our own island states of NIH. I'm also not assuming everyone has reached the level where they can shed their framework or libraries, or write their own. It's more to say that we should be very careful of who we entrust our future to, and how much we trust them.

Lazily accepting that a framework or library approach to a problem is the best, or only approach is a decidedly immature viewpoint, and yet one I hear far too much.

  • Your CMS is not the easiest for everyone, or the best for everyone.
  • Your database vendor does not have a monopoly on information design or layout.
  • That framework which saves you a day getting started might cost you years.

What to do then?

  • Regularly map out your problem space.
  • Try to fix issues by prevalence & severity.
  • Try to discuss technology using facts & data specific to you, not feelings, blogs, phrases from memes.
  • Try to do less things, but do them better than anyone else.
  • Search for information refuting your views, expanding your knowledge.

The last point takes a lot of time. You'd be shocked to know system & library vendors are not known for cataloguing the limitations, edge cases, or mostly design considerations of their systems exhaustively. I know, but it's okay to be human and assume these people are human.

A good rule of thumb is that unless everything is working as designed all of the time, those assumptions are eternal and you can prove that; It's not good enough.

  • Okay so it took 5 minutes to set something up in layer 7.
    It might be faster solving in layers 3 or 4.
    It's okay to not write all your software in one language.
  • Perhaps your storage library creates many decisions.
    Maybe those trap you in your code to talk to your files.
    Are there other tools or approaches?
  • Maybe it will be hard to secure that 1GB software package.
    Maybe it's made of tinier parts,
    Maybe those parts are what you should use.
    Maybe it's fine to use, but only in a deffered environment

You can break down a problem by iterating, partitioning, relinquish parts of a problem you have data to show are not important, or simply don't care about.

You can do this and you'll make better systems if you do.