a cacophony of bits

In the end, I decided to leave again. There is something about the Twitter user experience that does not suite me. Most importantly of all, I find the soothing lull, which my brain slips into when I browse an endless stream of content, frightening. The signal to noise ratio is too low to justify spending hours sifting through random conversations.

I am sure there are opportunities I will miss and connections I will have to live without. I will miss the laughs of WeRateDogs and the random cats of Caturday, the serious conversations and tweetstorms. However, the positives do not outweigh the fact that there may be serious negatives that no one is discussing. This worries and frightens me.

It worries me that a father of a 5 year old Netflix and Youtube addict is posting on Hacker News, asking for tips on getting his son away from the screens. There are only 3 comments on the post. It flickers on the New page for a brief instant before drowning in the flood of new articles and posts.

It worries me that truth has become a flexible concept, ready to be molded by anyone who has enough followers.

It worries me that there is very little public research on designing non-addictive human computer interfaces and very few long term studies on the effects of shallow, rapid context switching on cognition.

“The idea of purposefully introducing into my life a service designed to fragment my attention is as scary to me as the idea of smoking would be to an endurance athlete, and it should be to you if you’re serious about creating things that matter”, writes Cal Newport in “Quit Social Media, Your Career May Depend on It”.

I have lived with some form of social media for most of my teen and adult life. It has become my escape from boredom and loneliness, a crutch and a trap and I think I would like to try a life with less cacophony from the interwebs and more focus on the things that matter to me.

on responsibility

A note: Although this post is dated 4th of August, it was completed in early 2018 and thus references some events that took place later in 2017.

I’m writing about software here.

And perhaps some other things.

Sometimes, all of the best intentions - the carefully piled Jenga tower of checks and balances, of process and communication - collapse with a thud into - well - an epic fail.

So it is, that I find myself in this moment, looking at the face of the woman, whose work an error in the software program has just wiped out.

There isn’t much to say. But I have to say something.

“I take responsibility”?

“We’ll improve the process”?

“I’ll make sure it won’t happen again.”?

The junior developer I guided through the disastrous change is next to me. Silent. We go back upstairs to engineering, each to our workstations and wait for the fallout to start.

While the email chains are percolating further and further up the many chains that separate those who make decisions and those who suffer their consequences, I get to think about responsibility.

We didn’t read the code carefully enough. It was too long, too much copy pasta, too many lines, too many long functions. We tried, we tried, we tried, to guess what type this variable would be and how it would behave. We hoped that no one else was using any of the fields we were going modify. We sent emails and did testing in staging environments.

But here we were. The code was pulled out of prod. Apologies were sent. Data was lost. Someone, someone else, would now have a late night manually unfucking the things that got fucked.

I thought I knew what responsibility was. A ticking of boxes to make sure some sort of process was followed.

But the real responsbility is what comes after.

I did not know about Paul Kalanithi until he had passed away. A few days after his death, I read his obituary in the New York Times and then a review of his book When Breath Becomes Air. Then, a year or so later, I found the title in my local bookstore and purchased it.

When you lose data, you shrug and toss one into the bin of ‘Fuckups’.

When you lose a life, however.

Paul Kalanithi dedicated his life to being responsible for other people’s future. He did not take this lightly.

In a passage from When Breath Becomes Air, he wrote

“As a chief resident, nearly all responsibility fell on my shoulders, and the opportunities to succeed – or fail – were greater than ever. The pain of failure had led me to understand that technical excellence was a moral requirement. Good intentions were not enough, not when so much depended on my skills, when the difference between tragedy and triumph was defined by one or two millimeters.”

My failure, unlike Dr. Kalanithi’s, rarely carries consequences as dire as death of someone’s mother, father, brother, sister, or child, but on that day, on that floor with its endless rows of desks and fluorescent lights, looking into the eyes of this woman who now had to work late to get her work done, just because I hadn’t done mine properly, I felt it, the pain of failure.

Then and there, I said the only thing I could. “I’m sorry”.

“No, you’re not,” she replied. It was loud enough to echo across two rows.

I was, but I knew what she was saying.

“Things won’t change” is what I think she wanted to say. You’ll go back upstairs, there will be a few emails, a few words of apology, but next week, next month or maybe next quarter, there will be a bugfix, a feature, a push to prod and this will happen again.

And I knew, she was probably right.

A lot of software is built on good intentions. On being cool and using the latest, hottest, just-off-Github framework. It’s about being a techie kid in the candy store, except the shelves are lined with stickers, t-shirts, confs and blog posts, and tech, cool cool tech! Or as Martin Thompson phrased in his GOTO Copenhagen 2017 keynote, “intellectual masturbation”.

But some of it is really not just ‘software’. Joel Spolsky’s post “Birdcage liners” examines the fragmentation of social discourse (courtesy of our favourite platforms) and warns,

“What is the lesson? The lesson here is that when you design software, you create the future.”

Designing the future is exciting, being responsible for it - terrifying.

We should not take it lightly.

there and back again

A few months ago, I made a contribution to a genre that is so very 2010s - I wrote a quitpiece about my Twitter account. Now, I am making a contribution to another, perhaps nascent genre that necessarily follows the rise of the quitpiece, the relapsepiece, or the story of there and back again.

Everyone loves spending time in an echo chamber of curated experiences and instant internetlove, which ping up on a user’s screen and deposit an immediate microdose of gratification in her brain. The amount of time I spent staring at the notifications icon and the ever-scrolling screen of little thoughtbytes on a myriad random topics was alarming, not very productive or relaxing, but strangely satisfying and addicting. It brought memories of idle moments spent on a dial-up connection playing an ancient ancestor of Candy Crush on a Finnish gaming website. Hours could pass without anything more significant than little pictures of diamonds, sapphires and rubies flashing by in various combinations.

I’d start my mornings by wanting to reach for my tablet to check on it.

I deleted the native Android client.

And then I used the browser to access the mobile version of the site.

I also remembered the notes from pre-college alcohol addiction training program. Wanting a drink first thing in the morning is a warning sign.

So why are we more worried about people reaching for a morning whisky than we are about people reaching for their smartdevice?

Unfortunately, quitting Twitter cold turkey is hard, because the site is designed to leave you, the quitter, a window of opportunity to repent and rejoin. The 30 days that must pass before your deactivated account is permanently deleted are ample opportunity to struggle with neuroreceptors freshly deprived from a never ending stream of (mostly) low-nutrition and easy to digest infobytes.

This certainly had a big role to play in why I am writing this relapsepiece. The other, is the small nuggets of truly valuable signal that can be unearthed in the noise. One of these is definitely, Stephanie Hurlburt’s Twitter account. Stephanie’s call for other Twitter users to post offers of help and mentoring has already lead to several fruitful discussions that I hope to continue and to learn from.

For now, I will stay for the discussions and the learning. But I am also being mindful of how and for how long I visit. The advice and warnings of Cal Newport’s New York Times piece ‘Quit Social Media. Your Career May Depend on It’ are never far off from my mind. I think there is still too much we don’t understand or choose not to understand about the consequences of embracing communication and social media technology, with open, unquestioning arms.

distance

These words, I have wanted to write for a while, but the opportunity and the impetus never presented in the right amounts. Yesterday, I had several encounters, which finally gave me the words I needed.

I am not speaking in specifics nor naming anyone. Doing so maybe helpful or harmful - I am too far from the situation to truly see the details and to truly know the extent and so I choose to be silent about the details. Instead, I can talk about my part in this sometimes wonderful and terrifying effort of being a technical community organizer.

I became part of a small and welcoming technical community in my first month after moving to BigCity. I was alone and lost, feeling unwelcome in a job where there were more managers named Steven than there were women and the community gave me a space to be who I was - a person interested in programing, mathematics, science and technology who happened to be born a woman. The environment was friendly and grassroots - the blatant tech recruiting swag and marketing were absent (there is nothing wrong with events that have recruiting and swag, but there should also be a space for simple food-and-code meetups). Instead, it was just a group of people and laptops working on things ranging from Project Euler to web apps and eating food (usually pizza).

I wanted to give back so I became involved. In the beginning, by choosing topics and giving talks, then more extensively by organizing meetups. It was good fun for a while and I hope that what I did helped at least a few people learn more about programming.

However, as of late, something has changed. What started as a study group and friendly meetup has become a ‘training provider’, or so it seems based on some of the comments the organizers are receiving. It was and will hopefully always remain without charge to allow everyone, and especially underrepresented minorities, to benefit from the spirit of learning and respect. Even if the event itself does not cost anything, it does not mean that organizing it is free. Instead, the brunt of the ‘second shift’ is borne by the volunteers who generously give their time and employees at companies who support technical communities. This time is not free. This is time that someone could be using to learn a technical skill or spend with their loved ones.

For some reason, this fact, the simple fact of respect for the time put into the community and the events by the volunteers and the employees has been lost. I’m not speaking about veneration or even gratitude. Acts of volunteering are rarely done for applause. Nevertheless, the time that is invested in making a community flourish or an event happen deserves respect and this is the respect that I find is lacking.

Moreover, (and perhaps this part is the most painful to say), I have realised my work as a technical community organizer is no longer accomplishing what I want it to accomplish. The roots of the inequality and imbalance in the technical industry (and many others ) are too long and deep to shake.

That is the impetus.

The opportunity is the conversation that I have witnessed in a series of tweets and blog posts by designer Karolina Szczur and by technologist and engineer Cate Huston. Yesterday, Karolina tweeted, “We can’t have more women, people of colour, people with disabilities and other underrepresented groups in engineering, design and leadership if they’re all busy doing community work”. Further to this, she wrote up her thoughts in this blog post. The piece is an honest look at her own journey in her career and the toll that community organizing has taken on her progress. In many ways, the words are painful to read. Juggling the desire to build up ladders and safer spaces, a wish to see the industry embrace diversity beyond the shallow marketing campaigns, and the desire to build up skills to truly lead in the industry is exhausting.

As a result, we often become spread thin over the many responsibilities, some forced, some chosen.

I used to think that this was the price that was necessary for change.

But now I am no longer sure of the direction or the speed of change. Or if the acceleration my own actions are providing is helpful or harmful.

“I have mixed feelings about lists of women. Well I say mixed: my feelings on lists are, broadly, negative.”, writes Cate Huston, in her essay Lists of Women Don’t Change anything. She is speaking about the kind of lists that are frequently publicised in the various corners of the internet: lists of women speakers or women influencers, usually in tech or STEM fields and usually compiled for the purpose of avoiding the so-called “manel” phenomenon (an all-male panel). “ ‘Diversity attention’ is at best worthless and at worst harmful. The only valuable attention is to work and/or impact”, Cate concludes and I agree. I want to work toward a future where more minorities start technology companies and become leaders in their field.

However, I no longer believe that the kind of work I am doing in the community is steering toward this goal. Instead, the ripples I had hoped to make are dissolving into the lukewarm pool of “diversity attention” and fading out. I have to consider that what I am doing may be more harmful than helpful.

Fellow community organizers, you have taught me more about perseverance, empathy, engineering and kindness than anyone ever could and it is those lessons that I will cherish forever. For that I am forever grateful.

Tales from the Python Trenches

A year and a half ago, serious doubts, along with the occasional bug (or ten ) crept into my daily coding life. At the time, I was working for a startup company, which made specialised analytics products for operations and computer security. Most of my days were spent wielding Python against unruly CSV files with missing, incomplete or corrupted data. Although the job was enjoyable and I was absorbing knowledge, I felt that I was failing to develop an ability to understand how large software systems and products are built and maintained. For some reason, at the time, this seemed like a paramount step in the career of an aspiring software engineer ( I hesitated to call myself an engineer back and then and still do, even though it has been my designated job title for a while).

Memories of a recent visit to Google added more fuel to the fire. One of my hosts for the day was the lead developer on Google Chrome and I remember how he beamed with pride when he remarked that the best part about his job was ‘riding on the Tube in the morning and seeing people using your product’. I remember thinking how cool this must be: knowing about and working on the internals of a system used by millions of people. Yet, the event also showed how far I was from being able to work as a professional developer on anything remotely as large and complicated as Chrome. My background was in the world of chemistry laboratories and then the chalk and blackboard floors of the mathematics deparment. I had learned a bit Java on my own and then Python, but that was about it. My graduate degree blended machine learning and bioinformatics, not systems design and operating systems. Thus when my fellow attendees discussed binary search trees, stacks, caches and the Linux kernel all I could do was nod politely and add these items to my ‘To-Study’ list.

After a few months of interviewing while working in my role as Python data analyst, I was able to secure a role in the tech department of Big Corp, where I spent the next year and a half as part of a global team of a hundred or so developers building a large analytics platform in Python. In spite of the barrage of corporate politics and bureaucracy that I encountered, I learned a few very valuable lessons that I would like to note down. Granted not all of the lessons-learned are directly related to Python. Some are bugs with peopleware or process - but important bugs nonetheless!

Invest in Memory Profiling

For junior developers like me, Python’s automatic memory management is a blessing and a curse. A blessing, because with a few lines we can build an entire web application or a server, a curse, because our lack of battle scars with pointers and malloc can easily lead to a support call from hell. This is precisely the situation I found myself one unlucky Thursday morning. With a coffee in tow to jolt awake my still-sleeping brain cells, I logged into my work terminal at my desk and opened the developer chat. There were several messages marked as urgent from the team in Asia-Pacific. They were linking to server side jobs that were crashing and restarting with out-of-memory errors. Within half-an hour or so, the same issues started plaguing the instances serving European users. This had not happened previously and nothing in the environment had changed. There was no monitoring on the data volumes passing through the system, so I tried to eyeball the logfiles to size up if there had been a change in volume. Nothing seemed out of the orginary, yet like clockwork, the server side process kept ballooning and crashing every hour or so.

Eventually, the root cause was found: an innocent looking code change that had been released into production on the previous day. Since the CI system had no performance regression tests, there was no automated alert or failing build to use as a starting point for finding the root cause. Instead, our saving grace was luck - the engineer who had made the commit was online and saw the conversation on developer chat.

Performance regression testing is hard. How often do you run your regression tests? After every commit? Once a day? What data do you use and how do you predict production loads? After you have discovered a memory leak, how do you find which objects are being retained?

OOP is Great until it’s Not

At a recent ClojureBridge meeting, I met an engineer whose first programming language was ClojureScript. After spending many years in Java land, it was strange to discuss programming with someone who was ‘functional-first’. Unfortunately (or fortunately), my first language was Java and so my brain will now and probably far into the foreseeable future be, at least to a certain extent, influenced by the principles of OOP.

In any case, OOP is great if you’re working on a small codebase, but without discipline, OOP in large (>50 million lines) codebases becomes one large inheritance forest. It probably makes sense to the developers who have been working in the same code space for a while, but for a new person, the levels of indirection in giant inheritance hierarchies are overwhelming. As are object public interfaces that span hundreds of methods.

Name Your Variables Well

In 3 weeks time, you won’t remember what type of object is numberContainer. Use variable names and use them well.

Build a Robust Incident Response and Documentation culture

What happens after a major bug or production issue is solved? Is the issue logged anywhere? Does anyone write down the steps that were used to diagnose the root cause of the issue? How was the root cause fixed and what testing was done to make sure the fix will work?

This lesson is not Python specific, but important nonetheless. Too often, it’s easy to get sucked into the chase for the root cause and then, after the bug has been successfully fixed, forget everything about it. Only until the next time, when something similar happens and no one at hand remembers the details well enough. I witnessed this happen many, many times. I was one of the engineers who fixed a bug and then went off to have a midday latte instead of being disciplined and writing down a proper postmortem. Setup a documentation system (Jira, Trello, etc ) that is easy to use, allows code snippets, attachments and screenshorts and that can be easily searched. It will be a valuable record of how resilient your system is and how it evolves.

Coupling Client and Server Code

This one, is once again, not strictly Python specific, but makes ever single server or client release or bug fix into a nightmare. If you don’t use some kind of code-agnostic format of messaging between client and server, but rely on object serialisation, at some point there will be some user somewhere, who refuses to update his client program. If your change is not backwards compatible, the server side process will be in trouble when clients running on the old codebases make connections.

Concurrency is Hard

CPython makes it even harder. A. Jesse Jiryu Davis does a better job than I ever could explaining concurrency in CPython in this post.

Design Your Systems for Monitoring

If monitoring is an afterthought, it will always be an afterthought. In some places, monitoring means a few hastily added debug statements or a few loglines that time the execution of a code block. You might not be motivated during development time to invest in proper monitoring, but come production-issue-time, those hours invested in designing how to monitor a system will pay off. However, pure logging noise (such as logging all messages your program receives) is not helpful to anyone, least of all an on-call engineer who has never committed code into your program.

There will be situations when monitoring is your only hope for figuring out what’s going on. The incidents where a component is crashing are in many ways the easiest, because at least you have a concrete starting point for debug. What about situations where everything seems to be going well, but a user is telling you that a number is incorrect? Or that a number is updating too slowly? If that number is supplied by one service, you have a starting point. What if that number comes out of a pipeline that depends on 5 services interacting together? None of the services has crashed, so there is no obvious point of failure, no obvious point to begin an investigation.

I always strive to think about the metrics that I would find most useful in a incident debug situation. For example, if there is a critical piece of code that needs to executed before an update is sent to the client, I make sure to log this. I also log the number of updates received and sent by the application. In the absence of good metric collecting systems, these give me some ballpark estimates of load. Logging procedures can slow your system down, be mindful of this.

Starting off with what you, the designer of a system, considers useful is not always ideal, because someone else, someone who has never touched your system, may be on-call on the day that it breaks. This is why, ‘debug scenarios’ is one of my favourite team activities. Write down several real or hypothetical user complaints (for example, ‘number X is wrong on my display and it’s taking Y minutes to update. I expect the number to be A and the update to be immediate’) and then ask your team members to diagnose the issue. Observe how they interact with the logs of the application and make notes of log statements that are confusing, unhelpful or red herrings.

For a more persuasive argument about the important of good monitoring, see Nathan Marz’s excellent post “How becoming a pilot made me a better programmer”.