Linden Lab en route to Hell, at last

The road to hell, it is said, is paved with good intentions. The same might be true of other destinations held akin to hell by individuals and organizations, though it is a matter of debate if these destinations would be considered such by anyone but the traveller. Take Linden Lab and governance for instance. One could argue the point that for the bunch of laid back coders and start-up entrepreneurs that constitute LL’s core, the notion of governance is by its very nature indeed hell. If it is, LL has taken a big step down that road, the biggest to date, and — of course — out of good intentions.

Hell ? Governance ? Girl, what are you gabbing about ?

Well, you will not have missed the fact that, with its New Policy Regarding In-World “Banks”, effective from January 22nd, Linden Lab has decreed what amounts to a total ban of in-world banking activities. This has been hailed, or at least acknowledged as inevitable, by observers as diverse as Benjamin Duranske, Prokofy Neva, Nobody Fugazi, Tateru Nino and Dusan Writer; dissenters have been few and in between, easily dismissed as either obvious lobbyists (for instance Arbitrage Wise, « CEO » of JT Financial) or as missing the point (see Lillie Yifu’s angry retort to Robert Bloomfield’s plea for a continuation of SL Y-K-W banking activities as a sandbox for RL studies). Not surprisingly, the decision sat far less well with depositors, who, on hearing the news, panically flocked to the ATMs to withdraw their deposits en masse before the impending closure of their « bank » (see Prokofy Neva’s superb on the spot reportage ; also the articles on Your2ndPlace and Massively), as well as being accompanied by cries of « despotism » on the comments everywhere. The whole thing simply reeks of RL déjà-vu.

Now I’m not going to discuss the pros and cons of the banking ban, nor of its implementation (try aforementioned Arbitrage Wise on Your2ndPlace, or Aldon Hynes on SLNN, a man also on record for saying it needed Colgate to bring smiling to SL Y-K-W avatars, for criticism of that). There is little my gut feeling and homebrew reasoning can contribute to what brighter minds have written. Neither am I going to ponder the question what the balance between scammers and little boys playing bank might have been ; being neither a lawyer, nor a theologian, I find the degree of inherent malice pretty irrelevant to my opinion of the impact of people’s actions.

What I do find interesting is why Linden Lab felt necessary to step in — and what this means for the future of Second Life as a society, a topic I hold dear since the spectacular failure of my first and only recurrence to in-world policing.

Gwyneth Llevelyn has summarized the three current theories about LL’s motivations. In a nutshell, these are : Continue reading



I really, really will update this blog real soon now, any minute… uh… well, as soon as possible… some time. Duh. RL is a bit mad right now, so while the blog topics and snippets pile up in my closet, I’ll try to keep everybody (if that is anybody but me) distracted by pointing to my contribution to Nicholaz Beresford’s call for Linden jokes, located on this page [note the location has moved as I have reorganized the miscellaneous pages section of this blog].

Linden Lightbulb Re-Deploy Post Mortem

This one was my contribution to Nicholaz Beresford’s call for Linden lightbulb jokes

Post mortem

The Linden Lightbulb 1.18.5 release included updates for several systems, including new carbon filament libraries, alloy couplings (a piece of infrastructure which handles a variety of services, such as local fixation and capabilities, and proxies current between systems), and glass geometry. The deploy as planned for November 6th did not require any downtime – all components could be updated live. We planned to perform the rollout per our patch deploy sequences: updating central rooms one by one, then offices. Read on for the day-by-day, blow-by-blow sequence of events which followed…

Tuesday, November 6th

Prior to the 1.18.5 Lightbulb deploy, at around midnight (all times are Pacific Standard Time) we suffered an electricity outage to our restroom facilities, which caused many systems to drop offline. The system recovered on its own after about an hour, and our electricity provider’s initial investigation pointed to hardware issues with the network infrastructure.

Starting at 10:00 am we began the actual update of the lighting fixtures to the Linden 1.18.5 Lightbulbs. We started by updating the “backbone” fixtures on central facilities one by one, such as hall areas, tackling the “non risky” fixtures first. At 11:00 am we got to the “risky” fixtures, which handle emergency lighting (i.e. show the way in case of evacuation) as well as several other key services. Closely monitoring the load on the electrical grid (which usually shows increased load when something goes wrong) as well as internal graphs which closely track the number of appliances online, we started making updates. Everything seemed to be going well.

Towards about 11:15 am the various internal communication channels lit up with reports of appliance failures. We stopped updates of these central systems (7/8ths of the way through) and started to gather data. We have seen this problem in the past when hardware issues or bugs caused the grid monitoring systems to spin out of control, but this time there were no obvious failures; for unknown reasons they grid wasn’t responding to requests from the appliances. Hoping for a quick-fix (i.e. a simple configuration change that could be applied live) we spent about 30 minutes trying to determine the cause, then gave up and rolled back to the previous lightbulb generation.

(Fortunately, in this case, a rollback was straightforward, and simply resulted in “unknown” lighting status for about 10 minutes. Rollbacks are not always so easy – see below!)

Simultaneously, lighting in developer cubicles and coffee rooms failed. These were due to the update as well (but, as it turned out, for different reasons). Once the dust had settled on the rollback it was easy to roll back one more set of fixtures to restore the lights.

Completely unrelated to the update, the electrical load on the central systems required us to pause the Tuesday stipend payouts, delaying the payouts for several hours.

Wednesday, November 7th

Several Lindens continued the investigation, and determined a source of the issues seen on Tuesday: the “emergency lighting” system was updated to use eolian and solar sources to increase performance, but the capacity of these sources was set too low. After some work, we were able to replicate this failure in test environments to verify the fix. The updated bulbs were re-distributed to the fixtures making up the service, and we prepared to try again on Thursday.

(Little did we know that the insufficient electrical capacity was merely a symptom, not the root cause.)

Thursday, November 8th

On Thursday, we proceeded with the 1.18.5 Lightbulb update. The first half of the central fixtures were updated by 12:00 pm. We paused to ensure that the system was behaving as expected, then continued at about 12:30 pm completing the updates. Shortly thereafter, as the number of online lights in the building passed 46,000, the lighting began failing in a new way. Although most of Linden Lab was functioning properly, many light fixtures were slow to go on or failed to light altogether, and some other appliances failed as well. We diagnosed the problem as an unrecognized dependency – the central transformers were assuming that the fuses would shutdown on overload, but the fuse circuits (which had not yet been updated) were assuming the transformers would throttle down instead. Once this root cause was identified (by about 2:15 pm) we were able to change the breaker code in the central transformers’ controllers to resume throttling current consumption, since that was a faster fix. Restarting the transformers did cause employees to sit in the dark for a short period of time, which was unexpected (and is being investigated). Starting after 3 pm we initiated a rolling restart to update the electrical grid as well to complete the update, a process which took about 5 hours. During a rolling restart, in order to reduce electricity consumption and load on central systems, the service is in an unusual state – employees are not allowed to put lights or appliances back on in case of a crash. There was anecdotal evidence that some floors were crashing a lot, but we were unable to verify that this was not simply due to bad hardware until after the process was complete.

After the post-roll cleanup, it became clear that this was not an anomaly. A few contingency plans were discussed, including rollbacks for specific floors, but we were primarily in a data-gathering phase.

Friday, November 9th

As sleepy Lindens stumbled back into work, one incorrect (but ostensibly harmless) idea was tried; unfortunately, due to a typo, this accidentally knocked many employees off the electrical grid entirely around 9:40 am. Shortly thereafter, more testing including complete rollbacks on simulator offices showed that the new transformer controller code was indeed the culprit, but it took a while longer to identify the cause. By 12:00 pm the investigation had turned up a likely candidate – and an indication that a simple widespread rollback of the code would not, in fact, be safe or easy!

The crashing was caused by the transformer “message queue” getting backed up. A server-to-viewer message (related to the grid emergency control system) was updated and changed to move over TCP (reliable, but costly) instead of UDP (unreliable, but cheap and fast). On floors with many appliances and lights, this would cause the grid to become backed up (storing the “reliability” data) and eventually crash. We have a switchboard that allows us to toggle individual messages from TCP to UDP on the fly, but while testing we discovered a second issue – another circuit necessary for the UDP channel needed to be updated, and it could not be changed on the fly, and if we flipped the switch back from TCP to UDP the transformer would crash. (The TCP to UDP update on-the-fly worked, which is how we were able to do the rolling restart in the first place.)

By testing on individual floors, we were able to confirm that by switching back to UDP the problem was eliminated, although this required cutting off all electrical current before throwing the switch. We co-opted an existing engineer for “host-based” rolling restarts (which he had been employed for once in the past), and had him shut down offices on each floor (doing several in parallel), update the breaker circuits, and restart the transformers. After significant testing, we asked this engineer to perform another rolling restart of the service, which was completed by 11 pm on Friday, including subsequent cleanup.

Saturday, November 10th

Unrelated to the deploy (but included here to clear up any confusion), on Saturday at 5:20 pm we suffered another electrical outage, which resulted in hundreds of developers being offline for just under two hours. The cause was due to the expiration of a contract renewal term with our electricity provider. We extended the contract, and our DNOC team brought the affected floors back up.

What Have We Learned

Readers with technical backgrounds have probably said “Well, duh…” while reading the above transcription. There are obviously many improvements that can be made to our tools and processes to prevent at least some of these issues from occurring in the future. (And we’re hiring operations and release engineers and developers worldwide, so if you want to be a part of that future, head on over to the Linden Lab Employment page)

Here are a few of the take-aways:

  • Our load testing of systems is insufficient to catch many issues before they are deployed. Although we have talked about janitors and in-house technicians as a way to roll out changes to a small number of offices to find issues before they are widely deployed, this will not allow us to catch problems on central systems. We need better monitoring and reporting; our reliability track record is such that even problem such as electricity failures for 1/16th of employees aren’t noted for a significant period of time.
  • When problems are detected, we don’t do a good enough job internally in communicating what changes went into each release at the level of detail necessary for first responders to be most effective.
  • Our end-to-end deployment process takes long enough that responding to issues caused during the rollout is problematic.
  • Our tools for managing deploys have not kept pace with the scale of the service, and manual processes are error prone.
  • Track date-driven work (e.g. contract renewal expiry) more closely; build pre-emptive alerts into the system if possible.
  • Be more skeptical about doing updates while the office is live, especially when involving third-party providers.

Age and Treachery

On the set of « A Fish Called Wanda » then 77-year old director John Crichton was given a t-shirt by his co-director John Cleese. It read

Age and Treachery Will Always Overcome Youth and Skill

Though this was meant as a joke about the significant age difference between the director and the rest of his crew, there is a moment of philosophical truth in this. Faced with a world ruled mainly by elder people with an entirely different background and set of interests, youthful talent is doomed to fail. Witness the departure from the Second Life Teen Grid You-Know-Where-for-Underagers of its whizz kid Katharine Berry, as announced in her blog.

For those of you who have never heard the name before, Katharine Berry is probably best known as the developer of AjaxLife, a web based Second Life You-Know-Where client which has allowed many people to maintain their social network on SL Y-K-W even when unable to install the SL Y-K-W client on a machine. I have been using AjaxLife on and off, myself, and have been full of admiration for the work of one single dedicated developer, a female one at that, always a sure way to win my sympathy. What I was not aware of before the announcement of Katharine’s withdrawal and the accompanying media coverage is that she is « only » 15 years old (an ageist reaction I will come back to later).

As befits to her being underage, Katharine has religiously stuck to using the Teen Grid you know where alone, gaining a track record there as a forum moderator, volunteer and provider of web services (all of these by now defunct). Exemplary one would say, were it not for the fact that it must have been a very disheartening experience. As she explains in more detail here, her main reason for leaving Second Life You-Know-Where is the complete state of neglect Linden Lab seems to leave the Teen Grid you know where in, and the low resident count which follows from this sorry state of affairs.

Now, like most Main Grid Second Lifers You-Know-Where denizens I presume, I have been entirely oblivious of what goes on on the Teen Grid you know where. Judging from Katharine’s comments on her departure, it hardly looks like an interesting offering for bright young minds. In fact, it looks suspiciously like an excuse everybody can agree on when it comes to excluding teens from Second Life You-Know-Where in general. By everybody, I do mean that Linden Lab aren’t the only ones at fault here, though they are obviously responsible for the sorry state of the thing, but that all of us Grid Second Lifers You-Know-Where denizens share a part of this. After all, we always point to the Teen Grid you know where as the place for underage Second Lifers You-Know-Where denizens to go whenever we squabble with the Lindens and each other over the value and founding of age verification. It is such a convenient fig leaf when we basically agree that everybody under legal age (and that means anybody below the age of 21, in some cases) should be excluded from participating in Second Life You-Know-Where. They have their own grid you know where, haven’t they ?

Katharine’s withdrawal alerts us to the fact Continue reading

Wild, wild west 2.0

Two weeks ago, I had my first experience of griefing. Somebody entered my home unasked (and we are talking of a skybox over 500 m up, with closed privacy drapes at that) ; I hit « eject and ban » in reflex, and after some fruitless IM exchange my « visitor » started some silly scripted device flooding my screen in green chat. I then did what a good SL Y-K-W citizen is meant to do : I filed an Abuse Report in world, complete with the required screenshot, location and situation resume. Shortly afterwards, I got a polite if obviously boilerplate mail from the abuse report team telling me my report had been received, would be investigated and that I would be notified of the results.

That was two weeks ago.

Now, I could wave this off and tell myself I have learned the hard way never to forget re-enabling my alarm system after a visitor left. The fact is that I do use a scripted alarm and ejection system since I built my skybox — I never relied on the Linden’s enforcement of the TOS, my co-Second Lifers’ You-Know-Where denizens respect for other resident’s privacy and what should be common sense and good manners enough to believe I could leave my home unsecured.

But the Lindens’ complete failure to react rankles. All right, I can hear the old hands hooting with laughter at the thought of anybody in SL Y-K-W below concierge level expecting any kind of reaction to what was, ultimately, « only » a breach of privacy and some minor harassment. It still sucks. So much, actually, that it begs for some thought what kind of environment we live in. Continue reading