Dear Catherine Linden,

following up on your post on the Official Linden Blog on the Brand Center, I turn to you today in my quest for an instance able to certify that the measures I have taken on my blog do indeed make it conform to the new trademark policy of Linden Research. Not wishing to incur costly legal disputes, I have made the following amends:

  • All mentions of Linden Research’s trademark ‘Second Life’ have been replaced by ‘You-Know-Where’ throughout all posts, pages and comments.
  • All mentions of Linden Research’s trademark ‘SL’ have been replaced by ‘Y-K-W’ throughout all posts, pages and comments.
  • All mentions of Linden Research’s trademark ‘grid’ have been replaced by ‘you know where’ throughout all posts, pages and comments.
  • Common derivative terms have been adapted as fitted this  ‘Second Lifers’ for instance has been replaced by ‘You-Know-Where denizens’ throughout all posts, pages and comments, ‘SLURL’ by ‘Y-K-WURL’, etc.
  • Related uses of Linden Research trademarks have been corrected likewise: the subtitle of the blog now reads ‘Rheta Shan’s You-Know-Where blog’, and posts relating to the trademarked product are now tagged ‘you-know-where’.
  • The Disclaimer page has also been amended to conform to the required form.
  • Finally, all errant ‘s’ appended to Linden Lab have been removed throughout all posts, pages and comments.

I have left these edits documented for inspection, and will do so for the length of the grace period generously offered by Linden Research, or until a crack team of censors trademark lawyers from your office signifies the all-clear for this measure, preferably with an official seal of approval (a large watermark ‘TM’ might be fitting). I must however preemptively beg your leniency as to an issue that is bugging me: the title of the post Second Life Guess, which, as any expert you care to consult will probably be able to explain to you, is a pun, or play on words – I’d appreciate a proposal on how to handle the translation into the new terminology in this case. I also must apologize for not having converted the incriminated trademarked terms to the new form in URLs, as I have not found a solution to do so this without breaking the link to the targeted page. On this matter too, advice would be greatly welcome.

Once approval is given, I will extend this corrections to all other statements on the web I have made, as far as I have editorial rights to these (I am very much afraid there is not much I can do about old Twitter posts, barring deleting them all. Would that be the suggested course?).

If I may, I would urge you to accede to my request in a timely fashion, allowing me to turn this blog into a shiny example for the entire Second LifeYou-Know-Where-related blogosphere. I can promise that, once your and the censors’ trademark lawyers’ approval has been given, I will use my influence in blogger circles for the blogosphere to follow my example and conform to your expectations.

Do not hesitate to contact me, in-world or here, in case of questions.
Cordially yours
Rheta Shan (avatar)

Second Life Guess

[I apologize for not being able to correct the title of this post so as to conform to the new trademark policy of Linden Research. A suggestion by the office of C. Linden and / or Linden Research’s censorship trademark lawyers is however pending]

Linden Lab’s ban on SL Y-K-W banks is in effect as of today, on the dot two weeks after it has been announced. Two weeks in which, in the words of Massively’s Tateru Nino :

…we’ve had protests – some assuredly genuine, some apparently staged – runs on banks (a sort of game of musical chairs, where everyone hopes not to be the one left standing when the music stops), and the usual commingled mish-mash of cheering, screaming, jesting and angry outbursts…

— much of it happening in the SL Y-K-W blogosphere (see my own round-up of it here). Now the grace period is over, the economic effect of the ban turns to have fallen somewhat short of the sky caving in, in fact seeming to be near to negligible for the SL Y-K-W economy as a whole (though I am sure that on the personal level, for people having lost money in the crash, this is an entirely different story).

Seeing that and the fact that Linden Lab seem nowhere close to reverting their decision, the discussion of the pro and con seems to have petered out somewhat. Artur Fermi stating in essence « Good riddance, and keep the hype down » on Your2ndPlace and Aldon Huffhines / Hynes arguing on SLNN that Linden Lab’s decision is ill conceived and drives needed financial services out of the official grid, and that LL should reconsider are more or less the last ones to battle it out. Prokofy Neva and Benjamin Duranske on the other hand have added most welcome shades of grey to the often black and white discussion by casting some light on the people behind the banks. So, what am I up to, besides fawning for some more pats on the back for my diligent compiling work ?

Well, there is a twist on the debate I find fascinating : Continue reading

Linden Lab en route to Hell, at last

The road to hell, it is said, is paved with good intentions. The same might be true of other destinations held akin to hell by individuals and organizations, though it is a matter of debate if these destinations would be considered such by anyone but the traveller. Take Linden Lab and governance for instance. One could argue the point that for the bunch of laid back coders and start-up entrepreneurs that constitute LL’s core, the notion of governance is by its very nature indeed hell. If it is, LL has taken a big step down that road, the biggest to date, and — of course — out of good intentions.

Hell ? Governance ? Girl, what are you gabbing about ?

Well, you will not have missed the fact that, with its New Policy Regarding In-World “Banks”, effective from January 22nd, Linden Lab has decreed what amounts to a total ban of in-world banking activities. This has been hailed, or at least acknowledged as inevitable, by observers as diverse as Benjamin Duranske, Prokofy Neva, Nobody Fugazi, Tateru Nino and Dusan Writer; dissenters have been few and in between, easily dismissed as either obvious lobbyists (for instance Arbitrage Wise, « CEO » of JT Financial) or as missing the point (see Lillie Yifu’s angry retort to Robert Bloomfield’s plea for a continuation of SL Y-K-W banking activities as a sandbox for RL studies). Not surprisingly, the decision sat far less well with depositors, who, on hearing the news, panically flocked to the ATMs to withdraw their deposits en masse before the impending closure of their « bank » (see Prokofy Neva’s superb on the spot reportage ; also the articles on Your2ndPlace and Massively), as well as being accompanied by cries of « despotism » on the comments everywhere. The whole thing simply reeks of RL déjà-vu.

Now I’m not going to discuss the pros and cons of the banking ban, nor of its implementation (try aforementioned Arbitrage Wise on Your2ndPlace, or Aldon Hynes on SLNN, a man also on record for saying it needed Colgate to bring smiling to SL Y-K-W avatars, for criticism of that). There is little my gut feeling and homebrew reasoning can contribute to what brighter minds have written. Neither am I going to ponder the question what the balance between scammers and little boys playing bank might have been ; being neither a lawyer, nor a theologian, I find the degree of inherent malice pretty irrelevant to my opinion of the impact of people’s actions.

What I do find interesting is why Linden Lab felt necessary to step in — and what this means for the future of Second Life as a society, a topic I hold dear since the spectacular failure of my first and only recurrence to in-world policing.

Gwyneth Llevelyn has summarized the three current theories about LL’s motivations. In a nutshell, these are : Continue reading

Gasp

I really, really will update this blog real soon now, any minute… uh… well, as soon as possible… some time. Duh. RL is a bit mad right now, so while the blog topics and snippets pile up in my del.icio.us. closet, I’ll try to keep everybody (if that is anybody but me) distracted by pointing to my contribution to Nicholaz Beresford’s call for Linden jokes, located on this page [note the location has moved as I have reorganized the miscellaneous pages section of this blog].

Linden Lightbulb Re-Deploy Post Mortem

This one was my contribution to Nicholaz Beresford’s call for Linden lightbulb jokes

Post mortem

The Linden Lightbulb 1.18.5 release included updates for several systems, including new carbon filament libraries, alloy couplings (a piece of infrastructure which handles a variety of services, such as local fixation and capabilities, and proxies current between systems), and glass geometry. The deploy as planned for November 6th did not require any downtime – all components could be updated live. We planned to perform the rollout per our patch deploy sequences: updating central rooms one by one, then offices. Read on for the day-by-day, blow-by-blow sequence of events which followed…

Tuesday, November 6th

Prior to the 1.18.5 Lightbulb deploy, at around midnight (all times are Pacific Standard Time) we suffered an electricity outage to our restroom facilities, which caused many systems to drop offline. The system recovered on its own after about an hour, and our electricity provider’s initial investigation pointed to hardware issues with the network infrastructure.

Starting at 10:00 am we began the actual update of the lighting fixtures to the Linden 1.18.5 Lightbulbs. We started by updating the “backbone” fixtures on central facilities one by one, such as hall areas, tackling the “non risky” fixtures first. At 11:00 am we got to the “risky” fixtures, which handle emergency lighting (i.e. show the way in case of evacuation) as well as several other key services. Closely monitoring the load on the electrical grid (which usually shows increased load when something goes wrong) as well as internal graphs which closely track the number of appliances online, we started making updates. Everything seemed to be going well.

Towards about 11:15 am the various internal communication channels lit up with reports of appliance failures. We stopped updates of these central systems (7/8ths of the way through) and started to gather data. We have seen this problem in the past when hardware issues or bugs caused the grid monitoring systems to spin out of control, but this time there were no obvious failures; for unknown reasons they grid wasn’t responding to requests from the appliances. Hoping for a quick-fix (i.e. a simple configuration change that could be applied live) we spent about 30 minutes trying to determine the cause, then gave up and rolled back to the previous lightbulb generation.

(Fortunately, in this case, a rollback was straightforward, and simply resulted in “unknown” lighting status for about 10 minutes. Rollbacks are not always so easy – see below!)

Simultaneously, lighting in developer cubicles and coffee rooms failed. These were due to the update as well (but, as it turned out, for different reasons). Once the dust had settled on the rollback it was easy to roll back one more set of fixtures to restore the lights.

Completely unrelated to the update, the electrical load on the central systems required us to pause the Tuesday stipend payouts, delaying the payouts for several hours.

Wednesday, November 7th

Several Lindens continued the investigation, and determined a source of the issues seen on Tuesday: the “emergency lighting” system was updated to use eolian and solar sources to increase performance, but the capacity of these sources was set too low. After some work, we were able to replicate this failure in test environments to verify the fix. The updated bulbs were re-distributed to the fixtures making up the service, and we prepared to try again on Thursday.

(Little did we know that the insufficient electrical capacity was merely a symptom, not the root cause.)

Thursday, November 8th

On Thursday, we proceeded with the 1.18.5 Lightbulb update. The first half of the central fixtures were updated by 12:00 pm. We paused to ensure that the system was behaving as expected, then continued at about 12:30 pm completing the updates. Shortly thereafter, as the number of online lights in the building passed 46,000, the lighting began failing in a new way. Although most of Linden Lab was functioning properly, many light fixtures were slow to go on or failed to light altogether, and some other appliances failed as well. We diagnosed the problem as an unrecognized dependency – the central transformers were assuming that the fuses would shutdown on overload, but the fuse circuits (which had not yet been updated) were assuming the transformers would throttle down instead. Once this root cause was identified (by about 2:15 pm) we were able to change the breaker code in the central transformers’ controllers to resume throttling current consumption, since that was a faster fix. Restarting the transformers did cause employees to sit in the dark for a short period of time, which was unexpected (and is being investigated). Starting after 3 pm we initiated a rolling restart to update the electrical grid as well to complete the update, a process which took about 5 hours. During a rolling restart, in order to reduce electricity consumption and load on central systems, the service is in an unusual state – employees are not allowed to put lights or appliances back on in case of a crash. There was anecdotal evidence that some floors were crashing a lot, but we were unable to verify that this was not simply due to bad hardware until after the process was complete.

After the post-roll cleanup, it became clear that this was not an anomaly. A few contingency plans were discussed, including rollbacks for specific floors, but we were primarily in a data-gathering phase.

Friday, November 9th

As sleepy Lindens stumbled back into work, one incorrect (but ostensibly harmless) idea was tried; unfortunately, due to a typo, this accidentally knocked many employees off the electrical grid entirely around 9:40 am. Shortly thereafter, more testing including complete rollbacks on simulator offices showed that the new transformer controller code was indeed the culprit, but it took a while longer to identify the cause. By 12:00 pm the investigation had turned up a likely candidate – and an indication that a simple widespread rollback of the code would not, in fact, be safe or easy!

The crashing was caused by the transformer “message queue” getting backed up. A server-to-viewer message (related to the grid emergency control system) was updated and changed to move over TCP (reliable, but costly) instead of UDP (unreliable, but cheap and fast). On floors with many appliances and lights, this would cause the grid to become backed up (storing the “reliability” data) and eventually crash. We have a switchboard that allows us to toggle individual messages from TCP to UDP on the fly, but while testing we discovered a second issue – another circuit necessary for the UDP channel needed to be updated, and it could not be changed on the fly, and if we flipped the switch back from TCP to UDP the transformer would crash. (The TCP to UDP update on-the-fly worked, which is how we were able to do the rolling restart in the first place.)

By testing on individual floors, we were able to confirm that by switching back to UDP the problem was eliminated, although this required cutting off all electrical current before throwing the switch. We co-opted an existing engineer for “host-based” rolling restarts (which he had been employed for once in the past), and had him shut down offices on each floor (doing several in parallel), update the breaker circuits, and restart the transformers. After significant testing, we asked this engineer to perform another rolling restart of the service, which was completed by 11 pm on Friday, including subsequent cleanup.

Saturday, November 10th

Unrelated to the deploy (but included here to clear up any confusion), on Saturday at 5:20 pm we suffered another electrical outage, which resulted in hundreds of developers being offline for just under two hours. The cause was due to the expiration of a contract renewal term with our electricity provider. We extended the contract, and our DNOC team brought the affected floors back up.

What Have We Learned

Readers with technical backgrounds have probably said “Well, duh…” while reading the above transcription. There are obviously many improvements that can be made to our tools and processes to prevent at least some of these issues from occurring in the future. (And we’re hiring operations and release engineers and developers worldwide, so if you want to be a part of that future, head on over to the Linden Lab Employment page)

Here are a few of the take-aways:

  • Our load testing of systems is insufficient to catch many issues before they are deployed. Although we have talked about janitors and in-house technicians as a way to roll out changes to a small number of offices to find issues before they are widely deployed, this will not allow us to catch problems on central systems. We need better monitoring and reporting; our reliability track record is such that even problem such as electricity failures for 1/16th of employees aren’t noted for a significant period of time.
  • When problems are detected, we don’t do a good enough job internally in communicating what changes went into each release at the level of detail necessary for first responders to be most effective.
  • Our end-to-end deployment process takes long enough that responding to issues caused during the rollout is problematic.
  • Our tools for managing deploys have not kept pace with the scale of the service, and manual processes are error prone.
  • Track date-driven work (e.g. contract renewal expiry) more closely; build pre-emptive alerts into the system if possible.
  • Be more skeptical about doing updates while the office is live, especially when involving third-party providers.