Did you know that Wednesday Wisdom is also a podcast! Find it on Apple Podcasts or on Spotify. Did you know that there is also a custom GPT called Midweek Muse that has access to all of Wednesday Wisdom? Did you know that all Wednesday Wisdom videos are also available on YouTube?
There comes a time in every SRE team’s life when it has had enough of many late evenings and working through the night to deal with incidents. At that time, the thought of starting a sister team in a convenient location inevitably arises. Wouldn’t it be nice to clock out at six because your team in <somewhere else> just woke up and is able to bear the brunt of feeding cookies to the servers to keep them going?
Answer: Yes, that would be nice and for most of my time in Google Zürich, I was part of such a team <somewhere else>, combining Big Tech salaries and equity with Swiss public transport and alps, rather than wringing my way through hellish traffic on the US 101 to some godforsaken bayside corporate jungle. For most of that time, I was in happy and effective remote teams, but I have also seen more than my fair share of very ineffective and unhappy teams, which inspired me to write down some ground rules for starting remote SRE teams “the right way”.
Note: In the rest of the article, I am going to make the wild assumption that the main team is based in Silicon Valley.
Your first question is where to start your remote team. This is already not entirely obvious, because you want them to be far enough away (in terms of time zones) to provide effective night coverage, but close enough so that there are still enough opportunities for meetings at a somewhat comfortable time for everyone. Unfortunately, these goals are incompatible. Put your team 12 time zones away (in either direction, because the world is round and a day has approximately 24 hours) and the pain of covering the wee hours is equally distributed between both teams. Unfortunately, this will make it impossible to have meetings at reasonable times and you want at least an oncall handover meeting and maybe even a weekly team meeting. Plus, 12 hours from Silicon Valley puts you in Kazakhstan, which is unsuitable for reasons that will be discussed later. Timezone wise you want to make a compromise and put your remote team 8-9 hours ahead or behind your main team.
There are more considerations than time zone distance though. You will want to place your remote team at a location where you can hire, which means that there is either a local pool of talent or you can get people to move there. This is another reason why Kazakhstan is not a great place for a remote team. I have never been there and I assume it to be lovely (which in Dutch we call a “positieve grondhouding”, which translates to having a positive basic attitude), but as far as I know it is neither a hot bed of local IT talent, nor a desirable immigration destination for talent you find somewhere else.
For this reason, I don’t recommend Tokyo either. I have been there and love it dearly, and there is surely a decent pool of technical talent there, but it is a hard destination to immigrate into, given the still meagre understanding of English in the general population and general unfriendliness towards immigrants (although times are changing there too). If you want to go the Asia direction, Singapore or Australia are probably better bets.
Going in the easterly direction, you end up in the British isles (practically speaking Dublin or London) or continental Europe. All of these are amazing places, though London has shot itself in the foot somewhat with Brexit and of course the problem of London is that it considers itself the center of the world, so there is a non-zero chance that your remote team starts thinking that they should be calling the shots and who are these upstarts in the colonies anyway? Before you know it, your local site director will call themselves “CEO of <company> Europe” and you will have to let them go…
It would be ideal if you can embed your new remote team in an existing office. First of all, that precludes you from having to go through much bureaucracy to create a legal entity, set up payroll, figure out benefits, register with the tax authorities, and whatnot. Secondly, and this is the bigger benefit, an existing office forms an anchor in terms of mission, sense of belonging, and culture.
The biggest potential problem for a remote team is culture drift.
The life of a remote team can be difficult. There you are, at the ass end of your corporate world, disconnected from all the unofficial communication channels, all alone, in the night. In such circumstances, it is not uncommon for the remote team’s culture to start drifting from that of the main team, which will eventually lead to a strong “us versus them” feeling. This is not helpful. Effective teams are happy teams and, per Tuckman, have shared norms when it comes to the work they are doing. Having a grumpy subteam sitting somewhere in the corporate annex does not help with that.
This danger is all the greater when all the members of the remote team have a shared culture and language already, setting themselves even more apart from the main team. A good remote team is a culturally and linguistically diverse team so that there is a smaller chance that they are going to form an isolated sociotope. Plus it will force them to communicate in English among each other, which will improve everyone’s English language skills and makes all communication easier.
This is another reason why Dublin, London, Singapore, and Australia are favorites for placing remote teams, they kinda already speak English. I say “kinda”, because I once managed a team that contained an Englishman, an American, and Australian, and a Canadian. Let me tell you, these are four people divided by a common language.
This is also another strike against a location like Tokyo. When I worked at Google, transferring into that office was made more difficult because you really needed to speak, read, and write Japanese at some reasonable level of fluency just to be able to integrate into the office and the team. Compare this to Amsterdam, which is by now so international that I cannot speak Dutch there anymore to random people on the streets, in restaurants, or in stores.
Last time I was there, I went into a stroopwafel store. “Ik had graag een grote stroopwafel”, I said to the young lady behind the counter. “Sorry, I don’t speak Dutch”, she replied. “Well, I am in the stroopwafel store”, I said, “what do you think I just ordered? Really the only question is what size and the answer to that is obviously large.”
Creating your new remote team in an existing, already diverse, remote office is beneficial because it creates an anchor into company culture and it adds to the feeling that the new team is not just a bunch of mushrooms in some box somewhere, kept in the dark and fed a load of bullshit. Bonus points if you manage to create your new remote SRE team in a location where there are already engineers of other stripes, since that adds to the career opportunities available to everyone and it allows for professional as well as cultural connections inside the remote office.
To kickstart your new team, you need a few very strong hires. The start of a new remote team is a very delicate time and you want to make sure the first people, who are going to set the tone for the new team, are strong performers. Ideally speaking you would seed the new team with a senior engineer from your main team, who you bribed to move and help kickstart the team, if only for a year or maybe two.
Another good idea is to make your first hires for the new remote team spend 2-3 months with the main team right after their starting date. When I started in the brand new SRE team in Zürich in 2006, I spent my first two months of employment embedded with SRE in Mountain View and one month in New York, to learn the systems and make connections with all the right people. Say what you will about the effectiveness of remote work, but in my experience, people still work better with people whom they know personally. Apart from the direct value of being exposed in person to the work and to the main players, being in HQ for an extended amount of time is hugely inspiring. For instance, it enabled me to attend all-hands meetings with the founders bantering and answering questions. Plus our main office was incredibly lavish compared to our modest offices in the Freigutstrasse (though that got fixed later when we got a new and bigger office that was the envy of the world).
At Google SRE in Zürich, we sent our new hires to Mountain View for a few years after founding the remote team, until such time that it was felt that the our office and team was big and established enough that we could onboard new hires without running the risk of unwanted diversions in culture and quality.
The next thing to look out for is to make sure that the new remote SRE team does not end up being a troupe of pager monkeys. It is fairly obvious that the need for off-hours support is/was the main reason for founding this new team in a distant galaxy far, far, away, but that cannot and should not be the only work they do down there because that would be boring and demotivating. An enormous challenge for every team is to make sure that the remote team gets meaty projects of their own that are challenging, motivating, and inspiring. Without work of that nature, you will not be able to keep standards up, keep your strong performers in that team, and keep morale up.
This typically runs into resistance from the main team who are often very selfish when it comes to keeping the important and fun work for themselves, but that way lies ruin for the whole effort. There are tons of good reasons to keep the important work in headquarters: Typically that is where the major decision makers are, where the cross functional teams are, and where the big decisions get made. But all of these reasons are short-term reasons; if you want to have a sustainable reliability engineering effort in the long term, you need effective and happy remote teams and one of the ways to make them effective and keep them happy is to make sure that they have meaningful work. And I am not talking about some P2 items that nobody in the main team really wants to do anyway; I am talking about major pieces of work that would make an existing main team member move to be on the project.
The main problem here is trust.
The existing team in HQ is often reluctant to “give projects away” to a bunch of noobs in some far away location. This is one of the main reasons why you want your first remote hires to be strong and to start them off in the main office: To start building the required level of trust that this new remote team will eventually grow into a high quality sister team. I already mentioned Tuckman and his theory on team development applies when establishing a remote team too. You will need to go through the forming, storming, and norming stages with the entire team in order to get to the performing stage.
To guard this entire process of hiring only strong performers, ensuring that the team has enough good work, and creating the required connections with the home base, you want this new remote team to be managed by an exceptionally strong manager, who themselves will need to establish connections and trust with the leaders in the non-remote office. Hiring a great manager is the first step in getting everything going, preferably one who has ample experience working in/with Silicon Valley.
Next up is travel.
Once the remote team gets going a bit, you will need to institute a strong and regular program of travel in both directions to make sure that all team members get to work face-to-face with their colleagues on the other side of the world. This travel program must work both ways: The remote people need to be able to travel to HQ about once a quarter, but the main team members should also travel to the remote office at least once a year.
Time for an anecdote: In 2012 I was a founding member of the new remote YouTube SRE team in Zürich and of course I worked to do much everything I wrote about above, including asking people from the team in San Bruno to come to Zürich and work from our luxurious offices at the Hürlimann Areal. On Monday evening, from 6:30-7:30pm we had the weekly meeting with our colleagues in the Bay Area. For us, this was our last meeting of the day and we were always ready to get it over with and go home. For our colleagues in San Bruno, it was Monday morning and they had just got in, not yet accustomed to being back at work after the weekend and so they typically bantered around a bit and asked each other how their weekends had been.
Literally the first San Bruno colleague we got to come to Zürich to work with us for a week said the following a mere ten minutes into the meeting: “Oh my God, we are dicks!”, showcasing the importance of travel in order to experience both sides of the coin.
I know that travel budgets are managed separately from compensation and are continuously being cut, video conferencing technology being what it is today. And, agreed, a week-long trip to the sister team can easily cost $5,000 - $6,000 in flights, hotel, Ubers, and other expenses. But, looking at the total cost of employment, can you afford to skimp on this and run the risk of letting the whole remote team experiment go bust, at the cost of several million dollars? Penny wise, pound foolish!
I have seen many very successful remote SRE teams but I also have seen my fair share of failures too. As I outlined above, it is not rocket science to get it right, but it is real work and there is always pressure to get this done faster and with less hassle, which I would consider decidedly unwise, and therefore running against the whole raison d’être of Wednesday Wisdom.
See you next week!
