LinkedIn is a company that needs no introduction – with over 187 million users in more than 200 countries, it is one of the most important internet companies on the planet. Today we are very lucky to speak with Jim Brikman, staff software engineer at LinkedIn, about their development process.
Can you tell us about yourself? What is your role at LinkedIn and what do you do?
Sure. I have been at LinkedIn for a bit over three years. I am a software engineer and have worked on a number of teams, including the monetization team and the platform team. My current job is on our Presentation Infrastructure team. We are developing the infrastructure and the technologies that we actually use to build the site – the rest of the company builds and runs their code on top of our code. I also run several other programs at LinkedIn – the monthly HackDays, our [in]cubator program, open source, and the engineering blog.
Sounds good. Can you tell us a bit about the development team at LinkedIn? How many engineers do you have and how is it organized?
Good question. The company has grown enormously since I have joined – I don’t know the exact number, but it’s probably between 500-700 people in technology. The exact organization often varies as technologies and requirements evolve, but at a high level, you can break it down into roughly six groups. The Applications & Mobile Applications teams work on the user-facing products, features, and APIs. The Systems and Infrastructure team defines and builds the processing, storage, caching, and distributed systems solutions that are used by the Apps teams to serve and scale the site. The Front-End Engineering team works on the client-side pieces, including JavaScript, CSS, and HTML. The Tools Team builds the internal tools and processes, including how we store code, the build system, automated deployment, monitoring, and so on. The Data Team works on how we make sense of our huge data sets, including search, data mining, and machine learning. Finally, the Test Team figures out how to make our code robust, bug-free, and performant.
Within the large, overarching groups, there are much smaller teams. The breakdowns vary quite a bit. Usually, the teams own a certain product, and they work together with product managers, designers, and other folks. The average project that I have worked at LinkedIn has had between one and five people working on it – plus one product manager and one designer.
We push code to the site a few times a day. Not every team has something to push each day – you have to code something before pushing it. A lot of teams use an agile, Scrum process, where they do two to three-week sprints, and then at the end of each sprint, put something onto the live website. That does not necessarily mean that the users see it – we can shut features off or hide them, or only make them visible to employees.
The sprints vary a lot per team and per project – as some of the projects are much more long-term. But for the most significant part of the company – the Apps team – it is usually two to three-week sprints and then pushing something live.
Do you work autonomously within the small teams or is there a lot of interaction with the rest of the company?
It depends on which team you are on and what you are doing. In general, you do need a fair amount of interaction with the rest of the company to make sure we are all aligned, not repeating the same work, or contradicting each other. Usually, it is taken care of pretty well near the top.
But even beyond that, every new feature usually has a lot of dependencies on everything else. LinkedIn is built as a service-oriented architecture, so for each different type of data, I may need to talk to a different service. When you launch a new feature, it would be less than polite just to dump it out there, and not notify the teams that own that service. You have to work pretty closely with other teams to make sure it is going to scale before you put it in front of a couple of hundred million people. We also have to work with the operations team that gets the code onto the site. With the QA to make sure that somebody tests the code.
It is hard to do something entirely in isolation, but the closer you are to the front-end of the site, the easier that can often become. Right now, I am on the infrastructure team, which means that my primary customers are other developers, and I am working with other teams very closely. I used to also work on some of the Apps teams and the Monetization team, and there you have a lot more ability just to go and try things. It tends to be a very data-driven decision process. You put something out there, you are doing an A/B test or bunch of buckets – and see how it behaves.
And how do you communicate? You mentioned Scrum, do you do stand-up meetings or other communication methods?
Within your close team of one to five, the communication is usually done just in person and via stand-up meetings. A lot of teams have a ten-minute meeting in the morning, or at some point during the day. We also often have the big Scrum board with Post-It notes, and all the tasks and things we are working on.
Between teams, communication varies a lot. We use a lot the standard tools – such as email and in-person chats when it is local (most of the team is in Mountain View). If that does not work, we have an internal chat system, IRC, internal Q&A tool, JIRA for bug tracking, wiki pages – a pretty standard set of tools to either synchronously communicate with someone, or leave information for others for later.
What challenges do you encounter when organizing the development process, if any?
Oh, there are tons. There are plenty of challenges at any time you do any project. As described in The Mythical Man-Month, the more people there are working on a single project, the more time is spent just communicating things to them. So that often can be a challenge. Finding out who has given information can be a hard thing. For example, if I need the data from a certain service, it is not always obvious which services will even have the data that I need, how to use it correctly, what the authoritative source is, or what the scalable way of doing it is. Even knowing who knows the piece of information, as the company grows, can be a challenge.
There are a lot of exciting ways to solve it. I am not sure that we have cracked that puzzle entirely. But it comes down to everything from:
- How do you actually build the technologie?
- Are all of your APIs consistent, canonical, and reusable?
- How easy it is to find the documentation for these things?
- How is the code organized, is there some kind of pattern that everybody follows?
- And so on…
Also, how we share information in the organization is important – are things like wiki pages of a certain format, and can you search them, and the information would show up. Some teams do weekly status reports, which are filtered up to the top, and then there is a company-wide email with the information on what the whole technology group is working on.
Do you plan any changes in your process?
I am not aware of any humongous changes as a whole company, but within each team, we tweak little things almost every single sprint. After every sprint, we get together and say what worked and what did not – and make little adjustments, like changing how we file bugs, track priorities, etc.
OK. The final question that we have does you have any advice for startups, their CTOs or other developers?
I think that the last point about Scrum is the most important thing to keep in mind. No process just works, you won’t get it right the first time, you won’t get it right the tenth time. And as the company grows and the projects change, what used to work will no longer work. The most useful thing is to build your process in a way where you can change it and adapt it easily. Set your process to be this powerful thing that you can adjust as necessary, and make sure to listen to feedback from the people that are using the process. If they are not happy, then the process changes from something that helps you go faster to just pure overhead.
When we first started doing Scrum, it did not work perfectly – a lot of things were wrong and just were not sufficient for the way we had been trying to do the project. The only way we were able to fix it is after every sprint to honestly say – here are things we need to stop doing, here are things we need to start doing, and here are things that are working well. If you do that, after a few sprints, you will end up with something that helps a team go faster rather than just feeling like stupid overhead that some managers impose.
