Day 4- Facebonkers

Today (yesterday) was an exciting day, you could feel it. Everyone knew we were heading to Facebook and were so overly excited about it.
Facebook seems like so far away when you’re in Australia, so out of reach, but alas we were all going to see Zuck by day’s end (we didn’t see Zuck).
As I sit here typing this on the bus on our way to Equinix the hotel (yes, I know, I’m late posting this one. I wanted to make sure I got everything is), something that stands out to me (and makes me happy), is that looking out the window every few hundred metres, you will spot another startup or another tech company you’ve heard of before. This isn’t something you get in Australia. It’s usually petrol stations, supermarkets or electronics stores. There is also an astronomical amount of fibre optic cabling on all the utility poles, something we will unfortunately miss out on with NBN Co’s change of direction.

Before visiting Facebook, we had another couple of places to learn from. Our first visit was to ServiceRocket, founded by Australian Rob Castaneda.
Now, I’ll be honest. I hadn’t heard of Rob or ServiceRocket before, so I was very interested with what a company used by Facebook, Atlassian, Telstra, Suncorp Group and other big players had to offer.
Arriving at ServiceRocket was like nowhere we had been before. We were greeted and welcomed in like family. This is something that Rob and the team feel makes a great company and hold close to their core values. We were treated to breakfast, lunch and of course some free swag (#swag).
During the visit, we heard from Rob, who told us about his journey from university student teaching courses for a US company to building a business which spans the globe and is relied upon by just about every company you’ve ever heard of.

During Rob’s talk, I picked up a number of things which appear to push and build the entrepreneur above and beyond what an average employee would do. Some of these can be replicated, others not.

• Family and personal life experiences
• The need to fix a problem they’ve found
• Opportunity seizing
• Taking negativity and impossibility in their stride
• Building strong and lasting networks

Notice how I haven’t mentioned money, building solutions to non-existent problems or luck.

You create your own luck

Rob also had some other insights so far as developing a Silicon Valley back in Australia. He believes you should be where your customers are, which makes total sense, and if we’re targeting a global market, the US makes up a significant proportion of that. He doesn’t advocate building a SV in Sydney or anywhere else, he says the US is simply where that is. This view point, which we have heard a few times, is contrary to the goals of the mission we’re on. Interesting food for thought and something I will be spending a considerable amount of time reflecting on.

On a side note, Rob and the team had a working Atari and one of the largest computer history museums owned by an Australian based startup (that’s not really official, I made it up).

After ServiceRocket we headed to VMware, specifically to talk about the merger and acquisition of AirWatch. This was interesting as we had been hearing a lot about building startups and finding product and customer fit, we hadn’t yet heard much about the later stages. And whilst we shouldn’t get ahead of ourselves, it is great knowledge to have.
The main topic that came up was not bullshitting. This will be found under the enormous scrutiny that potential acquisitions are put under.
We were also treated to an awesome tour of the grounds and buildings, which provided for some awesome scenery.

Then it was time for the big one, Facebook. We had heard people say Facebook is ‘Disneyland’ and I thought it could be in the sense that it is big and a goal for so many companies. But when you see the fake streets, with themed restaurants, private bus services with digital info boards, bikes for cycling between buildings, under a highway, the sheer size of the campus and the rooftop gardens, you soon see that it is literally like Disneyland (or Movie World on steroids).
Something else I noticed was how nice everybody was. A true incorporation of the values and culture the company promotes. This wasn’t just regular employees, but security and parking attendants as well (oh yeah, they have valet parking).

We met up with Joel Pobar, Aussie, now managing one of Facebook’s engineering teams of over 100 people. He introduced us to this land and scoped out our plan for the evening. He introduced Chuck Rossi who took us through deployment at scale. He had some interesting stats and insights as follows:

• Every 30 minutes, Facebook processes
• 10 terabytes (10,000 gigabytes) of logs through Hadoop
• 6 million photo uploads
• 116 million Newsfeed stories
• 5 billion instant messages
• 10 billion profile pictures
• 108 billion MySQL queries
• 3.8 trillion cache operations
• 86% of Facebook users are outside of the US
• Around 1 billion daily active users on both mobile and web.

These are incredible numbers. To think that when you’re sitting on your phone scrolling through Facebook, billions of other things are happening in the background. There is so much complexity to this it really boggle the mind.
Due to the requirements of instant information and the sheer number of requests, Facebook requires an entire cache layer to sit above the database so it is ready to be served instantly. This cache layer sits on thousands of solid state drives in gigantic data centres all around the world. These data centres are six times the size of Facebook’s HQ, coming in at around 28,500 square metres!

Chuck also took us through their deployment methodology. For the web, every Sunday, all changes are wrapped up and at Tuesday 4pm (Pacific), they are pushed to live. They then continuously deploy by pushing to live changes three times per day. They work on only three branches- trunk (master), live and archive (ex-live). They also have infrastructure set up to automatically merge if there’s conflicts or changes since you started your work.

We then took a small tour around some of the Facebook campus, including eating areas, tech help areas and the hardware lab (we glanced at it)- f*cking amazing. It was pointed out that every building is purposefully incomplete- no roofing, polished tile floors, cables across the roof and the rear of the main sign out the front still has the old Sun Microsystems logo on the back (the original campus is built in where Sun’s campus used to be). This is to signify that their work is never done and that even giants can fall if they become complacent.
We made our way across under a highway to Facebook’s Building 20.
Making our way over there, the sheer enormity of the physical locations became apparent. Walking in, you could feel how open and energetic the place felt- even after 6pm. People are working on some cool shit, solving big problems and enjoying the heck out of it.

We were taken through a talk on machine deep learning with another Aussie, Andrew Tulloch. We also had pizza ordered in- just the quintessential Facebook hacker thing to do. I’ll be honest, for me the talk was a bit in the weeds, but fascinating nonetheless. Andrew clearly knows his shit- he’s known to be the guy asking ‘what’s the biggest problem we’ve got? I’m gonna solve it.’ Something interesting he brought up was that the goal setting at Facebook has a 50/50 arrangement- 50% of goals are expected to be so grand that they won’t be achieved.

After Andrew, it was finally Joel’s turn to take us on. He had a deep and highly insightful discussion on management of teams. This covered everything from team formation, psychology, tuning yourself as a manager, goals and planning and leadership. Everyone was in awe and I can attest to the fact deep reflecting occurred following this on the bus that night, in showers and on the bus again the following day.

We all then left the building (by this time just before 10pm) and made our way back, visiting the Like sign at 1 Hacker Way. Many photos were had.
The night ended after bussing back to the hotel and visiting a diner (we were hungry after having our minds blown).

‍