Thursday, December 21, 2017

Facebook Engineer Gorkana Bjedov : How to Prepare for Technical Interview



p.s. I was attending the Live Video on Facebook and missed on asking a question I had in mind. Hence I followed up with Gorkana on Linkedin with my question and she was most humble to reply back instantaneously. I understand that like me there are many other people who struggle with System Design Questions and hence I am sharing this with all of my readers. I think this is one way I can truly contribute to others life in a meaningful manner and I am happy to do it. This is truly the motive of this blog. If you have your own set of experiences you would like to share, you are welcome contribute as Guest Author.

Followup from LeanIn Facebook Live.

Hello Goranka, I listened to your Facebook live video today on LeanIn Channel. I wanted to ask you a technical question and missed the opportunity. I have been interviewing with companies and the one place I lack is System Design Questions. Due to lack of my experience in this developing systems from scratch, I feel quite lost. A questions as simple as design a highly available and reliable tinyurl.com or design a elevator system for 100 floor building. What would you recommend to me and how can I get good at them?

Response :

Hi Amit - we usually ask detailed systems design questions from more experienced candidates and not fresh graduates, but... Here is what I would do:

1. find resources that discuss design in general - there is, for example, a very good book by a Civil Engineering professor called "Why buildings fall down" that discussed design failures. You will actually learn a lot, even if it looks like this has "nothing to do" with programming.
2. When designing a system, you need to first understand what are the requirements, the risks and trade-offs. LEt's take a look at the highly available and reliable system - there are many of them out there that are described in a great level of detail: Google's search front end (GFE) is probably the best documented one. I would definitely read a paper on its design. Same, for example, for Hadoop and for Facebook photo and video storage. Basically, look at the examples of systems currently designed for those requirements, read about them and understand why they made the decisions they made. 
3. For any other design priority - let's say something that has to be super secure - I would look for examples of products that fulfill that requirement and see how they accomplished it. Ideally, two or three examples should be enough, and you will usually find a lot of software for pretty much any problem in the open source. 


Basically, you prepare for it like you would for the coding problem. You find examples, and you study. Again, in this case, I would say something like one or two study-cases per week, probably not one every day, but also, don't let a week go by without making some progress. Keep notes. Understand underlying issues and common solutions. Pretty soon, you will see patterns emerge and you can then focus on those.

Wednesday, December 20, 2017

Cybersecurity Topics


The cybersecurity domain makes a big use of Hashing Techniques.  Complete listing of the course is here : http://ocw.mit.edu/6-046JS15


Tuesday, December 19, 2017

What is Growth? When does Growth Happens?


I want to talk more about this video on a Philosophical level but for now in the interest of time, I have got this KEXP recording which you might light and also notice how every professional has to go through the deep hole of nothingness to finally find something Great. I really love this.


Monday, December 18, 2017

Where to learn more on Systems topic?


Operating Systems

  • Udacity's : Intro to Operating Systems


  • Operating Systems : The three easy pieces

############################################################

Distributed Systems

  • Cornell University's Distributed Systems Course CS5414

  • University of Waterloo Youtube Series


Distributed System in one Vide



MIT Algorithms OCW



MIT OCW needs no introduction in terms of quality and does a great job with teaching Algorithms Fundamentals. You will feel at home if you have studied Cormen 

An implementation to go along with each of the tutorial will be like Chai with Pakora on a Rainy Day . There's bit of cultural reference here if you know what I mean. :D




13. Graph Theory (Breadth First Search)



14. Graph Theory(Depth First Search)


Coursera Adjacency Matrix : Great quick recap.

GeeksforGeeks DFS and BFS : I found the explanation to be very simple and the Python/C++/Java code is a great for starters.

Network Fundamentals



DNS
  • Akamai Tech Talk : How DNS Protocol Works.
I found this youtube video to be quite exhaustive in the explanation of DNS Protocol. I don't the specifics but I know that the next time I need to do a quick review, I might just come back to this one.


Related to this topic is popular interview question : What happens when you type google.com?

Check out this article on Medium :
https://medium.com/@maneesha.wijesinghe1/what-happens-when-you-type-an-url-in-the-browser-and-press-enter-bb0aa2449c1a


IP Addressing
https://networkengineering.stackexchange.com/questions/7106/how-do-you-calculate-the-prefix-network-subnet-and-host-numbers


Network Virtualization


Network Security


etc..

Sunday, December 17, 2017

System Design : Message Broker




RabbitMQ : A 5 min walk through.

CDN : Content Delivery Network



CloudFlare Free Account Setup with an example site. Having a CDN can improve upon the Speed of Content Delivery.



To check your website performance go to: https://gtmetrix.com/analyze.html

I tried Times Of India website on this site to measure it performance and this is what I got.


Comparatively NYTimes results are as follows:


AWS : This is my Architecture

AWS : This is my Architecture Video Series is on Companies talking about their Architecture and how they have benefited using AWS tech stacks. Here's few I have come across.







High Scalibility Blogs



Plenty of Fish
http://highscalability.com/plentyoffish-architecture

A small calculation on storage I scribbled go like this:

  • Each User has avg 5 pictures of .5 MB each. = 2.5 MB of Data Per User. ~ Not counting Textual Data.
  • That means a 10TB Disk can host 4 million users. I think it is just amazing how storage is cheap. ~ 400$ Amazon Link   . This really opened my eyes when I think of how insignificant hardware cost is when compared to scale of application.

ButterCMS

I think article does a great job in going to details about where they made mistakes and what they did to overcome that. I personally wish to print it out and read it in more depth as I find it very valuable. I guess it is more important to understand one system design better than reading 100's. 

Datbase Clustering Tutorial by Caleb

On my way to further understand how Data Cluster is different from Database Cluster I stumbled upon CalebTheVideoMaker2 youtube channel and I found his video on this topic very useful. You might find it funny and waste of time at times but I think overall he does a great job. Specially with setting up of cluster which is always something i struggle with. I am putting it all here so that you can see how useful it is.

  • Lesson 1 : Intro to Database Clustering


  • Lesson 2 : Terminology


  • Lesson 3 : Setting up a Database Cluster

  • Lesson 4 : Galera Clustering

  • Lesson 5 : Setting up Ubuntu - Part 1



  • Lesson 6 : Setting up Ubuntu - Part 2


  • Lesson 7 : Passwordless SSH




  • Lesson 8 : Installing ClusterControl


  • Lesson 9 : Setting up a Cluster in ClusterControl




  • Lesson 10 : ProxySQL Load Balancer




What is Hadoop


I think this article from Oreilly does a great job in giving a short but sweet journey to this awesome Big Data Architecture. Just to understand what does Hadoop means deserves its own blog. Enjoy reading.

Specially loved the line :
"That’s MapReduce: you map the operation out to all of those servers and then you reduce the results back into a single result set."

Well, I didn't really tell you why I loved the line above. I have always been confused about this fancy word called MapReduce and wondered how does it work? Few days back I read about map-reduce python functions and thought.. "The Hadoop MapReduce wouldn't be far from this one" but then I never went to dig deeper on Hadoop side of story. Now that I read the line from the article, it all makes sense. It means that we pass the functions and data to Hadoop framework and it distributes the work within various nodes (called mapping here) and then collect results from various nodes and aggregates the output (called reducing here). The user of the Hadoop framework doesn't have to do the homework. He/She just need to give the work to Hadoop and wait for results. This is over simplifications of what goes in there but I am writing as to what I am thinking right now. I am sure going to come back and write more as I learn more about it. I like writing down my initial thoughts so that I can see how I have grown from knowing nothing to knowing something about the topic.


Go to the link here:
https://www.oreilly.com/ideas/what-is-hadoop

Saturday, December 16, 2017

SOLID Principle



MVC Pattern

The MVC pattern helps you create applications that separate the different aspects of the application (input logic, business logic, and UI logic), while providing a loose coupling between these elements. The pattern specifies where each kind of logic should be located in the application. The UI logic belongs in the view. Input logic belongs in the controller. Business logic belongs in the model. This separation helps you manage complexity when you build an application, because it enables you to focus on one aspect of the implementation at a time. For example, you can focus on the view without depending on the business logic.
Ref : https://msdn.microsoft.com/en-us/library/dd381412(v=vs.108).aspx

Big Data Architecture Take-ins





Design a Web Crawler

Online Courses : To Do Infinite List


System Design

  • Learn How to Develop Containerized Microservices Scalable Design in this Course.


Status : Not Started.
Thoughts : I really wish to do it. This will help me become better with design and architecture kind of works.

  • Learn about Programming Language Design


Status : Not Started
Thoughts : I have strong interest in learning more about CS Theory. This is one of the topic I don't have much knowledge in.

  • Developing Scalable Apps in Python

Status : Not Started
Thoughts : I think this course can help me walk through the latest System Design Flow in Google Cloud Framework

  • Hadoop

Status : Not Started
Thoughts : An important Big Data Framework to learn.


System Fundamentals

  • Introduction to Operating System


Status : In Progress
Thoughts : This is a great learning for me. I think my background in understanding of OS has always been week and I want to keep learning to make it well understood in my head.

  • Advanced Operating System


Status : Not Started.
Thoughts : Once I am done with Course I, I want to take this Course-II to make sure I have a good depth in OS Fundamentals.


Web Programming


  • Web Programming

https://classroom.udacity.com/courses/cs253

System Design : High Availability & Scalability




LB's can also be used for HA but it is not default setup.



Veritas Scalable System Architecture



Database Clusters with HA & Scalibility


How to choose your Database for HA and Scalability? This talk from Neha is an excellent overview of what goes in choosing a appropriate database for your application. One thing she said that struck me is "Don't start with Scalibility".  She also talks about how NoSQL is not the solution for all your High Scalibility Problems. 

Here's her talk :




A related article based on Neha's talk is from Pinetrust : 


Database Sharding with Django


System Design : Load Balancing


Why Load Balancers?



Load Balancing with NGINX and IP_HASH vs Round-Robin


  • Setup Nginx Webserver (Could have used Apache Webserver also) and Nginx LB Configuration.




Load Balancing with Nginx (Haven't seen this video but with likes I figure this one could be a good one)



System Design : Data Orchestration


A quick overview of what is Zookeeper. I liked the way this video explain important concepts and not get bogged down by details. I will update the details later.


Friday, December 15, 2017

My Frustrations with System Design - Try2

Prologue : The aptness of this title is known to me only after I lost my first draft of this article by accident. Sure enough it is frustrating but let's focus on my deal with Software System Design. The second writeup will not have any dramatic entries I had in my first one but focus on MVP.

The knowledge of Big Data System Design has many components which can be happily group into various categories. My plan is to write one blog per category so that I understand them in more detail. This is to help me store all my learning at one place and help as a quick guidebook when times comes for Job Interviews or Project Execution.

The various categories that come to my mind so far are:-
  1. Data Orchestration : Hbase, Flume, Zookeeper
  2. Search : Solr, Elasticsearch
  3. Data Access :  Pig, Hive, Sqoop
  4. Hadoop : HDFS, MapReduce, Yarn
  5. Spark : Stream Processing 
  6. NoSQL Database : MongoDB, Casandra
  7. Load Balancers : HAProxy, Nginx
  8. Message Broker : RabbitMQ, Kafka
  9. High Availability & Scalability
  10. Container Orchestration Engine : Mesos, Kubernetes or Swarm


Tuesday, December 12, 2017

Page Replacement Algorithm



This is in Hindi Language and surprisingly the best video on this topic. There are many others too but I found this to be most useful.


Monday, December 11, 2017

Where to start Learning Python?

If you have the same question I had few years back on where to start learning Python, I highly recommend going through Youtube tutorials by Corey Schafer. You will absolutely love it and learn the nuts and bolts needed.





Thursday, December 7, 2017

Hadoop Cluster Setup on VirtualBox




---

Best will be I create my own video of how I went about getting a cluster setup done. This will be useful for my reference in future. Hopefully someday soon. :)

Sunday, December 3, 2017

Lets Dock !



Linkedin Docker Course

Docker Cheat Sheet

Building a multi-host, multi-container orchestration and distributed system using Docker


Found this to be very good tour of getting started. Has details about Docker-Compose also.
Edureka : Get started with Docker.


What is Container for God Sake?



Bringing Docker and Django Together


Associated Link : https://bitbucket.org/markadams/pytexas-2015


Dockerizing a Django REST Framework Project



Scalability



https://www.infoq.com/presentations/Development-at-Google

How to build docker cluster with celery and RabbitMQ in 10 minutes


How Uber scaled its Real Time Infrastructure to Trillion events per day




Scaling Pinterest.




Scaling Dropbox




Golang : Oh, the Places You'll Go!


Seems like the right question to ask... Why learn GO?



https://github.com/ethereum/go-ethereum

https://eng.uber.com/go-geofence/

https://www.quora.com/How-do-I-learn-to-write-a-distributed-system-in-Go-language

Google Search Results for Go based Distributed Systems

https://www.goin5minutes.com/


http://www.confusedcoders.com/go/create-a-basic-distributed-system-in-go-lang-part-1

http://www.confusedcoders.com/go/create-a-basic-distributed-system-in-go-lang-part-2-http-server-json-requestresponse

Great talk on Go language and Distributed System Design using Go.



One of the best videos on understanding Go language from its creators.



This one I haven't seen but looks like a good one.


Learn Go

Official Go Guide. I found it succinct and focuses on essentials.
https://tour.golang.org

Tons of good resources.
https://github.com/golang/go/wiki/Learn

Learn X in Y Mins.
https://learnxinyminutes.com/docs/go/

Resources List
https://dave.cheney.net/resources-for-new-go-programmers

Reddit has it.
https://www.reddit.com/r/golang/comments/46f5gi/best_book_to_pick_up_go_for_an_advanced_developer/

What I learned in 2017 Writing Go
https://www.commandercoriander.net/blog/2017/12/31/writing-go/

It's OK to be NOT-OK.

02/22/2021  Last week I was just un-productive and was not doing enough to talk about. I also wanted to say that weeks before the last, I ha...