News Feed is the constantly updating list of stories in the middle of your home page. It includes status updates, photos, videos, links, app activity, and likes from people, pages, and groups that you follow.
There are two main components in the news feed,
- Publishing the posts
- Retrieving the posts
Let’s first concentrate on publishing the post.
Publishing a Post
It is pretty straightforward. You log in to the website/app. Under the post, you write the text that you want to post and hit the submit button. As soon as you click on the submit button, the request sends to the server where it first gets authenticated. The server checks whether the user who has to send the request is a valid user or not. The load balancer is used to distribute the traffic to web servers. The web server sends the request data to the post publisher where it gets saved into the database as well as in the post cache for fast retrieval.
Once the post has been saved into the database successfully. Now it’s time to fan out the post.
Fanout
The process of pushing a post to all the followers is called fanout.
There are two ways we can Fanout a post,
Fanout Read
Fanout Write
Fanout Read
When you request for news feed, you create a read request to the system. With fanout read, the read request is fanned out to all your followers to read their posts. The feed is created during the read time. Usually, it pulls recent posts of a user.
Pros
No need to pre-compute the posts for in-active users.
The cost of the write operations is low.
Cons
The time taken to load the feed is more as compare to the Fanout Write.
It can take a lot of time if a user has many followers.
You cannot update the user feed on a real-time basis. The user has to request the feeds in case he/she wants to see the latest posts.
Fanout Write
In this case, as soon as the post is published, it gets fanned out to all the friends immediately. Most of the work is done at the time of writing to figure out who will get this post. This significantly reduces the read operations on the system. This approach is not recommended if a user has a large number of followers.
Pros
The cost of the read operation is low.
Feed is generated in real-time and pushed to all the friends.
Displaying the feed is faster as compare to the fanout read because it’s pre-computed.
Cons
It can take significant time if a user has too many friends.
Hybrid
In this approach, we will take the benefits from both the above approaches (Fanout Read and Write). Allowing the users with a less number of followers to use the Fanout Write model whereas the users with a higher number of followers, a Fanout Read model will be used.
Retrieval
When a user sends a request to retrieve a news feed, it lands on the load balancer which distributes the request to different web servers. The news feed service gets the request from the web servers to fetch the news feed from the cache. The news feed cache returned the post ids for the specific user. User cache returns the user-related info and post cache returns the post-related info. Once everything is computed, we send back the response to the web servers that further deliver the content to the user.
Improvement
Scale the System
As the number of users and posts increases, the current system might now be able to handle the load. So we need to scale the system horizontally. There are different ways we can scale the system. We can use Master-Slave System to separate the read and write operations on the system. We can also use Sharding on the data. We can shard the system based on the feed data and posts. The other improvement we can do is to limit the number of requests that we are getting by implementing the Rate Limiting Module at the webserver level.
European best, cheap and reliable ASP.NET hosting with instant activation. HostForLIFE.eu is #1 Recommended Windows and ASP.NET hosting in European Continent. With 99.99% Uptime Guaranteed of Relibility, Stability and Performace. HostForLIFE.eu security team is constantly monitoring the entire network for unusual behaviour. We deliver hosting solution including Shared hosting, Cloud hosting, Reseller hosting, Dedicated Servers, and IT as Service for companies of all size.
