Making Browser Extension Smart By Supporting SPA websites
Browser Extension is a way to extend the nature of a webpage by modifying, adding, deleting, or/and by any other means. Think of it as a sword anyone can use to tweak the web the way they want for themselves. For example, you can get rid of annoying advertisements on a page, style the page you desire it to look like, block third-party libraries from stealing your private data and history, automate mundane work or anything you wish you can do with DOM and manipulate how front-end works with limited secured ways.
Content Scripts
Browsers provide an interface to read the details of a web page through a file, known as content-script, and DOM API to make changes. Each content-script has it’s own isolated environment.
As per the docs:
Content scripts live in an isolated world, allowing a content script to makes changes to its JavaScript environment without conflicting with the page or additional content scripts.
Read more here.
What is a Single Page Application (SPA)?
As the name suggests, there’s one single page for a particular website, and the same page gets re-rendered on page-URL change. All the critical resources, along with the content for the requested page, are downloaded before DOM construction. On viewing other pages, only the resources required to display them are dynamically downloaded and rendered on the same page. There’s no full reload of the website unless forcefully done. There are tons of resources available on the internet related to SPA pages to read.
We use a lot of SPA examples in our daily life like Gmail, Facebook, Twitter, Medium, etc. GitHub is also one of them which gradually moved to SPA entirely.
How SPA behaves differently from multiple-pages?
I can use content-script to make changes on a particular page. But the next question arises: When? When should we start modifying the page?
For non-SPA pages and for the pages where content is not loaded dynamically after page load, we can simply do our stuff after the DOMContentLoaded event is fired.
Being said so, you can now correlate the fact that for SPA pages, the DOM load event fires just once i.e. when the page is refreshed. All other times, data is fetched from the server using asynchronous AJAX calls. The callback executes once they are successful. It causes a problem for the content-script to determine when a new page loads. That’s where the content-script starts disappointing. Many hacks come into a cunning mind, but they too get failed at one or the other place.
So, how do we know that a new page has loaded without the whole website getting refreshed? That’s the “hook” I will be discussing in the next sections.
Before we come to the solution, I would like you to introduce to the Chrome Extension I developed and the exact problem I faced.
Enhanced-GitHub
It is a Chrome Extension that offers many benefits on the GitHub website, which are otherwise absent. Benefits like being able to copy contents of any file with just a single click, able to see the filesize before even opening it, download a file, directly access the raw link of a file, and view the size of the entire repository.
Link: enhanced-github
PS: The extension has 20k+ active users currently 😍
Below are the screenshots highlighting what changes would you observe on the GitHub website after installing it.
Display repo size, size of each file, raw link, and the option to download a file directly.
Copy button to copy the contents of the entire file to the clipboard.
Download the file directly to your system.
Everything worked perfectly fine on GitHub when it was a non-SPA website.
After the introduction of SPA on the GitHub website, users of the extension started facing issues while navigating back and forth on the website. On the initial page-load, everything seems to work as intended. But clicking on any hyperlink no longer was reloading the browser tab. Instead, data was fetched using AJAX calls and rendered by replacing the same page with the new content. That means neither DOM load event would fire nor the content-script would be aware of the page change. This is where my tech-mind started exploring ways to tackle it.
Approaches to tackling SPA behavior
Approach #1
The first thing that comes in mind is to find out when does page-URL changes and immediately tell the content-script to behave as it would on page reload. The drawback of this approach is when page-URL changes, it doesn’t guarantee whether the page has the new content available instantly. A lot of websites update the page-URL first, which in turn fires the code responsible for fetching data according to the updated page-URL. The time required to download resources will vary and will most likely be ahead in time than of content-script execution time. It makes content-script execution prone to errors as the latest DOM has still not constructed. Certain required elements would not be present, and some stale selectors would result in false-positive results.
Hence, discarded.
Approach #2
Should I add a timeout and then invoke content-script execution? Again, the drawback is the exact duration of fetching resources is not known, which would cause errors.
Hence, discarded.
Approach #3
Should I add an interval after getting a page-URL change event that checks whether the required element is visible on the page? It might seem like the perfect time to execute content-script, but it still not the right way.
Consider a case where your content-script adds an element to the DOM on every page. Initially, it works correctly. But as soon as the page-URL change event is triggered and we check for the visibility, we will again be fooled by the stale DOM content. New content has yet not fetched, whereas the old DOM is still on the page.
Hence, discarded.
The above approaches taught me that these never lead to a perfect all-in-one solution.
The only way is to know the hook when the page-URL has changed, and the page has successfully re-rendered. Reading the docs related to webRequest API, I knew I had found the sword now.
Background Script and hooking with webRequest and webNavigation APIs
The background script is a program running in the background to monitor events that a browser triggers, such as navigating to a new page, closing a page, etc.
webRequest API — Read the docs
<manifest.json>
"background": {
"scripts": ["background.js"],
"persistent": true
}As per docs:
The only occasion to keep a background script persistently active is if the extension uses chrome.webRequest API to block or modify network requests. The webRequest API is incompatible with non-persistent background pages.This API assists in observing and analyzing the traffic. One can spy on every single call that’s requested.
Using webNavigation API — Read the docs
Firstly, I had to add it to the list of permissions since any API prefixed with chrome.* has to be granted permission by the user to use it.
<manifest.json>
"permissions": ["*://*.github.com/*", "storage", "webRequest", "webNavigation"]<background.js>
chrome.webNavigation.onHistoryStateUpdated.addListener(details => {
tabId = details.tabId;
currentUrl = details.url;
// ...
});We are now sure that whenever the page-URL changes, we will always be notified by the above code and has the latest page-URL.
Using the onHistoryStateUpdated event, we can spy on every page-URL change event. This event also provides the tab-id and the new page-URL. Tab-id is required to communicate with the content-script. We will be discussing this in the next paragraph.
Using webRequest API — Read the docs
The only thing I needed to know was which call(s) are responsible for rendering the data on the GitHub website.
I was able to identify that GitHub has implemented server-side rendering. I was more than lucky to figure out that there is a single request to spy. I mentioned webRequest API earlier, which helps in analyzing the requests. So, I used the onCompleted event provided by webRequest API to mark the call has responded successfully.
I had to add it to the list of permissions since any API prefixed with chrome.* has to be granted permission by the user to use it.
<manifest.json>
"permissions": ["*://*.github.com/*", "storage", "webRequest"]<background.js>
chrome.webRequest.onCompleted.addListener(function(details) {
const parsedUrl = new URL(details.url);// TODO: filter and check if the desired URL has completed}, { urls: ['*://*.github.com/*'] });
The above code will spy on every single request. The aim is to know which request is responsible for fetching and re-rendering the page. Once we know, it has to be communicated to the content-script for its execution.
Moving on to the next section, I’ll tell how to determine the time to fire the content-script execution.
Communicating between background and content-script
Knowing when page-URL changes and the corresponding request completes, I simply had to convey the message to the content-script to behave as if the tab has refreshed with the new page-URL.
<background.js>
chrome.webRequest.onCompleted.addListener(function(details) {
const parsedUrl = new URL(details.url);if (currentUrl && currentUrl.indexOf(parsedUrl.pathname) > -1 && tabId) {
chrome.tabs.sendMessage(tabId, { type: 'page-rendered'});
}
}, { urls: ['*://*.github.com/*'] });
As I mentioned earlier, I was able to figure out the call which pulls data from the GitHub server and page gets re-rendered once the content has downloaded.
The above code filters the call which is responsible for fetching the data for the new page and convey the same to the content-script.
<content-script.js>
Listening to the messages being sent by background script to the content-script is easy and can be achieved by attaching an event listener.
chrome.runtime.onMessage.addListener(function(request) {
if (request && request.type === 'page-rendered') {
// call method which gets fired as if new page is opened
}
});Once a message is received by the content-script, I just need to invoke the function responsible for fetching and displaying the data.
Source Code — GitHub
References
- Chrome Extension — https://chrome.google.com/webstore/detail/enhanced-github/anlikcnbgdeidpacdbdljnabclhahhmd?hl=en
- Firefox Add-on — https://addons.mozilla.org/en-US/firefox/addon/enhanced-github/
- Microsoft Edge Add-on — https://microsoftedge.microsoft.com/addons/detail/enhanced-github/eibibhailjcnbpjmemmcaakcookdleon
- Source Code — https://github.com/softvar/enhanced-github
Other
- Chrome Extension content script — https://developer.chrome.com/extensions/content_scripts
- Chrome Extension background script — https://developer.chrome.com/extensions/background_pages
- webRequest API — https://developer.chrome.com/extensions/webRequest
- webNavigation API — https://developer.chrome.com/extensions/webNavigation
I hope you might have learned something new in this post. For any queries, drop me words at varun2902@gmail.com.
Please recommend, tweet and share to boost my confidence for more write-ups. Thanks a lot!
Would love to be connected via the following platforms:
GitHub | Twitter | LinkedIn | StackOverflow
