Infinite Scrolling for a Third Party Website Using Intersection Observer API

Recently, I've picked up treasure hunting on Carousell. With some luck, you might find some really cool stuffs made in a time where products were built to last. That meant tirelessly clicking on "Load more" buttons to dig through hundreds of items. Ultimately, I got lazy enough to decide that I should be able to keep browsing just by scrolling, so here we are.

Overview

The idea is to detect whether the "Load more" button has entered our viewport, and programmatically trigger a click on the button.

We will be building a Chrome extension to augment that behavior, which means our target browser will be the latest version of Chrome (or chromium-based browsers such as Brave). The full code can be found here on Github. Alternatively, you could also create a bookmarklet using the code and activate it everytime you browse the site.

Most libraries detect elements in viewport using window dimensions and scrolltop, probably due to cross browser and fallback support reasons. As we are only using it for Chrome browser, we can use better supported features such as the Intersection Observer API.

We will also be using the MutationObserver interface to listen for DOM changes, as we find out later on that a new "Load more" button renders after more items load in.

Code Breakdown

Let's take a look at the working code and how it works.

window.addEventListener(
  'load',
  () => {
    const observerOptions = {
      root: null,
      rootMargin: '0px',
      threshold: 0,
    }

    observeButton()

    function observeButton() {
      // Check button is in listing page or 'you may like' section
      const buttonElement =
        document.querySelector(
          'main > div > button:last-child'
        ) ??
        document.querySelector(
          '#root > div > div > div > button:last-child'
        )

      if (buttonElement) {
        const observer = new IntersectionObserver((entries) => {
          for (const entry of entries) {
            const { intersectionRatio, isIntersecting, target } = entry
            if (
              intersectionRatio > observerOptions.threshold ||
              isIntersecting
            ) {
              // Trigger click on load more button
              ;(target as HTMLButtonElement).click()

              observer.unobserve(target)
            }
          }
        }, observerOptions)
        observer.observe(buttonElement)

        const mutationObserver = new MutationObserver((mutationList) => {
          // Check if the button was removed/replaced
          const removed = mutationList.some((mutation) =>
            Array.from(mutation.removedNodes).includes(buttonElement)
          )

          if (removed) {
            mutationObserver.disconnect()
            observer.unobserve(buttonElement)
            observeButton()
          }
        })

        mutationObserver.observe(
          document.querySelector('#root'),
          {
            childList: true,
            subtree: true,
          }
        )
      } else {
        // Retry if button has not been mounted yet
        setTimeout(observeButton, 2000)
      }
    }
  },
  false
)

Line 4-6

For clarity purposes, observer options are explicity set even though some of them are set to the default values. Feel free to omit them.

root

Setting this to null defaults the root to browser viewport. Set this to an element you want to define as the viewport, usually the ancestor of the element you are observing

rootMargin

This set of values serves to grow or shrink each side of the root element's bounding box before computing intersections. Set to '0px', which is the default.

threshold

Either a single number or an array of numbers which indicate at what percentage of the target's visibility the observer's callback should be executed.

e.g. 0.5 : When visibility passes 50% of the target visibility, execute callback

e.g. [0.25, 5, 7.5, 1]: Execute callback every 50%

The callback should execute only once when the target enters the viewport, so set the threshold to 0 first. This means when even one pixel of the target enters would trigger the callback.

Line 23-45

One thing to note is that the callback is triggered twice; when the target enters and exit the viewport. To ensure that the we execute our code only on entering, add the following conditional check to the callback:

if (
	intersectionRatio > observerOptions.threshold ||
	isIntersecting
) {
    // ...
    observer.unobserve(target)
}

This ensures that once the intersectionRatio of the target is more then our threshold of 0, the target is unobserved to prevent further triggers.

Line 39-58

The "Load more" button is conditionally re-rendered everytime new items are added to the list (since it should not render if there are no more pages). This poses an issue where the IntersectionObserver would be listening to the previous button instead of the new one.

One way is to use the MutationObserver to track when the button is removed, and then observe the new button.

Line 61

Next, since the rendering of the new button depends on the loading time of the server response, create a polling timer to check the existence of the new button.

That's all, now you can simply scroll to load more items:

Using the Chrome extension

This was made for personal convenience. Feel free to grab the .zip file on the release page of the Github repository if you want.

1. Extract the contents to a directory
2. Go to chrome://extensions on your browser
3. Enable developer mode
4. Click on "Load unpacked" and select the directory in Step 1.
5. Enable the extension

Using as a bookmarklet

Also available as a bookmarklet, try it out dragging this button to your bookmarks bar and click on it when browsing on Carousell.