How to Handle Large Datasets in Frontend Applications

Originally posted on https://niteshseram.in.

Handling large datasets is a common challenge in frontend applications. As the amount of data grows, it can lead to performance issues, such as slow loading times and unresponsive user interfaces. In this blog, we will explore different methods to effectively handle large datasets in React applications. We will discuss techniques like pagination, infinite scroll, and windowing. By implementing these strategies, we can ensure that our frontend application remains fast and efficient, even when dealing with large amounts of data.

Understanding the Performance Problems with Large Datasets

Before we dive into the different methods of handling large datasets, let's first understand the performance problems associated with them. When an application tries to render or manipulate a large amount of data in a list, it can cause significant performance issues. This is because rendering a large number of DOM elements can be time-consuming and resource-intensive.

To illustrate this, let's create a sample React application that renders a list of 10,000 records. By examining the performance of this sample application, we can better understand the challenges of handling large datasets.

To get started, create a new React application using the create-react-app command in your terminal:

npx create-react-app large-dataset-app

Once installed, open the App.js file in the src directory and replace the existing code with the following:

const data = new Array(10000).fill().map((_, index) => ({
  id: index,
  name: `Temp Name ${index}`,
  email: `Temp Email ${index}`,
}));

function App() {
  return (
    <div>
      {data.map((item) => (
        <div key={item.id}>
          <h3>{item.name}</h3>
          <p>{item.email}</p>
        </div>
      ))}
    </div>
  );
}

export default App;

In this code, we generate an array of 10,000 objects, where each object represents a record in our dataset. We then use the map function to render each item in the array as a <div> element. Each <div> contains the name and email of the corresponding item.

Now, start the React application by running the following command in your terminal:

npm start

Open your browser and navigate to http://localhost:3000. You will notice that it takes some time for the page to load, and scrolling through the list may also be slow. This is because rendering 10,000 DOM elements at once can cause performance issues.

Pagination: Rendering Data in Pages

One way to handle large datasets is by implementing pagination. Pagination allows you to render data in pages, rather than all at once. By controlling the amount of data shown on the page, you can reduce the stress on the DOM tree and improve performance.

There are several UI libraries in React that provide pagination components, such as react-paginate. However, if you prefer not to use a UI library, you can implement pagination manually.

To illustrate this, let's modify our sample application to include pagination. First, install the react-paginate library by running the following command:

npm i react-paginate

Next, open the App.js file and replace the existing code with the following:

import { useState } from 'react';
import ReactPaginate from 'react-paginate';

const data = new Array(10000).fill().map((_, index) => ({
  id: index,
  name: `Temp Name ${index}`,
  email: `Temp Email ${index}`,
}));

function App() {
  const [currentPage, setCurrentPage] = useState(0);
  const itemsPerPage = 10;
  const pageCount = Math.ceil(data.length / itemsPerPage);
  const offset = currentPage * itemsPerPage;
  const currentData = data.slice(offset, offset + itemsPerPage);

  const handlePageChange = (selectedPage) => {
    setCurrentPage(selectedPage.selected);
  };

  return (
    <div>
      {currentData.map((item) => (
        <div key={item.id}>
          <h3>{item.name}</h3>
          <p>{item.email}</p>
        </div>
      ))}
      <ReactPaginate
        previousLabel={'Previous'}
        nextLabel={'Next'}
        breakLabel={'...'}
        pageCount={pageCount}
        marginPagesDisplayed={2}
        pageRangeDisplayed={5}
        onPageChange={handlePageChange}
        containerClassName={'pagination'}
        activeClassName={'active'}
      />
    </div>
  );
}

export default App;

In this code, we use the useState hook to manage the current page state. We calculate the number of pages based on the total number of records and the desired number of items per page. We then use the slice method to get the current data to be displayed on the page.

The ReactPaginate component renders a pagination UI with previous and next buttons, as well as page numbers. The onPageChange event handler updates the current page state when the user clicks on a page number.

Now, when you run the application, you will see that the data is rendered in pages, with only a subset of records shown at a time. This helps to improve the performance of the application by reducing the number of rendered DOM elements.

Infinite Scroll: Loading Data on Demand

Another approach to handling large datasets is through the infinite scroll. Infinite scroll involves loading data incrementally as the user scrolls down the page. Initially, only a subset of data is loaded, and more data is appended as the user reaches the end of the list.

There are various ways to implement infinite scroll in React, and one popular library for this purpose is react-infinite-scroll-component. To use this library, install it by running the following command:

npm i react-infinite-scroll-component

Next, open the App.js file and replace the existing code with the following:

import { useState } from 'react';
import InfiniteScroll from 'react-infinite-scroll-component';

const data = new Array(10000).fill().map((_, index) => ({
  id: index,
  name: `Temp Name ${index}`,
  email: `Temp Email ${index}`,
}));

function App() {
  const [items, setItems] = useState(data.slice(0, 20));

  const fetchMoreData = () => {
    setTimeout(() => {
      setItems((prevItems) => [
        ...prevItems,
        ...data.slice(prevItems.length, prevItems.length + 20),
      ]);
    }, 1500);
  };

  return (
    <InfiniteScroll
      dataLength={items.length}
      next={fetchMoreData}
      hasMore={items.length < data.length}
      loader={<h4>Loading...</h4>}>
      {items.map((item) => (
        <div key={item.id}>
          <h3>{item.name}</h3>
          <p>{item.email}</p>
        </div>
      ))}
    </InfiniteScroll>
  );
}

export default App;

In this code, we use the useState hook to manage the item's state. Initially, we load the first 20 items from our dataset. The fetchMoreData function is called when the user scrolls to the end of the list. It appends the next 20 items to the existing items using the spread operator.

The InfiniteScroll component from react-infinite-scroll-component wraps the list of items. It takes the current length of the items as the dataLength prop, the fetchMoreData function as the next prop, and a boolean value to indicate whether there is more data to be loaded.

When you run the application, you will notice that the data is loaded incrementally as you scroll down the page. This approach improves the user experience by providing a seamless scrolling experience while efficiently loading and rendering the data.

Windowing: Efficiently Rendering Large Lists

Another technique for handling large datasets is windowing. Windowing involves rendering only the visible portion of a list to the screen, rather than rendering all the items at once. This helps to reduce the number of DOM elements and improves performance.

One popular library for windowing in React is react-window. It provides a set of components for efficiently rendering large lists. To use react-window, install it by running the following command:

npm i react-window

Next, open the App.js file and replace the existing code with the following:

import { FixedSizeList as List } from 'react-window';

const data = new Array(10000).fill().map((_, index) => ({
  id: index,
  name: `Temp Name ${index}`,
  email: `Temp Email ${index}`,
}));

const Row = ({ index, style }) => (
  <div style={style}>
    <h3>{data[index].name}</h3>
    <p>{data[index].email}</p>
  </div>
);

function App() {
  return (
    <List height={400} itemCount={data.length} itemSize={80} width={300}>
      {Row}
    </List>
  );
}

export default App;

In this code, we define a Row component that renders each item in the list. The FixedSizeList component from react-window is used to render the list. It takes the height and width of the list, the total number of items, and the size of each item as props.

When you run the application, you will see that only a portion of the list is rendered at a time, based on the height of the list. As you scroll through the list, the windowing technique efficiently renders only the visible items, resulting in improved performance.

You might be wondering what’s the difference between what the react-infinite-scroll-component and react-window do. The difference is that in react-infinite-scroll-component load data incrementally as the user scrolls. It dynamically adds more items to the list as needed, creating an illusion of infinite content. react-window, on the other hand, renders only a subset of the list items that are currently visible in the viewport, reusing DOM elements as the user scrolls.

Due to its simpler API and automatic handling of scrolling, react-infinite-scroll-component may be easier to set up and use for basic infinite scrolling needs. However, it may not perform as well with extremely large data sets or complex list items since it keeps all rendered elements in the DOM. In contrast, react-window's windowing technique ensures that only the visible items are rendered, resulting in improved performance and reduced memory footprint for large lists.

Conclusion

Handling large datasets in frontend applications can be challenging, but there are various techniques available to address this issue. By implementing pagination, infinite scroll, windowing, or using specialized libraries like react-virtualized or react-window, you can effectively manage large amounts of data while maintaining optimal performance.

In this blog, we explored different methods of handling large datasets in React applications. We discussed pagination as a way to render data in pages, infinite scroll for loading data on demand, windowing for efficiently rendering large lists, and libraries like react-window that provide additional features for handling large datasets.

Remember to consider the specific requirements and constraints of your application when choosing a method for handling large datasets. Each approach has its advantages and trade-offs, so it's important to evaluate which technique best suits your use case.

By implementing these strategies, you can ensure that your frontend applications remain fast, responsive, and user-friendly, even when dealing with large amounts of data.