Setting Up Algolia Search for a Next.js Application

Introduction to Algolia

To get started with Algolia, you first need to create an account. This process is quick and easy. Once you've created an account, you can access Algolia's powerful features, including lightning-fast search and advanced analytics. Additionally, Algolia offers a variety of customization options that allow you to tailor the search experience to your specific needs.

Creating an Index

You will need to create an index to add your documents to. You may have one or multiple indexes depending on your application's needs. Different indexes for different environments may exist, such as Development, UAT, and Production. This concept could be used to separate content types or different language versions.

1. Create a new Index



2. Give your index and name in the pop-up window and click Create.

Creating an API Key

You will need to create a new API Key for your front-end application with the permissions to add new objects to the index.

1. First, find the Settings icon on the bottom left-hand corner of the dashboard.



2. Then click on API Keys on the screen that appears after.



3. Create a new API Key with the required permissions as we have done in our screenshot. You can also link it to particular indexes under the Indices field.

Connecting Algolia to Your Next.js Application

Create a Next.js application and install the necessary modules.

npm install algoliasearch cheerio node-fetch

Set up an .env file on the root of your solution to hold your Algolia variables. The ALGOLIA_SEARCH_API_KEY will be equal to the API key we just created above.

ALGOLIA_APP_ID = 'ALGOLIA_APP_ID';
ALGOLIA_SEARCH_API_KEY = 'ALGOLIA_SEARCH_API_KEY';
ALGOLIA_INDEX_NAME = 'ALGOLIA_INDEX_NAME';

Adding Documents to Algolia

First, we must define all the fields we want to see in our index. Then, create a type and store it in a separate file for application-wide referencing.

export type PageResult = {
    title: string,
    content: string,
    url: string,
    objectID: string
}

Here we have defined a simple endpoint that takes in a URL and passes it to a function that indexes the page. This file lives under pages/api. It is called crawler.ts.

import type { NextApiRequest, NextApiResponse } from 'next';
import { crawlPageAndIndex } from '@/util/crawlIndex';

export default async function handler(req: NextApiRequest, res: NextApiResponse) {
    const { url } = req.query as { url: string };

  if (!url) {
    res.status(400).send('Missing URL parameter');
    return;
  }

  await crawlPageAndIndex(url);

  res.status(200).send('OK');
}

We can hit this endpoint by going to the following URL.

https://<APPLICATION URL>/api/crawler?url=<URL TO CRAWL>

The following code was added to a subfolder called /util under the root of our application. We have included examples of how to fill the object fields we have defined above. However, how this happens in a production-ready application will likely need to be driven by the business requirements.

import cheerio from 'cheerio';
import fetch from 'node-fetch';
import algoliasearch from 'algoliasearch';
import { PageResult } from './types';

const APP_ID = process.env.ALGOLIA_APP_ID ?? '';
const API_KEY = process.env.ALGOLIA_SEARCH_API_KEY  ?? '';
const INDEX = process.env.ALGOLIA_INDEX_NAME ?? '';

export async function crawlPageAndIndex(url: string) {
  const content = await crawlPage(url);

  await addToAlgolia(url, content);
}

async function crawlPage(url: string) {
  const response = await fetch(url);
  const html = await response.text();
  const c$ = cheerio.load(html);

    // Use the page h1 tag as our title
  const title = c$('h1').map((i, el) => c$(el).text()).get();
    // Get all header content
  const headings = c$('h1, h2, h3, h4, h5, h6').map((i, el) => c$(el).text()).get();
    // Get all page paragraphs
  const paragraphs = c$('p').map((i, el) => c$(el).text()).get();
    // Combine headers and paragraphs and trim content to avoid going over the allowed field size
  const content = [headings.join(' '), paragraphs.join(' ').slice(0,500)].filter(s => s.trim().length > 0).join(' ');

  const page = {
    objectID: url,
    title: title[0],
    url: url,
    content: content,
  } as PageResult;

  return page;
}

async function addToAlgolia(url: string, page: PageResult) {
    const client = algoliasearch(APP_ID, API_KEY);

    const index = client.initIndex(INDEX);

    await (index as any).saveObject(page);
}

Note: We need to create a unique objectID to avoid duplicates and also to be able to update existing records. In this screenshot, we have used the URL but this could be any unique identifier.

Displaying and Searching Our Algolia Index

Algolia has made searching for our content very simple with it’s InstantSearch library of components. In our example, we have created a simple text based search component using these components.

import { InstantSearch, SearchBox, Hits, InstantSearchProps } from 'react-instantsearch-dom';
import algoliasearch from 'algoliasearch';
import { useState } from 'react';
import styles from './Search.module.css';

const ALGOLIA_APP_ID = process.env.ALGOLIA_APP_ID ?? '';
const ALGOLIA_SEARCH_API_KEY = process.env.ALGOLIA_SEARCH_API_KEY  ?? '';
const ALGOLIA_INDEX_NAME = process.env.ALGOLIA_INDEX_NAME ?? '';

const algoliaClient = algoliasearch(ALGOLIA_APP_ID, ALGOLIA_SEARCH_API_KEY);

function Search() {
  const [query, setQuery] = useState('');

  const handleSearch = (event: React.SyntheticEvent<HTMLInputElement, Event>) => {
    setQuery(event.currentTarget.value);
  };

  return (
        // Pass our index name and the search client into the InstantSearch components parameters
    <InstantSearch indexName={ALGOLIA_INDEX_NAME} searchClient={algoliaClient}>
      <div className={styles.input}>
        <SearchBox onChange={handleSearch} />
      </div>

      {query && (
                // This component displays all of the search results
        <div className={styles.results}>
          <Hits hitComponent={Hit} />
        </div>
      )}
    </InstantSearch>
  );
}
// Define our result template here
function Hit({ hit }: { hit: any }) {
  return (
    <div className={styles.hit}>
      <a href={hit.url}>{hit.title}</a>
      <p>{hit.content}</p>
    </div>
  );
}

export default Search;

Resolving Errors in Algolia by Viewing the Log

The first place to start when you run into errors when attempting to index content is in the logs. To view the logs:

1. Click on API Monitoring in the left-side navigation panel.



2. Click on Search API Logs in the panel that appears afterwards.



3. You can further narrow down your search by using the filter in the top right corner of the log results listing.

Some Common Algolia Indexing Errors

1. Duplicate objectIDs: Algolia requires each object in an index to have a unique objectID. If two or more objects have the same objectID, only one of them will be added to the index. Check to see if any of the pages in your sitemap share a URL path, and if so, make sure that your objectID field is set to a unique value for each page.
2. Indexing limits: Algolia has indexing limits that vary depending on your plan. If you've exceeded your indexing limit, only a portion of the pages in your sitemap will be added to your index. Check your Algolia dashboard to see if you've exceeded your indexing limit, and consider upgrading your plan if necessary.
3. Indexing rate limits: Algolia has rate limits on indexing requests to prevent abuse. If you're making too many indexing requests too quickly, some of your requests may be throttled. Check your Algolia dashboard to see if there are any rate limit errors or warnings.
4. Data formatting issues: Make sure that the data you're sending to Algolia is formatted correctly and matches the schema of your index. If there are data formatting issues, Algolia may reject some or all of the indexing requests.
5. Delay between indexing requests: To avoid triggering rate limit errors, make sure that there is a delay between indexing requests. Algolia recommends waiting at least 100ms between requests to avoid overloading their servers.
6. Field size is too big: The maximum size for a field is 10000 bytes. Hence why in the above example the Content fields value is trimmed to max 500 characters (this can be bigger, however, as long as it stays below 10000 bytes). Click here for more information on this.

---