Building Web API integrations that scale (5 key lessons)

I've used the Web API plugin with a wide range of APIs, and each one taught me something new. But before diving into building, I learned to pause and ask: What am I actually trying to display? Not what data the API can give me, but what would be useful on a dashboard?

That shift in thinking — from "fetch everything" to "fetch what matters" — shapes how I approach every integration. It’s really easy to get some data out of an API using a simple GET or PUT request — but if you want data that’s useful on a dashboard, you need to think about how to deal with complex and large datasets. That means considering paging, filtering, simplifying, and summarizing.

Building integrations that scale makes them easier to use for others as well, which is important to consider if you’re thinking about sharing your integration across your business or with the broader community.

Here are the 5 key lessons I learned:

1. It’s a matter of timeframe
2. Not all paths lead to Rome
3. Take it one page at a time
4. Patience has Its time limits
5. The whole is greater than the sum of its parts

1. It’s a matter of timeframe

With that mindset, The first thing I tackled was filtering by time. Instead of fetching everything the API has, I used SquaredUp's dynamic timeframe variables that automatically adjust based on whatever time range the user selects on the dashboard:

{{timeframe.start}}        >> "2025-03-01T00:00:00.000Z"
 {{timeframe.end}}          >> "2025-04-01T00:00:00.000Z"
 {{timeframe.unixStart}}    >> 1709251200
 {{timeframe.unixEnd}}      >> 1711929600

Most APIs are happy with {{timeframe.start}} and {{timeframe.end}}, or the Unix equivalents. But not all.

I once ran into a date format issue that showed me just how flexible these variables can be.

Following the API docs, I added the timeframe variables to my parameters:

created_at[gte] = {{timeframe.start}}
 created_at[lte] = {{timeframe.end}}

But the API had other ideas. I got a 422 error!

Turned out the API only accepts dates in YYYY-MM-DD format - and {{timeframe.start}} returns a full timestamp like 2025-12-02T14:30:00.000Z.

SquaredUp lets you use JavaScript directly inside your parameter values, so formatting the date is straightforward:

{{new Date(timeframe.start).toISOString().split('T')[0]}}
 {{new Date(timeframe.end).toISOString().split('T')[0]}}
 or
 {{new Intl.DateTimeFormat('en-CA').format(new Date(timeframe.start))}}
 {{new Intl.DateTimeFormat('en-CA').format(new Date(timeframe.end))}}

The API also supports date ranges, so my final parameters looked like this:

Parameter name: created_at[date_range]
 Parameter value: {{new Date(timeframe.start).toISOString().split('T')[0] + "~" + new Date(timeframe.end).toISOString().split('T')[0]}}

Now my tile dynamically fetches incidents within whatever time range I want without having to hardcode anything.

2. Not all Paths lead to Rome

Once I had my data filtered by time, the next question was: how do I actually get to the data I need in the API response?

Some APIs make this easy. The response is flat, the data is right there, and you just point to it. Others? Not so much.

The simple ones

Take a straightforward API that returns something like this:

{
  "results": [
    { "id": 109665, "slug": "Sev0", "status": "open" },
    { "id": 126478, "slug": "Sev2", "status": "closed" }
  ]
}

For these, SquaredUp's Path option is all you need. Just tell it where the data lives:

Select data: Path
Path: results

The nested ones

Then there are APIs whose response looks like this:

{
  "data": [
    {
      "id": "4e05f21d-6858-4cee-ac15-474e0bcaabe6",
      "type": "incidents",
      "attributes": {
        "sequential_id": 4,
        "title": "Database latency",
        "status": "closed",
        "severity": {
          "data": {
            "attributes": {
              "name": "Critical",
              "severity": 1
            }
          }
        },
        "user": {
          "data": {
            "attributes": {
              "name": "Noorul Khan",
              "email": "XXXXXXX"
            }
          }
        }
      }
    }
  ]
}

Everything I needed was buried three levels deep in nested objects. Using just a path wouldn't help - I needed to extract and flatten this data.

That's when I used a response script:

This script:

• Drills into the nested structure
• Extracts only the fields I need
• Flattens everything into clean, simple objects
• Makes the data usable in a table or chart

If your API returns flat data, use Path. If it's nested or you need to transform it, use Script. SquaredUp gives you both options.

3. Take it one page at a time

Now that I had filtered my data by time and trimmed it to only the fields I needed, there was one more thing to handle: paging.

Most APIs don't return all your data in one go. They split it into pages - typically 25, 50, or 100 results at a time. If you want everything, you need to tell SquaredUp how to fetch each page automatically.

Configuring paging

SquaredUp supports three common paging patterns. The right one depends on how your API handles pagination:

I was working with an API that only returned 25 results per page. Looking at the JSON response, I saw it included a continuation token:

"pagination_meta": {
    "page_size": 25,
    "after": "01KE9RCFT55VWMKBA390KS78KZ"
}

This is token-based paging - the API tells you where to pick up next. Here's how I configured it in SquaredUp:

Configuration:

• Mode: Token
• Page size: Query parameter → page_size
• Token in response: Body → pagination_meta.after
• Token in request: Query parameter → after

SquaredUp fetches each page, passes the token back to the API, and keeps going until there are no more pages.

Different APIs use different paging patterns - some use page numbers, some return a next URL. Check out the Web API KB for examples of other paging modes.

Here's something I learned: page size matters.

• Too small (like 25): You'll make tons of requests, and SquaredUp might time out before fetching all pages
• Too large (like 1000): The API might take a long to respond, or hit payload size limits

What worked for me: Start with 100 per page. If you're hitting timeouts, try 250 or 500 (check your API's max limit). The sweet spot is fetching as much as possible per page without making the API slow down.

4. Patience has its time limits

Now I had everything configured perfectly. Filtered by time, trimmed the response, paging set up. But I had an API that I worked with which kept timing out with a warning.

I tried everything - reduced the page size, tightened the date range, removed fields. Nothing worked.

So I tested the same query in PowerShell to see what was actually happening:

Received 831 records
 Time taken: 22.8 seconds

22.8 seconds for a single request. That's when I realized the problem wasn't my configuration - the API itself was slow.

For context, other APIs I'd worked with returned similar volumes in 10-12 seconds. This one was taking 2x longer.

SquaredUp has a time limit for fetching all pages to prevent requests from hanging indefinitely. When a single API call takes longer than that limit, you'll hit the timeout - no matter how well you've configured everything. Here's what actually worked:

Filter more aggressively. Instead of fetching a full month, I reduced it to one week. Response time dropped to 6 seconds and everything worked.

Accept the limits. Some APIs are just slow. If you can't make them faster, work within their constraints - fetch less data, update less frequently.

Test outside SquaredUp. If something feels off, test the same query in Postman or PowerShell. It'll tell you if the problem is your configuration or the API itself.

5. The whole is greater than the sum of its parts

Response scripts are great for transforming individual API responses - flattening nested data, extracting fields. But they run per page, not on your full combined dataset. And they can only work with one API at a time.

SQL Analytics runs after all your data is fetched and combined. This is where individual pieces of data become real insights.

Aggregate across all pages: Because paging processes results page-by-page, aggregations at the script stage operate on individual pages. To aggregate across the full combined dataset, use SQL Analytics.

Combine data from multiple APIs: Pull incidents from one API, on-call schedules from another, and join them together to show incidents by the person currently on-call. Or combine deployment data with incident data to see which releases caused issues.

Calculate, filter, transform the complete dataset: Sort across everything, find top/bottom performers, calculate percentages, deduplicate - anything SQL can do, you can do here on your complete, combined dataset.

Give it a shot

The Web API plugin isn't just a fallback for when your datasource isn't listed - it's a powerful and versatile datasource that handles whatever APIs throw at it.

Date formats? JavaScript in parameters handles it.
Nested responses? Response scripts flatten them.
Slow APIs? Test first, filter aggressively.
Multiple data sources? SQL Analytics brings them together.

Now its time to build your own 😉