How to handle pagination when retrieving large sets of data from a REST API in PowerShell?
I'm migrating some code and I'm sure I'm missing something obvious here, but I tried several approaches but none seem to work... I'm trying to retrieve a large dataset from a REST API that paginates its responses. The API uses standard pagination parameters like `page` and `pageSize`, and I'm working with issues with looping through the pages until I collect all the data. I've set up a script, but I'm not sure if I'm correctly incrementing the page number or managing the response data properly. The API returns a maximum of 100 records per page, and I'm running into issues when the results come back with an empty array, which indicates there are no more pages. Here’s the code snippet I've written so far: ```powershell $baseUrl = "https://api.example.com/data" $page = 1 $pageSize = 100 $data = @() while ($true) { $response = Invoke-RestMethod -Uri "$baseUrl?page=$page&pageSize=$pageSize" -Method Get if ($response.data.Count -eq 0) { break } $data += $response.data $page++ } ``` However, I'm receiving an behavior when I access `$response.data.Count`. The behavior message is: `Property 'data' want to be found on this object; make sure it exists and is set.` This indicates that the response structure might not be consistent. I've checked the API documentation, and it states that the structure should be consistent, but I'm wondering if there's something I missed. Is there a better way to handle the response or any best practices for managing this kind of pagination scenario in PowerShell? Any advice or improvements to my script would be greatly appreciated! My development environment is Windows. Is there a better approach? What am I doing wrong? The project is a application built with Powershell. Any suggestions would be helpful. The project is a application built with Powershell. What would be the recommended way to handle this?