Despite the nasty sounding title, this post will demonstrate an easy method to abstract the paging of an applications URL.
For my particular use-case, I need to retrieve many records from an application called Rapid7 for the purpose of exporting the data into Splunk. To do this successfully, I will need to query 5 or 10 separate tables, and I will need to page the results for each table.
I am sure many of you have written logic like this before, and included the paging logic directly in the method for each table. Myself- I am a huge fan of writing simple, clean abstracted code, which is easy to read… Since, this will be mostly the same logic, over and over… It makes sense
Application Logic
I like to have a strong separation between the “What” the workflow is doing, and “How” it is accomplished. I handle this by using copious amounts of abstraction. Here is the main logic:
public async Task Execute(IJobCancellationToken jobCancellationToken)
{
try
{
//var splunk = log.CreateSplunkTarget(splunk_Index, null, splunk_Source);
int TotalAssetCount = 0;
await foreach (var asset in PageAllResults(GetAsset, jobCancellationToken.ShutdownToken))
{
TotalAssetCount++;
}
int total_Exceptions = 0;
await foreach (var asset in PageAllResults(GetVulnerability_Exceptions, jobCancellationToken.ShutdownToken))
{
total_Exceptions++;
}
//For testing- Output results to console for validation testing.
Console.WriteLine(TotalAssetCount);
Console.WriteLine(total_Exceptions);
}
catch (OperationCanceledException)
{
if (jobCancellationToken.ShutdownToken.IsCancellationRequested)
{
log.Warning("Workflow was cancelled.");
return;
}
throw; //Let the normal exception handler handle this.... since cancellation was not requested.
}
catch (Exception ex)
{
//Log the exception as fatal error.
log.Fatal(ex.GetUsefulDetails());
//Throw the exception to allow the job scheduler to handle the retry logic.
throw;
}
finally
{
//This will invoke the logic to send out error emails to subscribers to this workflow.
log.SendLogSession();
}
}
For this article, it is only currently doing a count of the records returned.
You will notice the IJobCancellationToken. All of my “Automation Jobs” are configured with the idea in mind, that the server, or service, or job may need to be cancelled at any time. If any of those events happens to occur, The cancellationToken is passed down to allow a graceful stop of the application.
The logic executing this code contains the logic to automatically re-enqueue the job on another server when the previous job is cancelled.
How to abstract paging the results of multiple method calls
Next up- we will look at a method whose sole purpose, is to page the results of a method. This method is tailored specifically to this class, and prevents us from repeating the paging logic for each api(over 15 in total).
private async IAsyncEnumerable<T> PageAllResults<T>(Func<int, CancellationToken, Task<Wrapper<T>>> action, [EnumeratorCancellation]CancellationToken ct) where T : class
{
bool HasMoreRecords = true;
int Start = 0;
do
{
var Results = await action.Invoke(Start, ct).ConfigureAwait(false);
foreach (var a in Results.Resources)
{
yield return a;
}
HasMoreRecords = Start < Results.Page.TotalPages;
Start++;
}
while (HasMoreRecords);
}
While the method signature may look scary- It is actually very simple to utilize. This method contains the application API-specific logic to handle paging for all of the GET requests we will be performing.
Methods for specific APIs
For the individual APIs, While I could abstract this logic into a dictionary, I decided to leave everything into its own individual method to allow me to easily document and customize the parameters per call. As well- if I need to intercept the results, and perform modification on the fly, it would allow me to easily contain the logic here. There will be around 15+ of these in total.
private Task<Wrapper<Asset>> GetAsset(int Start, CancellationToken ct)
{
//https://help.rapid7.com/insightvm/en-us/api/index.html#operation/getAssets
var uri = new UriBuilder("https", baseUri, 443, "/api/3/assets");
uri.Query = $"page={Start}&size={50}&sort=idASC";
return QueryRapid7_GET<Asset>(uri.Uri, ct);
}
private Task<Wrapper<Vulnerability_Exception>> GetVulnerability_Exceptions(int Start, CancellationToken ct)
{
//https://help.rapid7.com/insightvm/en-us/api/index.html#operation/getVulnerabilityExceptions
var uri = new UriBuilder("https", baseUri, 443, "/api/3/assets");
uri.Query = $"page={Start}&size={500}&sort=idASC";
return QueryRapid7_GET<Vulnerability_Exception>(uri.Uri, ct);
}
You will notice- The entire purpose of this method, is focused specifically how to obtain the results for the provided API. It does not contain the paging logic, or even the logic on how to perform the GET request. This allows for less code-duplication overall, which leads to a simpler product.
Lastly- In the current implementation of these two tables, it does not use Async. Rather, it passes the Task back to the calling method, without waiting for it to complete.
Lastly- Abstractions for performing the request.
private async Task<Wrapper<T>> QueryRapid7_GET<T>(Uri Uri, CancellationToken ct) where T : class
{
string credentials = Convert.ToBase64String(Encoding.ASCII.GetBytes(creds.UserName + ":" + creds.Password));
var obj = await rest.GetObjectAsync<Wrapper<T>>(Uri, HttpMethod.Get, ct, RequestBody: null, Configure: o =>
{
o.Headers[HttpRequestHeader.Authorization] = $"Basic {credentials}";
}).ConfigureAwait(false);
if (obj.Exception != null)
throw obj.Exception;
return obj.Object;
}
The purpose of this method, is similarly to perform a GET query against the rapid7 API. To perform the query, it only needs to know the Uri, and the credentials (which are set by the constructor.).
Again- there is no logic for creating the URI specific to the endpoint, There is no logic for paging. The only logic contained in the method, is to perform a GET query against the provided URI.
Both error handling and CancellationTokens are built into my RestHelper class. This allows graceful cancellation of the workflows, as well as- allowing me to seperate my logic, from my error handling.
Why Would I do that? That looks complicated!
Because when this workflow is expanded to perform logic on each of the hundred+ APIs offered- The code will remain very readable with minimal repeated logic.
Each method is very specific to its own purpose, with as little overlap as possible. The end result, will be cleaner code, with less wasted room.
Now- if you job rates you based on lines of code written, by all means- Don’t abstract at all. Copy and paste everything!
However- if you like building supportable software, Separating areas of functionality will allow you to build a much more supportable product. As well, it is much easier to write a unit test for your methods, when they are responsible for performing a single task.
