I wanted to build a simple page, which would allow our NOC to trigger certain workflows, automatically (assuming they have proper permission)

My requirements:

1. Jobs should be easily added to the list, just by specifying an Attribute, and Implementing an interface (to ensure the proper method exists)
2. Jobs needs to be kicked off on my back-end servers and not on the web-front end.
3. Everything needs to be properly logged.
4. The front-end needs visual-feedback a job has been executed.

Here is the web page to trigger jobs.

As you can see, the above page is very simple.

In the above image, if a job fails, it will turn red, with direction to call the developers (me).

How it works (Short Version)

At startup, Reflection is used to build a dictionary containing types which implement IRemoteJob interface, and the NocJob attribute.

A webpage displays the values. Permissions are implemented at the controller level.

When a job is triggered, the Type’s full name is passed back to the method. The method will compare the provided name, against the dictionary of valid workflows. If the workflow is found, and verification passes, Hangfire.Io is then used to execute the workflow on my backend- automation services away from the web front end.

Since- we cannot execute a string containing the name of a type… We utilize Ninject’s kernel to resolve a instance of the class. We then Invoke the execute method, which is defined by the IRemoteJob interface, on the class instance.

Error handling is handed by Hangfire.

In short- that is it.

How it works (Long Version)

RuntimeTypeResolver.cs // IRunTimeTypeResolver

    public class RuntimeTypeResolver : IRuntimeTypeResolver
    {
        private IKernel kernel;
        public RuntimeTypeResolver(IKernel Kernel)
        {
            this.kernel = Kernel;
        }
        public object GetInstanceOfType(Type type)
        {
            return kernel.Get(type);
        }
    }

This class, resolves a class instance from the Ninject kernel and returns it.

JobHelper.cs

//Note - Singleton class.
    public class JobHelper
    {
        //IRunTimeTypeResolver passes a Type into a dependancy injection kernel. The kernel will return a instance of the Type.
        private IRuntimeTypeResolver rtr;
        public JobHelper(IJobScheduler JOB, IRuntimeTypeResolver RTR)
        {
            this.rtr = RTR;
            //Note- Dictionary only needs to be populated at startup.

            //Create a dictionary containing all eligible classes to be executed ad-hoc.
            RemoteJobs = this.GetType() //Types are filtered to this current assembly.
                .Assembly
                .GetTypes()
                //Type/Class implements IRemoteJob interface, and is NOT an interface or abstract class.
                .Where(x => (typeof(IRemoteJob).IsAssignableFrom(x) && !x.IsInterface && !x.IsAbstract))
                //Class has the NocJobAttribute, which contains display information.
                .Where(x => Attribute.IsDefined(x, typeof(NocJobAttribute)))
                //Cast the types, and create a dictionary.
                .ToDictionary(o => (NocJobAttribute)Attribute.GetCustomAttribute(o, typeof(NocJobAttribute)), o => o);
        }

        //A read-only dictionary containing the information. 
        public IReadOnlyDictionary<NocJobAttribute, Type> RemoteJobs { get; }

        /// <summary>
        /// This is intended as an entry point for background jobs. 
        /// </summary>
        /// <param name="JobName"></param>
        public void EnqueueRemoteJob(string JobName)
        {
            //Get the Type from the list.
            Type Rec = RemoteJobs
               .First(o => o.Value.FullName.Equals(JobName, StringComparison.OrdinalIgnoreCase))
               .Value;

            //Get the Execute method. It should exist, because all items in this list, implement IRemoteJob.
            MethodInfo ExecuteMethod = Rec.GetRuntimeMethod("Execute", new Type[] { });

            //Get a instance of the type, from the RuntimeTypeResolver class.
            object Instance = rtr.GetInstanceOfType(Rec);

            //Invoke the Execute Method on the class instance.
            ExecuteMethod.Invoke(Instance, null);
        }
    }

The primary purpose of this job, is to both hold a list of “valid” workflows which can be executed, and to expose a method which the Hangfire.io can trigger with the Type’s name, to invoke the workflow.

JobController.cs – Trigger_NOC_Job method

        [HttpPost]
        [CheckPermissionAttribute(Permission.Trigger_AutomationJobs)]
        public ActionResult Trigger_NOC_Job(string JobType)
        {
            log.Component = "Trigger";
            var L = new JobTriggerLog
            {
                JobName = new string[] { JobType },
                Time = DateTime.Now,
                UserID = User.Identity.Name,
                UserName = User.GetFullName()
            };
            try
            {
                //now the FUN part...
                if (!jobHelper.RemoteJobs.Any(o => o.Value.FullName.Equals(JobType, StringComparison.OrdinalIgnoreCase)))
                {
                    L.Error = new Exception("Job was not located.").GetUsefulDetails(false);
                    log.Error(L);
                    return new HttpStatusCodeResult(System.Net.HttpStatusCode.NotFound, "Job was not found.");
                }

                //Get the Type from the list.
                Type Rec = jobHelper
                   .RemoteJobs
                   .First(o => o.Value.FullName.Equals(JobType, StringComparison.OrdinalIgnoreCase))
                   .Value;

                //Sanity check to double-check the Execute method exists, without required parameters.
                if (Rec.GetRuntimeMethod("Execute", new Type[] { }) == null)
                    throw new NullReferenceException("Execute method was not found on job.");

                //Use IJobScheduler to enqueue the job on the backend automation services.
                job.Enqueue<JobHelper>(o => o.EnqueueRemoteJob(JobType));

                log.Success(L);

                return Json(true);
            }
            catch (Exception ex)
            {
                L.Error = ex.GetUsefulDetails();
                log.Error(L);
                return new HttpStatusCodeResult(System.Net.HttpStatusCode.InternalServerError, $"Something went wrong. {ex.Message}");
            }
            finally
            {
                log.SendLogSession();
            }
        }

This is the api which is called when a user would like to trigger a workflow.

Why is this interesting?

Creating a method for each action is easy to do. Creating a single method which can dynamically invoke class instances at runtime, while resolving all of their dependencies, and following your current error handling strategies- isn’t quite so easy.

This process- allows me as the developer, to quickly, and easily add new workflows / jobs to the NOC visible page. The NOC can easily trigger the jobs…. and the process of creating the class instances, resolving dependencies…. is all auto-magically handled by the dependency injection framework.

As well, I have not posted anything in a while, and I felt this was interesting enough to share.

For my particular use-case, I need to retrieve many records from an application called Rapid7 for the purpose of exporting the data into Splunk. To do this successfully, I will need to query 5 or 10 separate tables, and I will need to page the results for each table.

I am sure many of you have written logic like this before, and included the paging logic directly in the method for each table. Myself- I am a huge fan of writing simple, clean abstracted code, which is easy to read… Since, this will be mostly the same logic, over and over… It makes sense

Application Logic

I like to have a strong separation between the “What” the workflow is doing, and “How” it is accomplished. I handle this by using copious amounts of abstraction. Here is the main logic:

        public async Task Execute(IJobCancellationToken jobCancellationToken)
        {
            try
            {
                //var splunk = log.CreateSplunkTarget(splunk_Index, null, splunk_Source);

                int TotalAssetCount = 0;
                await foreach (var asset in PageAllResults(GetAsset, jobCancellationToken.ShutdownToken))
                {
                    TotalAssetCount++;
                }


                int total_Exceptions = 0;
                await foreach (var asset in PageAllResults(GetVulnerability_Exceptions, jobCancellationToken.ShutdownToken))
                {
                    total_Exceptions++;
                }

                //For testing- Output results to console for validation testing. 
                Console.WriteLine(TotalAssetCount);
                Console.WriteLine(total_Exceptions);
            }
            catch (OperationCanceledException)
            {
                if (jobCancellationToken.ShutdownToken.IsCancellationRequested)
                {
                    log.Warning("Workflow was cancelled.");
                    return;
                }
                throw; //Let the normal exception handler handle this.... since cancellation was not requested.
            }
            catch (Exception ex)
            {
                //Log the exception as fatal error.
                log.Fatal(ex.GetUsefulDetails());

                //Throw the exception to allow the job scheduler to handle the retry logic.
                throw;
            }
            finally
            {
                //This will invoke the logic to send out error emails to subscribers to this workflow.
                log.SendLogSession();
            }
        }

For this article, it is only currently doing a count of the records returned.

You will notice the IJobCancellationToken. All of my “Automation Jobs” are configured with the idea in mind, that the server, or service, or job may need to be cancelled at any time. If any of those events happens to occur, The cancellationToken is passed down to allow a graceful stop of the application.

The logic executing this code contains the logic to automatically re-enqueue the job on another server when the previous job is cancelled.

How to abstract paging the results of multiple method calls

Next up- we will look at a method whose sole purpose, is to page the results of a method. This method is tailored specifically to this class, and prevents us from repeating the paging logic for each api(over 15 in total).

private async IAsyncEnumerable<T> PageAllResults<T>(Func<int, CancellationToken, Task<Wrapper<T>>> action, [EnumeratorCancellation]CancellationToken ct) where T : class
        {
            bool HasMoreRecords = true;
            int Start = 0;

            do
            {
                var Results = await action.Invoke(Start, ct).ConfigureAwait(false);

                foreach (var a in Results.Resources)
                {
                    yield return a;
                }

                HasMoreRecords = Start < Results.Page.TotalPages;
                Start++;
            }
            while (HasMoreRecords);
        }

While the method signature may look scary- It is actually very simple to utilize. This method contains the application API-specific logic to handle paging for all of the GET requests we will be performing.

Methods for specific APIs

For the individual APIs, While I could abstract this logic into a dictionary, I decided to leave everything into its own individual method to allow me to easily document and customize the parameters per call. As well- if I need to intercept the results, and perform modification on the fly, it would allow me to easily contain the logic here. There will be around 15+ of these in total.

        private Task<Wrapper<Asset>> GetAsset(int Start, CancellationToken ct)
        {
            //https://help.rapid7.com/insightvm/en-us/api/index.html#operation/getAssets
            var uri = new UriBuilder("https", baseUri, 443, "/api/3/assets");
            uri.Query = $"page={Start}&size={50}&sort=idASC";

            return QueryRapid7_GET<Asset>(uri.Uri, ct);
        }

        private Task<Wrapper<Vulnerability_Exception>> GetVulnerability_Exceptions(int Start, CancellationToken ct)
        {
            //https://help.rapid7.com/insightvm/en-us/api/index.html#operation/getVulnerabilityExceptions
            var uri = new UriBuilder("https", baseUri, 443, "/api/3/assets");
            uri.Query = $"page={Start}&size={500}&sort=idASC";

            return QueryRapid7_GET<Vulnerability_Exception>(uri.Uri, ct);
        }

You will notice- The entire purpose of this method, is focused specifically how to obtain the results for the provided API. It does not contain the paging logic, or even the logic on how to perform the GET request. This allows for less code-duplication overall, which leads to a simpler product.

Lastly- In the current implementation of these two tables, it does not use Async. Rather, it passes the Task back to the calling method, without waiting for it to complete.

Lastly- Abstractions for performing the request.

        private async Task<Wrapper<T>> QueryRapid7_GET<T>(Uri Uri, CancellationToken ct) where T : class
        {
            string credentials = Convert.ToBase64String(Encoding.ASCII.GetBytes(creds.UserName + ":" + creds.Password));

            var obj = await rest.GetObjectAsync<Wrapper<T>>(Uri, HttpMethod.Get, ct, RequestBody: null, Configure: o =>
           {
               o.Headers[HttpRequestHeader.Authorization] = $"Basic {credentials}";
           }).ConfigureAwait(false);


            if (obj.Exception != null)
                throw obj.Exception;

            return obj.Object;
        }

The purpose of this method, is similarly to perform a GET query against the rapid7 API. To perform the query, it only needs to know the Uri, and the credentials (which are set by the constructor.).

Again- there is no logic for creating the URI specific to the endpoint, There is no logic for paging. The only logic contained in the method, is to perform a GET query against the provided URI.

Both error handling and CancellationTokens are built into my RestHelper class. This allows graceful cancellation of the workflows, as well as- allowing me to seperate my logic, from my error handling.

Why Would I do that? That looks complicated!

Because when this workflow is expanded to perform logic on each of the hundred+ APIs offered- The code will remain very readable with minimal repeated logic.

Each method is very specific to its own purpose, with as little overlap as possible. The end result, will be cleaner code, with less wasted room.

Now- if you job rates you based on lines of code written, by all means- Don’t abstract at all. Copy and paste everything!

However- if you like building supportable software, Separating areas of functionality will allow you to build a much more supportable product. As well, it is much easier to write a unit test for your methods, when they are responsible for performing a single task.