I wanted to build a simple page, which would allow our NOC to trigger certain workflows, automatically (assuming they have proper permission)

My requirements:

1. Jobs should be easily added to the list, just by specifying an Attribute, and Implementing an interface (to ensure the proper method exists)
2. Jobs needs to be kicked off on my back-end servers and not on the web-front end.
3. Everything needs to be properly logged.
4. The front-end needs visual-feedback a job has been executed.

Here is the web page to trigger jobs.

As you can see, the above page is very simple.

In the above image, if a job fails, it will turn red, with direction to call the developers (me).

How it works (Short Version)

At startup, Reflection is used to build a dictionary containing types which implement IRemoteJob interface, and the NocJob attribute.

A webpage displays the values. Permissions are implemented at the controller level.

When a job is triggered, the Type’s full name is passed back to the method. The method will compare the provided name, against the dictionary of valid workflows. If the workflow is found, and verification passes, Hangfire.Io is then used to execute the workflow on my backend- automation services away from the web front end.

Since- we cannot execute a string containing the name of a type… We utilize Ninject’s kernel to resolve a instance of the class. We then Invoke the execute method, which is defined by the IRemoteJob interface, on the class instance.

Error handling is handed by Hangfire.

In short- that is it.

How it works (Long Version)

RuntimeTypeResolver.cs // IRunTimeTypeResolver

    public class RuntimeTypeResolver : IRuntimeTypeResolver
    {
        private IKernel kernel;
        public RuntimeTypeResolver(IKernel Kernel)
        {
            this.kernel = Kernel;
        }
        public object GetInstanceOfType(Type type)
        {
            return kernel.Get(type);
        }
    }

This class, resolves a class instance from the Ninject kernel and returns it.

JobHelper.cs

//Note - Singleton class.
    public class JobHelper
    {
        //IRunTimeTypeResolver passes a Type into a dependancy injection kernel. The kernel will return a instance of the Type.
        private IRuntimeTypeResolver rtr;
        public JobHelper(IJobScheduler JOB, IRuntimeTypeResolver RTR)
        {
            this.rtr = RTR;
            //Note- Dictionary only needs to be populated at startup.

            //Create a dictionary containing all eligible classes to be executed ad-hoc.
            RemoteJobs = this.GetType() //Types are filtered to this current assembly.
                .Assembly
                .GetTypes()
                //Type/Class implements IRemoteJob interface, and is NOT an interface or abstract class.
                .Where(x => (typeof(IRemoteJob).IsAssignableFrom(x) && !x.IsInterface && !x.IsAbstract))
                //Class has the NocJobAttribute, which contains display information.
                .Where(x => Attribute.IsDefined(x, typeof(NocJobAttribute)))
                //Cast the types, and create a dictionary.
                .ToDictionary(o => (NocJobAttribute)Attribute.GetCustomAttribute(o, typeof(NocJobAttribute)), o => o);
        }

        //A read-only dictionary containing the information. 
        public IReadOnlyDictionary<NocJobAttribute, Type> RemoteJobs { get; }

        /// <summary>
        /// This is intended as an entry point for background jobs. 
        /// </summary>
        /// <param name="JobName"></param>
        public void EnqueueRemoteJob(string JobName)
        {
            //Get the Type from the list.
            Type Rec = RemoteJobs
               .First(o => o.Value.FullName.Equals(JobName, StringComparison.OrdinalIgnoreCase))
               .Value;

            //Get the Execute method. It should exist, because all items in this list, implement IRemoteJob.
            MethodInfo ExecuteMethod = Rec.GetRuntimeMethod("Execute", new Type[] { });

            //Get a instance of the type, from the RuntimeTypeResolver class.
            object Instance = rtr.GetInstanceOfType(Rec);

            //Invoke the Execute Method on the class instance.
            ExecuteMethod.Invoke(Instance, null);
        }
    }

The primary purpose of this job, is to both hold a list of “valid” workflows which can be executed, and to expose a method which the Hangfire.io can trigger with the Type’s name, to invoke the workflow.

JobController.cs – Trigger_NOC_Job method

        [HttpPost]
        [CheckPermissionAttribute(Permission.Trigger_AutomationJobs)]
        public ActionResult Trigger_NOC_Job(string JobType)
        {
            log.Component = "Trigger";
            var L = new JobTriggerLog
            {
                JobName = new string[] { JobType },
                Time = DateTime.Now,
                UserID = User.Identity.Name,
                UserName = User.GetFullName()
            };
            try
            {
                //now the FUN part...
                if (!jobHelper.RemoteJobs.Any(o => o.Value.FullName.Equals(JobType, StringComparison.OrdinalIgnoreCase)))
                {
                    L.Error = new Exception("Job was not located.").GetUsefulDetails(false);
                    log.Error(L);
                    return new HttpStatusCodeResult(System.Net.HttpStatusCode.NotFound, "Job was not found.");
                }

                //Get the Type from the list.
                Type Rec = jobHelper
                   .RemoteJobs
                   .First(o => o.Value.FullName.Equals(JobType, StringComparison.OrdinalIgnoreCase))
                   .Value;

                //Sanity check to double-check the Execute method exists, without required parameters.
                if (Rec.GetRuntimeMethod("Execute", new Type[] { }) == null)
                    throw new NullReferenceException("Execute method was not found on job.");

                //Use IJobScheduler to enqueue the job on the backend automation services.
                job.Enqueue<JobHelper>(o => o.EnqueueRemoteJob(JobType));

                log.Success(L);

                return Json(true);
            }
            catch (Exception ex)
            {
                L.Error = ex.GetUsefulDetails();
                log.Error(L);
                return new HttpStatusCodeResult(System.Net.HttpStatusCode.InternalServerError, $"Something went wrong. {ex.Message}");
            }
            finally
            {
                log.SendLogSession();
            }
        }

This is the api which is called when a user would like to trigger a workflow.

Why is this interesting?

Creating a method for each action is easy to do. Creating a single method which can dynamically invoke class instances at runtime, while resolving all of their dependencies, and following your current error handling strategies- isn’t quite so easy.

This process- allows me as the developer, to quickly, and easily add new workflows / jobs to the NOC visible page. The NOC can easily trigger the jobs…. and the process of creating the class instances, resolving dependencies…. is all auto-magically handled by the dependency injection framework.

As well, I have not posted anything in a while, and I felt this was interesting enough to share.

No worries- I just made that exact same mistake, and I am going to demonstrate how to resolve the issue….

In the below image, you can see…. I accidentally made my commits to the master branch, instead of my feature/dev/test branch. Instead of redoing all of the commits manually, I am going to demonstrate the cherry-pick method for manually moving the commits over.

Method: Cherry Pick Commits to new branch

For method one, I will demonstrate cherry-picking the specific commits, into the proper branch.

Step 1: Take note of the specific commits.

In my example, I need to move these commits:

• dda291d4 – Update LIB.Interfaces
• d46a4441 – The lastExecutionIdParameter….

I am leaving out the “merge branch ‘master'”commit, because, the new branch is already up to date with master.

Step 2: Create new branch, or switch to the proper branch.

Step 3. Cherry pick the individual commits

While this functionality is likely built into visual studio somewhere, I chose the command line terminal to perform the cherry-pick.

If you do not have git in your path variable, you may use the visual studio Nuget package manager console to perform these changes.

Done!

After cherry-picking the individual commits to the new branch, you are done!

Make sure to rebuild with the updated code, to ensure you didn’t leave something out.

For my particular use-case, I need to retrieve many records from an application called Rapid7 for the purpose of exporting the data into Splunk. To do this successfully, I will need to query 5 or 10 separate tables, and I will need to page the results for each table.

I am sure many of you have written logic like this before, and included the paging logic directly in the method for each table. Myself- I am a huge fan of writing simple, clean abstracted code, which is easy to read… Since, this will be mostly the same logic, over and over… It makes sense

Application Logic

I like to have a strong separation between the “What” the workflow is doing, and “How” it is accomplished. I handle this by using copious amounts of abstraction. Here is the main logic:

        public async Task Execute(IJobCancellationToken jobCancellationToken)
        {
            try
            {
                //var splunk = log.CreateSplunkTarget(splunk_Index, null, splunk_Source);

                int TotalAssetCount = 0;
                await foreach (var asset in PageAllResults(GetAsset, jobCancellationToken.ShutdownToken))
                {
                    TotalAssetCount++;
                }


                int total_Exceptions = 0;
                await foreach (var asset in PageAllResults(GetVulnerability_Exceptions, jobCancellationToken.ShutdownToken))
                {
                    total_Exceptions++;
                }

                //For testing- Output results to console for validation testing. 
                Console.WriteLine(TotalAssetCount);
                Console.WriteLine(total_Exceptions);
            }
            catch (OperationCanceledException)
            {
                if (jobCancellationToken.ShutdownToken.IsCancellationRequested)
                {
                    log.Warning("Workflow was cancelled.");
                    return;
                }
                throw; //Let the normal exception handler handle this.... since cancellation was not requested.
            }
            catch (Exception ex)
            {
                //Log the exception as fatal error.
                log.Fatal(ex.GetUsefulDetails());

                //Throw the exception to allow the job scheduler to handle the retry logic.
                throw;
            }
            finally
            {
                //This will invoke the logic to send out error emails to subscribers to this workflow.
                log.SendLogSession();
            }
        }

For this article, it is only currently doing a count of the records returned.

You will notice the IJobCancellationToken. All of my “Automation Jobs” are configured with the idea in mind, that the server, or service, or job may need to be cancelled at any time. If any of those events happens to occur, The cancellationToken is passed down to allow a graceful stop of the application.

The logic executing this code contains the logic to automatically re-enqueue the job on another server when the previous job is cancelled.

How to abstract paging the results of multiple method calls

Next up- we will look at a method whose sole purpose, is to page the results of a method. This method is tailored specifically to this class, and prevents us from repeating the paging logic for each api(over 15 in total).

private async IAsyncEnumerable<T> PageAllResults<T>(Func<int, CancellationToken, Task<Wrapper<T>>> action, [EnumeratorCancellation]CancellationToken ct) where T : class
        {
            bool HasMoreRecords = true;
            int Start = 0;

            do
            {
                var Results = await action.Invoke(Start, ct).ConfigureAwait(false);

                foreach (var a in Results.Resources)
                {
                    yield return a;
                }

                HasMoreRecords = Start < Results.Page.TotalPages;
                Start++;
            }
            while (HasMoreRecords);
        }

While the method signature may look scary- It is actually very simple to utilize. This method contains the application API-specific logic to handle paging for all of the GET requests we will be performing.

Methods for specific APIs

For the individual APIs, While I could abstract this logic into a dictionary, I decided to leave everything into its own individual method to allow me to easily document and customize the parameters per call. As well- if I need to intercept the results, and perform modification on the fly, it would allow me to easily contain the logic here. There will be around 15+ of these in total.

        private Task<Wrapper<Asset>> GetAsset(int Start, CancellationToken ct)
        {
            //https://help.rapid7.com/insightvm/en-us/api/index.html#operation/getAssets
            var uri = new UriBuilder("https", baseUri, 443, "/api/3/assets");
            uri.Query = $"page={Start}&size={50}&sort=idASC";

            return QueryRapid7_GET<Asset>(uri.Uri, ct);
        }

        private Task<Wrapper<Vulnerability_Exception>> GetVulnerability_Exceptions(int Start, CancellationToken ct)
        {
            //https://help.rapid7.com/insightvm/en-us/api/index.html#operation/getVulnerabilityExceptions
            var uri = new UriBuilder("https", baseUri, 443, "/api/3/assets");
            uri.Query = $"page={Start}&size={500}&sort=idASC";

            return QueryRapid7_GET<Vulnerability_Exception>(uri.Uri, ct);
        }

You will notice- The entire purpose of this method, is focused specifically how to obtain the results for the provided API. It does not contain the paging logic, or even the logic on how to perform the GET request. This allows for less code-duplication overall, which leads to a simpler product.

Lastly- In the current implementation of these two tables, it does not use Async. Rather, it passes the Task back to the calling method, without waiting for it to complete.

Lastly- Abstractions for performing the request.

        private async Task<Wrapper<T>> QueryRapid7_GET<T>(Uri Uri, CancellationToken ct) where T : class
        {
            string credentials = Convert.ToBase64String(Encoding.ASCII.GetBytes(creds.UserName + ":" + creds.Password));

            var obj = await rest.GetObjectAsync<Wrapper<T>>(Uri, HttpMethod.Get, ct, RequestBody: null, Configure: o =>
           {
               o.Headers[HttpRequestHeader.Authorization] = $"Basic {credentials}";
           }).ConfigureAwait(false);


            if (obj.Exception != null)
                throw obj.Exception;

            return obj.Object;
        }

The purpose of this method, is similarly to perform a GET query against the rapid7 API. To perform the query, it only needs to know the Uri, and the credentials (which are set by the constructor.).

Again- there is no logic for creating the URI specific to the endpoint, There is no logic for paging. The only logic contained in the method, is to perform a GET query against the provided URI.

Both error handling and CancellationTokens are built into my RestHelper class. This allows graceful cancellation of the workflows, as well as- allowing me to seperate my logic, from my error handling.

Why Would I do that? That looks complicated!

Because when this workflow is expanded to perform logic on each of the hundred+ APIs offered- The code will remain very readable with minimal repeated logic.

Each method is very specific to its own purpose, with as little overlap as possible. The end result, will be cleaner code, with less wasted room.

Now- if you job rates you based on lines of code written, by all means- Don’t abstract at all. Copy and paste everything!

However- if you like building supportable software, Separating areas of functionality will allow you to build a much more supportable product. As well, it is much easier to write a unit test for your methods, when they are responsible for performing a single task.

Normally- I would say this should be an easy request… Query sys_audit where tablename=alm_asset, fieldname=assigned_to, filter by the values… and build a report based on this data.

However- data in sys_audit, stays forever, for every single table. So, if you attempt to run that query on any service-now instance that has been up for a few years, You will notice, it crashes.

This is because neither fieldname, nor tablename are indexed columns (on the underlying database)

However- if you look at an asset, and select history, this data is being pulled from sys_audit.

You may ask, well- why does that work when I cannot query the table at all??

The answer is- sys_audit.documentkey is an indexed field. So- when the related query looks for sys_audit where document_key = sys_id of the alm_asset, it returns very quickly, because it is indexed.

Now that we have gotten that part out of the way…. What about the report?

Well- while I am very knowledgeable on the BACKEND of service-now, and its available APIs, I do not know enough knowledge to build the query within service-now. BUT- I did create the logic in c# using a service-now library which I have created.

If you are a experienced service-now individual, you may be able to build a report based on my logic.

var Users = sn.OfType<sys_user>().Where(o => (Insert lamba expression here to locate the specific users)).ToList();

foreach (var rec in sn.OfType<alm_asset>())
{
    var history = sn.OfType<sys_audit>()
        .Where(o => o.documentkey == (object)rec.sys_id)
        .Where(o => o.fieldname == nameof(alm_asset.assigned_to) || o.fieldname == nameof(alm_asset.u_custom_assigned_to))
        //Force query to list.
        .ToList()
        //Local Expression
        .Where(o => Users.Any(usr => usr.sys_id.Equals(o.newvalue)));

    foreach (var match in history)
    {
        //Do Something with you results.
    }
}

The above query is quite inefficient, because it pulls down ALL assigned_to history change records for every asset to evaluate locally. However- it is many times faster then attempting to query sys_audit directly.

sys_history only contains 28 days of data, before any mentions it.

Summary:

1. Query all assets
2. For each asset, query sys_audit where documentkey = asset.sys_id
3. Perform additional filtering on sys_audit, for specific field names, or values.
4. Profit.

Edit- as a last note, this could easily be implement in service-now javascript.