Orchestration Services (Complex Higher Order Logic)

2.3.0 Introduction

Orchestration services are the combinators between multiple foundation or processing services to perform a complex logical operation. Orchestrations main responsibility is to do a multi-entity logical operation and delegate the dependencies of said operation to downstream processing or foundation services. Orchestration services main responsibility is to encapsulate operations that require two or three business entities.

public async ValueTask<LibraryCard> CreateStudentLibraryCardAsync(LibraryCard libraryCard) =>
TryCatch(async () =>
{
    ValidateLibraryCard(libraryCard);

    await this.studentProcessingService
        .VerifyEnrolledStudentExistsAsync(libraryCard.StudentId);

    return await this.libraryCardProcessingService.CreateLibraryCardAsync(libraryCard);
});

In the above example, the LibraryCardOrchestrationService calls both the StudentProcessingService and LibraryCardProcessingService to perform a complex operation. First, verify the student we are creating a library card for does exist and that they are enrolled. Then second create the library card.

The operation of creating a library card for any given student cannot be performed by simply calling the library card service. That's because the library card service (processing or foundation) does not have access to all the details about the student. Therefore a combination logic here needed to be implemented to ensure a proper flow is in place.

Its important to understand that orchestration services are only required if and only if we need to combine multiple entities operations primitive or higher-order. In some architectures, orchestration services might not even exist. That's simply because some microservices might be simply responsible for applying validation logic and persisting and retrieving data from storage, no more or no less.

2.3.1 On The Map

Orchestration services are one of the core business logic components in any system. They are positioned between single entity services (such as processing or foundation) and advanced logic services such as coordination services, aggregation services or just simply exposers such as controllers, web components or anything else. Here's a high level overview of where orchestration services may live:

As shown above, orchestration services have quite a few dependencies and consumers. They are the core engine of any software. On the right hand side, you can see the dependencies an orchestration service may have. Since a processing service is optional based on whether a higher-order business logic is needed or not - orchestration services can combine multiple foundation services as well.

The existence of an orchestration service warrants the existence of a processing service. But thats not always the case. There are situations where all orchestration services need to finalize a business flow is to interact with primitive-level functionality.

From a consumer standpoint however, orchestration service could have several consumers. These consumers could range from coordination services (orchestrators of orchestrators) or aggregation services or simply an exposer. Exposers are like controllers, view service, UI components or simply another foundation or processing service in case of putting messages back on a queue - which we will discuss further in our Standard.

2.3.2 Characteristics

In general, orchestration services are concerned with combining single-entity primitive or higher-order business logic operations to execute a successful flow. But you can also think of them as the glue that ties multiple single-entity operations together.

2.3.2.0 Language

Just like processing services, the language used in orchestration services define the level of complexity and the capabilities it offers. Usually, orchestration services combine two or more primitive or higher-order operations from multiple single-entity services to execute a successful operation.

2.3.2.0.0 Functions Language

Orchestration services have a very common charateristic when it comes to the language of it's functions. Orchestration services are wholistic in most of the language of its function, you will see functions such as NotifyAllAdmins where the service pulls all users with admin type and then calls a notification service to notify each and every one of them.

It becomes very obvious that orchestration services offer functionality that inches closer and closer to a business language than perimitive technical operation. You may see almost an identical expression in a non-technical business requirement that matches one for one a function name in an orchestration service. The same pattern continues as one goes to higher and more advanced categories of services within that realm of business logic.

2.3.2.0.1 Pass-Through

Orchestration services can also be a pass-through for some operations. For instance, an orchestration service could allow an AddStudentAsync to be propagated through the service to unify the source of interactions with the system at the exposers level. In which case, orchestration services will use the very same terminology a processing or a foundation service may use to propagate the operation.

2.3.2.0.2 Class-Level Language

Orchestration services mainly combine multiple operations to support a particular entity. So, if the main entity of support is Student and the rest of the entities are just to support an operation mainly targetting a Student entity - then the name of the orchestration service would be StudentOrchestrationService.

This level of enforcement of naming conventions ensures that any orchestration service is staying focused on a single entity responsibility with respect to multiple other supporting entities.

For instance, if creating a library card requires ensuring the student referenced in that library card must be enrolled in a school. Then an orchestration service will then be named after it's main entity which is LibraryCard in this case. Our orchestration service name then would be LibraryCardOrchestrationService.

The opposite is also true. If enrolling a student in a school has an accompanying operations such as creating a library card, then in this case a StudentOrchestrationService must be created with the main purpose to create a Student and then all other related entities once the aforementioned succeeds.

The same idea applies to all exceptions created in an orchestration service such as StudentOrchestrationValidationException and StudentOrchestrationDependencyException and so on.

2.3.2.1 Dependencies

As we mentioned above, orchestration services might have a bit larger range of types of dependencies unlike processing and foundation services. This is only due the optionality of processing services. Therefore, orchestration services may have dependencies that range from foundation services or processing services and optionally and usually logging or any other utility brokers.

2.3.2.1.0 Dependency Balance (Florance Pattern)

There's a very important rule that govern the consistency and balance of orchestration services which is the 'Florance Pattern'. the rule dictates that any orchestration service may not combine dependencies from different categories of operation.

What that means, is that an orchestration service cannot have a foundation and a processing services combined together. The dependencies has to be either all processings or all foundations. That rule doesn't apply to utility brokers dependencies however.

Here's an example of an unbalanced orchestration service dependencies:

An additional processing service is required to give a pass-through to a lower-level foundation service to balance the architecture - applying 'Florance Pattern' for symmetry would turn our architecture to the following:

Applying 'Florance Pattern' might be very costly at the beginning as it includes creating an entirely new processing service (or multiple) just to balance the architecture. But its benefits outweighs the effort spent from a maintainability, readability and pluggability perspectives.

2.3.2.1.1 Two-Three

The 'Two-Three' rule is a complexity control rule. This rule dictates that an orchestration service may not have more than three or less than two processing or foundation services to run the orchestration. This rule, however, doesn't apply to utility brokers. And orchestration service may have a DateTimeBroker or a LoggingBroker without any issues. But an orchestration service may not have an entity broker, such as a StorageBroker or a QueueBroker which feeds directly into the core business layer of any service.

The 'Two-Three' rule may require a layer of normalization to the categorical business function that is required to be accomplished. Let's talk about the different mechanisms of normalizing orchestration services.

2.3.2.1.1.0 Full-Normalization

Often times, there are situations where the current architecture of any given orchestration service ends up with one orchestration service that has three dependencies. And a new entity processing or foundation service is required to complete an existing process.

For instance, let's say we have a StudentContactOrchestrationService and that service has dependencies that provide primitive-level functionality for Address, Email and Phone for each student. Here's a visualization of that state:

Now, a new requirement comes in to add a student SocialMedia to gather more contact information about how to reach a certain student. We can go into full-normalization mode simply by finding the common ground that equally splits the contact information entities. For instance, we can split regular contact information versus digital contact information as in Address and Phone versus Email and SocialMedia. this way we split four dependencies into two each for their own orchestration services as follows:

As you can see in the figure above, we modified the existing StudentContactOrchestrationService into StudentRegularContactOrchestrationService, then we removed one of its dependencies on the EmailService.

Additionally, we created a new StudentDigitalContactOrchestrationService to have two dependencies on the existing EmailService in addition to the new SocialMediaService. But now that the normalization is over, we need an advanced order business logic layer, like a coordination service to provide student contact information to upstream consumers.

2.3.2.1.1.1 Semi-Normalization

Normalization isn't always as straightforward as the example above. Especially in situations where a core entity has to exist before creating or writing in additional information towards related entities to that very entity.

For instance, let's say we have a StudentRegistrationOrchestrationService which relies on StudentProcessingService, LibraryCardProcessingService and BookProcessingService as follows:

But now, we need a new service to handle students immunization records as ImmunizationProcessingService. We need all four services but we already have a StudentRegistrationOrchestrationService that has three dependencies. At this point a semi-normalization is required for the re-balancing of the architecture to honor the 'Two-Three' rule and eventually to control the complexity.

In this case here, a further normalization or a split is required to re-balance the architecture. We need to think conceptually about a common ground between the primitive entities in a student registration process. A student requirements contain identity, health and materials. We can, in this scenario combine LibraryCard and Book under the same orchestration service as books and libraries are somewhat related. So we have StudentLibraryOrchestrationService and for the other service we would have StudentHealthOrchestrationService as follows:

To complete the registration flow with a new model, a coordination service is required to pass-in an advanced business logic to combine all of these entities together. But more importantly, you will notice that each orchestration service now has a redundant dependency of StudentProcessingService to ensure no virtual dependency on any other orchestration service create or ensuring a student record exists.

Virtual dependencies are very tricky. it's a hidden connection between two services of any category where one service implicitly assumes that a particular entity will be created and present. Virtual dependencies are very dangerous and threaten the true autonomy of any service. Detecting virtual dependencies at early stage in the design and development process could be a daunting but necessary task to ensure a clean Standardized architecture is in place.

Just like model changes as it may require migrations, and additional logic and validations, a new requirement for an entirely new additional entity might require a restructuring of an existing architecture or extending it to a new version, depending in which stage the system is receiving these new requirements.

It may be very enticing to just add an additional dependency to an existing orchestration service - but that's where the system starts to diverge from 'The Standard'. And that's when the system starts off the road of being an unmaintainable legacy system. But more importantly, it's when the engineers involved in designing and developing the system are challenged against their very principles and craftmanship.

2.3.2.1.1.2 No-Normalization

There are scenarios where any level of normalization is a challenge to achieve. While I believe that everything, everywhere, somehow is connected. Sometimes it might be incomprehensible for the mind to group multiple services together under one orchestration service.

Because it's quite hard for my mind to come up with an example for multiple entities that have no connection to each other, as I truly believe it couldn't exist. I'm going to rely on some fictional entities to visualize a problem. So let's assume there are AService and BService orchestrated together with an XService. The existence of XService is important to ensure that both A and B can be created with an assurance that a core entity X does exist.

Now, let's say a new service CService is required to be added to the mix to complete the existing flow. So, now we have four different dependencies under one orchestration service, and a split is mandatory. Since there's no relationship whatsoever between A, B and C, a 'No-Normalization' approach becomes the only option to realize a new design as follows:

Each one of the above primitive services will be orchestrated with a core service X then gathered again under a coordination service. This case above is the worst case scenario, where a normalization of any size is impossible. Note, that the author of this Standard couldn't come up with a realistic example unlike any others to show you how rare it is to run into that situation, so let's a 'No-Normalization' approach be your very last solution if you truly run out of options.

2.3.2.1.1.3 Meaningful Breakdown

Regardless of the type of normalization that you will need to follow. You have to make sure that your grouped services represent a common meaning. For instance, putting together a StudentProcessingService and LibraryProcessingService must require a commonality in function between the two. A good example of that would be StudentRegistrationOrchestrationService for instance. The registration process requires adding a new student record and creating a library card for that very same student.

Implementing orchestration services without intersection between two or three entities per operation defeats the whole purpose of having an orchestration service. This condition is satisfied if at least one intersection between two entities has occurred. An orchestration service may then have other operations that we call 'Pass-Through' where we propagate certain routines from their processing or foundation origins if they match the same contract.

Here's an example:

public class StudentOrchestrationService
{
    public async ValueTask<Student> RegisterStudentAsync(Student student)
    {
        Student addedStudent =
            await this.studentProcessingService.AddStudentAsync(student);
    
        LibraryCard libraryCard = 
            await this.libraryCardPorcessingService.AddLibraryCardAsync(
                addedStudent.Id);

        return addedStudent;
    }


    public async ValueTask<Student> ModifyStudentAsync(Student student) =>
        await this.studentProcessingService.ModifyStudentAsync(student);
}

In the example above, our StudentOrchestrationService had an orchestration routine where it combined adding a student and creating a library card for that very same student. But additionally it also offers a 'Pass-Through' function for a low-level processing service routine to modify a student.

'Pass-Through' routines must have the exact same contract as the rest of all other routines in any orchestration service. That's our 'Pure Contract' principle which dictates that any service should allow the very same contract as input and output or primitive types.

2.3.2.2 Contracts

Orchestration services may combine two or three different entities and their operations to achieve an advanced business logic. There are two situations when it comes to contract/models for orchestration services. One that stays true to the main entity of purpose. And one that is complex - a combinator orchestration service that tries to explicitly expose it's inner target entities.

Let's talk about these two scenarios in detail.

2.3.2.2.0 Physical Contracts

Some orchestration services are still single-purposed even though they may be combining two or three other higher order routines from multiple entities. For instance, an orchestration service that reacts to messages from some queue then persists these messages are single-purposed and single-entity orchestration services.

Let's take a look at this code snippet:

public class StudentOrchestrationService
{
    private readonly IStudentEventProcessingService studentEventProcessingService;
    private readonly IStudentProcessingService studentProcessingService;

    public StudentOrchestrationService(
        IStudentEventProcessingService studentEventProcessingService,
        IStudentProcessingService studentProcessingService)
    {
        this.studentEventProcessingService = studentEventProcessingService;
        this.studentProcessingService = studentProcessingService;
        ListenToEvents();
    }

    public void ListenToEvents() =>
        this.studentEventService.ListenToEvent(UpsertStudentAsync);

    public async ValueTask<Student> UpsertStudentAsync(Student student)
    {
        ...
        await this.studentProcessingService.UpsertStudentAsync(student);

        ...
    }
}

In the above example, the orchestration service still exposes a functionality that honors the physical model Student and internally communicates with several services that may provide completely different models. These are the scenarios where there's a main purpose single entity and all other services are supporting services to ensure a successful flow for that very entity succeeds.

In our example here, the orchestration services listens to a queue where new student messages will be placed, then use that event to persist any incoming new students in the system. So the physical contract Student is the same language the orchestration service explicitly use as a model to communicate with upper stream services/exposers or others.

But there are other scenarios where a single entity is not the only purpose/target for an orchestration service. Let's talk about that in detail.

2.3.2.2.1 Virtual Contracts

In some scenarios, an orchestration service may be required to create it's own non-physical contracts to complete a certain operation. For instance, consider an orchestration service that is required to persist a social media post with a picture attached to it. The requirement here is to persist the picture in one database table, and the actual post (comments, authors and others) into a different database table in a relational model.

Now, the incoming model might be significantly different from what the actual physical models would look like. Let's take a look at how would that look like in the real-world.

Consider having this model:

public class MediaPost
{
    public Guid Id {get; set;}
    public string Content {get; set;}
    public DateTimeOffset Date {get; set;}
    public IEnumerable<string> Base64Images {get; set;}
}

The above contract MediaPost contains two different physical entities combined. The first is the actual post, including the Id, Content and Date and the second is the list of images attached to that very post.

here's how an orchestration service would react to this incoming virtual model:

public async ValueTask<MediaPost> SubmitMediaPostAsync(MediaPost mediaPost)
{
    ...

    Post post = MapToPost(mediaPost);
    List<Media> medias = MapToMedias(mediaPost);

    Post addedPost =
        await this.postProcessingService.AddPostAsync(post);
    
    List<Medias> addedMedias = 
        await this.mediaProcessingService.AddMediasAsync(medias);

    return MapToMediaPost(addedPost, addedMedias); 
}

public Post MapToPost(MediaPost mediaPost)
{
    return new Post
    {
        Id = mediaPost.Id,
        Content = mediaPost.Content,
        CreatedDate = mediaPost.Date,
        UpdatedDate = mediaPost.Date
    };
}

public List<Media> MapToMedias(MediaPost mediaPost)
{
    return mediaPost.Base64Images.Select(image =>
        new Media
        {
            Id = Guid.NewGuid(),
            PostId = mediaPost.Id,
            Image = image,
            CreatedDate = mediaPost.Date,
            UpdatedDate = mediaPost.Date
        });
}

The above code snippet shows the orchestration service deconstructing a given virtual model/contract MediaPost into two physical models, each one of them has it's own separate processing service that handles it's persistence. There are scenarios also where the virtual model gets deconstructed into one single model with additional details that are used for validation and verification with downstream processing or foundation services.

There are also hybrid situations where the incoming virtual model may have nested physical models in it. Which is something we can only allow with virtual models. Physical models shall stay anemic (contains no routines or constructors) and flat (contains no nested models) at all times to control complexity and focus responsibility.

In summary, Orchestration services may create their own contracts. These contracts may be physical or virtual. And a virtual contract may be a combination of one or many physical (or nested virtual) contracts or simply has it's own flat design in terms of properties.

2.3.2.2 Cul-De-Sac

There are times where Orchestration services and it's equivalent (coordination, management ... etc.) may not need an exposer component (controller for instance). That's because these services may be listeners to certain events and communicating the event back into a processing or a foundation service at the same level where the event started or was received.

For instance, imagine building a simple application where it gets notified with messages from a queue then maps these messages into some local model to persist it in storage. In this case here, the incoming or input for these services isn't necessarily through an exposer component anymore. The incoming messages can be received from a subscription to an event service or a queue. In this case the orchestration service would look something like the following:

The StudentEventOrchestrationService in this case, listens to the messages for new students coming in and immediately converts that into models that can be persisted in the database.

Here's an example:

Let's start with a unit test for this pattern as follows:

[Fact]
public void ShouldListenToProfileEvents()
{
   // given . when
   this.profileEventOrchestrationService.ListenToProfileEvents();

   // then
   this.profileEventServiceMock.Verify(service =>
       service.ListenToProfileEvent(
           this.profileEventOrchestrationService.ProcessProfileEventAsync),
               Times.Once);

   this.profileEventService.VerifyNoOtherCalls();
   this.profileServiceMock.VerifyNoOtherCalls();
   this.loggingBrokerMock.VerifyNoOtherCalls();
}

[Fact]
public async Task ShouldAddProfileAsync()
{
   // given
   ProfileEvent randomProfileEvent =
       CreateRandomProfileEvent();

   ProfileEvent inputProfileEvent =
       randomProfileEvent;

   this.profileServiceMock.Setup(service =>
       service.AddProfileAsync(inputProfileEvent.Profile));

   // when
   await this.profileEventOrchestrationService
       .ProcessProfileEventAsync(inputProfileEvent);

   // then
   this.profileServiceMock.Verify(service =>
       service.AddProfileAsync(inputProfileEvent.Profile),
           Times.Once);

   this.profileServiceMock.VerifyNoOtherCalls();
   this.loggingBrokerMock.VerifyNoOtherCalls();
   this.profileEventServiceMock.VerifyNoOtherCalls();
}

The test here indicates an event listening has to occur first, then a persistence in the student service must match the outcome of mapping an incoming message to a given student.

Let's try to make this test pass.

public partial class ProfileEventOrchestrationService : IProfileEventOrchestrationService
{
   private readonly IProfileEventService profileEventService;
   private readonly IProfileService profileService;
   private readonly ILoggingBroker loggingBroker;

   public ProfileEventOrchestrationService(
       IProfileEventProcessingService profileEventService,
       IProfileProcessingService profileService,
       ILoggingBroker loggingBroker)
   {
       this.profileEventService = profileEventService;
       this.profileService = profileService;
       this.loggingBroker = loggingBroker;
   }

   public void ListenToProfileEvents() =>
   TryCatch(() =>
   {
       this.profileEventService.ListenToProfileEvent(
           ProcessProfileEventAsync);
   });

   public ValueTask ProcessProfileEventAsync(ProfileEvent profileEvent) =>
   TryCatch(async () =>
   {
       ...

       await this.profileService.AddProfileAsync(profileEvent.Profile);
   });
}

In the above example, the constructor of the Orchestration service subscribes to the events that would come from the StudentEventService, when an event occurs, the orchestration service will call the ProcessingIncomingStudentMessageAsync function to persist the incoming student into the database through a foundation or a processing service at the same level as the event service.

This pattern or characteristic is called the Cul-De-Sac. Where an incoming message will be a turn and head in a different direction for a different dependency. This pattern is very common is large enterprise-level applications where eventual-consistency is incorporated to ensure the system can scale and become resilient under heavy consumption. This pattern also prevents malicious attacks against your API endpoints since it allows processing queue messages or events whenever the service is ready to process them. We will discuss the details in the 'The Standard Architecture'.

2.3.3 Responsibilities

Orchestration services provide an advanced business logic. It orchestrates multiple flows for multiple entities/models to complete a single flow. Let's discuss in detail what these responsibilities are:

2.3.3.0 Advanced Logic

Orchestration services cannot exist without combining multiple routines from multiple entities. These entities may be different in nature, but they share a common flow or purpose. For instance, a LibraryCard as a model is fundamentally different from a Student model. However, they both share common purpose when it comes to student registration process. Adding a student record is required to register a student, but also assigning a library card to that very same student is a requirement for a successful student registration process.

Orchestration services ensures the correct routines for each entity are integrated, but also ensures these routines are called in the right order. Additionally, orchestration services are responsible for rolling back in case of a failing operation if needed. These three aspects are what constitutes an orchestration effort across multiple routines, entities or contracts.

Let's talk about those in detail.

2.3.3.0.0 Flow Combinations

We spoke earlier about orchestration services combining multiple routines to achieve a common purpose or a single flow. This aspect of orchestration services can serve as both a fundamental characteristic but also a responsibility. And orchestration service without at least one routine that combines two or three entities is not considered truly an orchestration. Integrating with multiple services without a common purpose is a better fit definition for Aggregation services which we will discuss later in this services chapter.

But within the flow combination comes the unification of contract. I call it mapping and branching. Mapping an incoming model into multiple lower-stream services models then branching the responsibility across these services.

Just like the previous services, Orchestration services are responsible during their flow combination to ensure the purity of the exposed input and output contracts. Which becomes a bit more complex when combining multiple models. Orchestration services will continue to be responsible for mapping incoming contracts to their respective downstream services, but also ensures to map back the returned results from these services into the unified model.

Let's bring back a previous code snippet to illustrate that aspect:

public async ValueTask<MediaPost> SubmitMediaPostAsync(MediaPost mediaPost)
{
    ...

    Post post = MapToPost(mediaPost);
    List<Media> medias = MapToMedias(mediaPost);

    Post addedPost =
        await this.postProcessingService.AddPostAsync(post);
    
    List<Medias> addedMedias = 
        await this.mediaProcessingService.AddMediasAsync(medias);

    return MapToMediaPost(addedPost, addedMedias); 
}

private Post MapToPost(MediaPost mediaPost)
{
    return new Post
    {
        Id = mediaPost.Id,
        Content = mediaPost.Content,
        CreatedDate = mediaPost.Date,
        UpdatedDate = mediaPost.Date
    };
}

private List<Media> MapToMedias(MediaPost mediaPost)
{
    return mediaPost.Base64Images.Select(image =>
        new Media
        {
            Id = Guid.NewGuid(),
            PostId = mediaPost.Id,
            Image = image,
            CreatedDate = mediaPost.Date,
            UpdatedDate = mediaPost.Date
        });
}

private MediaPost MapToMediaPost(Post post, List<Media> medias)
{
    return new MediaPost
    {
        Id = post.Id,
        Content = post.Content,
        Date = post.CreatedDate,
        Base64Images = medias.Select(media => media.Image)
    }
}

As you can see in the above example, the mapping and branching doesn't just happen on the way in. But a reverse action has to be taken on the way out. It's a violation of The Standard to return the same input object that was passed in. That takes away any visibility on any potential changes that happened to the incoming request during persistence. The duplex mapping should substitute the need to dereference the incoming request to ensure no unexpected internal changes have occurred.

Note that it's also recommended to break the mapping logic into its own aspect/partial class file. Something like StudentOrchestrationService.Mappings.cs to make sure the only thing left there is the business logic of orchestration.

2.3.3.0.1 Call Order

Calling routines in the right order can be crucial to any orchestration process. For instance, a library card cannot be created unless a student record is created first. Enforcing the order here can split into two different types. Let's talk about those here for a bit.

2.3.3.0.1.0 Natural Order

The natural order here refers to certain flows that cannot be executed unless a prerequisite of input parameters is retrieved or persisted. For instance, imagine a situation where a library card cannot be created unless a student unique identifier is retrieved first. In this case here we don't have to worry about testing certain routines were called in the right order because it comes naturally with the flow.

here's a code example of this situation:

public async ValueTask<LibraryCard> CreateLibraryCardAsync(LibraryCard libraryCard)
{
    Student student = await this.studentProcessingService
        .RetrieveStudentByIdAsync(libraryCard.StudentId));

    return await this.libraryCardProcessingService
        .CreateLibraryCardAsync(libraryCard, student.Name);
}

In the example above, having a student Name is a requirement to create a library card. Therefore, the orchestration of order here comes naturally as part of the flow without any additional effort.

Let's talk about the second type of order - Enforced Order.

2.3.3.0.1.1 Enforced Order

Imagine the very same example above, but instead of the library card requiring a student name it instead just needs the student Id which is already enclosed in the incoming request model. Something like this:

public async ValueTask<LibraryCard> CreateLibraryCardAsync(LibraryCard libraryCard)
{
    await this.studentProcessingService.VerifyEnlistedStudentExistAsync(
        libraryCard.StudentId);

    return await this.libraryCardProcessingService.CreateLibraryCardAsync(libraryCard);
}

ensuring a verify enlisted student exists has happened before creating a library card might become a challenge to achieve naturally because there's no dependency between a return value of one routine and the input parameters of the next. In other words, there is nothing that the VerifyEnlistedStudentExistAsync function returns that the CreateLibraryCardAsync function cares about in terms of input parameters.

In this case here an enforced type of order must be implemented through unit tests. A unit test for this routine would require to verify not just the dependency have been called with the right parameters, but also that they are called in the right order let's take a look at how that would be implemented:

[Fact]
public async Task ShouldCreateLibraryCardAsync()
{
    // given
    Student someStudent = CreateRandomStudent();
    LibraryCard randomLibraryCard = CreateRandomLibraryCard();
    LibraryCard inputLibraryCard = randomLibraryCard;
    LibraryCard createdLibraryCard = inputLibraryCard;
    LibraryCard expectedLibraryCard = inputLibraryCard.DeepClone();
    Guid studentId = inputLibraryCard.StudentId;
    var mockSequence = new MockSequence();

    this.studentProcessingServiceMock.InSequence(mockSequence).Setup(service =>
        service.VerifyEnlistedStudentExistAsync(studentId))
            .Returns(someStudent);

    this.libraryCardProcessingServiceMock.InSequence(mockSequence).Setup(service =>
        service.CreateLibraryCardAsync(inputLibraryCard))
            .ReturnsAsync(createdLibraryCard);

    // when
    LibraryCard actualLibraryCard = await this.libraryCardOrchestrationService
        .CreateLibraryCardAsync(inputLibraryCard);

    // then
    actualLibraryCard.Should().BeEquivalentTo(expectedLibraryCard);

    this.studentProcessingServiceMock.Verify(service =>
        service.VerifyEnlistedStudentExistAsync(studentId),
            Times.Once);

    this.libraryCardProcessingServiceMock.Verify(service =>
        service.CreateLibraryCardAsync(inputLibraryCard),
            Times.Once);

    this.studentProcessingServiceMock.VerifyNoOtherCalls();
    this.libraryCardProcessingServiceMock.VerifyNoOtherCalls();
    this.loggingBrokerMock.VerifyNoOtherCalls();
}

From the example above, the mock framework here is being used to ensure a certain order is being enforced when it comes to calling these dependencies. This way we enforce a certain implementation within any given method to ensure a non-naturally connected dependencies are being sequentially called in the exact right order.

It's more likely that the form of ordering leans more towards enforced than natural when orchestration services reach the maximum number of dependencies they may have at any point in time.

2.3.3.0.2 Exceptions Mapping (Wrapping & Unwrapping)

This responsibility is very similar to flow-combinations. Except that in this case orchestration services unify all the exceptions that may occur out of any of its dependencies into one unified categorical exception model. Let's start with an illustration of what that mapping may look like:

In the illustration above, you will notice that validation and dependency validation exceptions thrown from downstream dependency services are being mapped into one unified dependency exception at the orchestration level. This practice allows upstream consumers of that very orchestration service to determine the next course of action based on one categorical exception type instead of four or in the case of three dependencies it would be six categorical dependencies.

Let's start with a failing test to materialize our idea here:

public static TheoryData DependencyValidationExceptions()
{
    string exceptionMessage = GetRandomMessage();
    var innerException = new Xeption(exceptionMessage);

    var studentValidationException =
        new StudentValidationException(innerException);

    var studentDependencyValidationException =
        new StudentDependencyValidationException(innerException);

    var libraryCardValidationException =
        new LibraryCardValidationException(innerException);

    var libraryCardDependencyValidationException =
        new LibraryCardDependencyValidationException(innerException);

    return new TheoryData<Xeption>
    {
        studentValidationException,
        studentDependencyValidationException,
        libraryCardValidationException,
        libraryCardDependencyValidationException
    };
}


[Theory]
[MemberData(nameof(DependencyValidationExceptions))]
public async Task ShouldThrowDependencyValidationExceptionOnCreateIfDependencyValidationErrorOccursAndLogItAsync(
    Xeption dependencyValidationException)
{
    // given
    Student someStudent = CreateRandomStudent();

    var expectedStudentOrchestrationDependencyValidationException =
        new StudentOrchestrationDependencyValidationException(
            dependencyValidationException.InnerException as Xeption);

    this.studentServiceMock.Setup(service =>
        service.AddStudentAsync(It.IsAny<Student>()))
            .ThrowsAsync(dependencyValidationException);

    // when
    ValueTask<Student> addStudentTask =
        await this.studentOrchestrationService.AddStudentAsync(someStudent);

    // then
    await Assert.ThrowsAsync<StudentOrchestrationDependencyValidationException>(() =>
        addStudentTask.AsTask());

    this.studentServiceMock.Verify(service =>
        service.AddStudentAsync(It.IsAny<Student>()),
            Times.Once);

    this.loggingBrokerMock.Verify(broker =>
        broker.LogError(It.Is(SameExceptionAs(
            expectedStudentOrchestrationDependencyValidationException))),
                Times.Once);

    this.libraryCardServiceMock.Verify(service =>
        service.AddLibraryCard(It.IsAny<Guid>()),
            Times.Once);

    this.studentServiceMock.VerifyNoOtherCalls();
    this.loggingBrokerMock.VerifyNoOtherCalls();
    this.libraryCardServiceMock.VerifyNoOtherCalls();
}

In the test above, we are verifying that any of the four aforementioned exceptions on occurrence, they would be mapped into a StudentOrchestrationDependencyValidationException. We maintain the original localized exception as an inner exception. But we unwrap the categorical exception at this level to maintain the original issue as we go upstream.

These exceptions are mapped under a dependency validation exception because they originate from a dependency or a dependency of a dependency downstream. For instance, if a storage broker throws an exception that is deemed to be a dependency validation (something like DuplicateKeyException). The broker-neighboring service would map that into a localized StudentAlredyExistException then wrap that exception in a categorical exception of type StudentDependencyValidationException. When that exception propagates upstream to a processing or an orchestration service, we lose the categorical exception as we have already captured it under the right scope of mapping. Then we continue to embed that very localized exception under the current service dependency validation exception.

Let's try to make this test pass:

public partial class StudentOrchestrationService
{
    private delegate ValueTask<Student> ReturningStudentFunction();

    private async ValueTask<Student> TryCatch(ReturningStudentFunction returningStudentFunction)
    {
        try
        {
            return await returningStudentFunction();
        }
        catch (StudentValidationException studentValidationException)
        {
            throw CreateAndLogDependencyValidationException(studentValidationException);
        }
        catch (StudentDependencyValidationException studentDependencyValidationException)
        {
            throw CreateAndLogDependencyValidationException(studentDependencyValidationException);
        }
        catch (LibraryCardValidationException libraryCardValidationException)
        {
            throw CreateAndLogDependencyValidationException(libraryCardValidationException);
        }
        catch (LibraryCardDependencyValidationException libraryCardDependencyValidationException)
        {
            throw CreateAndLogDependencyValidationException(libraryCardDependencyValidationException);
        }
    }

    private StudentOrchestrationDependencyValidationException CreateAndLogDependencyValidationException(Xeption exception)
    {
        var studentOrchestrationDependencyValidationException =
            new StudentOrchestrationDependencyValidationException(exception.innerException as Xeption);

        this.loggingBroker.LogError(studentOrchestrationDependencyValidationException);

        throw studentOrchestrationDependencyValidationException;
    }
}

Now we can use the TryCatch as follows:

public async ValueTask<Student> AddStudentAsync(Student student) =>
TryCatch(async () => 
{
    ...
    Student addedStudent = await this.studentService.AddStudentAsync(student);
    LibraryCard libraryCard = await this.libraryCard.AddLibraryCard(addedStudent.Id);

    return addedStudent;  
});

You can see now in the implementation that we mapped all four different types of external downstream services validation exceptions into one categorical exception and then maintained the inner exception for each one of them.

The same rule applies to dependency exceptions. Dependency exceptions can be both Service and Dependency exceptions from downstream services. For instance, in the above example, calling a student service may produce StudentDependencyException and StudentServiceException. Both of these categorical exceptions will be unwrapped from their categorical layer and have their local layer wrapped in one unified new orchestration-level categorical exception under StudentOrchestrationDependencyException. The same thing applies to all other dependency categorical exceptions like LibraryCardDependencyException and LibraryCardServiceException.

It's extremely important to unwrap and wrap localized exceptions from downstream services with categorical exceptions at the current service layer to ensure at the Exposers layer these exceptions can be easily handled and mapped into whatever the nature of the exposer component dictates. In the case of an Exposer component of type API Controller, the mapping would produce Http Status Codes. In the case of UI Exposer components it would be mapped to meaningful text to the end users.

We will discuss further upstream in this Standard when to determine to expose localized inner exceptions details where end-users are not required to take any action. This is exclusive to dependency and service level exceptions.

2.3.4 Variations

Orchestration services have several variations depending on where they stand in the overall low-level architecture. For instance, an orchestration service depending on downstream orchestration services is called a Coordination Service. An orchestration service working with multiple coordination services as dependencies is called a Management Service. These variants are all in essence an orchestration service with an uber-level business logic.

2.3.4.0 Variants Levels

Let's take a look at the possible variants for orchestration services and where they would be positioned:

In my personal experience, I've rarely had to resolve to an Uber Management service. The idea of the limitation here in terms of dependencies and variations of orchestration-like services is to help engineers re-think the complexity of their logic. But admittedly there are situations where the complexity is an absolute necessity, therefore Uber-Management services exist as an option.

The following table should guide the process of developing variants of orchestration services based on the level:

Variant	Dependencies	Consumers	Complexity
Orchestrations Services	Foundation or Processing Services	Coordination Services	Low
Coordination Services	Orchestration Services	Management Services	Medium
Management Services	Coordination Services	Uber Management Services	High
Uber Management Services	Management Services	Aggregation, Views or Exposer Components	Very High

Working beyond Uber Management services in an orchestration manner would require a deeper discussion and a serious consideration of the overall architecture. Some future versions The Standard might be able to address this issue in what I call "The Lake House" but that is outside of the scope of this version of The Standard.

2.3.4.1 Unit of Work

With the variations of orchestration services, I highly recommend staying true to the unit of work concept. Where every request can do one thing and one thing only including it's pre-requisites. For instance, if you need to register a student in some school, You may require also adding a guardian, contact information and some other details. Eventing these actions can greatly decrease the complexity of the flow and lower the risk of any failures in downstream services.

Here's a visualization for a complex single-threaded approach:

The solution above is a working solution for registering a student. We needed to include guardian information, library cards, classes ... etc. These dependencies can be broken down in terms of eventing and allowing other services to pick up where the single-threaded services leaves off to continue the registration process. Something like this:

The incoming request in the above example would be turned into events, where each one of these events would be notifying its own orchestration services in a cul-de-sac manner as we discussed above in section 2.3.2.2. What that means is that a single thread is no longer responsible for the success of each and every dependency in the system. Instead, every event-listening broker would handle its own process in a much, much simplified way.

That approach does not guarantee an immediate response of success or failure to the requestor. It's an eventual-consistency pattern where the client would get an Accepted message or its equivalent based on the communication protocol to let them know that a process has started but there's no guarantee of any results yet until it's done.

It's also important to mention that an extra layer of resiliency can be added to these events by temporarily storing them in Queue-like components or memory-based temporary storages; depending on the criticality of the business.

But an eventual consistency approach isn't always a good solution if the client on the other side is waiting for a response. Especially in a much critical situations where an immediate response is required. This problem can be solved through Fire-n-Observe queues that we will discuss in future version of The Standard.

[*] Introduction to Orchestration Services

[*] Cul-De-Sac Pattern for Orchestration Services

[*] Cul-De-Sac Pattern for Coordination Services