Enterprise architecture

Cloud as code

A key part of this recommendation is to do all cloud setup and configuration as code - that is, automate everything from application infrastructure to account creation, to compliance monitoring. Automating everything allows the enterprise to ensure scale, compliance, security, and so on, in a repeatable and easily evolvable fashion. IBM Cloud provides a number of deployable architectures in the IBM Cloud catalog that can be used to automate these recommendations with minimal effort. These pre-made solutions can also be customized or extended to match the needs of the enterprise.

Managing infrastructure as code at scale requires more than just Terraform in a Git repo. Infrastructure as code is the DNA of an enterprise-wide cloud solution because it must be subject to both governance and management. Infrastructure as code ensures that only validated, compliant, and approved code is deployed. You can also track how deployments correspond to internal projects, teams, cost centers, and more.

IBM Cloud provides a number of tools to assist with this process. The IBM Cloud catalog allows an enterprise to advertise approved IaC solutions to users across the cloud. IBM Cloud projects allow an enterprise to ensure that only approved IaC is deployed and to track how the resulting resources correspond to projects, teams, and cost centers.

Separate development and production

Development environments must be separated from production environments as they have different accounting, resource utilization, access, availability, networking, and compliance requirements. However, at the same time development environments should be kept in sync with production environments so that users testing in development can be confident that the application will behave similarly in production.

This recommendation creates a clean separation between development and production environments, but uses infrastructure as code to ensure that development and production are aligned.

Hub and spoke model

In cloud, it is common to talk about using a hub and spoke model for networking. In this model, a centralized network hub links many spoke networks (VPCs) together. A centralized hub is typically used to reduce costs and increase scale while still allowing private network connectivity within the cloud.

In this recommendation, we also generalize this hub and spoke pattern, applying it to several different areas of concern including networking, managing infrastructure as code, secrets, shared services, and more.

Shared application infrastructure

Creating secure, compliant, robust, and scalable application infrastructure can be difficult to set up and costly to maintain. VMs or clusters must be deployed, logging and monitoring configured, firewalls and vulnerability checks enabled, and all of this must be monitored and kept up to date. In addition, this infrastructure must be scaled to accommodate application burst loads and as a result some aspects of the infrastructure are underutilized most of the time.

Standardizing application infrastructure where possible, and then hosting many applications on the same infrastructure, is essential to reduce operations costs. This set up allows the infrastructure to be better utilized and shares the operations cost across many applications. This also makes it practical to use infrastructure as code to deploy many copies of the infrastructure, amortizing the development and maintenance cost of that infrastructure as code across potentially thousands of applications.

As shown in the volumetric analysis, from 1-100 applications should be hosted per account on this shared infrastructure with an average of perhaps 10-20 applications per account. With hundreds of accounts (some that are designated for development and test and some that are designated for production workloads), this model can support many thousands of applications and their development.

A good starting point for developing the standardized application hosting infrastructure is IBM Cloud's VPC landing zone deployable architecture. This IBM Cloud for Financial Services compliant solution sets up everything that is needed to host VM or Container based applications in a fully secure and compliant fashion. Also, the deployable architecture is supported and maintained by IBM. As new compliance requirements arrive or vulnerabilities are discovered, the solution is updated to maintain security and compliance.

Separate management from workload

In the IBM Cloud Framework for Financial Services reference architecture, management workloads are separated from application workloads to support the different access models, data privacy requirements, network access requirements, and so on. Because these principles apply more broadly than just in the Financial Services industry, they have been applied in this general recommendation.

Work within IBM Cloud operating limits and constraints

IBM Cloud, like all public clouds, have certain quotas, limits, and constraints on deployment patterns, scale, and so forth. This recommendation must align to and keep within all hard limits and constraints.

Examples of such quotas and limits include:

• VPC quotas and limits
• IAM limits
• Enterprise account limits
• Transit Gateway limits

Rationale for separate production and nonproduction enterprises

A separate enterprise is used for production and nonproduction for several reasons:

• Separation of concerns. It is easier to reason about the overall system when nonproduction and production resources are clearly separated.
• Accounting. Ensures that R&D costs are separated from production costs that are required to be handled differently under GAAP and other accounting standards.
• Networking. Makes it possible to isolate the production and nonproduction networks and install rules that ensure that production applications don't accidentally depend on nonproduction resources.
• Scale. Doubles the maximum number of accounts and applications that can be managed.
• Alignment with IBM use. IBM Cloud uses this pattern internally, so it is well tested.

There are some limitations to using multiple enterprises, for example subscription commitments and credits are separately calculated for each enterprise at this time.

User Access

In general, users should not be directly provisioned within this account structure. Instead, use Federated identities in combination with trusted profiles or dynamic access groups to grant access permissions in accounts as required. Use of dynamic access groups or trusted profiles ensures cloud privileges stay in sync with the user's role as captured in the corporate directory and reduce the number of access policies that need to be managed.

Scaling down

Not all enterprises will require maximum scale immediately. In this case there are some options to consider:

• Single enterprise

  Place both production and nonproduction in a single enterprise. In this case, create a top-level production and nonproduction account group that contains the remaining accounts. This makes it easier to separate production and nonproduction resources for accounting, compliance, security policy, and so on. A single enterprise cuts the maximum scalability of the overall solution in half and results in a single bill.

• Reduced number of business units

  Deploy only as many business unit account groups as required. At a minimum, an enterprise can use a single business unit account. Some enterprises might want to consolidate multiple real-world business units into a single business unit account. The single BU approach should be avoided at scale as it does not provide individual business unit autonomy or separation of concerns.

• Reduced number of workload accounts

  Deploy only as many workload accounts in each business unit as required. However, it is not recommended to consolidate production and nonproduction into a single account.

Infrastructure as code

The central administration account hosts the infrastructure as code and configuration to perform the initial setup and ongoing maintenance of the key accounts in the enterprise architecture.

Authorization

Rather than authorizing users to deploy infrastructure as code directly, authorize project instances within the central administration account to deploy resources and create child accounts. Authorizing projects to deploy resources and create child accounts ensures that only changes to the enterprise account structure and key shared infrastructure can be made through projects. As projects are subject to the governance provided by catalog onboarding and project configuration management, this helps implement the zero trust security best practice that is required by many compliance programs, including Financial Services Cloud.

User access to the projects and resources within the Central Administration account is best performed with Access Groups or Trusted Profiles. This keeps the number of policies low, making it easier to reason about access permissions and reducing the opportunity for error. A typical organization should be able to operate their central administration account with on the order of 10 access groups or trusted profiles.

Rationale for centralized management across development and production

Centralizing management of deployable architectures and their configuration into an admin account provides the following benefits:

• Single place to find IaC related to enterprise administration. The admin catalog ensures that only approved, tested, and compliant deployable architectures are available to the user. Using a single catalog allows administrators to find and use these deployable architectures easily.
• One catalog to publish IaC related to enterprise administration. Having a single catalog to maintain makes it easier to ensure that the correct set of deployable architectures is exposed and maintained. This includes ensuring that updates to deployable architectures are available and deployed in a timely fashion.
• Centralized access control and monitoring for key Enterprise infrastructure. Placing the catalogs and projects in a centralized account means that only a single set of access policies must be audited to ensure the correct (limited) set of users have access to manipulate this infrastructure.
• Use of projects with trusted profiles also ensures that credentials with the capability to manipulate this fundamental configuration are not directly accessible to users and thus cannot be misused. Projects also ensure configuration governance, making it impossible for a single individual to modify key infrastructure.
• Only one schematics agent is needed.

Bootstrapping the central administration account

As the central administration account is responsible for laying down the majority of the enterprise account architecture, the enterprise accounts with IBM Cloud and the Central Administration account must be created first.

Steps to bootstrap the Central Administration Account:

1. Manually create a development and production enterprise based on two freshly created subscription accounts (docs).
2. Manually create a child account for centralized administration in the production enterprise (docs)
3. Onboard deployable architectures for the network and services account and for the central administration account to a private catalog (docs). The deployable architecture source can be stored in a private Git repo accessible over the public network during the bootstrap procedure.
4. Deploy the network and services account through a project (docs). This includes setting up direct link to provide connectivity to on-premises networking.
5. Populate the central administration account through a project. This includes provisioning the schematics agent to enable access to private Git repos stored on premises (required if these private Git repos are not accessible from IBM Cloud through the public internet).
6. (Optional) Update the private catalog entries for previously onboarded deployable architectures to point to a Git repo on the private network for future maintenance.

Rationale for centralizing shared resources

Some network infrastructure and certain key resources are costly and require lengthly lead times and manual labor to set up. These resources benefit from enterprise-level centralization to avoid the costs and delays that would be incurred if deployed at an account level. These needs must be balanced against the desire for separation of concerns and blast radius reduction.

Hyper Protect Crypto Services

Hyper Protect Crypto Services (HPCS) instances are centrally located so they can be shared across the enterprise to reduce costs. Two instances are required to reach the scale requirements of this recommendation. Smaller organizations can start with a single instance and grow as needed. When services are provisioned in workload accounts, service to service policies (in the network and services hub account) enable those services to access HPCS.

Centralized network services

A Transit Gateway in the network and services hub account connects all VPCs in the environment to each other with private networking. This unified private network is then linked to the corporate network through Direct Link. Network isolation is achieved within this flat network through VPC security groups and network ACLs. For more detailed recommendations on securely configuring VPC networking, see Financial Services Cloud Networking.

Make sure that VPC network addresses do not overlap across the enterprise.

A firewall might be deployed in this account or within the corporate network to ensure that traffic from the production environment cannot reach the development environment.

Virtual Private Endpoints (VPE) for shared IBM Cloud services are configured within a VPC in the network and services hub account as VPEs might be attached at only one point in the flat address space. This VPC also includes a custom DNS resolver to ensure the centralized VPEs are properly DNS resolved from all VPCs in the environment.

Some details on how VPEs should be centralized are still being worked out. This document will be updated when that information is available.

Security and Compliance Monitoring

Security and Compliance Center is most valuable when all relevant resources are scanned with an appropriate profile. Reasoning about resource coverage can be difficult in the face of independent Security and Compliance Center instances, complex scope configuration, and complex account structures.

Centralizing the Security and Compliance Center and configuring a broad scope that covers the business unit account group, makes reasoning about resource coverage easy while still allowing business unit autonomy over compliance objectives. This in turn, makes it easy to prove to internal and external auditors that all cloud resources within an audit scope are covered by appropriate Security and Compliance Center scans.

The instance of Security and Compliance Center deployed to the business unit administration account is configured to monitor all accounts in the business unit accoung group in both production and nonproduction enterprises (docs). It's also configured to send alerts with Event Notifications so that compliance problems are detected early and can be resolved rapidly.

Management / Edge VPC

The Financial Services Cloud reference architecture separates management and public internet access functions from the application hosting functions - see the VPC reference architecture for IBM Cloud for Financial Services and the Variation with edge or transit VPC for public internet access. These management and edge functions are centralized into the BU administration account to reduce cost and ease management and control.

Infrastructure as code

The Business Unit administration account hosts the infrastructure as code and configuration to perform the initial setup and ongoing maintenance of the workload accounts for the Business Unit. Workload accounts are created and managed in a self-serve fashion that is limited to the deployable architectures made available within the private catalogs for the Business Unit.

Authorization

Rather than authorizing BU account users to deploy infrastructure as code directly, authorize IBM Cloud project instances within the business unit administration account to deploy resources and create child accounts. As projects deploy only approved architectures from the catalog, this ensures changes to workload accounts are subject to the governance provided by catalog onboarding and project configuration management.

This authorization model helps implement the zero trust security best practices that are required by many compliance programs, including Financial Services Cloud.

Users should not be granted privileges to modify resources in the workload accounts.

As is best practice, users should be granted access to projects, catalogs, and observability tools within the account through access groups, trusted profiles, or both. An additional level of security can be obtained by ensuring that users normally do not have access to deploy changes to production. Users needing production deployment privileges must switch to a production trusted profile (which has privileges to deploy to production through projects), greatly reducing the opportunity for mistakes that can affect production. Production trusted profiles should be configured with a short session duration and with privileges only for production deployments, eliminating the possibility that users will simply use this profile all the time.

Rationale for centralized infrastructure as code management

Centralizing management of deployable architectures and their configuration into a production administration account for each BU provides the following benefits:

• BU operators can manage their own workloads within the constraints that are imposed by the centralized organization.

  The application and infrastructure catalogs ensure that only approved, tested, and compliant deployable architectures are available. Using two catalogs makes it easy to set an IAM policy such that users have access to the correct set of deployable architectures according to their roles. For example, DevOps users get access to infrastructure and application developers get access to application development tools.

• Centralized access control and monitoring for the BU

  Placing the catalogs and projects in a centralized account makes it easier to ensure that the principle of least privilege is applied. Use of projects also ensures that credentials with the capability to manipulate applications and infrastructure are not accessible to users and thus cannot be misused. Finally, keeping these related projects in the BU account makes it easy to monitor deployments and ensure that the infrastructure is up to date and compliant.

• Ensures that development, test, and production are aligned.

  Using a single project across nonproduction and production makes it possible to align development and test environments with production environments. The single project helps reduce the chance of defects that are related to environmental differences while providing control to the team over testing new deployable architecture versions in different environments. It also ensures that the lifecycle of these resources is properly managed across all environments. For example, if a project is no longer needed, it is easy to clean up all resources across nonproduction and production environments.

• Allows all project resources to be tracked for accounting and configuration management.

  Using a single project across nonproduction and production ensures that all project (or application) resources are tracked and allocated to the project. Projects enforce resource tagging and track resource providence, approvals, and so on. This ensures that accounting and configuration management needs are covered.

• Only one schematics agent is needed per BU.

Rationale for shared application infrastructure

Benefits of using shared application infrastructure:

• Less over-provisioning reduces cost

  Application hosting infrastructure must be sized to handle burst loads for an application. With a single application, it might be necessary to provision as much as 10x the steady state capability to handle bursts. However, when many applications are hosted in the cloud, the net load across many applications tends to smooth out as bursts rarely coincide. As a result, it might be necessary to only provision 10-50% extra capacity when you are hosting 10 or 20 applications thus providing considerable cost savings.

• Fixed cost elements can be shared across applications

  Application hosting infrastructure typically includes that certain resources can be considered a fixed cost per cluster or per VPC. Consolidating many applications onto the same infrastructure allows this cost to be spread across many applications for further cost savings.

• Operations savings

  Monitoring and maintaining an instance of application hosting infrastructure is approximately the same cost regardless of the size of that infrastructure. For example, maintaining a Kubernetes cluster is a fixed cost regardless of how many worker nodes it contains. Consolidating many applications onto the same infrastructure allows the cost of monitoring and maintaining infrastructure to be spread across many applications for operations savings.

Other considerations

Consider the following items when you are using shared application infrastructure:

• Application isolation is not binary

  Applications can be network that is isolated but still share resources such as compute, RAM, and storage within a cluster. Policy around allocation of such shared resources must be considered with tradeoffs between cost and isolation in mind. For more information about network isolation, see FS Cloud Boundary Protection.

• Not all application models can share infrastructure

  Shared application infrastructure makes sense only for sets of applications that have similar architectures. For example, sets of applications that use stateless microservices, containers, and managed database services can easily be deployed together. Other applications that are based on VM images with local storage will need separate application infrastructure. It is best practice to identify 2 or 3 common architectural patterns so that a small number of application hosting deployable architectures can be built and used for many applications.

• Interactions with centralized networking and services

  • Transit gateway

  Connecting the shared infrastructure VPC to the central transit gateway requires a process where both the centralized operations team and the BU operations team coordinate to connect the VPC.

  • Hyper Protect Crypto Services and other centralized services

    Authorizing the connection across accounts requires a process where both the centralized operations team and the BU operations team coordinate to create a service-to-service authorization for the consuming applications.

• Hosting multiple applications on Kubernetes

  Red Hat Open Shift and Kubernetes are designed for efficient hosting of multiple applications. However, it is best practice to leverage kubernetes names spaces for application isolation and istio for application network ingress and egress controls. The cluster configuration to support an application must be automated and that automation owned by the shared infrastructure hosting project, however, application owners can provide information to tailor the automation to there needs. Note that application owners should not have permission to modify the configuration of the shared infrastructure directly.

Sharing deployable architectures

After a deployable architecture version is ready for use, it can be shared with the central administration account and BU administration accounts for consumption. Leverage the catalog's ability to share private offerings to make the deployable architectures available within the enterprise.

Cost allocation

Projects have built in resource tagging mechanics that ensure all resources within the enterprise are tagged with appropriate project tags. When combined with IBM Cloud usage reporting by tag, enterprises can rollup all your cloud costs across many IBM Cloud accounts to your project teams.

Further cost allocations can be accomplished by tagging projects rather than individual cloud resources and allocating costs to organizational units or cost centers. From here, chargebacks and showbacks can readily be accomplished.

By using enterprise separation between nonproduction and production environments, project costs can be allocated between R&D and Cost of Goods accounting categories.

Because of the way projects tag resources, deallocated resource costs can be identified by using the Security and Compliance Center by scanning for untagged resources, indicating manually provisioned resources. Any untagged resources that are found indicate a gap in an access control policy and improper configuration management that should trigger a root cause analysis.

Resource utilization and efficiency

IBM Cloud monitoring can be used to monitor resource utilization across your various compute instances by allowing your enterprise to identify infrastructure right-sizing opportunities or to assess whether the performance, availability, or other quality metrics are of value for the expense incurred.

Centralizing monitoring can also be used to identify infrastructure consolidation opportunities where multiple applications can be on bare minimum hardware yet struggle to efficiently use the resources available. Consolidating the hosting of many small applications can minimize over-provisioning and improve availability through added redundancy.

High availability deployments

For high availability deployments, additional copies of infrastructure and applications should be deployed in the same workload account and by the same projects as the primary infrastructure. This makes it easier to ensure that additional regions are kept in sync and simplifies configuring access for DevOps teams.

Backups

Backups must be stored in a different region from the workload, and ideally in a separate backup account. This separation in location and management domain (account) minimizes the chances that an outage, operational error, or security breach that affect the workload will also affect the backup. Backup regions should be multi-zone IBM Cloud regions or highly available Satellite regions. However, each family of services on IBM Cloud has its own backup mechanism that introduces some limitations that impact this overall strategy. For more information, see service considerations.

In addition, a "disaster recovery kit" should be developed and stored outside of IBM Cloud. The disaster recovery kit is an off-site and protected repository that includes hardware, software, and system secure configurations, one-time keys, and the disaster recovery plan. Use this disaster recovery kit during an incident response to restore services.