Building, managing and operating a permissioned blockchain network is no trivial task. What appears on the surface to be little more than a few nodes and some networking configurations, is actually a pandora’s box of technical complexities and challenges when the veneer is peeled back. Issues like high availability, scalable infrastructure, disaster recovery, rolling upgrades, systems integration, identity schemes, security models, firewalls, member on-boarding, etc. exist as first-class concerns and can quickly morph into significant barriers, and even more so when taking into account that these networks are being built on rapidly evolving open source protocols. Because organizations have shared IT responsibilities to collaboratively govern and operate these business networks, solving these problems across a group of IT Ops professionals from multiple companies becomes even more difficult. This article will go through a high-level exploration of the various engineering roadblocks that typically emerge with a “do it yourself” type approach and share how successful business networks can leverage Blockchain Software as a Service (BaaS) to drive business value across new business models and cross-industry ecosystems.
How Local Blockchain Deployments Have Evolved
A few years ago, simply spinning up a local blockchain instance was an immense challenge. Technologies like Docker and Kubernetes were still in their infancy and the underlying code bases of the three core blockchain protocols (Ethereum, Fabric and Corda) were still being developed and hardened. It took some serious technical chops and heavy knowledge of crypto, hashing algorithms, curves and peer-to-peer protocols to even patch together a quasi-functional system.
Fast forward to 2019, and you can quickly pull or build a few Docker images (i.e. the runtime components) and use a lightweight YAML file (human-readable data serialization language) to bring some containers to life on your personal machine. Thanks to hardened shell scripts and pre-mounted config files, the ease with which this is accomplishable is pretty awesome, but it can also lead to a false sense of confidence. Devs and power users alike can fall into a mindset of, “Wow, that wasn’t so difficult, we could totally run this ourselves.” And while it may seem that way initially, these miniature sandboxes are in no way representative of an enterprise-grade, multi-org, production-caliber decentralized business network.
Authentication and Permissioning
When operating in a permissioned or private blockchain environment, nodes need some trust mechanism to determine who they are willing to communicate with. This typically involves some combination of transport layer security (TLS) certificates along with IP addresses and/or unique node identifiers. In a centralized or local deployment, this doesn’t present too many challenges. Simply mount the file systems with the appropriate certs and static configuration files, and you’re off and running.
However, in a decentralized architecture where nodes are running on different servers and potentially different cloud providers, this authentication/trust paradigm becomes a lot more cumbersome. You’re now tasked with coordination amongst separate parties in different places, all of whom aren’t inherently trusting of their counterparts. The necessary out-of-band communication streams can be costly and time consuming, and are heavily susceptible to error. Moreover, the manual transfer of configuration data such as certs introduces an attack surface that can be exploited by a malicious actor.
A managed blockchain service helps alleviate this tedium of cross-organizational reconciliation and greatly reduces potential attack boundaries. The ensuing overhead and management savings are immediately quantifiable, as production-grade consortia networks are now able to be securely bootstrapped in a matter of minutes, as opposed to weeks or months.
When evaluating BaaS platforms, look for a provider that makes no compromises on the integrity of the critical blockchain layer trust components (keys, certificates and config definitions). These pieces should be solely isolated to the runtime processes that require them and should remain strongly encrypted at rest. Certain platforms might even provide extensible encryption techniques, allowing for organizations to leverage hardware-protected master keys to apply an additional layer of organizational-controlled security.
Along with this security due diligence, seek out platforms that also offer techniques for transparent and self-evident identity assertions. These tend to be cryptographic proofs in the form of digital x509 certificates, that accommodate KYC (know your customer) and AML (anti money laundering) concerns, and allow fellow members of the network to authoritatively identify their counterparts by inspecting an uploaded certificate chain. Preferably, these PKI-based proofs should be woven into the blockchain layer such that the enterprise identity trust paradigm accompanies any on-chain transaction.
Decentralized Networking Made Easy
A local blockchain network is easy enough to wrap your head around. All of the containers are likely running in a virtual machine (VM) on a single server, and can seamlessly peer with each other. The dilemma arises when you start to introduce more complex peer-to-peer networks, with nodes in different cloud vendors forming virtual networks through virtual tunneling and firewalling. The subject matter expertise required to make this work should not be understated. The world of VLANs, secure tunnels, asymmetric encryption, IPsec, routing, Kubernetes, Helm Charts and microservice architecture becomes very real, very fast.
A managed blockchain service can remove the headaches associated with the minutiae of cloud deployments and virtual networking by abstracting the technical complexities and allowing a user to quickly and confidently deploy nodes and resources via an intuitive and straightforward approach. An exemplary platform will provide multi-cloud and hybrid deployment options, thereby delivering business and software continuity with engrained processes and hardened stacks. And while cross-cloud compatibility is incredibly valuable, aim for a provider that delivers this functionality within a single unified experience. The upside of a hybrid network is greatly diminished if organizations are operating in a fragmented and isolated fashion.
The Importance of High Availability, Disaster Recovery (HA/DR) and Scaling
A blockchain network is only as useful as its performance, accessibility and robustness. If your node is down or the network is becoming suffocated due to heavy traffic or an overstrained consensus algorithm, the entire apparatus can crumble. Scalability, high availability and disaster recovery are critical tenets that must be accounted for in any legitimate production deployment.
Immense protections need to be in place that not only offer failover support, but also account for the unlikely but potential disaster scenario. Moreover, networks need to be designed to support dynamic scaling and contraction, without interrupting ongoing business processes. If five new nodes join a network, they should be able to quickly synchronize their ledgers and participate in the configured consensus algorithm if they are properly designated. Lastly, robust checks and balances that safeguard against unsound consensus configurations must be implemented. For example, when running certain consensus algorithms there can only be n number of nodes that are participating as signers. Exceeding the “n” threshold would implode the system due to overwhelming traffic.
An enterprise-grade BaaS platform should prioritize HA, DR and scalability as first class citizens. Look for a platform that has features such as auto-scaling Kubernetes clusters, data redundancy, consensus protections and automatic failover across multiple cloud availability zones. In a non-managed service, the costs associated with ensuring these safeguards can easily reach into seven figures and likely involve multi-year contractual agreements with integrators or IT services firms. A managed service can slash these costs and remove the responsibility from the network’s stakeholders.
Assistance with Advanced Troubleshooting
If you’ve ever looked at the logs for a Geth node or constellation module, you’re aware that this is basically the weeds of the protocol. You’re bombarded with information about hashes, blocks, tries, uncles, canonical state, stale transactions, etc…. And even for the advanced blockchain developers out there, these logs can be esoteric. So the burden of debugging any number of problems that can arise in a private Ethereum blockchain (e.g. queued transactions, stalled nodes, gas limits, certificate mismatches) now falls upon the operators of the network; and anyone who has been tasked with this type of troubleshooting can attest that it is not a pleasant exercise.
An enterprise-grade BaaS provider can minimize countless blockchain errors by optimizing protocol layer configurations. This is incredibly valuable, because many of these low-level nuances are immensely esoteric and can prove to be programmatically challenging to even an advanced collection of blockchain sys admins. BaaS helps prevent misconfigurations from metastasizing into stalled and unusable networks. An ideal platform will not only provide a reliable and consistent protocol tier, but expose 24/7 service support to assist with human-manufactured chain layer errors (e.g. queued transactions due to nonce gap).
Shared IT and Rolling Upgrades
Believe it or not, we’re still in the early days of blockchain and the protocols are continuing to evolve at a blistering pace. Consensus algorithms grow more robust, modules are rewritten in more accessible languages, APIs are added, privacy techniques are heightened, etc…. Organizations naturally want to leverage these enhancements, rather than remaining stagnant with out-of-date and less-performant code bases. Imagine you see an update for an app on your phone fixing a security vulnerability, chances are you’ll want to download it. The same methodology applies to blockchain. Enterprise organizations want and need the latest and greatest
Coordinating runtime updates across a network’s stakeholders can be a formidable endeavor. You need assurances that all parties are in possession of the exact same software stack and that the library has been packaged properly and securely in a deployable framework. You then need to account for any reconfiguration changes that accompany this drastic alteration to the network (e.g. do certs need to be refreshed?). And finally, once all of the data and has been exchanged and vetted, you need precise coordination to prevent extended downtime of the network. It’s hard to enumerate the number of problems that can arise with this scope of shared IT.
Look for an enterprise-grade BaaS where rolling upgrades take place as a single collective experience, and where the ensuing network downtime is truncated to its absolute minimum. This unification and administrative simplicity is beyond valuable for business continuity and persisted availability.
Growing your Consortia and Instituting Governance
Perhaps the most difficult challenge to overcome when attempting to craft a blockchain network is the formation of a consortium. At its core, this is a human problem that requires picking up the phone and/or getting in a room with the various stakeholders and reaching some level of agreement. This can be an incredibly drawn-out process and it oftentimes requires a broker or arbitrator to help the group reach consensus. In some cases, an entirely separate legal entity is formed just for this purpose, as was seen with komgo’s commodity trade finance network.
Unfortunately, the difficulties don’t subside once the consortia is formed and incorporated. Now you’re tasked with on-boarding the individual organizations to the network and ideally implementing some degree of governance and access control. In a consortium, it’s atypical for every party to exist on equal footing. Some may require super-user degrees of authority and some may only need a lower number of permissions. Configuring and enforcing these various tiers of governance is incredibly complicated, and many consortia actually see their networks stall as they attempt to stand up such an elaborate tapestry.
A managed service can deliver incredible collective value to a consortium by embedding governance and permissions directly into the workflow. An ideal platform will allow for any out-of-band agreements on access control to be woven into the consortia’s configuration in a radically straightforward manner. This aspect is absolutely critical because the quicker all parties can be on-boarded to the platform, the quicker the business value can be extracted.
This article could likely meander on for another five pages as we explore features such as firewalls, network isolation, distributed denial of service (DDoS) prevention, risk compliance, cipher suites and more. And these are only the chain layer and networking pieces! In order for a bonafide production solution to crafted, there needs to be componentry for systems integration, user management, privacy, wallets, storage, oracles, legal agreements, asset modeling, etc…
The synopsis is that there are hundreds of pieces hiding beneath the surface or in plain sight that are required for a network to assert itself as “enterprise-grade.” And compliance with these requirements can be not only technically challenging, but also astronomically expensive and time consuming. Look for a BaaS that has accounted for these mission-critical pieces of scaffolding, and surfaces them in a consumable and pluggable manner.
For a recent industry assessment of BaaS providers, check out the article “A No-BS Guide to the Blockchain as a Service Space” by Invector Labs.
About the Author
Nick Gaski leads client enablement at Kaleido, the Blockchain Business Cloud. Nick has worked extensively across numerous enterprise blockchain protocols and is an ardent evangelist for the revolutionary potential of the technology.