As with everyone currently working remotely during lockdown, I became aware of how a broadband connection really has become the most important utility coming into our house. I’d just moved a WiFi booster box to try to extend the range of signal to the top floor of the house where the kids were online with their remote learning from school, but in the process somehow destabilised the whole setup. The WiFi was intermittently dropping in and out, with download speeds being fine one minute then a snail’s pace the next. Chaos broke out immediately – my work Zoom calls were impossible, the kids’ online classes were being affected and, worst of all, it was severely impacting their Xbox and YouTube activity. An interruption of our running water supply would’ve been met with fewer complaints. Indeed, a survey across 8 European countries last year found that broadband was more important to people than holidays, favourite TV channels, chocolate and even sex!
After a couple of unsuccessful attempts at rebooting the router I decided the only option was to try to disconnect everything in the network and add each component back one at a time. As I did so, it struck me that the situation I had got myself into was completely analogous to the problems faced by so many capital markets players.
Firstly, as I looked at the rats nest of cables and boxes in my AV cupboard I realised how complex our setup had become as we had added more components (Sonos speakers, multi-room connected TV boxes, WiFi extenders, Powerline adapters etc). Some devices were connected via ethernet cables, others relying on a WiFi signal, but it was impossible to tell what the heck was going on. I remember when we had first moved to the house and set up the AV system it was very simple and straightforward – a few wires from the router to a couple of devices. Years of additional new gadgets and component upgrades now meant that I had no understanding of how everything interacted. It had worked fine until I had made what I’d thought was a minor change and now I wish I’d never touched it. This situation is exactly analogous to that which investment banks and asset managers currently find themselves in. Their platforms have grown organically over time with bolt on components added as a solution to short term business requirements. However, what they are left with is a set of fragile integrations which make every change highly risky. The consequences of this are change becomes ever more costly as careful manual checking is required across the stack, the pace of change slows and hence each release gets loaded with bigger requirements which makes each upgrade ever more risky and needing increasingly rigorous testing.
As I carefully continued to trace each cable back from device to router, I was reminded of another issue plaguing capital markets businesses – that of data lineage. Data is passed from one system to the next in a complex chain of interdependencies and it becomes impossible to trace derived results back to their source inputs. Results are aggregated at points along the processing pipeline before they are pushed to systems further up the stack and these lossy transmissions mean that it becomes very difficult to trace back to the root causes of problems with the end numbers – enter the world of manual data downloads and excel sheets to piece together a reconciliation. A related problem is that of data consistency – if you can’t ensure that the inputs to the various calculations being made by the disparate systems are identical then when you join those results back up you will be prone to getting nonsensical analysis. The classic example of this is where a trade is booked at around the cutoff time for the risk and P+L runs and makes it into one but not the other – when you bring the results together in your P+L explain report you are going to have issues.
Most organisations attempt to work around these issues by using Event Driven Systems to carefully orchestrate the run order problem. However, current “bleeding edge” data processing systems suffer from incorrect results when processing times fall outside of the expected range. Edge cases are commonplace and must be dealt with via manual intervention and consistency issues must be isolated before being offset with adjustments elsewhere. At this point problems are unfixable without careful major surgery and dealing with this complexity is the root cause of the industry cost problem.
At Ediphy we have built our platform from the ground up to avoid these issues by design, allowing us to embed huge efficiencies into our own operations and that of our clients.
Please get in touch if you want to find out more.
PS. I solved my broadband issue. As I carefully reconnected everything, I found I had created a double NAT issue (unintended data inconsistency issue!!) and once this was fixed a healthy broadband speed resumed and a happy household was restored.
