Data integration in a South African landscape

The past few years have taught us a few things, one of them being agility. South African companies that adopted the more innovative approach have been better able to react quickly to changing market conditions and adapt to the current situation. As a result, more of these dynamic companies are moving away from a centralised approach which has proven to be a safer option for data protection.

With COVID permitting, Altron Systems Integration in collaboration with Teradata hosted their annual data Innovation Forum. The event included a discussion on an engineering approach for Data Mesh deployment that provides a practical architecture and governance framework to deliver connectivity across domains while encouraging a decentralised approach to stewardship and implementation.

For Stephen Brobst, Chief Technology Officer at Teradata, an agile approach plays a vital role when it comes to implementing the Data Mesh Concept. “A monolithic approach to building systems rarely succeeds as opposed to a domain-driven design approach. Companies should deploy autonomous, decentralised teams while avoiding anarchy in architecture and deployment,” he said. Key benefits of this approach is scalability and focus as it allows domain experts to optimise time spent on solution delivery.

Also speaking on the state of the data landscape in South Africa, Theo Spickett, Data Platform Practice Lead at Altron Systems Integration said, “South Africa has been on our own unique journey with Data in the past 5 years.  In some areas, we are lagging behind the rest of the world, in other areas we are able to leap-frog as we don’t have the legacy to hold us back. Given this, we have our own unique challenges and opportunities for solving problems businesses may face.”

According to Chris Hillman, Data Science Director at Teradata, numerous organisations battle with the “proof of concept to production gap” when it comes to scaling Machine Learning and Artificial Intelligence (AI). He outlined a few case studies that show how a ModelOps approach can address this issue. He shared some practical recommendations from the field for Machine Learning data architecture, technical approaches for scaling and model deployment.

Brobst also spoke about the co-existence and collaboration of AI and humanity. There is a far greater need to explore ways that ensure that Al is used ethically. Questions that often arise include, “Will AI replace humankind? What skills should a person develop in order to remain relevant given the prevalence of neural networks?  What will happen with AI in 10 years?”

For AI to work, there will always be an effort from people. After all, 50-80% of the time is taken preparing raw data. This includes Data integration, Data access and exploration, Data cleansing, Feature engineering, and Feature selection.

South Africa is rapidly adopting public cloud, machine learning and AI. Given our unique challenges and opportunities, it is more important than ever to direct investment and limited resources towards a sustainable architecture, process and organisation.