Customer Data Onboarding – Automated, Recurring Data Exchanges with Customers

These days, business data is the lifeblood of any organization or business. Companies exchange business information daily with their ecosystem of customers, partners, vendors and suppliers. Automated, recurring data flows, where data is exchanged with external customers and partners, is called customer data onboarding.

This is especially relevant and mission-critical for businesses that provide data-driven services to deliver their services, depending on their ability to set up data integrations quickly – without putting an undue burden on their customers. These data integration connections are often difficult and time-consuming to set up and hard-to-maintain with changes, errors, and new additions.

For example,

• An employee benefits provider needs to get employee information and benefits plans enrollment information from its client employers for whom it is providing its services.
• A healthcare patient billing processing company needs to get invoices information from its customers who may be hospitals and clinics.
• A business services company that provides payroll processing services needs to get employee data from its client customers.

All these companies need to set up automated and recurring data onboarding flows with their customer organizations.

Customer Data Onboarding is a Key Process

Customer data onboarding bolsters existing data integration infrastructure to enable business users establish, monitor, and manage bi-directional data flow so that business data and information can flow seamlessly, securely, and accurately between the companies without interruptions and exceptions. It forms the essential foundation for business processes such as selling and delivery of products and services as well as invoicing and getting paid.

These data feeds, where business data flows in both directions between a company and its network of customers and partners, is critical for the company to efficiently provide services to its clients, especially to ensure the services are quality-driven, timely – and deliver a delightful experience to customers. Problems in the customer data onboarding data flows can lead to mistakes, errors, missed deliverables, inefficiencies, increasing customer churn and revenue loss. Many companies treat customer data onboarding as a part of Business-to-Business Integration (B2Bi) that typically involves data exchanges using standards-based (EDI X12) and non-standard (Excel, Text, PDFs) data formats. Customer data onboarding is sometimes considered a part of ecosystem integration initiatives undertaken by organizations.

Types of Business Data and Use Cases

Exchanging business data with the ecosystem of customers and partners is essential for most companies in almost all industries. Here are some examples in different industries of business processes and use cases where customer data onboarding is utilized:

• Manufacturing –Purchase orders coming from customers through online and offline channels need to be entered into ERP systems while invoices, shipping information, product catalogs and pricing information needs to be sent out to customers and distributors, etc. EDI X12 standards are often used.
• Healthcare –Plan enrollment information from online healthcare exchanges needs to flow to insurance carriers, hospitals need to verify enrollment with carriers, and patient data needs to be aggregated from multiple providers for analytics of trends etc. HIPAA and HL7 standards are common.
• Financial Services – Mutual funds needs to reconcile transaction data daily with the exchanges, retirement benefits providers and payroll processors exchange data with their customers, banks need to share money transfer and reconciliation data, mortgage companies exchange data with title companies and banks using MISMO messages etc. SWIFT and non-standard formats are often used.
• Insurance – Policies and claims data are exchanged between insurance companies and outsourced Third-Party Administrators (TPAs) or brokers (Managing General Agents MGAs), reinsurance companies exchange books of business and premium data with insurance companies etc. ACORD, X12 and non-standard formats are often used.
• Technology – Companies share customers and marketing data with external vendors and services providers, inventory, pricing, and discount information is exchanged with distributors etc. Multiple standards and non-standard data formats are often used.
• Logistics – Companies exchange orders, invoices, delivery and location updates with their clients. Logistics companies move products from point to point and have automated data flows with retailers on one side and manufacturers on the other side to share orders, inventories, deliveries, and shipping date information. EDI X12 standard and other non-standard data formats are often used.

Typical Data Flows

The electronic data flows for customer or partner data onboarding involve data moving bi-directionally from the company to its customers and partners and from the external entities to the company. Internally this data goes into critical applications, databases and systems which act as a system of record for storing and managing this critical business data.

These backend applications could be Enterprise Resource Planning (ERP) applications for manufacturing companies, policy and claims management systems for insurance companies, Customer Relationship Management (CRM) and marketing automation applications for consulting companies, etc. So, customer data onboarding requires a significant level of data and application integration activities using Extract Transform and Load (ETL) and Enterprise Service Bus (ESB) technologies. This internal integration activity focuses on data validations to ensure the quality and integrity of critical business data going into backend applications as that is “golden data”. Application integration is needed for data synchronization to ensure correct and relevant information goes into all the right places and there are no inconsistencies.

The far bigger challenge is external data sharing as that is characterized by a large variety of formats and protocols, hundreds or thousands of end-points, unpredictable failure causes and little ability to control the quality of the data. This is what makes customer data onboarding a huge challenge.

Data flows with external entities such as customers, partners, vendors and suppliers can happen via multiple channels such as:

• Automated File transfers –Data in different standard or non-standard formats is sent and received using different protocols such as FTP/SFTP/FTPS, AS2, HTTPS etc. Used for batch data and often scheduled as hourly, daily or weekly transfers.
• Email communication –Important business data is sent and received in emails either in the body of the email or attachments. Used for ad-hoc data exchange scenarios.
• Manual file uploads and downloads – Companies provide websites and portals for their external partners to manually send and get files. This is usually done for infrequent data exchanges.
• Application Programming Interfaces (APIs) – Lately many companies have published APIs for their external partners to utilize for data exchanges. Typically, applicable for frequent, real-time data flows.
• Cloud applications – Recently companies have started using cloud applications and cloud data stores for sharing data with external entities. These cloud applications are file storage apps like Box, Dropbox, Google Drive, Amazon S3 or cloud business applications such as Salesforce, HubSpot or cloud databases such as Amazon RDS, Azure SQL Database, Google Cloud Datastore etc.
• Agent software – Many companies want to reduce the burden on their customers and partners to connect with them, and to simplify this process they provide pre-configured agent or bot software. This lite software component is installed behind the firewall of the external entity and it can securely collect the relevant data from internal application or staging database and send it to the company via API calls.

Key Elements of Customer Data Onboarding

There are a number of common characteristics of data integration scenarios and use cases that define customer data onboarding. The business and technical needs for customer data onboarding use cases are:

1. 1.Automated data sharing – Since the ecosystem of external customers and partners may range from scores to hundreds or thousands of external companies, it is not viable to have manual, ad-hoc data exchanges. The automated, event-triggered data flows are certainly needed for customer data onboarding.
2. 2.Recurring data flows – The business needs of interfacing with the ecosystem of external customers and partners almost always requires that the data exchanges happen in a recurring manner either in real-time or over regularly scheduled time periods.
3. 3.Multiple formats and protocols – As normalized data formats cannot be mandated to customers and sometimes even to partners, there is often a need to handle multiple different data formats and delivery protocols.
4. 4.Both Standards and Non-Standards based – In some cases the data formats may be standard-based but they are non-standard formats in the majority of cases.
5. 5.Both batch and real-time – Data flows that involve exchanging of business data with customers and partners could be for delivering and processing bulk data (batch-mode) or transactional data (real-time mode), and so the solution needs to have flexibility to handle both scenarios.
6. 6.Both inbound and outbound data flows – Data exchanges with customers and partners always involve bi-directional flow of data which could be incoming data sent from the external entities in the business ecosystem or outbound data sent from the company to its network of customers and partners.
7. 7.Implemented by IT – Data flows with external companies and internal integrations with backend systems are implemented by the Information Technology (IT) staff. These are often complex projects that require extensive discussions with business users, operational teams and coordination with external contacts at customers or partners and testing before going live. Gartner estimates that on average a data integration between a company with its customer may take up to 8-12 weeks. During this time, business users and external customers wait for IT do finish these jobs.
8. 8.Data Integrity – Since quality and accuracy of data coming from outside cannot be predicted, inbound data flows often require extensive data validations and checks before it can be processed further into internal systems of record to ensure pristine quality of the golden data.
9. 9.Significant interactions and work by end-customers –Set up of data exchanges with external companies usually requires multiple interactions and IT development work on the end-customer side to be able to deliver and receive data. This requires setting up data extractions for pulling data and then sending to the company and also receiving as well as processing incoming data from company.
10. 10.Data exchange platform – Business users believe they have a need for a common platform where they can manage, monitor, and troubleshoot all the data exchanges with their external customers and partners. IT teams similarly dream of a single platform to create, configure and test hundreds of data flows and manage changes easily over time rather than working with disparate FTP servers, event brokers, integration tools, APIs and spaghetti code of scripts, programs, etc.

Customer data onboarding is one of the most difficult challenges for IT organizations to manage as new customers signed bringing forth requirements that are increasingly disparate and involve more complex data processing requirements. Until now, IT teams had to rely on toolboxes of different integration technologies and a significant implementation of custom code to yield data feeds as there was no single business application available to meet the requirements while offering both the scale of massively high volumes connections and data throughput as well as the incredible ease-of-use to offer a differentiated business user and customer experience. Fortunately, now there is a solution. Adeptia Connect is a self-service business application that dramatically accelerates customer data onboarding by up to 80%, providing delightful customer experiences and shortening the time-to-revenue.