Features - Video Block - Left Vimeo V2

Any Integration, Anywhere. Securely, in minutes.

If you'd like to enable your business users to create secure Any-to-Any integration within minutes, ask us for a short demo.

Request Demo

Adeptia Connect - Large File - Challenge of Processing Large Files

Adeptia Connect - Large File - Challenge of Processing Large Files

How Adeptia Handled the Challenge of Processing Large Files 

Our customers base is a strong support for us. While talking to them, we found that most of our large scale customers, which includes giants in the Insurance and Finance domain, were facing a similar challenge. They were getting data in multiple formats, such as XML, CSV, or PDFs, and had to process files that ranged from a few KBs to 100s of MBs to 10s of GBs per file.

They not only needed data ingestion for large, flat, or hierarchical files, they needed streaming transformation in parallel to process the data and deposit in a data warehouse. The data needed to be cleansed of errors, validated with business rules, transformed by normalizing into a common format. The challenge was multi-fold.

When our customers ran these large multi-GB files thru their existing data integration software, those applications immediately crashed. Our customers increased the memory and system requirements, reran the large files for processing, and the applications crashed again. Then our customers wrote custom scripts and programs to process these large multi-GB files, but still some of them were processed but most crashed the custom programs. These mundane efforts of data preparation and data feeds transformation were complex and difficult to operationalize.

A software solution that was free from the limitations of available solutions was needed, and Adeptia recognized that need.

Adeptia built a large file data ingestion feature that processes multi-GB files, ingests and transforms large volume of data, and delivers that data in a common format timely and reliably. Adeptia’s software solution processes both flat and hierarchical files in any format - XML, CSV, Text, or PDF - and delivers to a normalized format or data warehouse.

Adeptia data feeds ingestion solution is a fully managed, simple and extendable model for efficiently extracting, and moving large amounts of data. Our solution supports many use cases: real-time analytics, continuous computation, Data Lake, etc. It is scalable, fault-tolerant, and easy to setup & operate. 

Adeptia Connect - Large File - Stunning Results

Stunning Results

The Adeptia Large File Data Ingestion software solution went through rigorous benchmarking and testing, and the results were remarkable and better than anything we’ve seen in the industry.

  • A single 25GB XML file with insurance claims information is successfully processed with complex transformation rules in 33 minutes.
  • A single 200GB XML file with insurance claims information is successfully processed with complex transformation rules in 4 hours.
  • 50 different XML files of 25GB each with insurance policy information are successfully processed in parallel in 10 hours.
  • 10 different text files of 5GB each with application log data are successfully processed in parallel in less than an hour.

These performance tests were run on an X-Large instance (m4.xlarge) on Amazon AWS that has 4 cores and 16GB RAM with 8GB allocated to the Adeptia application.

Adeptia Connect - Large File - Data Ingestion Feature

Adeptia Connect - Large File - Data Ingestion Feature

Real World Application of Adeptia’s Large File Data Ingestion Feature

Our large file data ingestion capability has multiple real world applications, including how our clients are currently using this functionality to drive business intelligence.

A US Department of Health and Human Services backed medical research agency aggregates sensitive medical data from all around the country. The research agency interacts with medical centers, health clinics, and medical insurance providers to receive medical records in multiple formats and from multiple sources. Adeptia’s solution acts as the central receiver of this data, ingests it, transforms it while streaming it into the agency’s data warehouse for driving analytics, research and decisions.

A large North American credit union has connectivity with smaller credit unions across the country to exchange and aggregate data. The data comes from multiple source applications and databases and in multiple non-standardized formats including large multi-GB XML and CSV files. This data is ingested by Adeptia’s software, streamed and transformed in parallel, and ultimately sent to the data lake at the credit union.

As a general scenario, our Large File Data Ingestion feature helped in handling large incoming volume of data at large enterprises (hubs) from multiple external or internal sources (spokes). This feature processes files that are multi-GB in size, accepts all formats and file sizes, including flat or hierarchical files, and ingests and streams data in parallel to deposit in a central data warehouse or data lake at the hub company. Adeptia solution is proven in production environments for ingesting data feeds which are continuous or asynchronous and real-time or batched with no data loss and no human intervention. 


Adeptia Connect - Large File - Benefits

Adeptia Connect - Large File - Benefits


Adeptia’s software approach for handling large multi-GB files offers many benefits over traditional solutions for large data file processing.

  • The software-based approach (rather than relying on hardware appliance) allows non-technical business users to process large files easily without manually coding or relying on specialized IT staff.
  • With reduced load on specialized IT staff, Adeptia’s solution reduces manual effort and system resource costs, ultimately accelerating delivery time.
  • Hubs and large enterprises do not need to support expensive infrastructure or specialized servers for supporting appliances for large file data ingestion

Adeptia’s unique approach of parallel ingestion of large data along with runtime data transformation and streaming is a competitive edge that lets you save time, accelerate service delivery, and fast forward revenues.

Testimonial - TEC Services

  • Image
    The bottom line is that Adeptia enabled us to successfully bring a new service offering to the marketplace, generating millions in revenue for our clients, and rapidly grow our business.
    – Tom Sweat, President, TEC Services Group

Cloud - Help your Org

See how Adeptia can help your organization.

Request Demo

Stay in Touch
Be the first to know about product updates, press releases and news.