AI-based Data Mapping

How does AIMap work?

AIMap works by utilizing existing data maps in your library. It learns mapping patterns between existing data fields and trains a neural network to suggest future data mappings. To understand how AIMap works, think of 3 distinct steps:

Step 1: Training

• Data Maps are put in the AIMap Library. These data map objects in Adeptia have been tested in a business environment and are confirmed to contain validated and correct data transformation rules.
• AIMap uses these data maps in its library to train the neural net and learn from these mapping rules.

Step 2: Suggesting

• AIMap is triggered by a user creating a new data map.
• AIMap scans the source and destination schemas and uses its algorithms and neural net to find mapping rules for each target field. Upon finding most appropriate rules, it ranks them in the order of decreasing probability.
• It then selects the suggestions with highest probabilities and displays them by High, Medium, Low confidence levels.

Step 3: Reviewing

• User views the suggestions and is guided by the confidence levels predicted by AIMap.
• Accept/Reject options in the review process allows user to work through the suggestions sequentially.
• User can load and run a data file to see actual results of the mappings and correct them.

Real world use cases for AIMap

A large manufacturing company had to create 1300 data maps of different source order types for processing automatically by the Adeptia application. Although there were 1300 different source formats, the destination format was always the same normalized schema that had about 60 data fields.

After the manufacturing company had created the first 25-30 data maps and put them in the AIMap library, the subsequent data maps were created very quickly and efficiently. AIMap was able to learn from those initial data maps and then suggest data mapping rules for subsequent mappings. This significantly reduced the effort required for developing and testing the 1300 maps by a factor of 10.

Another large mutual insurance company had 6000 legacy data maps that were created in the old Mercator product. These were not being maintained and were impossible to edit or change as there was little documentation. They decided to move all of these to the Adeptia application and a large project was started to migrate and implement them. The source formats were different and varied, but the target format comprised a few types of normalized schema of destination applications. In this case, AIMap neural net was able to effectively learn after the first few dozen data mappings and then was able to give high-quality suggestions that dramatically reduced the expected effort to migrate the thousands of data maps.