Summary / Context
Data migration and remediation are critical steps in transitioning to Fenergo. This article provides an overview of a structured approach to migrating data from existing systems, including key entities, documents, related parties, reference data, and other critical domain data. The article examines leveraging different remediation approaches depending on volume, impact, and complexity. It also incorporates guidance on selecting the appropriate migration tools or methods (Manual UI, Bulk Load, Custom APIs/Ingress, ETL) based on volume, complexity, and operational requirements to ensure successful migration and minimal disruption.
Common Challenge or Scenario
Many institutions face challenges in migrating large data and documents from existing systems to Fenergo. Typical issues include duplicate records, incomplete datasets, misaligned entity mappings, missing documents, unclassified documents, incorrect related party relationships, and inconsistent reference data. All of these can lead to delays, rework, and compliance risks if handled incorrectly.
Root Cause / Contributing Factors
- Lack of early alignment on migration scope and data ownership.
- Insufficient data quality assessment and remediation planning.
- Complexity in mapping source data, documents, related parties, and reference data to the Target Operating Model in Fenergo.
- Resourcing dependencies – technical and operational staff
- Testing environments with suitable data
When considering Data Migration and the associated Data Remediation, use the following framework heading to analyse your specific situation and create a plan. These headings are built on many years of working with our clients on migrations as part of the wider Fenergo deployments.
Best Practice / Recommended Approaches
- Migration Strategy aligned to Business Outcome.
The following overview is a standard way to plan for and execute the Data Migration and Remediation process.
- Migration Approach
- Adopt a Big Bang migration approach where feasible, minimizing the need to reconcile two systems simultaneously.
- If Big Bang is not possible, use a segmentation approach with minimal overlap between datasets.
- Avoid migration of In-flight cases. They introduce significant complexity which can be avoided by arranging for cases to be closed prior to migration.
- Consider remediation approaches ahead of time, defining volume and impact thresholds to decide between manual updates, remigration, or custom API solutions.
- Data Cleanse
- Conduct a completeness and quality assessment for all items to be migrated including entities, documents, related parties, reference data, and other domain-specific data. Developing an early understanding of your current state sets a strong data driven foundation.
- Determine which data can be cleans in advance, during the migration and which data can only be cleansed post migration. The objective is to minimize the operational activity post go-live.
- Validate business rules, including active/inactive status and client-association relationships.
- Perform de-duplication across the source system agreeing on unique identifiers like Legal Name and Date of Birth.
- Involve the location operational and business team early and throughout the process. Obtain sign-off from those team on the cleansing and deduplication results.
- Migration Tooling
- Based on your migration and cleansing strategy, select the appropriate combination of methods to use depending on data volume and complexity:
- Manual via UI: Suitable for small-scale updates (<1000 records), executed by Client Operations Teams, using existing journeys and manual reconciliation.
- Bulk Load via UI: Suitable for medium-scale remediation (<10,000 records), executed by platform admins/maintenance teams, using BL Policy/Journey and XLSX files.
- Custom APIs / Ingress: Suitable for complex or high-volume updates, requires expert knowledge and custom client application or Postman; performance is relative to solution complexity.
- Migration via ETL: Suitable for mass ingestion of domain data; large updates can be performed using Alternate IDs or Entity IDs, with throughput up to 70k entities/hour, 70k products/hour, 60k associations/hour, 180k documents/hours.
- Utilize migration tooling for documents, products, related parties, reference data, and other domain data, leveraging SQL scripts to create staging tables in existing data schema.
- Testing and Validation
- Implement multi-stage testing: development (unit), UAT (acceptance), and Gold tenant (production-data validation).
- Anonymized datasets are much less effective than working with production data. Plan to work with product data from early in the process and ensure clearance to work with this data is also obtained early.
- Review reconciliation reports after each batch to identify failures and take corrective action. This is a highly iterative process plan on frequent test runs.
- Determine remediation approach per issue severity and volume, ranging from UI-based fixes for low-volume, low-impact issues to API-driven remigration for high-volume/high-impact scenarios.
- Build reconciliation into all actions to ensure full evidence of all data actions.
- Runbook and Reconciliation
- Maintain a detailed migration runbook assigning responsibilities, dates, and durations for each step.
- Generate reconciliation files to track successful vs unsuccessful migrations, including entity IDs, status, and last update date.
- Resource Planning
- Plan that Data Migration and Remediation activities will start as early as possible.
- Define responsibilities for both client and Fenergo teams using a RACI approach.
- Ensure stakeholders from Enterprise Architecture, Security, Product Owners, programme management, and business teams are engaged.
- Secure technical resources for database access, ETL execution, and troubleshooting.
- A smaller knowledgeable detailed-focused technical migration team is preferable to larger generic technical team.
- Existing Systems Decommissioning and Historical Data
- Plan to retire existing post-migration systems, potentially maintaining report-only access.
- Historical data not tied to active processes may remain outside migration scope; ensure audit trails and compliance requirements are captured.
Key Takeaways
- Confirm migration strategy early; Big Bang minimizes reconciliation complexity.
- Conduct thorough data cleanse and deduplication before migration.
- Include documents, related parties, reference data, and other domain-specific data in migration scope
- Choose migration/remediation method based on data volume, complexity, and operational requirements (UI, Bulk Load, Custom APIs, ETL).
- Establish a structured runbook and validate through staged testing.
- Assign clear roles and responsibilities for client and Fenergo teams.
- Plan remediation strategy for post go-live issues, defining thresholds and approach depending on volume, impact, and complexity.
- Reconciliation and error management are critical to ensure integrity.
- Plan for existing systems decommissioning and consider historical data handling.
Future Guidance
For more tailored guidance based on your specific functional, technical and data circumstances please contact the Advisory Services Team through your Client Success Manager.