Feature proposal
Purpose:
Every customer using hyperportal cms needs to interact with legacy services, which are called 3rd party services in this document. To provide ability to use 3rd party data sources hyperportal has to have functionality of sending/receiving data to/from external erp/crm/dms systems. Additionally customer wants to use the only services he needs. Currently hyperportal does not support setup of particular service. And import export in most cases has to be done manually by support team.
XDMS integration description
The high level diagram of basic data pulling implementation is below:
The core entry point is a module definition, which is an entity with several fields:
- serviceId - identifier of the service, in this case it going to be XDMS
- moduleId - identifier of the particular data source in scope of service, can be organizations, contacts, vehicles etc...
- token - auto-generated unique security token for client identification
- configuration - data mapping configuration - hard coded during bootstrapping the module and can be changed
- availableFields - fields that are exposed by the data source - hard coded during bootstraping the module, can not be changed by the user
- enabled - defines if particular module is enabled for the current tenant
Basic workflow
XDMS calls API endpoint:
https:xxx.xxx.xxx/api-integration-push-data/{token}/{tenantId}/{serviceId}/{moduleId}
where:
- token - security token
- tenantId - current tenant id, where the data has to be consumed
- serviceId - defines which service is used
- moduleId - defines which module is used
Request validation process:
After successful request validation the data is saved using already existing storage API. The pattern of data key is tenantId/serviceId/module. Once data saved in S3 bucket - new topic is triggered and SQS queues notification for background importer service. Background importer service gives notification, according to file name, gets integration entity and maps data according to mapping configuration. Then the entity is saved inside the scope of current tenant.
Challenges:
- As puled data can be JSON/XML/CSV - we have to convert everything to CSV. In that case on data saving stage, we need to define mime type of the payload and convert data accordingly.
- We need to define rules to convert not flat JSON data.
- We need to define rules to convert not flat XML data.
- Can we run additional ECS task for each import procedure to make it run quirkier.
- We need to define who will receive the logs for each import iteration.
- How to move from existing implementation of XDMS import to proposed one.