Long term bidirectional migration via event stream
:::tip Guideline only As a rule of thumb, bidirectional sync fits user volumes above ~500k and a step-by-step app cutover. Volume alone is not enough — you need events from both sides. These are guidelines, not hard limits. :::
When we are talking about a long term bidirectional migration via event streams, we are referring to a scenario where data from the legacy system is synced to cidaas and from cidaas to the legacy system. This scenario is usually running for a few weeks. This approach is often preferred when:
- the amount of user data is quite high (guideline: above ~500k users)
- multiple apps (webshops, mobile apps, websites…) cannot be migrated at the same time
- authentication against the legacy and cidaas system should be possible at the same time
Here we can make sure, that updates which are happening in the legacy system are populated in real time to cidaas and the other way around. This scenario is interesting for all customers who want to migrate from the legacy system to cidaas, but who are not able to go-live with all applications at the same time. This means that the switch can start by for example only to a go-live with the webshop, which uses cidaas for the authentification. All other applications like mobile apps, will remain to authenticate against the legacy system.
How is this possible?
During the migration phase we will create a setup, which allows us to keep the data in cidaas and the legacy system in sync. This means all data of the user accounts like email, given_name, family_name and as even the password (better: password hashes) will be synchronized in case of any event, either triggered by cidaas or the legacy system. To understand this more clearly, we will take a look at the next graph:
On the left hand side, we see the legacy system, a custom wrapper in the middle and on the right hand side cidaas. Right at the beginning there will be a setup of the bidirectional sync between both systems. Therefore, as a precondition it is mandatory, that the legacy system can generate events for the changed data. E.g., if user changes his email or password, the legacy system should be able to inform about this change and deliver the data necessary to update the data in cidaas. If the legacy system is able to create this data, the magic of the mapping will happen in a custom wrapper.
What is a custom wrapper?
The responsibility of the custom wrapper is, to map the data of both systems, the legacy one and cidaas. Let’s say, the legacy system uses the field key "first_name" for the given name and in cidaas is uses the key "given_name". The custom wrapper will now take the data from both sources and map it to the correct key, depening, who should receive this. In general, cidaas will provide the events via webhooks, which can easily be consumed with a REST endpoint. The data itself from cidaas can then be accessed by using the sub from the webhook and using a token from a client_credentials app. Note: the data is not included in the webhook, to not extend the size of those too much, but the user can always be identified by the unique sub.
Hint: Since in those scenarios many data is being consumed, it is highly recommended to add some messaging queue in between, which can handle incoming messages.
After sync is in place, the first application can be moved to cidaas. By this the first app will now be switched to authenticate against cidaas, but the remaining apps will still authenticate against the legacy system. After this was successful, the apps will be moved from time to time to cidaas. During this phased rollout, legacy and cidaas act as co-sources so users can authenticate against either IdP with the same credentials. After the last application is switched, stop the sync and treat cidaas as the identity master.
Now after the last application is switched, the sync can be stopped and the full feature set of cidaas can be used in live mode!
Technical implementation
cidaas provides a REST Bulk Migration API:
| Path | {your-domain}/users-srv/user/bulk |
| Max batch size | 1000 users per request (recommend smaller batches, e.g. 50–100, via a queue for large migrations) |
| API reference | Migrate (add or update) user entities |
- API scope — The endpoint requires
cidaas:bulk_user_create. migrationMode— Only needed if you must overwrite passwords on users that were already migrated. Ask support to enable this configuration when required.
See the User Migration Guide for token setup and payload mapping.