ExternalLocalizationStore doesn't remove old values from AbpLocalizationTexts #9346

fjinc created 22 days ago

Hi
We're trying to hide certain localisations from the users in the frontend and ran into a problem and the eventual cause seems to be the ExternalLocalizationStore service not deleting old localisations from the ExternalLocalizationStore table. We found this out after we deployed a fix to our staging environment that removed a localization key value - however this did not get propagated to the database or the api/abp/application-localization api, meaning we could see the value we wanted to removed in the response.
I have solved this by disabling the ExternalStore feature and deleting the values from the AbpLocalizationTexts database table however this isn't ideal as it means we no longer have the ability to edit the localizations using the Language Management UI (since our localisations are distributed across different webhosts).
There is a similar problem with the AbpLocalizationResources table where the resource does not get removed if you delete it from the application altogether.

Steps:

Create a localization resource in your application with some en.json
Run the host and ensure AbpExternalLocalizationOptions.SaveToExternalStore is set to true in the host configuration.
Observe the resource and the values from your en.json gets added to the AbpLocalizationTexts table
Remove several keys from the en.json file
Rerun the host (ensure db is kept online to simulate prod/stg environment).
Observe the entry in AbpLocalizationTexts table still shows the keys you have just deleted.
This can also be checked by looking at the LanguageManagement UI and seeing the value there still.
I have cleared redis to ensure I'm not seeing old cached values and I can see they get repopulated upon page refresh so it isn't this.

If this is the intended behaviour and just requires manual patching each time we make changes to the .json files it is disappointing as it adds additional overhead to deployments.
Is there a fix for this that deletes older keys from the db to ensure only the latest is shown to the users? We are using ABP commercial 7.3.3.
Thank you ! :)

maliming created 21 days ago

Support Team Fullstack Developer

I will check this case.

Thanks.

maliming created 21 days ago

Support Team Fullstack Developer

Please send an email to liming.ma@volosoft.com

I will share the latest ExternalLocalizationSaver with you. then you can use it to override the default one(IExternalLocalizationSaver).

Thanks.

fjinc created 21 days ago

Thanks Maliming.
I can see this implements the option to delete old resources however the deleted text records in a resource dont get removed from the db still. Is this something that is being added?
Thanks.

maliming created 21 days ago

Support Team Fullstack Developer

These code will remove the text records

fjinc created 21 days ago

Thanks Maliming. I see the issue I had when I QA'ed this which is why I thought they didn't get deleted. I think it might be something wrong with the way the hash of the text records caching is created.

My steps were:

Run a version of our application where we add a new localization resource with text records
Run our application with changes to the text records localization resource.
Run our application with the new ExternalLocalizationSaver code in it.

When I run step 2, there are changes to the localization resource so the hashes of the db versus the .json file are different so it doesn't exit early from the UpdateResourceTextChangesAsync method. However, because I'm using an older version of the ExternalLocalizationSaver, the localizations in the db field do not get deleted. The hash for the resource gets updated into the cache, using the hash calculated from the localizedStrings - the raw .json text records.
When I run step 3 with the new ExternalLocalizationSaver, it exits early because the hashes are the same. This means it doesn't delete the old text records from the db field.

I've fixed this by clearing the cache between steps 2 and 3 which ensures it does not return early from the UpdateResourceTextChangesAsync method.

A fix on your side could be regenerating the hash that is to be stored in the cache at the end of the method from the textsInDatabase to ensure it captures any existing texts records stored in the db.

Screenshot 2025-05-22 at 12.52.50.png

maliming created 21 days ago

Support Team Fullstack Developer

ok, is the issue solved?

fjinc created 21 days ago

Yes I think so more or less, but I would suggest this fix could improve + reduce issues like this in the future.

A fix on your side could be regenerating the hash that is to be stored in the cache at the end of the method from the textsInDatabase to ensure it captures any existing texts records stored in the db.