[horde] Migrating user data from an old horde instance to a new installation

Sun Mar 5 14:40:14 UTC 2017

Zitat von Michael Menge <michael.menge at zdv.uni-tuebingen.de>:

> Hi,
>
> Quoting Christian Schoepplein <c.schoepplein at musin.de>:
>
>> Hi,
>>
>> On Fr, Mär 03, 2017 at 04:56:43 +0200, Vilius Sumskas/LNK wrote:
>>> "horde" <horde-bounces at lists.horde.org> wrote on 2017.03.03 16:48:06:
>>>> I have a damn old horde installation which has to be migrated to a
>>>> current setup.
>>>>
>>>> Old installation:
>>>>
>>>>  * Gollem: H3 (1.0.2)
>>>>  * Horde: 3.1.4
>>>>  * Imp: H3 (4.1.4)
>>>>  * Ingo: H3 (1.1.3)
>>>>  * Kronolith: H3 (2.1.4)
>>>>  * Turba: H3 (2.1.5)
>>>>
>>>> New setup:
>>>>
>>>>  * Horde: 5.2.13
>>>>  * Content: 2.0.5
>>>>  * Gollem: H5 (3.0.10)
>>>>  * Imp: H5 (6.2.17)
>>>>  * Ingo: H5 (3.2.13)
>>>>  * Kronolith: H5 (4.2.19)
>>>>  * Nag: H5 (4.2.13)
>>>>  * Timeobjects: 2.1.3
>>>>  * Turba: H5 (4.2.18)
>>>>
>>>> Migrating and adapting the config from the old to the new system causes
>>>> me no trouble, but I am not able to migrate all the userspecific data
>>>> stored in the mysql db backend.
>>>>
>>>> for that reason I see two possebilities now:
>>>>
>>>> 1. Try again to migrate the database from the old horde setup to the new
>>>
>>>>   installation. Because I have no idea what went wrong during the
>>>>   first try, it is very likely that I will run into the same issues
>>>>   again :-(.
>
> We did the migration a few years ago. But we don't use Gollum, and we
> use Postgresql. There where some problems we encountered.
>
> As far as I remember most common causes where:
>
> 1. memory usage of horde-db-migrate was to big, and the migration  
> process was canceled
> 2. the script was taking too long to finish
> 3. corrupted, inconsistent, or wrong data in the database
>    e.g. Encoding problems in serialized data

FWIW this shouldn't be a problem anymore with current releases of  
Horde. We migrated several installations with 10k+ users during the  
last few years and improved memory usage and execution times during  
that time. The migration still may run a few hours on large  
installations though.

Any problems we experienced during those migrations where due to not  
completely up-to-date Horde 3 installations, or incorrect charset  
settings in Horde 3. Once you get the database to a correct Horde 3  
state, the Horde 4/5 migrations should run smoothly.

> We tailored the migration process. Some steps where done with one  
> postgresql specific
> SQL update statements instead of horde/php doing a loop, some where  
> rewritten to
> use less memory (don't load the whole table in one select) etc.
>
> We did automate the migration, and run them every night. Each  
> morning we searched
> for errors in the logs, fixed errors and optimized (e.g. running  
> some independent
> migrations steps from different horde apps in parallel) the migration.
> It still took several hours of downtime on the final migration day.
>
>>>> 2. Start the new installation with an empty database which will be OK,
>>>>   if I at least can migrate the addressbook entries from the old to
>>>>   the new installation. What steps are necessary to do this? Would it
>>>>   also be possible to export the adressbook entries into e.g. vCards,
>>>>   csv files or whatever and import them into the new system via a
>>>>   script or so?
>
> This was not an option for us, but horde has many options to export/import
> this data, so it should be possible to do.
>
>>>>
>>>> Any help or hints how to migrate the adresses would be great!
>>>
>
> I can take a look if I still have my old migration scripts customizations.
>
>>> Without telling us what exactly went wrong (steps you did, errors,
>>> screenshots, etc.) I'm afraid we won't be able to help much.
>>
>> I understand that I'd need to provide more information, but doing a
>> manual upgrade for horde and all application with all upgrades for the
>> DB again, will take hours :-(. So first let me explain in general what
>> steps I performed to get out if I did the right thing...
>>
>> First of all I upgraded the horde framework to the latest 3.x version
>> step by step for every patchlevel. The same I did for every application.
>> I applyed the database patches and followed the steps mensioned in the
>> doc/UPGRADING file.
>>
>> When I was at the latest patchlevel for horde 3.x and all applications
>> working with this horde version, I dumped the database, changed the
>> table charset from ISO-8859-1 to UTF8 and the table engine from MyISAM
>> to InnoDB.
>
> As Vilius already pointed out the serialized data is a problem as  
> length in bytes
> of strings is encoded in the serialized data, and this my change on  
> converting
> the charset it will break this data.
>
> I have provided our script for this conversion on the horde wiki.
> https://wiki.horde.org/ConvertSerializedDataToUTF8
>
>>
>> In the next step I created a new installation with horde and the
>> applications I need via pear to have a setup with the latest horde and
>> applications versions and a current database setup. Then I replaced the
>> database of this new setup with the dump from the old setup, so I had an
>> old database but a current setup regarding to the framework and the
>> apps.
>>
>> In the next step I ran the horde-db-migrate script to update the old
>> database schema to the current version.
>>
>
> you may need to run the horde-db-migrate script multiple times
>
>> This were the steps I performed to upgrade my old system, I hope they
>> were right :-).
>>
>> I again could do the database migration and provide the logfile, if this
>> would help, and ofcourse I can provide the sql errors I get in the log
>> after the upgrade. But first of all it would be interesting to know if
>> I followed the right way during I upgraded the system...
>>
>> Kind regards,
>>
>>  Christian
>>
>> --
>> Christian Schoepplein
>>
>
>
>
> --------------------------------------------------------------------------------
> M.Menge                                Tel.: (49) 7071/29-70316
> Universität Tübingen                   Fax.: (49) 7071/29-5912
> Zentrum für Datenverarbeitung          mail:  
> michael.menge at zdv.uni-tuebingen.de
> Wächterstraße 76
> 72074 Tübingen

-- 
Jan Schneider
The Horde Project
https://www.horde.org/