[horde] Migrating user data from an old horde instance to a new installation

[Michael Menge]

Hi,

Quoting Christian Schoepplein <c.schoepplein at musin.de>:

> Hi,
>
> On Fr, Mär 03, 2017 at 04:56:43 +0200, Vilius Sumskas/LNK wrote:
>> "horde" <horde-bounces at lists.horde.org> wrote on 2017.03.03 16:48:06:
>>> I have a damn old horde installation which has to be migrated to a
>>> current setup.
>>>
>>> Old installation:
>>>
>>>   * Gollem: H3 (1.0.2)
>>>   * Horde: 3.1.4
>>>   * Imp: H3 (4.1.4)
>>>   * Ingo: H3 (1.1.3)
>>>   * Kronolith: H3 (2.1.4)
>>>   * Turba: H3 (2.1.5)
>>>
>>> New setup:
>>>
>>>   * Horde: 5.2.13
>>>   * Content: 2.0.5
>>>   * Gollem: H5 (3.0.10)
>>>   * Imp: H5 (6.2.17)
>>>   * Ingo: H5 (3.2.13)
>>>   * Kronolith: H5 (4.2.19)
>>>   * Nag: H5 (4.2.13)
>>>   * Timeobjects: 2.1.3
>>>   * Turba: H5 (4.2.18)
>>>
>>> Migrating and adapting the config from the old to the new system causes
>>> me no trouble, but I am not able to migrate all the userspecific data
>>> stored in the mysql db backend.
>>>
>>> for that reason I see two possebilities now:
>>>
>>> 1. Try again to migrate the database from the old horde setup to the new
>>
>>>    installation. Because I have no idea what went wrong during the
>>>    first try, it is very likely that I will run into the same issues
>>>    again :-(.

We did the migration a few years ago. But we don't use Gollum, and we
use Postgresql. There where some problems we encountered.

As far as I remember most common causes where:

1. memory usage of horde-db-migrate was to big, and the migration  
process was canceled
2. the script was taking too long to finish
3. corrupted, inconsistent, or wrong data in the database
    e.g. Encoding problems in serialized data

We tailored the migration process. Some steps where done with one  
postgresql specific
SQL update statements instead of horde/php doing a loop, some where  
rewritten to
use less memory (don't load the whole table in one select) etc.

We did automate the migration, and run them every night. Each morning  
we searched
for errors in the logs, fixed errors and optimized (e.g. running some  
independent
migrations steps from different horde apps in parallel) the migration.
It still took several hours of downtime on the final migration day.

>>> 2. Start the new installation with an empty database which will be OK,
>>>    if I at least can migrate the addressbook entries from the old to
>>>    the new installation. What steps are necessary to do this? Would it
>>>    also be possible to export the adressbook entries into e.g. vCards,
>>>    csv files or whatever and import them into the new system via a
>>>    script or so?

This was not an option for us, but horde has many options to export/import
this data, so it should be possible to do.

>>>
>>> Any help or hints how to migrate the adresses would be great!
>>

I can take a look if I still have my old migration scripts customizations.

>> Without telling us what exactly went wrong (steps you did, errors,
>> screenshots, etc.) I'm afraid we won't be able to help much.
>
> I understand that I'd need to provide more information, but doing a
> manual upgrade for horde and all application with all upgrades for the
> DB again, will take hours :-(. So first let me explain in general what
> steps I performed to get out if I did the right thing...
>
> First of all I upgraded the horde framework to the latest 3.x version
> step by step for every patchlevel. The same I did for every application.
> I applyed the database patches and followed the steps mensioned in the
> doc/UPGRADING file.
>
> When I was at the latest patchlevel for horde 3.x and all applications
> working with this horde version, I dumped the database, changed the
> table charset from ISO-8859-1 to UTF8 and the table engine from MyISAM
> to InnoDB.

As Vilius already pointed out the serialized data is a problem as  
length in bytes
of strings is encoded in the serialized data, and this my change on converting
the charset it will break this data.

I have provided our script for this conversion on the horde wiki.
https://wiki.horde.org/ConvertSerializedDataToUTF8

>
> In the next step I created a new installation with horde and the
> applications I need via pear to have a setup with the latest horde and
> applications versions and a current database setup. Then I replaced the
> database of this new setup with the dump from the old setup, so I had an
> old database but a current setup regarding to the framework and the
> apps.
>
> In the next step I ran the horde-db-migrate script to update the old
> database schema to the current version.
>

you may need to run the horde-db-migrate script multiple times

> This were the steps I performed to upgrade my old system, I hope they
> were right :-).
>
> I again could do the database migration and provide the logfile, if this
> would help, and ofcourse I can provide the sql errors I get in the log
> after the upgrade. But first of all it would be interesting to know if
> I followed the right way during I upgraded the system...
>
> Kind regards,
>
>   Christian
>
> --
> Christian Schoepplein
>



--------------------------------------------------------------------------------
M.Menge                                Tel.: (49) 7071/29-70316
Universität Tübingen                   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung          mail:  
michael.menge at zdv.uni-tuebingen.de
Wächterstraße 76
72074 Tübingen