[stable11] Add DAV repair step to fix calendar data

[rullzer]

Backport of #2273

[mention-bot]

@rullzer, thanks for your PR! By analyzing the history of the files in this pull request, we identified @nickvergessen and @LukasReschke to be potential reviewers.

[MorrisJobke]

Tested and works 👍

[michauk]

Hi,
I think the live repair of incorrect cal entries isn't correct (complete?).

I switched from OC 9.1.3 to NC 10.0.x, no problem
Then I upgraded from 10.x to 11.0, calendar broken with this "Unsupported VALUE parameter for DTEND property" in the logs. After some googling, I found all these github issues 2741, this one, 2273 and so on.

I've done again the upgrade (restored to oc 9 and so on) several times, to be sure (I wasn't sure if I had to have the calendar plugin reactivated while in transition NC 10, before upgrading to 11.0.1). Same problem, every time.

I mass-edited some cal entries with this bad DTEND property. Now the logs (logging menu in NC adminitration) don't show any error (all debug levels enabled) but I have even more calendars loading endlessly. I don't think I have bad entries because of the mass-editing. But in any case, the upgrade from 10 to 11 doesn't correct the entries that make NC 11 load endlessly.

Can't move to NC with this bug. I switched back to OC 9.1.3
I can help find the incorrect entries if needed. My different calendars are many years old, a looooong time ago in google calendars, moved to owncloud (7 if I'm not wrong, some years ago) and upgraded little by little to 9.1.3. My cal entries have been created with different tools (thunderbird, some android apps, directly in the OC web interface...), I might have some entries NC 11.0.1 doesn't like any more.

Regards,

JAcques M.

[jokakilla]

Hi,
I had some trouble with the calendar parsing too.
After upgrading from 10.0.3 to 11.0.1 the calendar was endlessly loading.
The logs showed something like this:

Sabre\VObject\InvalidDataException: Unsupported VALUE parameter for DTEND property. You supplied ""

/var/www/html/owncloud/3rdparty/sabre/vobject/lib/Parser/MimeDir.php - line 453: Sabre\VObject\Document->createProperty('DTEND', NULL, Array)
/var/www/html/owncloud/3rdparty/sabre/vobject/lib/Parser/MimeDir.php - line 234: Sabre\VObject\Parser\MimeDir->readProperty('DTEND;TZID=Euro...')

I've exported the calendar entries from the DB and checked the format. All entries seem to have a consistent DTEND format.

DTEND;TZID=Europe/Berlin;VALUE=:20140517T180000

To me it looks like the readProperty method has an issue parsing the line.
I hope someone can dig deeper into this issue. I don't have an experimental installation to do it and needed to go back to 10.0.3.

[jokakilla]

Ok after having a look at pull request #2807 it seems like the repair step would convert

DTEND;TZID=Europe/Berlin;VALUE=:20140517T180000

into

DTEND;TZID=Europe/Berlin:20140517T180000

But how is the repair triggered?

[nickvergessen]

@rullzer

[jokakilla]

Second try upgrading today. It's working now after manually repairing the calendar entries.

• Upgrade
• Export oc_calendarobjects table as CSV
• Replace "Berlin;VALUE=:" by "Berlin:" (suggested solution of my last comment)
• Flushed database and reimported from CSV
• All fine

Just wonder why the repair job of NC didn't do that automatically.

[michauk]

Thanks to @jokakilla , I managed to upgrade to nextcloud 10 then 11. But some problems should still be considered by the team, I suppose:

You can correct your faulty calendar entries with something like :
UPDATE oc_calendarobjects SET calendardata = REPLACE(calendardata, "DTEND;TZID=Europe/Paris;VALUE=:", "DTEND;TZID=Europe/Paris:");
Someone might use a correct regular expression to replace Europe/Paris by anything like "some/city"
Be careful, test it on a specific calendar row if needed (add AND id=xxx), spot an faulty entry with something like :
SELECT id, calendardata FROM oc_calendarobjects WHERE calendardata LIKE '%DTEND;TZID=Europe/Paris;VALUE:%';
It fixed approx 250 entries in my case O_O

But, new problem, as I couldn't migrate from OC to NC, some days ago, I upgraded my OC 9.1.3 to 9.1.4.
The OC->NC migration doc says you can migrate from OC9.1.x to NC10.x but it's not entirely true, the latest 9.1.4 is not correctly handled. I decided to test this suggestion : https://help.nextcloud.com/t/owncloud-9-1-4-2-migration-to-nextcloud-not-working/8630 ; in a few words : fake the version so the upgrade command doesn't complain.

Then I could migrate to NC10 and 11 right after, disabled maintenance mode and my calendars aren't loading forever anymore. whoaaaa.
I tried some sync, both ways, from my android, thunderbird, nextcloud ;used my carddav/caldav app on android. Everything seems OK.

I hope all this will help.

Regards,

DO A CODE AND DB BACKUP BEFORE PLAYING WITH IT ALL.

[rullzer]

@jokakilla @michauk sorry for the late reply (but I'm glad you figured it out). Since we wanted to be a bit smarted when migrating and making upgrade faster the fix is not run on upgrade. Since if you have a large large enough instance that would make it all painfully slow.

So if cron would have been triggered a few times it should have launched the repair job.

Anyway hope all is fixed already!