Files getting set to 0 in data folder

[mikelupe]

Steps to reproduce

1. find /srv/ncdata/ -size 0 -type f | wc -l
2. Result: 215 files found
3. No other way to reproduce, I suddenly realized I had lost a file to zero and did some investigation, as some other nextcloud forum's users wrote about the same issue in https://help.nextcloud.com/t/files-become-zero-bytes/7214

Expected behaviour

Files should not randomly getting set to 0

Actual behaviour

Hi guys,

I'm one of those losing files to zeros. I thought it's better to continue in here, rather than spamming the forum's thread about this issue.

I'm on NC11RC1 since yesterday and I do have 215 files in the data folder set to zero - but I for sure already had one file on NC11 final that was set to zero and I had to restore it from a backup. Maybe the other 214 were lost too already on NC11 final.

a) 13 files related to one (and only one) frequent user's data
10 files in ncdata/ncuser01/files/01_Daten/
File extensions: .ini .txt .docx .exe .cfg .bin
3 files:
ncdata/ncuser01/files_versions/MyCloud/Tesfile01.txt.v1474618386
ncdata/ncuser01/files_trashbin/files/New Text Document.txt.d1475085009
ncdata/ncuser01/files_trashbin/files/New Text Document.txt.d1480781891

b) 199 files related to "updater-data" in the data folder
187 files in ncdata/updater-50793ab339aa5/
6 files in ncdata/updater-data/checkpoint/9.0.0.19-570dfac02c766/apps/
6 files in ncdata/updater-data/checkpoint/9.0.1.3-573246e809595/apps/

c) 3 other files in data root:
ncdata/.ocdata
ncdata/index.html
ncdata/appdata_50793ab339aa5/preview/16024/64-64-crop.png

It's midnight now, and I came home from work at 10pm, so please don't ask for more infos now.

Thanks, kind regards

Server configuration

Debian Jessie 8.6 (x64) - SMP Debian 4.8.11-1~bpo8+1 (2016-12-14) x86_64 GNU/Linux
Apache/2.4.25 (Debian)
mysql Ver 15.1 Distrib 10.0.28-MariaDB
PHP 5.6.29-0+deb8u1
NC 11.0.1 RC1 (beta)
Updated from NC11 final with Webgui updater
**Original clean manual installation of Nextcloud 9 (from nextcloud.com download) and then additionally migrated data from an old owncloud installation (which itself came the long way from OC 3.X) **

Signing status:

Signing status

Integrity checker has been disabled. Integrity cannot be verified.`

List of activated apps:

App list

Enabled:

• activity: 2.4.1
• admin_audit: 1.1.0
• announcementcenter: 3.0.0
• apporder: 0.3.3
• audioplayer: 1.4.0
• bookmarks: 0.9.1
• calendar: 1.4.1
• comments: 1.1.0
• contacts: 1.5.2
• dav: 1.1.1
• external: true
• federatedfilesharing: 1.1.1
• federation: 1.1.1
• files: 1.6.1
• files_accesscontrol: 1.1.2
• files_automatedtagging: 1.1.1
• files_external: 1.1.2
• files_pdfviewer: 1.0.1
• files_retention: 1.0.1
• files_sharing: 1.1.1
• files_texteditor: 2.2
• files_trashbin: 1.1.0
• files_versions: 1.4.0
• files_videoplayer: 1.0.0
• gallery: 16.0.0
• gpxedit: 0.0.3
• gpxpod: 2.0.0
• keeweb: 0.3.0
• logreader: 2.0.0
• lookup_server_connector: 1.0.0
• mozilla_sync: true
• nextcloud_announcements: 1.0
• notes: 2.1.0
• notifications: 1.0.1
• ocsms: 1.11.4
• password_policy: 1.1.0
• polls: 0.7.2
• provisioning_api: 1.1.0
• qownnotesapi: 16.12.0
• rainloop: 4.26.0
• richdocuments: 1.1.25
• serverinfo: 1.1.1
• sharebymail: 1.0.1
• spreed: 1.1.2
• spreedme: 0.3.5
• survey_client: 0.1.5
• systemtags: 1.1.3
• tasks: 0.9.4
• templateeditor: 0.2
• theming: 1.1.1
• twofactor_backupcodes: 1.0.0
• updatenotification: 1.1.1
• user_ldap: 1.1.1
• workflowengine: 1.1.1
  Disabled:
• encryption
• files_markdown
• firstrunwizard
• mail
• user_external
• user_saml

The content of config/config.php:

Config report

{
    "system": {
        "datadirectory": "\/srv\/ncdata",
        "dbtype": "mysql",
        "version": "11.0.1.1",
        "installedat": "1331029597.1919",
        "lastupdatedat": "1343136183.1441",
        "installed": true,
        "loglevel": 2,
        "instanceid": "50793ab339aa5",
        "ldapIgnoreNamingRules": false,
        "maintenance": false,
        "memcache.local": "\\OC\\Memcache\\APCu",
        "theme": "",
        "mail_smtpmode": "smtp",
        "mail_smtphost": "valid.smtp.host.xxx",
        "mail_smtpport": "465",
        "mail_smtptimeout": 30,
        "mail_smtpauth": 1,
        "mail_smtpauthtype": "LOGIN",
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "allow_user_to_change_display_name": true,
        "enable_avatars": true,
        "knowledgebaseenabled": false,
        "enable_certificate_management": true,
        "trusted_domains": [
            "localhost",
            "*someIP*",
            "some.dom.tld"
        ],
        "appcodechecker": false,
        "custom_csp_policy": "default-src 'self'; script-src 'self' 'unsafe-inline' 'unsafe-eval'; style-src 'self' 'unsafe-inline'; frame-src *; img-src *; font-src 'self' data:; media-src *",
        "share_folder": "\/Shared",
        "mail_from_address": "somevalidemail",
        "mail_domain": "dom.tld",
        "dbname": "nextcloud",
        "dbhost": "localhost",
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "forcessl": true,
        "defaultapp": "files",
        "mail_smtpsecure": "ssl",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "appstoreenabled": true,
        "appstoreurl": "https:\/\/apps.nextcloud.com\/api\/v0",
        "appstore.experimental.enabled": true,
        "trashbin_retention_obligation": "auto",
        "updater.release.channel": "beta",
        "ldapProviderFactory": "\\OCA\\User_LDAP\\LDAPProviderFactory",
        "updater.secret": "***REMOVED SENSITIVE VALUE***"
    }
}

Are you using external storage, if yes which one: No

Are you using encryption: no

Are you using an external user-backend, if yes which one: ActiveDirectory (but not really used, still local users)

LDAP config

</details>

### Client configuration
**Browser:** Firefox 50.1.0
**Operating system:** Debian 8.6 KDE Plasma / Windows 7 

### Logs
#### Web server error log
<details>
<summary>Web server error log</summary>

Errorlog from when?

</details>

#### Nextcloud log (data/nextcloud.log)
<details>
<summary>Nextcloud log</summary>

Nextcloud log from when?

</details>

#### Browser log
<details>
<summary>Browser log</summary>

From when?```

[nickvergessen]

Thanks for the detailed report!

Is there anything in the nextcloud.log or in your apache/nginx error.log?

[mikelupe]

Sorry the formatting mess...

I'll have to investigate the logs when I'll find some time. The problem is, the only trigger for me to realize there's zeros coming was when opening a file in Collabora, CODE got stuck. Then I saw the file went zero, but didn't have the time to investigate further (besides of restoring the file from backup). The "big" research I did with finding the 215 files was much later, like 2 days later or so...

[Sanookmakmak]

I also have that issue.

Unfortunately i can't say when it happened. Therefore i'm unable to look at the log files.

I only can warn everyone about this behavior!
Check from time to time if files in your data folder becomes 0 bytes.
find $nextcloud_data_folder -size 0 -type f

Here are some more users of Nextcloud with that zero bytes bug:
https://help.nextcloud.com/t/files-become-zero-bytes/7214

[tflidd]

For one of the users it could have been the client which overwrites the file:
https://help.nextcloud.com/t/files-become-zero-bytes/7214/3

Does this only concern users who sync their files with the nc-client? All longer existing setups with upgrade history? Zero files also happen on local storage, non shared files & folders? If you share a bit more about the other configs, it's easier to find a pattern.

[mikelupe]

@tflidd - well, for me it's indeed just for the only user syncing with desktop clients (Linux & Windows). All other user's files haven't been touched appearently. Local storage only, non shared files or folders. Quite "simple and small" personal setup. I didn't do the check on the company's NC yet though.

@Sanookmakmak - let's try to approach this issue in a calm way. Creating some panic doesn't help. Please give some closer insight on the zeroed files. I know it's boring - that's how it felt when I created the issue yesterday. But maybe this way we can advance faster on it.

thanks

[Sanookmakmak]

I can only say that i use the Linux and Windows client, sorry. I've uploaded the files about half a year ago and i really don't have an idea when and why the files have been zeroed. I found them coincidentally. But i will keep an eye on it if it happens again.

And i don't have a backup because Nextcloud is my backup solution. That's why i feel i little bit panic :-(

[mikelupe]

I see - Be careful, it's "dangerous" and not recommended to use OC or NC as the sole backup solution, even if you would use the "versions" app. The NC server data should always be backupped. That's what the big cloud providers also do. Otherwise they would be in big trouble.

Can you provide some information about the system setup?

Operating System
Kernel (uname -a)
Webserver engine ("apachectl -V" e.g.)
Database ("mysql --version" e.g.)
PHP Version ("php --version")

Thanks

[Sanookmakmak]

This information is from today, but the issue could also happened half a year ago and of course there were older versions installed. As i told before, i can't say when those files were zeroed.

Operating System
Ubuntu 16.041 x64 Server
Clients: Linux Mint 18 with Owncloud client 2.2.4 of SuSE repo
Windows 7 with Nextcloud client 2.2.4 build 2

Kernel (uname -a)
Linux server 4.4.0-59-generic #80-Ubuntu SMP Fri Jan 6 17:47:47 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Webserver engine (apachectl -V)

Server version: Apache/2.4.18 (Ubuntu)
Server built:   2016-07-14T12:32:26
Server's Module Magic Number: 20120211:52
Server loaded:  APR 1.5.2, APR-UTIL 1.5.4
Compiled using: APR 1.5.2, APR-UTIL 1.5.4
Architecture:   64-bit
Server MPM:     prefork
  threaded:     no
    forked:     yes (variable process count)
Server compiled with....
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
 -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
 -D APR_USE_SYSVSEM_SERIALIZE
 -D APR_USE_PTHREAD_SERIALIZE
 -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
 -D APR_HAS_OTHER_CHILD
 -D AP_HAVE_RELIABLE_PIPED_LOGS
 -D DYNAMIC_MODULE_LIMIT=256
 -D HTTPD_ROOT="/etc/apache2"
 -D SUEXEC_BIN="/usr/lib/apache2/suexec"
 -D DEFAULT_PIDLOG="/var/run/apache2.pid"
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_ERRORLOG="logs/error_log"
 -D AP_TYPES_CONFIG_FILE="mime.types"
 -D SERVER_CONFIG_FILE="apache2.conf"

Database (mysql --version)
mysql Ver 15.1 Distrib 10.0.28-MariaDB, for debian-linux-gnu (x86_64) using readline 5.2

PHP Version (php --version)

PHP 7.0.13-0ubuntu0.16.04.1 (cli) ( NTS )
Copyright (c) 1997-2016 The PHP Group
Zend Engine v3.0.0, Copyright (c) 1998-2016 Zend Technologies
    with Zend OPcache v7.0.13-0ubuntu0.16.04.1, Copyright (c) 1999-2016, by Zend Technologies

[mikelupe]

Great. If you find the time, could you analyze your "find $nextcloud_data_folder -size 0 -type f" command in an editor and filter down the major folders of the affected files (more or less as I did in the issue description).

And "sudo -u www-data php /var/www/your_nc_folder/occ app:list" - so the devs and ops got even more stuff to work with :)

[Sanookmakmak]

find $nextcloud_data_folder -size 0 -type f

./Archiv/Musikarchiv/Artists/K-O/Miles Davis/Blue Bird - Legendary Savoy Sessions/Miles Davis - Parker's Mood (feat. Charlie Parker).mp3

php occ app:list

Enabled:
  - activity: 2.4.1
  - admin_audit: 1.1.0
  - announcementcenter: 3.0.0
  - apporder: 0.3.3
  - audioplayer: 1.4.0
  - bookmarks: 0.9.1
  - calendar: 1.4.1
  - comments: 1.1.0
  - contacts: 1.5.2
  - dav: 1.1.1
  - direct_menu: 0.10.0
  - federatedfilesharing: 1.1.1
  - federation: 1.1.1
  - files: 1.6.1
  - files_accesscontrol: 1.1.2
  - files_automatedtagging: 1.1.1
  - files_downloadactivity: 1.0.0
  - files_external: 1.1.2
  - files_markdown: 1.0.0
  - files_pdfviewer: 1.0.1
  - files_reader: 0.8.1
  - files_retention: 1.0.1
  - files_sharing: 1.1.1
  - files_texteditor: 2.2
  - files_trashbin: 1.1.0
  - files_versions: 1.4.0
  - files_videoplayer: 1.0.0
  - firstrunwizard: 2.0
  - gallery: 16.0.0
  - gpxedit: 0.0.3
  - gpxpod: 2.0.0
  - keeweb: 0.3.0
  - logreader: 2.0.0
  - lookup_server_connector: 1.0.0
  - mail: 0.6.2
  - news: 10.1.0
  - nextcloud_announcements: 1.0
  - notifications: 1.0.1
  - ocr: 2.0.0
  - ocsms: 1.11.4
  - ojsxc: 3.0.2
  - password_policy: 1.1.0
  - provisioning_api: 1.1.0
  - richdocuments: 1.1.25
  - serverinfo: 1.1.1
  - sharebymail: 1.0.1
  - spreed: 1.1.2
  - spreedme: 0.3.5
  - survey_client: 0.1.5
  - systemtags: 1.1.3
  - tasks: 0.9.4
  - theming: 1.1.1
  - twofactor_backupcodes: 1.0.0
  - twofactor_totp: 0.5.0
  - twofactor_u2f: 0.1.0
  - updatenotification: 1.1.1
  - user_ldap: 1.1.1
  - user_saml: 1.2.2
  - workflowengine: 1.1.1

[nickvergessen]

@icewind1991 @rullzer any idea? Seems like filesystem or syncing causes an issue.

[feisar]

I've posted a script here https://help.nextcloud.com/t/files-become-zero-bytes/7214/17 which will help alert close to the time it happens. The Apache logs don't seem to help much, I caught them when one file was zero'd and they just showed one client putting and the other getting.

Anyone know which log file is going to be best to look at and would turning on debugging help?