Tags are lost when uploading or downloading files

[riparise]

Files tagged on Windows (possibly Linux as well) and uploaded do not show their tags on Nextcloud.
Tags generated on Nextcloud do not show in Windows when Downloaded.

Steps to reproduce

1. Create some tags in a file in windows.
2. Upload the file
   or
3. Create some tags in a file in Nextcloud.
4. Download the file

Expected behaviour

The tags should be kept while downloading and uploading into Nextcloud.

Actual behaviour

Tags are lost. See images bellow

Server configuration

Operating system:
Debian Stretch

Web server:
NGINX

Database:
MariaDB

PHP version:
PHP 7.0

Nextcloud version:
13.0.2

Updated from an older Nextcloud/ownCloud or fresh install:
Updated

Where did you install Nextcloud from:
https://download.nextcloud.com/server/releases

Signing status:

Signing status

No errors have been found.

List of activated apps:

App list

Enabled: - activity: 2.6.1

• apporder: 0.4.1
• audioplayer: 2.3.1
• bruteforcesettings: 1.1.0
• calendar: 1.6.1
• caniupdate: 0.1.2
• checksum: 0.4.0
• comments: 1.3.0
• contacts: 2.1.5
• dav: 1.4.6
• deck: 0.4.0
• drawio: 0.8.9
• federatedfilesharing: 1.3.1
• federation: 1.3.0
• files: 1.8.0
• files_automatedtagging: 1.3.0
• files_downloadactivity: 1.2.0
• files_frommail: 0.1.0
• files_pdfviewer: 1.2.1
• files_rightclick: 0.8.4
• files_sharing: 1.5.0
• files_texteditor: 2.5.1
• files_trashbin: 1.3.0
• files_versions: 1.6.0
• files_videoplayer: 1.2.0
• firstrunwizard: 2.2.1
• gallery: 18.0.0
• githubmergetracker: 0.0.18
• gpxpod: 2.2.2
• groupfolders: 1.2.2
• logreader: 2.0.0
• lookup_server_connector: 1.1.0
• mail: 0.8.2
• metadata: 0.6.0
• mindmaps: 0.1.0
• news: 12.0.4
• nextcloud_announcements: 1.2.0
• notes: 2.3.2
• notifications: 2.1.2
• oauth2: 1.1.0
• passman: 2.1.4
• password_policy: 1.3.0
• polls: 0.8.1
• previewgenerator: 1.0.9
• provisioning_api: 1.3.0
• quota_warning: 1.2.0
• radio: 0.6.1
• rainloop: 5.1.0
• serverinfo: 1.3.0
• sharebymail: 1.3.0
• spreed: 3.2.4
• survey_client: 1.1.0
• systemtags: 1.3.0
• tasks: 0.9.6
• theming: 1.4.5
• twofactor_backupcodes: 1.2.3
• updatenotification: 1.3.0
• weather: 1.5.1
• workflowengine: 1.3.0
  Disabled:
• admin_audit
• encryption
• files_external
• user_external
• user_ldap

Nextcloud configuration:

Config report

{

"system": {

    "passwordsalt": "***REMOVED SENSITIVE VALUE***",
    "secret": "***REMOVED SENSITIVE VALUE***",
    "trusted_domains": [
        "cloud.parise.space"
    ],
    "datadirectory": "***REMOVED SENSITIVE VALUE***",
    "overwrite.cli.url": "https:\/\/cloud.parise.space",
    "dbtype": "mysql",
    "version": "13.0.2.1",
    "dbname": "***REMOVED SENSITIVE VALUE***",
    "dbhost": "***REMOVED SENSITIVE VALUE***",
    "dbport": "",
    "dbtableprefix": "oc_",
    "dbuser": "***REMOVED SENSITIVE VALUE***",
    "dbpassword": "***REMOVED SENSITIVE VALUE***",
    "installed": true,
    "instanceid": "***REMOVED SENSITIVE VALUE***",
    "memcache.local": "\\OC\\Memcache\\APCu",
    "memcache.locking": "\\OC\\Memcache\\Redis",
    "filelocking.enabled": true,
    "redis": {
        "host": "***REMOVED SENSITIVE VALUE***",
        "port": 0,
        "timeout": 0,
        "password": "***REMOVED SENSITIVE VALUE***"
    },
    "maintenance": false
}

}

Are you using external storage, if yes which one:
NO

Are you using encryption:
NO

Are you using an external user-backend, if yes which one:
NO

Client configuration

Browser:
Chrome

Operating system:
Windows 10

[nextcloud-bot]

GitMate.io thinks possibly related issues are #1371 (No file-uploading possible at all), #9801 (Error downloading or uploading large files), #4678 ("Connection to server lost" error when uploading/downloading large files), #5438 (Nextcloud won't upload larger files), and #231 (Advanced File Tagging).

[andrewborell]

I tested this in my install and found that tags are retained for jpg images, even if I edit the tags in NextCloud before downloading the image.
Server: Ubuntu 16; NC 13.0.4; Apache 2.4; PHP 7.0;
Client: Windows 10; Chrome 67.0.3396.99;

Surely this is related to some sort of content compression. I know when sending images over MMS that ALL metadata is lost for me. I can test tethering from my mobile to see if it happens when I upload using a cellular data connection.

Have you tried to check directly in the OS with exiftool?
e.g.) your_linux_user@host_name:/var/www/domain.tld/web/nextcloud/data/your_nc_user_name/files# exiftool myimage.jpg
I can see the fields in the file on disk.

XP Keywords : TestTag;Another tag
Subject : TestTag, Another tag
Last Keyword XMP : TestTag, Another tag

If you cannot see tags directly on disk you should maybe try narrowing the scope by doing an SSH transfer directly to your home directory and see if the tags are retained with exiftool.

[riparise]

Andrew,

Thank you for your time and your answer.

I checked the file directly on my server (via SSH) using the exiftool tool you suggested and I can confirm that the tags are preserved on the metadata of my file when they are uploaded in my Nextcloud website. I also checked transferring the file using SCP and there was no problem as well.

When I re-download the files the original Tags are also preserved. So no tags are lost on any transfer. Only the Tags that I generate on the Nextcloud are lost.

The title of the issue is misleading: Apparently the Tags of the Nextcloud are not related at all with the Tags of the metadata of a file. Can someone please confirm so? So what are tags in Nextcloud?

If this it true I would suggest (maybe close this issue and open a request) to use the metadata tags on Nextcloud.

I organize all my photos and musics using Tags and I would love to import them to my Nextcloud to use it as a central storage.

[andrewborell]

Oh! Gotcha. The tags are not written to the property sheet of the file. From cli ( putty ) you can see them in the database:
mysql -u your_user -p
enter_your_password
show databases;
connect your_nextcloud_db_name;
show tables;
select * from oc_systemtag;
But they do not get written to the property sheet. It's a nice enhancement because I could see this being very useful in auto-tagging and exchanging data between systems, but I wouldn't say this is a defect so long as the intended functionality is not broken.

[MorrisJobke]

Files tagged on Windows (possibly Linux as well) and uploaded do not show their tags on Nextcloud.
Tags generated on Nextcloud do not show in Windows when Downloaded.

This was not yet implemented.

Also I think it is not possible to set tags from a browser download or read it from a browser upload. Thus it is a pure client feature.

There is already a ticket over there and I will close this one here as duplicate of nextcloud/desktop#206

[riparise]

Morris,

I did not understand why it should be impossible. Couldn't the same function that reads/write the oc_systemtag in the data base, reads (and write if they are not the same) the tags from the metadata of the file on the server?

This seems like a server feature, not client. I am tying to understand it better to open an enhancement request (or even try to code myself).

[MorrisJobke]

This seems like a server feature, not client. I am tying to understand it better to open an enhancement request (or even try to code myself).

The server file system was not designed to hold all the information that is in Nextcloud. Otherwise we would run into other issues, if they are updated on the filesystem and not inside Nextcloud. So this is highly unlikely to be implemented. The tags are stored in the DB of Nextcloud. Same way as most of the other metadata is. Also there are multiple different file backends available for Nextcloud that don't support this anyways.

[andrewborell]

To get the meta tags in php you can use a extension function ( for extracting them on upload) which would be a nice feature of auto-tagging.

PHP has something to handle this called "xattr". The functions are:

• xattr_get
• xattr_list
• xattr_remove
• xattr_set

the general request is a feature ( presumably for auto-tagging) where upon upload some logic does the following after the file is written to disk and successfully added to the document management system. Just some psuedo-code to get the idea ( I wrote this in about 2 minutes without any testing):

$file = 'some_file.jpg';
$root_attributes = xattr_list($file, XATTR_ROOT);
if(!empty($root_attributes['Tags'])){
$tags_attribute = xattr_get($file, 'Tags', XATTR_ROOT);
$tags_attributes_parts = implode(';',$tags_attribute);
foreach($tags_attributes_parts as $tags_attributes_parts_key => $tags_attributes_parts_val){
// add the tags to the database
}
}

and of course if the tags are set in nextcloud then to set them against the file Tags property.

Other notes:

• a block to check if the extension is enabled might be a good idea.
• I always like to use associative arrays to avoid duplicates, but some sort of dupe checking to make sure the tag doesnt get added more than one time to the same file etc.

[MorrisJobke]

To get the meta tags in php you can use a extension function ( for extracting them on upload) which would be a nice feature of auto-tagging.

This only works for files on the server and not uploaded files.

[MorrisJobke]

Feel free to write an app that does this, but the server will most likely not integrate this.

[riparise]

Andrew,

Thank you for the suggestions and the pseudo-code, I didn't know the xattr function. I will experiment a little during this week. I am learning PHP and I think this can be an interesting challenge.

Morris,

your point about not every back-end being compatible is valid, however if the server is using some filesystem that does not allow tags in the metadata of the file, this function would just be useless. I do not see any problem or drawback (except maybe for some performance decrease in up- and downloading).

Feel free to write an app that does this, but the server will most likely not integrate this.

Thank you for the warning. I try to code it as a personal challenge / learning. This function would be useful for me, even if it is not merged.

[MorrisJobke]

your point about not every back-end being compatible is valid, however if the server is using some filesystem that does not allow tags in the metadata of the file, this function would just be useless. I do not see any problem or drawback (except maybe for some performance decrease in up- and downloading).

There are quite some:

• it needs to keep in sync
• what happens if it the tag disappears on the filesystem and the user in the meantime added some tags - conflicts? delete all?
• what about renames of tags? should they happen on all files with that tag? or just on that tag? when to sync and what happens if there is concurrent change on the filesystem before the sync happened completely

Just to name a few ;)

[andrewborell]

The concept in itself is not much more complicated than my trashy mock-up code, but there are indeed a half dozen caveats I can cite off the top @Of my head as this relates to other applications. if @riparise starts the code I'd be happy to assist where I can. I have some of my own issues Im trying to pound out still however so I cannot lead such an effort. ;-\

[spoorun]

See also: #4216

[spoorun]

I do not see any problem or drawback (except maybe for some performance decrease in up- and downloading).

There are quite some:

* it needs to keep in sync

Indeed

* what happens if it the tag disappears on the filesystem and the user in the meantime added some tags - conflicts? delete all?

Deal with them the same way as current conflicts - such as file name change, sync by updating the most recent tag addition/removal/change for that file.

* what about renames of tags?  should they happen on all files with that tag? or just on that tag? when to sync and what happens if there is concurrent change on the filesystem before the sync happened completely

Renames of tags should only happen for the file concerned. After all, if a User renames a tag, then the server or client should apply the rename to all files with the tag, and that change would be synced per-file.

Just to name a few ;)

[spoorun]

Andrew,

Thank you for the suggestions and the pseudo-code, I didn't know the xattr function. I will experiment a little during this week. I am learning PHP and I think this can be an interesting challenge.

Morris,

your point about not every back-end being compatible is valid, however if the server is using some filesystem that does not allow tags in the metadata of the file, this function would just be useless. I do not see any problem or drawback (except maybe for some performance decrease in up- and downloading).

Feel free to write an app that does this, but the server will most likely not integrate this.

Thank you for the warning. I try to code it as a personal challenge / learning. This function would be useful for me, even if it is not merged.

Did you get anywhere with this @riparise ?
Keen to support anything that allows xattr tags to be synced...