Wiki: Archiving Research Data

Topics:

Overview of iRODS
Accesing iRODS
Example Process
iRODS Commands
Simplified iRODS Wrappers
Adding Metadata

Overview of iRODS

iRODS is a remote virtual filesystem separate from your normal directories in Linux or Windows. In iRODS, directories are called collections and they may contain further sub-collections (sub-directories) or data objects (files). In practice you can treat collections and data as directories and files for convenience. Files and folders in your own local and mounted directories can be Archived (copied) to the remote iRODS filesystem that the ZDV provides. Archiving is done by first creating a remote path directory (collection) and then copying your local files and directories to the remote path in iRODS. Archived files can be copied back or downloaded years later. Archives can be private for long term storage only, restricted to research groups or public for download.

Accessing iRODS

Archiving to iRODS is currently possible using the command line from the ZDV Linux Work-PC's, the linux.zdv.uni-mainz.de web server, and the MOGON II supercomputer. MOGON II also has a specialized iRODS wiki. New and more user friendly ways to access iRODS will be available in the future. Currently only users with a university account can use iRODS. If your current work computer does not have a standard ZDV Linux installation, you can remotely access the Linux servers using SSH. From there you can use the available iRODS iCommands to Archive data. Personal computers with an iRODS iCommand client installation could also access the ZDV iRODS Archive.

Before accessing the iRODS Server a configuration file must be present and you must authenticate yourself with your JGU-Account. The configuration file"irods_environment.json"  is locate in the hidden folder ".irods" with the content:

{
  "irods_authentication_scheme": "pam_password",
  "irods_client_server_negotiation": "request_server_negotiation",
  "irods_client_server_policy": "CS_NEG_REQUIRE",
  "irods_encryption_algorithm": "AES-256-CBC",
  "irods_encryption_key_size": 32,
  "irods_encryption_num_hash_rounds": 16,
  "irods_encryption_salt_size": 8,
  "irods_host": "irods.zdv.uni-mainz.de",
  "irods_port": 1247,
  "irods_user_name": "FILL IN YOUR USERNAME HERE",
  "irods_zone_name": "zdv"
}

Don't forget to fill in your correct username. Then enter the commant "iinit" and enter your JGU-Password.

Example Process

All commands to interact with iRODS start with an 'i' and are similar to UNIX commands. An example process workflow would look like this:

ipwd
shows the current directory;

imkdir -p /zdv/home/MUSTERPERSON/project123/
creates a new remote iRODS path to copy data to;

icd /zdv/home/MUSTERPERSON/project123/
changes the current remote iRODS directory to the newly created one;

ipwd
shows and confirms that it is the current directory;

ils
shows the (empty) contents of the remote directory;

iput -K some.file --metadata="Title;Publications data;;ResourceType;DataSet Short Description;;Title;Scientific Data;" 
archives a local some.file to the current remote iRODS path by copying it.

iget /zdv/home/MUSTERPERSON/project123/some.file
downloads some.file from iRODS to the local directory.

iticket create read /zdv/home/MUSTERPERSON/project123/some.file
creates a ticket number (i.e: ABCDEFG123456890) used to download some.file from the web:
wget https://irods-web.zdv.uni-mainz.de/irods-rest/rest/fileContents/zdv/home/MUSTERPERSON/some.file?ticket=ABCDEFG123456890

iRODS Commands.

Here is a short summary over the most important iRODS commands with some important command line parameters. Note that they start with an ' i ':

Simplified Wrapper Commands for iRODS.

Here is a short summary of our wrapper iRODS commands that simplify the archiving process of research data. Note that wrappers start with an ' i_ ' unlike pure iRODS commands that start with 'i ':

A simplified procedure to Archive research data would use the simplified wrappers and look like this:

source /usr/local/bin/i_init.sh for the linux.zdv.uni-mainz.de systems
Makes the wrapper commands available and initializes iRODS

imkdir -p /zdv/home/MUSTERPERSON/project123/
creates a new remote iRODS path to copy data to;

icd /zdv/home/MUSTERPERSON/project123/
changes the current remote iRODS directory to the newly created one;

i_archive /fullpath/some.file metadata.json
archives a local some.file to the current remote iRODS path by copying it and applying the metadata from the metadata.json files.

i_publish /zdv/home/MUSTERPERSON/project123/some.file
creates a ticket number (i.e: ABCDEFG123456890) and prints the generated web download link.

Adding Metadata

Metadata is accepted by iRODS as triplets (Attribute [i.e: Title], Value [i.e: "Research Data from X publication"], Unit [mostly always left empty]).
The first two fields Attribute and Value are mandatory and must not be empty, the unit is optional.

You could manually add metadata to every remote iRODS file and directory after uploading them by:
imeta set -d some.file Title "My research data from Publication X"

This would be very effort and time consuming so recommended alternative is to define a .json Metadata file for all your files and directories of a single Archive tree.

i_archive some.file metadata.json
Can be used to upload your local files with Metadata all at once.

i_metaupdate /zdv/home/MUSTERPERSON/project123/some.file metadata.json
Can be used to update a single remote file/directory with Metadata.

This flat metadata.json example needs a minimum set of Metadata Attributes:

{
 "Title":"",
 "ResourceType":"",
 "Project":"",
 "Keywords":"",
}

{
 "Title":"My Scientific Data from XYZ publication",
 "ResourceType":"Tables, Texts, Images",
 "Project":"BMBF-12345, DFG-67890",
 "Keywords":"Thermodynamics, Simulation, HPC, MPI , XYZ, ,BMBF, DFG",
}

The following Attributes are set automatically upon uploading files/directories and are not needed within the .json file:

Creator, Publisher, Location, Date, ExpiryDate(Date + 10 years), protected (default: “false”) .

The following Attributes are recommended within your .json Metadata file:

{
 "Title":"",
 "ResourceType":"",
 "Project":"",
 "Keywords":"",
 "Contributor":"",
 "Reference":"",
 "License":""
}

{
 "Title":"My Scientific Data from XYZ publication",
 "ResourceType":"Tables, Texts, Images",
 "Project":"BMBF-12345, DFG-67890",
 "Keywords":"Thermodynamics, Simulation, HPC, MPI , XYZ, ,BMBF, DFG",
 "Contributor":"Co-author1, Co-author2, Co-author3",
 "Reference":"",
 "License":"GPLX, CC0, CC-BY"
}

You can confirm the metadata was set by using:
imeta ls -d /zdv/home/MUSTERPERSON/project123/some.file for files
imeta ls -C /zdv/home/MUSTERPERSON/project123/ for directories.

Summary of imeta:

Command Description:

Publishing research data

For public access a ticket needs to be created for downloading files.
With this ticket and the path to the remote iRODS location, anyone can download information and the content.
Publishing already archived data can be done by using i_publish

i_publish /zdv/home/MUSTERPERSON/project123/some.file
creates a ticket number (i.e: ABCDEFG123456890) and prints the generated web download link.

You can also manually perform the publishing process with iRODS commands instead:
iticket create read /zdv/home/MUSTERPERSON/project123/some.file
creates a ticket number (i.e: ABCDEFG123456890) used to download some.file from the web.
Then you need to add the path such that it looks like this:
wget https://irods-web.zdv.uni-mainz.de/irods-rest/rest/fileContents/zdv/home/MUSTERPERSON/some.file?ticket=ABCDEFG123456890

Data Policy and Licensing

The “Creator” is responsible for taking care that reusing of third party data is legal (Urheberrechtsgesetz) and that personal data is handled correctly (DSGVO).
This applies for all data Archived, even if the “Creator” is not employed at the university any more or if the data is not public.
This a decision guide can help you decide if your data can be published(German).

If your data can indeed be published, Different kinds of Licenses exist for various cases:

• Software
  • Apache
  • GPL, GPLv2, GPLv3
  • MIT
  • BSD
• Arts, Images, Text, etc.
  • Creative Commons (Text, Arts, Photos, ...)
• Data sets
  • Open Data Commons

The applicability of CC-BY licenses for Software is not recommended: CC-recommendation and discussion. The same applies for datasets, their publication under a CC-license other than CC0 is doubtful. For other dataset licenses search at Open Definition Licenses Service.

Proprietary file formats should be avoided, since you don't know if the software to open them still exists in a few years. Try to stick to open standards.