... | ... | @@ -20,29 +20,6 @@ Server side orchestration of the process: |
|
|
|
|
|
The boxes with red borders and labels represent new patterns which rely on b2safe metadata stored on json files.
|
|
|
|
|
|
**System metadata**
|
|
|
|
|
|
data_name: data-object.dat
|
|
|
data_id: 20449
|
|
|
coll_id: 19547
|
|
|
data_repl_num: 0
|
|
|
data_version:
|
|
|
data_type_name: generic
|
|
|
data_size: 2151618
|
|
|
resc_group_name:
|
|
|
resc_name: cinecaRes1
|
|
|
data_path : /phisycal/path/to/data-object.dat
|
|
|
data_owner_name: proirod1
|
|
|
data_owner_zone: CINECA01
|
|
|
data_repl_status: 1
|
|
|
data_status:
|
|
|
data_checksum : 076aad9622fa3118f006927f05222817
|
|
|
data_expiry_ts (expire time): : None
|
|
|
data_map_id: 0
|
|
|
r_comment:
|
|
|
create_ts: 01430831283: 2015-05-05.15:08:03
|
|
|
modify_ts: 01430831283: 2015-05-05.15:08:03
|
|
|
|
|
|
**B2SAFE metadata**
|
|
|
|
|
|
attribute: eudat_dpm_checksum_date:cinecaRes1
|
... | ... | @@ -69,17 +46,30 @@ The boxes with red borders and labels represent new patterns which rely on b2saf |
|
|
INPUT *source="/My collection of registered data/datum.txt",*parent_pid="None",*ror="None"
|
|
|
OUTPUT ruleExecOut
|
|
|
|
|
|
It is possible to get back some basic metadata even with file system only browser clients like those supporting the protocols WebDAV and GridFTP. In fact the following json file is stored within the special collection ".metadata" for each registered object:
|
|
|
{
|
|
|
"**checksum**": "[sha2:nmDjK/7k1D5jjMUFoWHjX5qZmke9vpQbR6FaY9sk6eI=](http://sha2nmDjK)",
|
|
|
"**ror**": "None",
|
|
|
"**pid**": "842/6cff9eb8-47ef-11e5-a889-fa163e62896a",
|
|
|
"**checksum_timestamp**": "01440065311"
|
|
|
}
|
|
|
**server side rule: pid creation**
|
|
|
|
|
|
PID_DO_reg {
|
|
|
on (($objPath like "My collection of registered data/*") {
|
|
|
*parent_pid = "None";
|
|
|
*ror = "None";
|
|
|
*iCATCache = bool("true");
|
|
|
EUDATCreatePID(*parent_pid, $objPath, *ror, *iCATCache, *newPID);
|
|
|
}}
|
|
|
|
|
|
**function preview**
|
|
|
|
|
|
It is possible to get back some basic metadata even with file system only browser clients like those supporting the protocols WebDAV and GridFTP. In fact the following json file is stored within the special collection ".metadata" for each registered object:
|
|
|
|
|
|
{
|
|
|
"checksum": "[sha2:nmDjK/7k1D5jjMUFoWHjX5qZmke9vpQbR6FaY9sk6eI=](http://sha2nmDjK)",
|
|
|
"ror": "None",
|
|
|
"pid": "842/6cff9eb8-47ef-11e5-a889-fa163e62896a",
|
|
|
"checksum_timestamp": "01440065311"
|
|
|
}
|
|
|
|
|
|
The json file is stored with the following path:
|
|
|
|
|
|
**<span style="color: rgb(128,0,0);">/path/to/data set/</span>.metadata/** _object-name__metadata.json
|
|
|
**<span style="color: rgb(128,0,0);">/path/to/data set/</span>.metadata/** object-name_metadata.json
|
|
|
|
|
|
|
|
|
***
|
... | ... | @@ -88,20 +78,94 @@ The json file is stored with the following path: |
|
|
|
|
|
The flow is the same as depicted above for a single digital object, however there is no way to get back the list of PIDs yet.
|
|
|
|
|
|
**replication**
|
|
|
|
|
|
PID_Coll_reg {
|
|
|
on (($objPath == "My collection of registered data") {
|
|
|
EUDATPidsForColl($objPath);
|
|
|
}}
|
|
|
|
|
|
The aforementioned mechanism based on json files allows to retrieve asynchronously the PIDs of all the objects of a collection.
|
|
|
|
|
|
***
|
|
|
|
|
|
### Replication
|
|
|
|
|
|
Replication process triggered client-side without PID registration ( **registered data boolean flag** = False):
|
|
|
|
|
|
<ac:image ac:width="700"><ri:attachment ri:filename="b2safe_flows_replication_v2.png"></ri:attachment></ac:image>
|
|
|
![replication](https://raw.githubusercontent.com/wiki/EUDAT-B2SAFE/B2SAFE-core/b2safe_flows_replication_v2.png)
|
|
|
|
|
|
The red box with the label "Messaging system" is an experimental feature to provide back the results in case of asynchronous (server side triggered) process. The messages are posted to a queue which can be accessed via HTTP interface. The queue is provided by dweet.io.
|
|
|
|
|
|
**System metadata Zone A**
|
|
|
|
|
|
data_name: data-object.dat
|
|
|
data_path : /phisycal/path/to/data-object.dat
|
|
|
|
|
|
**B2SAFE Zone A metadata**
|
|
|
|
|
|
attribute: eudat_dpm_checksum_date:resource1 Zone A
|
|
|
value: 01431422755
|
|
|
units:
|
|
|
|
|
|
**System metadata Zone B**
|
|
|
|
|
|
data_name: data-object-replica.dat
|
|
|
data_path : /phisycal/path/to/data-object-replica.dat
|
|
|
|
|
|
**B2SAFE Zone B metadata**
|
|
|
|
|
|
attribute: eudat_dpm_checksum_date:resource1 Zone B
|
|
|
value: 1431529743
|
|
|
units:
|
|
|
|
|
|
Replication process triggered client-side with PID registration ( **registered data boolean flag** = True):
|
|
|
|
|
|
<ac:image ac:width="700"><ri:attachment ri:filename="b2safe_flows_registered-replication_v2.png"></ri:attachment></ac:image>
|
|
|
![replication 2](https://raw.githubusercontent.com/wiki/EUDAT-B2SAFE/B2SAFE-core/b2safe_flows_registered-replication_v2.png)
|
|
|
|
|
|
**System metadata Zone A**
|
|
|
|
|
|
data_name: data-object.dat
|
|
|
data_path : /phisycal/path/to/data-object.dat
|
|
|
|
|
|
**B2SAFE Zone A metadata**
|
|
|
|
|
|
attribute: eudat_dpm_checksum_date:resource1 Zone A
|
|
|
value: 01431422755
|
|
|
units:
|
|
|
----
|
|
|
attribute: PID
|
|
|
value: 842/f5188714-f8b8-11e4-a506-fa163e62896a
|
|
|
units:
|
|
|
|
|
|
**PID record metadata (842/f5188714-f8b8-11e4-a506-fa163e62896a)**
|
|
|
|
|
|
URL irods://hostname/Zone A/path/to/data-object.dat
|
|
|
10320/LOC <locations><location href="irods://hostname:1247/Zone A/path/to/data-object.dat" id="0"/><location href="http://hdl.handle.net/843/7d4c7ad2-24ab-11e5-bf18-0050569ed64e" id="1"/></locations></locations>
|
|
|
CHECKSUM md5:076aad9622fa3118f006927f05222817
|
|
|
|
|
|
**System metadata Zone B**
|
|
|
|
|
|
data_name: data-object-replica.dat
|
|
|
data_path : /phisycal/path/to/data-object-replica.dat
|
|
|
|
|
|
**B2SAFE Zone B metadata**
|
|
|
|
|
|
attribute: eudat_dpm_checksum_date:resource1 Zone B
|
|
|
value: 1431529743
|
|
|
units:
|
|
|
----
|
|
|
attribute: PID
|
|
|
value: 843/7d4c7ad2-24ab-11e5-bf18-0050569ed64e
|
|
|
units:
|
|
|
|
|
|
**PID record metadata (843/7d4c7ad2-24ab-11e5-bf18-0050569ed64e)**
|
|
|
|
|
|
URL irods://hostname/Zone B/path/to/data-object-replica.dat
|
|
|
10320/LOC <locations><location href="irods://hostname:1247/Zone B/path/to/data-object-replica.dat" id="0"/></locations>
|
|
|
CHECKSUM md5:076aad9622fa3118f006927f05222817
|
|
|
EUDAT/ROR 842/f5188714-f8b8-11e4-a506-fa163e62896a
|
|
|
EUDAT/PPID 842/f5188714-f8b8-11e4-a506-fa163e62896a
|
|
|
|
|
|
The **registered data boolean flag** is an input parameter which can be True or False. If True, the replication mechanism assumes that the data have PIDs. If it is not the case, the procedure creates and registers automatically new PIDs for the data.
|
|
|
|
... | ... | @@ -111,7 +175,7 @@ The **status** output parameter is True if the replication has been successful, |
|
|
|
|
|
The **response** output parameter contains the error message in case of failure.
|
|
|
|
|
|
Error messages
|
|
|
Error messages:
|
|
|
|
|
|
* no access permission (on the source)
|
|
|
* empty PID (of the source)
|
... | ... | @@ -121,15 +185,30 @@ Error messages |
|
|
* different size (between source and destination)
|
|
|
* different checksum (between source and destination)
|
|
|
|
|
|
**Replication**
|
|
|
|
|
|
Replication {
|
|
|
*source = /My/source/irods/path;
|
|
|
*destination = /My/destination/irods/path;
|
|
|
*registered = bool("true");
|
|
|
*recursive = bool("true");
|
|
|
*status = EUDATReplication(*source, *destination, *registered, *recursive, *response);
|
|
|
if (*status) {
|
|
|
writeLine("stdout", "Success!");
|
|
|
}
|
|
|
else {
|
|
|
writeLine("stdout", "Failed: *response");
|
|
|
}
|
|
|
}
|
|
|
OUTPUT ruleExecOut
|
|
|
|
|
|
***
|
|
|
|
|
|
### Replication with asynchronous PID registration
|
|
|
|
|
|
In this case the replication is perfomed with **registered data boolean flag = False**. And the PID registration is triggered in a second step.
|
|
|
|
|
|
|
|
|
|
|
|
<ac:image ac:width="700"><ri:attachment ri:filename="b2safe_flows_registration.png"></ri:attachment></ac:image>
|
|
|
![replication async](https://raw.githubusercontent.com/wiki/EUDAT-B2SAFE/B2SAFE-core/b2safe_flows_registration.png)
|
|
|
|
|
|
|
|
|
|
... | ... | |