Commit ce230382 authored by Claudio Cacciari's avatar Claudio Cacciari
Browse files

sorted a bit the dir tree of the package

parent 380f1ace
B2SAFE package 3.0.0 [2015/10/23]
This version is compatible with iRODS v4.x .
It is still backward compatible with iRODS 3.3.1 .
B2SAFE package 3.0 [2014/12/18]
This is the first version compatible with iRODS v4.0.x .
It is still backward compatible with iRODS 3.3.1 .
......
......@@ -56,7 +56,7 @@ class EpicClient(object):
self.cred = cred
self.debug = cred.debug
self.http = httplib2.Http(disable_ssl_certificate_validation=True)
self.http = httplib2.Http(disable_ssl_certificate_validation=False)
self.http.add_credentials(cred.username, cred.password)
# do not throw exceptions for connection errors
self.http.force_exception_to_status_code = True
......
*** Dependencies ***
*** Installation ***
This setup describes the procedure to enable the B2SAFE module for iRODS 3.3.x
There are two automatic ways to deploy the module.
One is based on the python script:
<irods>/modules/B2SAFE/b2safe_install.py
And on its configuration file:
<irods>/modules/B2SAFE/b2safe.conf
The other is based on the shell script:
<irods>/modules/B2SAFE/install.sh
And on its configuration file:
<irods>/modules/B2SAFE/install.conf
And a manual way, reported in the following paragraphs.
Enable the module
Automatically:
0. download the git repository
from https://github.com/EUDAT-B2SAFE/B2SAFE-core/releases
1. copy source conf dir (/whre_you_have_downloaded/B2SAFE-core/conf)
to the future config directory of the b2safe module
(for irods4: /etc/irods/b2safe-core; for irods3: /home/user/b2safe-core)
2. create a symbolic link to the eudat rulebase
for irods3: ln -s /whre_you_have_downloaded/B2SAFE-core/rulebase/*.re <irods>/server/config/reConfigs/*.re
for irods4: ln -s /whre_you_have_downloaded/B2SAFE-core/rulebase/*.re /etc/irods/*.re
replacing the jolly character "*" with each file name available in the directory "rulebase".
3. edit <irods>/server/config/server.config (irods3) or /etc/irods/server.config (irods4)
and append to "reRuleSet" the list of ruleset file names written
in a single line and without the suffix".re", in this way ",filename1,filename2,filename3 ..."
(make sure to include the comma and no spaces)
4. configure iRODS hooks.
edit the <irods>/server/config/reConfigs/core.re (irods3) or /etc/irods/core.re (irods4) file
and add the following two acPostProcForPutHooks:
acPostProcForPut {
ON($objPath like "\*.replicate") {
processReplicationCommandFile($objPath);
}
}
acPostProcForPut {
ON($objPath like "\*.pid.create") {
processPIDCommandFile($objPath);
}
}
5. properly configure the default resource in <irods>/server/config/reConfigs/core.re (irods3)
or /etc/irods/core.re (irods4)
6.1 install the python scripts
for irods3: <irods_dir> is usually /home/user/iRODS
for irods4: <irods_dir> is usually /var/lib/irods/iRODS
ln -s /whre_you_have_downloaded/B2SAFE-core/cmd/* <irods_dir>/server/bin/cmd/
check permissions on the scripts and make sure they are executable by the irods user
e.g.chmod u+x cmd/*
6.2 update the "getEpicApiParameters" rule in "./server/config/reConfigs/euloc.re" (irods3)
or /etc/irods/euloc.re (irods4)
- Configure the credential storage type: "os": stored on the local filesystem
or "irods": stored on de irods namespace.
- Set the path to the credentials file
- set the correct serverID to include the fully qualified hostname.
For instance: "irods://node.domain.com:1247"
- Set the proper values in the credentials file
(see /whre_you_have_downloaded/B2SAFE-core/cmd/credentials_example for an example)
6.3 update the "getAuthZParameters" rule in "./server/config/reConfigs/euloc.re" (irods3)
or /etc/irods/euloc.re (irods4)
- Set the proper values in <irods>/server/bin/cmd/authz.map.json
6.4 update the "getLogParameters" rule in "./server/config/reConfigs/euloc.re" (irods3)
or /etc/irods/euloc.re (irods4)
- Set the proper values in <irods>/server/bin/cmd/log.manager.conf
7. create a shared space in all zones (local and remote), for example: "<zone>/replicate".
Grant all users involved in the replication write access to this collection.
8.0.1 change "#!/usr/bin/env python" in the python scripts in modules/B2SAFE/cmd/ to your python installation
8.0.2 install httplib2, simplejson, lxml, defusedxml, queuelib, and pylint:
httplib2
download from http://code.google.com/p/httplib2
python setup.py install
simplejson
download from http://pypi.python.org/pypi/simplejson/
python setup.py install
pip install lxml
pip install defusedxml
apt-get install pylint
yum install pylint
queuelib
download from https://pypi.python.org/pypi/queuelib
python setup.py install
8.1 test the epic api interaction by running
"./cmd/epicclient.py <os_or_irods> <path_to_credentials> test" script manually
and with "iexecmd epicclient.py"
8.2 test the module by changing and triggering rules in <irods>/modules/B2SAFE/rules
For example, eudatRepl.r to replicate a data set. See the hints inside the rule.
DATACENTER only:
it is necessary to configure the <irods>/server/bin/cmd/authz.map.json file
in order to allow user to execute rules, which calls external scripts (see documentation).
it is necessary to configure the <irods>/server/bin/cmd/log.manager.conf file
in order to enable the logging system (see documentation).
*** Command files ***
The command files are written to the shared iRODS space and are used to control replications and PID management
1) *.replicate
Contains a list of replication commands separated by commas:
command1,command2,command3
Each replication command contains three parameters, separated by a semicolon:
"*pid;*source;*destination"
*pid the handle of the source object
*source the iRODS location of the source object
*destination the iRODS location of the destination object (will be created if it does not exist)
2) *.pid.create
Contains a command to create a new PID:
"create;*pid;*destination"
*pid the handle of the source object (the parent)
*destination the iRODS location of the destination object
3) *.pid.update
Contains a command to update a PID:
"update;*pid;*new_pid"
*pid the handle of the source object (the parent)
*new_pid the (generated) handle of the destination object (the child)
*** Know bugs ***
- the rule set string in the file <irods>/server/config/server.config has a limit of 64 characters.
Hence the error message:
ERROR: rstrcpy not enough space in dest, slen:65, maxLen:64
means that the string has to be made shorter.
- the character '&' is not managed by the function "EUDATSearchPID": the epic client invocation fails.
*** Installation ***
The following describes the procedure to enable B2SAFE pre-release-3.0 for iRODS v4.0.3.
** Through python script **
Assumptions:
iRODSv4.0.3 (Aug'14) installed.
<irods> /var/lib/irods
<absolute-irods-path> /var/lib/irods/iRODS
<etcirods> /etc/irods
- Install B2SAFE files (e.g. git, zip, etc.)
For example, under the <irods>/ directory.
Set permissions (ref. <etcirods>/service_account.config).
cd <irods>
git clone, unzip ...
configure b2safe.config
- choose a conf dir for the B2SAFE package (ex. CONF_DIR: /<etcirods>/b2safe-core)
- choose a log dir (ex. /<etcirods>/b2safe-core/log)
- choose the iRODS shared dir (ex. /myZone/replicate) AND create it
- run b2safe_install.py
- Install httplib2, simplejson, lxml, defusedxml, queuelib and pylint:
- test epic client
- change the following rule in eudat.re, setting the *collectionPath
according to the shared dir set before:
getSharedCollection(*zonePath, *collectionPath) {
EUDATGetZoneNameFromPath(*zonePath, *zoneName)
*collectionPath = "*zoneName/replicate/";
}
** Through binary packages **
It works as follows:
- clone the b2safe project under the user which runs irods. NOT root
git clone …
- go to the directory where the packaging files are:
cd B2SAFE-core/packaging
* RPM *
- create package for the user who run’s the irods instance:
./create_rpm_package.sh
- login as root and install package:
rpm -ivh /home/rods/rpmbuild/RPMS/noarch/irods-eudat-b2safe-3.0-0.noarch.rpm
Preparing... ########################################### [100%]
1:irods-eudat-b2safe ########################################### [100%]
* DEB *
- create package
./create_deb_package.sh # an example follows:
robertv@gitlab:~/git/B2SAFE-core/packaging$ ./create_deb_package.sh
dpkg-deb: building package `irods-eudat-b2safe' in `/home/robertv/debbuild/irods-eudat-b2safe_3.0-0.deb’
- login as root or use sudo to install package:
[gitlab]:~
root# dpkg -i --ignore-depends irods-icat /home/robertv/debbuild/irods-eudat-b2safe_3.0-0.deb
Selecting previously unselected package irods-eudat-b2safe.
(Reading database ... 48938 files and directories currently installed.)
Unpacking irods-eudat-b2safe (from .../irods-eudat-b2safe_3.0-0.deb) ...
Setting up irods-eudat-b2safe (3.0-0) ...
The package b2safe has been installed in /opt/eudat/b2safe.
To install/configure it in iRODS do following as the user rods :
su - rods
cd /opt/eudat/b2safe/packaging
# update install.conf with correct parameters with your favorite editor
./install.sh
- login as user who run’s iRODS and update config and install as above.
*** Dependencies ***
*** Installation ***
This setup describes the procedure to enable the B2SAFE module for iRODS 3.3.x
There are two automatic ways to deploy the module.
One is based on the python script:
<irods>/modules/B2SAFE/b2safe_install.py
And on its configuration file:
<irods>/modules/B2SAFE/b2safe.conf
The other is based on the shell script:
<irods>/modules/B2SAFE/install.sh
And on its configuration file:
<irods>/modules/B2SAFE/install.conf
And a manual way, reported in the following paragraphs.
Enable the module
Automatically:
0. download the git repository
from https://github.com/EUDAT-B2SAFE/B2SAFE-core/releases
1. copy source conf dir (/whre_you_have_downloaded/B2SAFE-core/conf)
to the future config directory of the b2safe module
(for irods4: /etc/irods/b2safe-core; for irods3: /home/user/b2safe-core)
2. create a symbolic link to the eudat rulebase
for irods3: ln -s /whre_you_have_downloaded/B2SAFE-core/rulebase/*.re <irods>/server/config/reConfigs/*.re
for irods4: ln -s /whre_you_have_downloaded/B2SAFE-core/rulebase/*.re /etc/irods/*.re
replacing the jolly character "*" with each file name available in the directory "rulebase".
3. edit <irods>/server/config/server.config (irods3) or /etc/irods/server.config (irods4)
and append to "reRuleSet" the list of ruleset file names written
in a single line and without the suffix".re", in this way ",filename1,filename2,filename3 ..."
(make sure to include the comma and no spaces)
4. configure iRODS hooks.
edit the <irods>/server/config/reConfigs/core.re (irods3) or /etc/irods/core.re (irods4) file
and add the following two acPostProcForPutHooks:
acPostProcForPut {
ON($objPath like "\*.replicate") {
processReplicationCommandFile($objPath);
}
}
acPostProcForPut {
ON($objPath like "\*.pid.create") {
processPIDCommandFile($objPath);
}
}
5. properly configure the default resource in <irods>/server/config/reConfigs/core.re (irods3)
or /etc/irods/core.re (irods4)
6.1 install the python scripts
for irods3: <irods_dir> is usually /home/user/iRODS
for irods4: <irods_dir> is usually /var/lib/irods/iRODS
ln -s /whre_you_have_downloaded/B2SAFE-core/cmd/* <irods_dir>/server/bin/cmd/
check permissions on the scripts and make sure they are executable by the irods user
e.g.chmod u+x cmd/*
6.2 update the "getEpicApiParameters" rule in "./server/config/reConfigs/euloc.re" (irods3)
or /etc/irods/euloc.re (irods4)
- Configure the credential storage type: "os": stored on the local filesystem
or "irods": stored on de irods namespace.
- Set the path to the credentials file
- set the correct serverID to include the fully qualified hostname.
For instance: "irods://node.domain.com:1247"
- Set the proper values in the credentials file
(see /whre_you_have_downloaded/B2SAFE-core/cmd/credentials_example for an example)
6.3 update the "getAuthZParameters" rule in "./server/config/reConfigs/euloc.re" (irods3)
or /etc/irods/euloc.re (irods4)
- Set the proper values in <irods>/server/bin/cmd/authz.map.json
6.4 update the "getLogParameters" rule in "./server/config/reConfigs/euloc.re" (irods3)
or /etc/irods/euloc.re (irods4)
- Set the proper values in <irods>/server/bin/cmd/log.manager.conf
7. create a shared space in all zones (local and remote), for example: "<zone>/replicate".
Grant all users involved in the replication write access to this collection.
8.0.1 change "#!/usr/bin/env python" in the python scripts in modules/B2SAFE/cmd/ to your python installation
8.0.2 install httplib2, simplejson, lxml, defusedxml, queuelib, and pylint:
httplib2
download from http://code.google.com/p/httplib2
python setup.py install
simplejson
download from http://pypi.python.org/pypi/simplejson/
python setup.py install
pip install lxml
pip install defusedxml
apt-get install pylint
yum install pylint
queuelib
download from https://pypi.python.org/pypi/queuelib
python setup.py install
8.1 test the epic api interaction by running
"./cmd/epicclient.py <os_or_irods> <path_to_credentials> test" script manually
and with "iexecmd epicclient.py"
8.2 test the module by changing and triggering rules in <irods>/modules/B2SAFE/rules
For example, eudatRepl.r to replicate a data set. See the hints inside the rule.
DATACENTER only:
it is necessary to configure the <irods>/server/bin/cmd/authz.map.json file
in order to allow user to execute rules, which calls external scripts (see documentation).
it is necessary to configure the <irods>/server/bin/cmd/log.manager.conf file
in order to enable the logging system (see documentation).
*** Command files ***
The command files are written to the shared iRODS space and are used to control replications and PID management
1) *.replicate
Contains a list of replication commands separated by commas:
command1,command2,command3
The following describes the procedure to enable B2SAFE release-3.0.0
Each replication command contains three parameters, separated by a semicolon:
"*pid;*source;*destination"
*pid the handle of the source object
*source the iRODS location of the source object
*destination the iRODS location of the destination object (will be created if it does not exist)
2) *.pid.create
It works as follows:
- clone the b2safe project under the user which runs irods. NOT root
git clone …
- go to the directory where the packaging files are:
cd B2SAFE-core/packaging
Contains a command to create a new PID:
* RPM *
"create;*pid;*destination"
*pid the handle of the source object (the parent)
*destination the iRODS location of the destination object
- create package for the user who run’s the irods instance:
./create_rpm_package.sh
- login as root and install package:
rpm -ivh /home/rods/rpmbuild/RPMS/noarch/irods-eudat-b2safe-3.0-0.noarch.rpm
Preparing... ########################################### [100%]
1:irods-eudat-b2safe ########################################### [100%]
3) *.pid.update
* DEB *
Contains a command to update a PID:
"update;*pid;*new_pid"
*pid the handle of the source object (the parent)
*new_pid the (generated) handle of the destination object (the child)
- create package
./create_deb_package.sh # an example follows:
robertv@gitlab:~/git/B2SAFE-core/packaging$ ./create_deb_package.sh
dpkg-deb: building package `irods-eudat-b2safe' in `/home/robertv/debbuild/irods-eudat-b2safe_3.0-0.deb’
- login as root or use sudo to install package:
[gitlab]:~
root# dpkg -i --ignore-depends irods-icat /home/robertv/debbuild/irods-eudat-b2safe_3.0-0.deb
Selecting previously unselected package irods-eudat-b2safe.
(Reading database ... 48938 files and directories currently installed.)
Unpacking irods-eudat-b2safe (from .../irods-eudat-b2safe_3.0-0.deb) ...
Setting up irods-eudat-b2safe (3.0-0) ...
The package b2safe has been installed in /opt/eudat/b2safe.
To install/configure it in iRODS do following as the user rods :
*** Know bugs ***
su - rods
cd /opt/eudat/b2safe/packaging
# update install.conf with correct parameters with your favorite editor
./install.sh
- the rule set string in the file <irods>/server/config/server.config has a limit of 64 characters.
Hence the error message:
ERROR: rstrcpy not enough space in dest, slen:65, maxLen:64
means that the string has to be made shorter.
- the character '&' is not managed by the function "EUDATSearchPID": the epic client invocation fails.
- login as user who run’s iRODS and update config and install as above.
......@@ -118,10 +118,12 @@ EUDATisMetadata(*path) {
# Author: Claudio Cacciari, Cineca
#
EUDATMessage(*queue, *message) {
getMessageParameters(*msgLogPath)
logInfo("sending message '*message' to the queue '*queue'");
msiExecCmd("messageManager.py", "-l *msgLogPath send *queue *message",
"null", "null", "null", *out);
getMessageParameters(*msgLogPath, *enabled);
if (*enabled) {
logInfo("sending message '*message' to the queue '*queue'");
msiExecCmd("messageManager.py", "-l *msgLogPath send *queue *message",
"null", "null", "null", *out);
}
}
......@@ -556,23 +558,25 @@ getCollectionName(*path_of_collection,*Collection_Name){
EUDATStoreJSONMetadata(*path, *pid, *ror, *checksum, *modtime) {
*extraMetaData = "";
if (*checksum == "" || *checksum == 'None') {
EUDATiCHECKSUMget(*path, *checksum, *modtime);
}
if (*checksum != "") {
*extraMetaData = "-c *checksum -t *modtime";
}
if (*ror == "" || *ror == 'None') {
EUDATGeteRorPid(*pid, *ror);
}
if (*ror != "") {
*extraMetaData = *extraMetaData ++ " -r *ror";
getMetaParameters(*metaConfPath,*enabled);
if (*enabled) {
*extraMetaData = "";
if (*checksum == "" || *checksum == 'None') {
EUDATiCHECKSUMget(*path, *checksum, *modtime);
}
if (*checksum != "") {
*extraMetaData = "-c *checksum -t *modtime";
}
if (*ror == "" || *ror == 'None') {
EUDATGeteRorPid(*pid, *ror);
}
if (*ror != "") {
*extraMetaData = *extraMetaData ++ " -r *ror";
}
msiExecCmd("metadataManager.py","*metaConfPath $userNameClient store '*path'"
++ " -i *pid *extraMetaData", "null", "null", "null", *out);
msiGetStdoutInExecCmdOut(*out, *resp);
}
getMetaParameters(*metaConfPath);
msiExecCmd("metadataManager.py","*metaConfPath $userNameClient store '*path'"
++ " -i *pid *extraMetaData", "null", "null", "null", *out);
msiGetStdoutInExecCmdOut(*out, *resp);
}
......
......@@ -55,22 +55,28 @@ getLogParameters(*logConfPath) {
#
# Arguments:
# *metaConfPath [OUT] the file path to the metadata tool configuration.
# *enabled [OUT] boolean value to enable (if true) globally the
# json metadata writing
#
# Author: Claudio Cacciari (Cineca)
getMetaParameters(*metaConfPath) {
getMetaParameters(*metaConfPath, *enabled) {
*metaConfPath="/srv/irods/current/modules/B2SAFE/cmd/metadataManager.conf";
*enabled=bool("false");
}
#Provides parameters for the message management mechanism
#
# Arguments:
# *msgLogPath [OUT] the file path to the message log file.
# *enabled [OUT] boolean value to enable (if true) globally the
# the usage of the messaging system
#
# Author: Claudio Cacciari (Cineca)
#
getMessageParameters(*msgLogPath) {
getMessageParameters(*msgLogPath, *enabled) {
*msgLogPath="/var/log/irods/messageManager.log";
*enabled=bool("false");
}
# Provides version of the B2SAFE
......
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment