Commit 75e2c58f authored by Beatriz Sanchez's avatar Beatriz Sanchez
Browse files

B2safe 4.3

parent 577ff6d1
*** Installation/Upgrade ***
The following describes the procedure to enable B2SAFE release-4.x.y with the
B2SHARE publication option.
The connection between B2SAFE and B2SHARE enables a B2SAFE user to trigger a set if IRods rules that call a python script (b2shareclientCLI) that connects to the HTTP API of b2share and performs the necesary acction to publis a document or collection.
INSTALLATION
The B2SHARE connection component is an extension of the B2SAFE core package (https://gitlab.eudat.eu/b2safe/B2SAFE-core), so following software are expected to be in place before installation:
iRODS
B2HANDLE or msi-persistent
B2SAFE
python 3
NOTE: B2HANDLE can be found at: https://github.com/EUDAT-B2SAFE/B2HANDLE
NOTE: iRODS is running as a normal user process. NOT as root. The package can
be build by any user. During installation of the package it will use:
"/etc/irods/service_account.config" to set the ownership of the files.
NOTE: iRODS needs to be installed AND configured before installing/upgrading B2SAFE
To install the B2SHARE-connect component you have to do following steps:
- clone the b2safe-b2share-connect project as any user. NOT root
$ git clone https://github.com/EUDAT-co.git
- add following scripts to \<your path to B2SAFE\>/B2SAFE-core/cmd:
* b2shareclient.py
* b2shareclientCLI.py
* configuration.py
* create_md_file.py
* irodsUtility.py
- add the rule “b2share.re” from B2SAFE-core/rulebase folder to the "rulebase" folder of your iRODS instance \<your path to B2SAFE\>/B2SAFE-core/rulebase
- add to your configuration folder \<your path to B2SAFE\>/B2SAFE-core/conf and modify the configuration file “b2share_client_example.json” according to your environment as described in "Configuration" section of this wiki page and rename it to “b2share_client.json”
- check for missing python libraries trying to run the major scripts with -d (dry run) option
* b2shareclientCLI.py
* create_md_file.py
This are mostly: python-irodsclient jsonpatch requests configparser mock pytest a list in included in requisites.txt
Try to install missing packages with the standard package manager like pip (sudo pip3.6 install), apt, yum, zypper etc.
As described in the “Example Workflow” section, the iRODS rules will trigger these scripts according to the flags user specify in the imeta of the collection.
CONFIGURATION
There are two major scripts for the B2SHARE connection component b2shareclientCLI.py and create_md_file.py that are using the configuration stored in file “b2share_client.json”. The configuration has of 3 parts.
The first two sections `logging` and `b2share_http_api` are prefilled with default values.
* `logging` - with 2 values needed to be specified: log level, default is `"loglevel": "DEBUG"`, and the file where to safe the logging information, default value is `"logfile": "/opt/eudat/b2safe/log/b2share_connection_client.log"`, so in a file named b2share_connection_client.log in folder log under the installation path.
* `b2share_http_api` - with 1 value needed to be specified: host_name of the B2SHARE instance, default is the address of the training instance of B2SHARE `"host_name": "https://trng-b2share.eudat.eu/"`. Do not change the attribute `"access_parameter": "?access_token"` or any other attributes of `b2share_http_api`. It is a part of the string needed to build the URL for the B2SHARE HTTP API and need to be changed only if the B2SHARE HTTP API will change.
* `irods` - connection information of the iRODS instance like the name of the iRODS zone `"zone_name": "YOUR_ZONE"` and `"irods_env": "/home/irods/.irods/irods_environment.json"`.
```
{
"configurations": {
"b2share_http_api": {
"host_name": "https://trng-b2share.eudat.eu/",
"access_parameter": "?access_token",
"list_communities_endpoint": "api/communities/",
"get_community_schema_endpoint": "/schemas/last",
"records_endpoint": "api/records/"
},
"irods": {
"zone_name": "YOUR_ZONE",
"irods_env": "/home/irods/.irods/irods_environment.json",
"resources": "",
"irods_home_dir": "",
"irods_debug": ""
},
"logging": {
"loglevel": "DEBUG",
"logfile": "/opt/eudat/b2safe/log/b2share_connection_client.log"
}
}
}
```
Example Workflow
Following workflow was considered during the component development:
1. The user is registered at B2SHARE and has an B2SHARE access token. The user provides the b2share token to the B2SAFE administrator, who adds it to the user metadata: `imeta add
-u <irods_user_name> access_token <token_value>`
2. The user adds a specific metadata attributes to the collection, that will be used to create a draft:
`imeta add -C \<collection_X\> EUDAT_B2SHARE_PUBLISH EUDAT` with community name as the value, to publish the collection unter
`imeta add -C \<collection_X\> EUDAT_B2SHARE_TITLE Some_Title` and the title of the publication.
The `imeta ls -C \<collection_X\>` will then deliver:
```
AVUs defined for collection collection_X:
attribute: EUDAT_B2SHARE_PUBLISH
value: EUDAT
units:
\----
attribute: EUDAT_B2SHARE_TITLE
value: Some_Title
units:
```
3. As preparation the user can create a meta data file **b2share_metadata.json** in the collection. The helper script **create_md_file.py** creates a sceleton file according to th
e B2SHARE schema of the community.
4. The B2SAFE administrator executes a rule to scan the B2SAFE repository for that meta data tag or configure a cron job or a script to do so.
* The workflow the rule executes will then create a draft in B2SHARE with a list of files the collection contains. (see worflow diagram below)
* Then it will add a meta data to the draft if the collection is containing the file **b2share_metadata.json** with describing meta data. For this it will compare the meta data in B2SHARE and that in the file and override the ones that user has added online in B2SHARE.
* Finaly it will publish the draft and "freez" the collection in B2SAFE by taking away the users access rights for the collection and giving the rights to the B2SAFE administrator.
5. The user has then the possibility to fill out the meta data online in the B2SHARE in the published record. The record id will be add to the imeta (EUDAT_B2SHARE_PUBLISHED_ID) of the collection and the record in B2SHARE will be accessable to the user as it will be published with his access token.
Following assumptions were made:
* there is just one owner for each collection.
* the drafts are registered in B2SHARE just one time, even if the rule to scan for draft is executed multiple times.
* the collections are copied and published just one time, even if the rule to publish them is executed multiple times.
NOTE: following needs to be updated/added in "/opt/eudat/b2safe/packaging/install.conf":
- DEFAULT_RESOURCE
- SERVER_ID
- HANDLE_SERVER_URL (needed for epicclient2)
- PRIVATE_KEY (needed for epicclient2)
- CERTIFICATE_ONLY (needed for epicclient2)
- PREFIX (needed for epicclient2)
- HANDLEOWNER (needed for epicclient2)
- REVERSELOOKUP_USERNAME (needed for epicclient2)
- HTTPS_VERIFY (needed for epicclient2)
- AUTHZ_ENABLED (default=true)
- MSG_QUEUE_ENABLED (default=false)
# update /opt/eudat/b2safe/conf/b2share_client.json with the parameters of your b2safe/irods intallation
and with your access parameters for your b2share instance
- check the python scripts for missing python libraries.
$ cd /opt/eudat/b2safe/cmd
$ ./authZmanager.py -h
$ ./epicclient2.py --help
$ ./logmanager.py -h
$ ./messageManager.py -h
$ ./metadataManager.py -h
$ ./timeconvert.py epoch_to_iso8601 2000000
Try to install missing packages with the standard package manager like apt, yum, zypper etc.
If packages are not within the standards use pip and install the missing packages with pip.
DONE
== B2SHARE ==
find information about how to use it in
https://eudat.eu/services/userdoc/the-b2share-http-rest-api#UserDocumentation-B2SHARE-Authentication
......@@ -25,14 +25,66 @@ Known issues can be found in [https://github.com/EUDAT-B2SAFE/B2SAFE-core/wiki/K
## Install
NOTE: iRODS needs to be installed AND configured before installing/upgrading B2SAFE
--------------
Deployment
---------------
* [Deployment on Centos 7, see install_centos7.md](install_centos7.md)
* [Deployment on other systems, see install_other.txt](install_other.txt)
1. on your **SAFE** server go to the irods user home directory
2. git clone this repository
```
git clone .git B2SAFE-core
```
## b2share
Information about b2share can be found in https://eudat.eu/services/userdoc/the-b2share-http-rest-api
The following describes the procedure to enable B2SAFE release-4.x.y with the B2SHARE publication option
Install the python packages listed in /opt/eudat/b2safe/cmd/requirements.txt
Create or update /opt/eudat/b2safe/conf/b2share_client.json with the parameters of your b2safe/irods intallation and with your access parameters for your b2share instance
```
{
"configurations": {
"b2share_http_api": {
"host_name": "https://trng-b2share.eudat.eu/",
"access_parameter": "?access_token",
"list_communities_endpoint": "api/communities/",
"get_community_schema_endpoint": "/schemas/last",
"records_endpoint": "api/records/"
},
"irods": {
"zone_name": "tempZone",
"irods_env": "/home/irods/.irods/irods_environment.json",
"resources": "",
"irods_home_dir": "",
"irods_debug": ""
},
"logging": {
"loglevel": "DEBUG",
"logfile": "/opt/eudat/b2safe/log/b2share_connection_client.log"
}
},
"record_id": "5f4beef03bf8445a8214fd13bdda3398"
```
The value of record_id correspond with the user access tocken
How to get a access tocken
To install the B2SHARE connectors you need to use pipenv (install using pip3 install pipenv)cat
```
......
#!/usr/bin/env python
#!/usr/bin/env python3
#
# authZ.manager.py
#
......@@ -36,7 +36,7 @@ class AuthZClient(object):
def __init__(self, mapFilePath):
"""Initialize object with connection parameters."""
self.mapfile = mapFilePath
self.mapfile = mapFilePath
def _debugMsg(self, method, msg):
"""Internal: Print a debug message if debug is enabled."""
......
#!/usr/bin/env python
#!/usr/bin/env python3
#
# epicclient.py
#
......@@ -381,7 +381,7 @@ def search_execution(client, search_key=None, search_value=None):
# set default return value
result = None
json_result = "None"
if search_key is not None and search_value is not None:
try:
kvpairs = dict([(search_key, str(''.join(search_value)))])
......@@ -406,7 +406,7 @@ def read_execution(client, read_handle, read_key=None):
# set default return value
result = None
json_result = "None"
if read_key is None:
try:
# retrieve whole handle
......@@ -478,7 +478,8 @@ def create_execution(client, create_handle, create_location, create_overwrite=Fa
result = 'error'
except HandleSyntaxError:
result = 'error'
except GenericHandleError as ex:
result = 'error'
return(result)
def delete_execution(client, delete_handle, delete_key=None):
......
#!/usr/bin/env python
#!/usr/bin/env python3
#
# authZ.manager.py
#
......@@ -48,7 +48,7 @@ class LogManager(object):
"""Internal: Print a debug message if debug is enabled."""
if self.debug:
print "[", method, "]", msg
print("[", method, "]", msg)
def _parseConf(self):
"""Internal: Parse the configuration file.
......@@ -59,8 +59,8 @@ class LogManager(object):
tmp = eval(filehandle.read())
filehandle.close()
except (OSError, IOError) as e:
print "problem while reading configuration file %s" % self.file
print "Error:", e
print("problem while reading configuration file %s" % self.file)
print("Error:", e)
self.log_level_value = self.log_level[tmp['log_level']]
self.log_dir = tmp['log_dir']
......@@ -72,7 +72,7 @@ class LogManager(object):
self.logger.setLevel(self.log_level_value)
self.logfile = self.log_dir+'/b2safe.log'
rfh = logging.handlers.RotatingFileHandler(self.logfile,
maxBytes=2097152,
maxBytes=2097152,
backupCount=9)
formatter = logging.Formatter('%(asctime)s %(levelname)s:%(message)s')
rfh.setFormatter(formatter)
......@@ -83,7 +83,7 @@ class LogManager(object):
queuedir = self.log_dir+'/b2safe.queue'
self.queue = FifoDiskQueue(queuedir)
def getLogger(self):
"""get the logger instance."""
......@@ -91,12 +91,12 @@ class LogManager(object):
def getLogFile(self):
"""get the log file."""
return self.logfile
def getQueue(self):
"""get the queue instance."""
return self.queue
......@@ -128,17 +128,17 @@ def log(args):
def push(args):
'''perform push action'''
logManager = LogManager(args.conffilepath, args.debug)
logManager.initializeQueue()
queue = logManager.getQueue()
queue.push(' '.join(args.message))
queue.push(' '.join(args.message).encode())
queue.close()
def pop(args):
'''perform pop action'''
logManager = LogManager(args.conffilepath, args.debug)
logManager.initializeQueue()
queue = logManager.getQueue()
......@@ -147,66 +147,66 @@ def pop(args):
i = 0
messages = []
while i < int(args.number):
message = queue.pop()
message = queue.pop().decode()
if message is not None:
messages.append(message)
else:
break
i += 1
print messages
print(messages)
else:
message = queue.pop()
print message
message = queue.pop().decode()
print(message)
queue.close()
def queuesize(args):
'''get the current size of the queue'''
logManager = LogManager(args.conffilepath, args.debug)
logManager.initializeQueue()
queue = logManager.getQueue()
length = str(len(queue))
print length
print(length)
queue.close()
def test(args):
'''do a serie of tests'''
def read_log_last_line(logger, logfile):
'''local helper func: read the last line of the log file'''
logger.info('test log message')
try:
filehandle = open(logfile, "r")
linelist = filehandle.readlines()
filehandle.close()
except (OSError, IOError) as er:
print "problem while reading file %s" % logfile
print "Error:", er
print("problem while reading file %s" % logfile)
print("Error:", er)
return False
lastline = linelist[len(linelist)-1]
if 'test log message' in lastline:
return True
return False
def test_result(result, testval):
"""local helper func: print OK/FAIL for test result
Returns 0 on OK, 1 on failure"""
if type(result) != type(testval):
print "FAIL (wrong type; test is bad)"
print("FAIL (wrong type; test is bad)")
return 1
if result == testval:
print "OK"
print("OK")
return 0
else:
print "FAIL"
print("FAIL")
return 1
try:
......@@ -215,55 +215,55 @@ def test(args):
logger = logManager.getLogger()
logfile = logManager.getLogFile()
except IOError as er:
print "IOError, directory 'log' might not exist"
print "Error: ", er
print("IOError, directory 'log' might not exist")
print("Error: ", er)
return False
logManager.initializeQueue()
queue = logManager.getQueue()
fail = 0
print
print "logging info to the file " + logfile
print "Test: log message"
print()
print("logging info to the file " + logfile)
print("Test: log message")
fail += test_result(read_log_last_line(logger, logfile), True)
print
print "Test: queue size before push"
print("Test: queue size before push")
fail += test_result(len(queue), 0)
print
print "Test: push info to the queue"
queue.push('test message')
print "OK"
print()
print("Test: push info to the queue")
queue.push('test message'.encode())
print("OK")
print
print "Test: queue size after push "
print()
print("Test: queue size after push ")
fail += test_result(len(queue), 1)
print
print "Test: pop info from queue"
fail += test_result('test message', queue.pop())
print
print "Test: check queue size, again (should be 0)"
print()
print("Test: pop info from queue")
fail += test_result('test message', queue.pop().decode())
print()
print("Test: check queue size, again (should be 0)")
fail += test_result(len(queue), 0)
queue.close()
if fail == 0:
print
print "All tests passed OK"
print()
print("All tests passed OK")
else:
print
print "%d tests failed" % fail
print()
print("%d tests failed" % fail)
if __name__ == "__main__":
parser = argparse.ArgumentParser(description='EUDAT log manager. ')
parser.add_argument("conffilepath", default="NULL",
help="path to the configuration file")
parser.add_argument("-d", "--debug", help="Show debug output",
parser.add_argument("-d", "--debug", help="Show debug output",
action="store_true")
subparsers = parser.add_subparsers(title='Actions',
......@@ -273,12 +273,12 @@ if __name__ == "__main__":
parser_log = subparsers.add_parser('log', help='log a message')
parser_log.add_argument("level", help='the value of the log level')
parser_log.add_argument("message", nargs='+',
parser_log.add_argument("message", nargs='+',
help='the message to be logged')
parser_log.set_defaults(func=log)
parser_push = subparsers.add_parser('push', help='push a message')
parser_push.add_argument("message", nargs='+',
parser_push.add_argument("message", nargs='+',
help='the message to be queued')
parser_push.set_defaults(func=push)
......
#!/usr/bin/env python
#!/usr/bin/env python3
import dweepy
import argparse
import sys
import logging
import logging.handlers
import json
import dweepy
logger = logging.getLogger('messageManager')
......@@ -31,11 +31,11 @@ def getlast(args):
try:
jsonPayLoad = dweepy.get_latest_dweet_for(args.queue)
except dweepy.DweepyError as e:
logger.exception('Failed to get the last message from queue %s',
args.queue)
logger.exception('Failed to get the last message from queue %s',
args.queue)
sys.exit(1)
print json.dumps({jsonPayLoad[0]['created']:
jsonPayLoad[0]['content']['message']})
print(json.dumps({jsonPayLoad[0]['created']:
jsonPayLoad[0]['content']['message']}))
def getall(args):
"""get all the messages from the queue"""
......@@ -43,13 +43,13 @@ def getall(args):
try:
jsonPayLoad = dweepy.get_dweets_for(args.queue)
except dweepy.DweepyError as e:
logger.exception('Failed to get all the message from queue %s',
args.queue)
logger.exception('Failed to get all the message from queue %s',
args.queue)
sys.exit(1)
payLoad = {}
for message in jsonPayLoad:
payLoad[message['created']] = message['content']['message']
print json.dumps(payLoad)
print(json.dumps(payLoad))
def _initializeLogger(args):
"""initialize the logger instance."""
......@@ -60,7 +60,7 @@ def _initializeLogger(args):
logger.setLevel(logging.INFO)
if (args.log):
han = logging.handlers.RotatingFileHandler(args.log,
maxBytes=6000000,
maxBytes=6000000,
backupCount=9)
else:
han = logging.StreamHandler()
......@@ -74,7 +74,7 @@ if __name__ == "__main__":
parser = argparse.ArgumentParser(description='EUDAT message manager.')
parser.add_argument("-l", "--log", help="Path to the log file")
parser.add_argument("-d", "--debug", help="Show debug output",
parser.add_argument("-d", "--debug", help="Show debug output",
action="store_true")
subparsers = parser.add_subparsers(title='Actions',
......
#!/usr/bin/env python
#!/usr/bin/env python3
#
# * use 4 spaces!!! not tabs
# * See PEP-8 Python style guide http://www.python.org/dev/peps/pep-0008/
......@@ -7,7 +7,7 @@
import json
user_map_file="/etc/irods/user_map.json"
user_map_file = "/etc/irods/user_map.json"
def parseUserMap():
"""Parse the user map file"""
......@@ -19,5 +19,5 @@ def parseUserMap():
if __name__ == "__main__":
parseUserMap()
#!/usr/bin/env python
#!/usr/bin/env python3
import re
import sys
......@@ -11,11 +11,11 @@ def search_pattern_in_string(pattern, string):
"""
searchObj = re.search(pattern, string, re.U|re.I)
if searchObj:
print searchObj.group()
print(searchObj.group())
else:
print "no match found!"
print("no match found!")
if __name__ == "__main__":