Unverified Commit 7d80b2dd authored by michec81's avatar michec81 Committed by GitHub
Browse files

Merge pull request #142 from EUDAT-B2SAFE/devel

Robert requested to merge again devel into the new branch v3.2-beta5
parents 2cb7ed5d f71a206b
# Building the package locally
![](dev-workflow.png)
## Requirements
* docker
* docker-compose
## Building and testing
The script can be used *./ci/build.sh* to build the RPM package.
### Build the RPM
Build the B2SAFE package for CentOS 7 and iRODS 4.2.6.
```
./ci/build.sh centos7_4_2_6
```
If the build process was successful, the RPM will be copied to the shared target folder:
```
ls ci/RPMS/Centos/7/irods-4.2.6/
irods-eudat-b2safe-4.3.0-0.noarch.rpm
```
### Configure the functional test
In order to run the tests make sure that you have access to a handle server with a dedicated test prefix.
You should have received a private key and a certificate PEM file
(in this case *308_21.T12995_TRAINING_privkey.pem* and *308_21.T12995_TRAINING_certificate_only.pem*)
Configure the following files:
*ci/secret/epic2_credentials*:
```
{
"handle_server_url": "https://epic5.storage.surfsara.nl:8003",
"private_key": "/etc/irods/308_21.T12995_TRAINING_privkey.pem",
"certificate_only": "/etc/irods/308_21.T12995_TRAINING_certificate_only.pem",
"prefix": "21.T12995",
"handleowner": "200:0.NA/21.T12995",
"reverselookup_username": "21.T12995",
"reverselookup_password": "<INSERT_PASSWORD_HERE>",
"HTTPS_verify": "False"
}
```
*ci/secret/install.json*
```
{
"server_id": "irods://{HOSTNAME}:1247",
"server_api_reg": "irods://{UUID}:1247",
"server_api_pub": "irods://{UUID}:1247",
"handle_server_url": "https://epic5.storage.surfsara.nl:8003",
"handle_private_key": "/etc/irods/308_21.T12995_TRAINING_privkey.pem",
"handle_certificate_only": "/etc/irods/308_21.T12995_TRAINING_certificate_only.pem",
"handle_prefix": "21.T12995",
"handle_owner": "200:0.NA/21.T12995",
"handle_reverse_lookup_name": "21.T12995",
"handle_reverse_lookup_password": "<INSERT_PASSWORD_HERE>",
"handle_https_verify": "False",
"handle_users": [ "*" ],
"handle_groups": []
}
```
### Run the functional test
```
./ci/test.sh centos7_4_2_6
```
### Automation
The details can be found in the Jenkinsfile in the root of the repository.
......@@ -56,29 +56,45 @@ To install/configure it in iRODS do following as the user who runs iRODS:
### Install required python modules
As the user who runs iRODS do following:
```bash
cd /opt/eudat/b2safe/cmd
sudo pip install -r requirements.txt
pip install --user -r requirements.txt
```
### update install.conf with correct parameters with your favorite editor.
### convert install.conf to install.json if needed
When going from version 4.2.x or lower of B2SAFE to version 4.3 or higher convert the installation configuration.
As the user who runs iRODS do following:
```bash
sudo vi /opt/eudat/b2safe/packaging/install.json
cd /opt/eudat/b2safe/packaging
./convert_b2safe_conf_to_json.sh
```
| parameter | comment |
|------------------------|------------------------|
| DEFAULT_RESOURCE | |
| SERVER_ID | |
| HANDLE_SERVER_URL | needed for epicclient2 |
| PRIVATE_KEY | needed for epicclient2 |
| CERTIFICATE_ONLY | needed for epicclient2 |
| PREFIX | needed for epicclient2 |
| HANDLEOWNER | needed for epicclient2 |
| REVERSELOOKUP_USERNAME | needed for epicclient2 |
| HTTPS_VERIFY | needed for epicclient2 |
| AUTHZ_ENABLED | default=true |
| MSG_QUEUE_ENABLED | default=false |
Notice all the warnings and take them in to account.
### update install.json with correct parameters with your favorite editor.
As the user who runs iRODS do following:
```bash
sudo vi /opt/eudat/b2safe/packaging/install.json
```
| parameter | comment |
|--------------------------------|---------------------------------------------|
| irods_default_resource | |
| server_id | |
| server_api_reg | if no htp api make it same as server_id |
| server_api_pub | if no htp api make it same as server_id |
| handle_server_url | needed for msi_pid uService |
| handle_private_key | needed for msi_pid uService |
| handle_certificate_only | needed for msi_pid uService |
| handle_prefix | needed for msi_pid uService |
| handle_owner | needed for msi_pid uService |
| handle_reverse_lookup_name | needed for msi_pid uService |
| handle_reverse_lookup_password | needed for msi_pid uService |
| handle_https_verify | needed for msi_pid uService |
| handle_users | needed for msi_pid uService. Users in iRODS |
| handle_groups | needed for msi_pid uService. Group in iRODS |
| authz_enabled | default=true |
| msg_queue_enabled | default=false |
### install/configure it as the user who runs iRODS
......@@ -90,7 +106,7 @@ sudo su - $IRODS_SERVICE_ACCOUNT_NAME -s "/bin/bash" -c "cd /opt/eudat/b2safe/pa
DONE
## confgiure B2HANDLE
## configure B2HANDLE
* Ask the handle hosting service which user to use for a certificate.
* Create a private/public keypair and create a derived certificate as described
in http://eudat-b2safe.github.io/B2HANDLE/creatingclientcertificates.html.:
......@@ -100,7 +116,7 @@ in http://eudat-b2safe.github.io/B2HANDLE/creatingclientcertificates.html.:
It can be found on: http://www.handle.net/download_hnr.html.
ii> Execute ./hdl-keygen from hsj-8.1.1/bin directory
b> Send public key (.bin file) to your hosting service.
c> Step 2: Upload the user’s public key to the.... is executed by the hosting service
c> Step 2: Upload of the user’s public key to the appropiate handle is executed by the hosting service
```
* Ask hosting service which username/password to use for reverselooup.
* Test using curl
......
......@@ -53,12 +53,38 @@ MSG_QUEUE_ENABLED=false
if [ -e $INSTALL_CONFIG ]
then
source $INSTALL_CONFIG
source $INSTALL_CONFIG
else
echo "ERROR: $INSTALL_CONFIG not present!"
STATUS=1
echo "ERROR: $INSTALL_CONFIG not present!"
STATUS=1
fi
# remove trailing /iRODS or /iRODS/ from IRODS_DIR
IRODS_DIR=${IRODS_DIR%/iRODS}
IRODS_DIR=${IRODS_DIR%/iRODS/}
# empty server_api_reg
if [ "x" == "x${SERVERAPIREG}" ]
then
echo "WARNING: server_api_reg is empty in install.json. Please give it a reasonable value."
echo " if the http api is not implemented make it the same as the server_id."
echo ""
fi
# empty server_api_pub
if [ "x" == "x${SERVERAPIPUB}" ]
then
echo "WARNING: server_api_pub is empty in install.json. Please give it a reasonable value."
echo " if the http api is not implemented make it the same as the server_id."
echo ""
fi
#empty handle_reverse_lookup_password
echo "WARNING: handle_reverse_lookup_password is empty in install.json. Please give it a reasonable value."
echo " It can be retrieved from /opt/eudat/b2safe/conf/credentials"
echo ""
# set handle_https_verify
string=$(echo "$HTTPS_VERIFY" | tr '[:upper:]' '[:lower:]')
if [[ $string =~ .*true.* ]]
then
......@@ -70,11 +96,18 @@ else
HTTPS_VERIFY_STRING="\"${HTTPS_VERIFY}\""
fi
# set handle_users
let count=0
handle_users_array=(`echo ${USERS} | sed 's/[\t ]+/\n/g'`)
handle_users_string=
for each in ${handle_users_array[@]}
do
if [[ $each =~ .*user0#Zone0.* || $each =~ .*user1#Zone1.* ]]
then
echo "WARNING: handle_users has a default value: \"$each\" in install.json. Please give it a reasonable value."
echo " This will be enforced with the msi_pid uService."
echo ""
fi
if [ $count -eq 0 ]
then
handle_users_string=$(echo -n "\"$each\"")
......@@ -84,7 +117,6 @@ do
let count=$count+1
done
cat > install.json << EOT
{
"b2safe_package_dir": "${B2SAFE_PACKAGE_DIR}",
......@@ -102,6 +134,7 @@ cat > install.json << EOT
"handle_prefix": "${PREFIX}",
"handle_owner": "${HANDLEOWNER}",
"handle_reverse_lookup_name": "${REVERSELOOKUP_USERNAME}",
"handle_reverse_lookup_password": "",
"handle_https_verify": ${HTTPS_VERIFY_STRING},
"handle_users": [ ${handle_users_string} ],
"handle_groups": [ ],
......
......@@ -233,7 +233,7 @@ def update_authz_map(json_config):
##write_json_config(authz_map_config, authz_map_file)
def update_epicclient_credentials(json_config):
''' create (if needed) and update epicclient2.py credentials file '''
''' create (if needed) and update epicclient2.py credentials file. Do not update existing file '''
credentials_file = json_config["cred_file_path"]
......@@ -243,24 +243,24 @@ def update_epicclient_credentials(json_config):
credentials_file)
secure_file(credentials_file)
# save credentails file
save_config_file(credentials_file)
# save credentails file
save_config_file(credentials_file)
# read credentials file
epicclient2_config = read_json_config(credentials_file)
# read credentials file
epicclient2_config = read_json_config(credentials_file)
# modify epicclient2 credentials
epicclient2_config["handle_server_url"] = json_config["handle_server_url"]
epicclient2_config["private_key"] = json_config["handle_private_key"]
epicclient2_config["certificate_only"] = json_config["handle_certificate_only"]
epicclient2_config["prefix"] = json_config["handle_prefix"]
epicclient2_config["handleowner"] = json_config["handle_owner"]
epicclient2_config["reverselookup_username"] = json_config["handle_reverse_lookup_name"]
epicclient2_config["reverselookup_password"] = json_config["handle_reverse_lookup_password"]
epicclient2_config["HTTPS_verify"] = json_config["handle_https_verify"]
# modify epicclient2 credentials
epicclient2_config["handle_server_url"] = json_config["handle_server_url"]
epicclient2_config["private_key"] = json_config["handle_private_key"]
epicclient2_config["certificate_only"] = json_config["handle_certificate_only"]
epicclient2_config["prefix"] = json_config["handle_prefix"]
epicclient2_config["handleowner"] = json_config["handle_owner"]
epicclient2_config["reverselookup_username"] = json_config["handle_reverse_lookup_name"]
epicclient2_config["reverselookup_password"] = json_config["handle_reverse_lookup_password"]
epicclient2_config["HTTPS_verify"] = json_config["handle_https_verify"]
# write credentials config
write_json_config(epicclient2_config, credentials_file)
# write credentials config
write_json_config(epicclient2_config, credentials_file)
def update_flat_file_parameter(modify_file, mod_key, mod_value, irods_file=False):
''' update a file and make it 'key = "value";' '''
......@@ -472,6 +472,10 @@ def update_pid_uservice_config(json_config):
pid_uservice_config["permissions"]["groups_delete"] = json_config["handle_groups"]
pid_uservice_config["permissions"]["groups_write"] = json_config["handle_groups"]
# lookup value change if NO http api. This happens if three values are the same.
if json_config["server_id"] == json_config["server_api_reg"] == json_config["server_api_pub"]:
pid_uservice_config["lookup"]["value"] = "{IRODS_URL_PREFIX}{OBJECT}"
# print json.dumps(pid_uservice_config, indent=2, sort_keys=True)
# write pid uService config
write_json_config(pid_uservice_config, pid_uservice_conf_file)
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment