Commit 6c8e08fe authored by Agustin Pane's avatar Agustin Pane
Browse files

(Release) V2.3.1

parent bda5835e
## PySVMON
PySVMON is a develoment project for SVMON client. It migrates bash-based client code to python.
The development is based on python 2.7.13 but runs also in Python3.0.
One has two options to install the client.
1. By root privilege.
2. In virtualenv. In such a case, the report can be only sent via --send options; we are searching for solutions
from pakiti-client.
PySVMON is a develoment project for SVMON client. It migrates bash-based client code to python. The development is based on python 2.7.13 but runs also in Python3.0.
<b> Before installation, check your installation of pakiti-client, see whether it support --svmonreport option. </b>
If unsupported, please remove it. A new version of pakiti-client will be installed with svmon client(rpm or pip).
Make sure you have installed perl-libwww-perl, perl-LWP-Protocol-https packages. <
## Description
## Installation
Note: for more detailed installation explaination please go to [installation section](https://gitlab.eudat.eu/EUDAT-TOOLS/SVMON/pysvmon/-/blob/master/docs/InstallClient.md)
Then one can use pip tools directly ( with root privileges ),
pip install svmon-python-client
<b>Update pip client to the latest version</b>
pip install --upgrade pip
Install python-dotenv:
pip install python-dotenv
---
For test, just run the command svmon, the shell will return you help information. One can also try
svmon -T (--test)
One can see an output with svmon client version. This could cause error with old pip versions.
After test, the service owner could specify the parameters of its own service via command options, and save them.
For example (currently not all service types are supported),
svmon --site KIT --host svmon.eudat.eu --type svmon --dump
the --dump option will save the configuration to config.json file.
<b>Make sure you have write access to the configuration files. </b>
Currently, for B2HANDLE service, one need
to further input epic version file and handle server executable directory into the client.
To check the current configuration, type
svmon --show-config
To see output of svmon client, use
svmon --print
<b>For services without well-defined automated way to extract version information,
please specify the service component, as well as specify the versions. </b> For instance,
svmon --site KIT --host svmon.eudat.eu --type svmon --component spring --with-version 1
--component angular --with-version 2 --dump
and save the specification with --dump options. The --with-version option should be always with --component option, otherwise it makes no sense. And in principal, the combination of these two options can be repeated as many times as the service components.
Then try
pakiti-client --svmonreport
to see the report collected by svmon client.
To send svmon report to our server, run pakiti client:
pakiti-client --svmonreport --url https://svmon.eudat.eu:8443/api/serviceComponent/pakiti/report
One can add it to crontab. Or alternatively, direct post to the svmon server is also supported but unrecommended, try:
svmon --send
. <b>For such option, you need to create an API token after you login in svmon.eudat.eu; and then configure it in the
client with --token yourapitoken --dump
. </b>However, you need install python requests library (http://docs.python-requests.org/en/master/user/install/#install).
In addition, to establish secure link, you will also need SSL trusted CA chain file.
<b> Note: we also enclose the trusted CA file within the client: chain_TERENA_SSL_CA_3.pem.</b>
---
## Troubleshooting
#### Request package not updated
- `/usr/lib/python2.7/site-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.24.1) or chardet (2.2.1) doesn't match a supported version!`
- For solving this you need to upgrade to the last version of requests with the command:
<pre>
pip uninstall urllib3
pip uninstall chardet
pip install requests
</pre>
- PySVMON is the python svmon client app. This client collections the versions of [different services](#available-services) installed on your host and sends them to svmon backend automatically.
## Installation
- See more [here](https://stackoverflow.com/questions/50202238/python-pip-requestsdependencywarning-urllib3-1-9-1-or-chardet-2-3-0-doe)
---
- Please go to [installation section](https://gitlab.eudat.eu/EUDAT-TOOLS/SVMON/pysvmon/-/blob/master/docs/InstallClient.md)
#### Sending report failed, please check your configuration
## Available Services
- In this case could be that your token expired, in that case you need to generate a new apiToken on [svmon.eudat.eu](svmon.eudat.eu)
- Once you have your token you just need to do this:
- Not every service is supported by svmon python client, before installing the client check the list above to see if you service is supported.
- Please note that we aim to support more services, if your service is missing and you want from us to receive support please write us at agustin.pane@kit.edu
<pre>
svmon --token yourtoken --dump
</pre>
---
### We are supporting following service types:
#### You have no token configured, please go to svmon.eudat.eu webpage to create a token and configure it
- Same as previous step
- b2safe
- gitlab
- svmon
- b2handle
- b2access
- b2share
- b2drop
----
#### ImportError: No module named dotenv
You can check the current available list also using svmon --list-service-type command
<pre>
pip install python-dotenv
</pre>
---
In case of problems please contact agustin.pane@kit.edu together with log of the error.
PySVMON
## PySVMON
PySVMON is a develoment project for SVMON client. It migrates bash-based client code to python. The development is based on python 2.7.13 but also runs on python 3.* version.
PySVMON is a develoment project for SVMON client. It migrates bash-based client code to python. The development is based on python 2.7.13 but runs also in Python3.0.
Before installation, check your installation of pakiti-client, see whether it support --svmonreport option.
If unsupported, please remove it. A new version of pakiti-client will be installed with svmon client(rpm or pip).
Then one can use pip tools directly (root privileges if no virtualenv),
## Description
pip install svmon-client
- PySVMON is the python svmon client app. This client collections the versions of [different services](#available-services) installed on your host and sends them to svmon backend automatically.
Update pip client to the latest version (pip 21.1.3)
## Installation
pip install --upgrade pip
- Please go to [installation section](https://gitlab.eudat.eu/EUDAT-TOOLS/SVMON/pysvmon/-/blob/master/docs/InstallClient.md)
Install python-dotenv:
## Available Services
pip install python-dotenv
- Not every service is supported by svmon python client, before installing the client check the list above to see if you service is supported.
- Please note that we aim to support more services, if your service is missing and you want from us to receive support please write us at agustin.pane@kit.edu
For test, just run the command svmon, the shell will return you help information. One can also try
### We are supporting following service types:
svmon -T (--test)
- b2safe
- gitlab
- svmon
- b2handle
- b2access
- b2share
- b2drop
One can see an output with svmon client version. This could cause error with old pip versions.
After test, the service owner could specify the parameters of its own service via command options, and save it
For example for svmon (currently not all service types are supported),
svmon --site KIT --host svmon.eudat.eu --type svmon --dump
the --dump option will save the configuration to config.json file. Currently, for B2HANDLE service, one need
to further input epic version file and handle server excutable directory into the client.
To check the current configuration, type
svmon --show-config
To see output of svmon client,
svmon --print
Make sure you have installed perl-libwww-perl, perl-LWP-Protocol-https packages.
Then try
pakiti-client --svmonreport
to see the report collected by svmon client.
To send svmon report to our server, run pakiti client:
pakiti-client --svmonreport --url https://svmon.eudat.eu:8443/api/serviceComponent/pakiti/report
One can add it to crontab. Or alternatively, direct post to the svmon server is supported, try:
svmon --send
. However, you need install python requests library (http://docs.python-requests.org/en/master/user/install/#install).
In addition, to establish secure link, you will also need SSL trusted CA chain file.
Note: we also enclose the trusted CA file within the client: chain_TERENA_SSL_CA_3.pem.
You can check the current available list also using svmon --list-service-type command
......@@ -43,14 +43,36 @@
![ApiTokens2](apiTokens3.png)
4) Set your service you have interest to report using "svmon --type <serviceName> --dump". Please notice that you can read the available service types list using "svmon ---list-service-type"
4) Run the following command to access our current list of supported services and select the one you have interest to report:
5) Set your site and hostname with: "svmon --site <SITE> --host <HOST> --dump". For example: "svmon --site KIT --host svmon.eudat.eu --dump"
svmon --list-service-type
6) Check that the installation was success by typing "svmon --test"
### The current supported services are:
- b2safe
- gitlab
- svmon
- b2handle
- b2access
- b2share
- b2drop
7) Then send the report using "svmon --send"
5) Set your service you have interest to report using "svmon --type <serviceName> --dump".
svmon --type svmon --dump
6) Set your site and hostname with: "svmon --site <SITE> --host <HOST> --dump".
svmon --site KIT --host svmon.eudat.eu --dump
7) Check that the installation was success by typing:
svmon --test
8) Then send the report using:
svmon --send
Notes: you can see which service and versions will be send to the server by using "svmon --p"
......@@ -77,7 +99,43 @@ Notes: you can see which service and versions will be send to the server by usin
````
pip uninstall svmon-client
````
---
## Troubleshooting
- For troubleshooting please go to the [readme troubleshooting section](https://gitlab.eudat.eu/EUDAT-TOOLS/SVMON/pysvmon/-/blob/master/README.md#Troubleshooting)
#### Request package not updated
- `/usr/lib/python2.7/site-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.24.1) or chardet (2.2.1) doesn't match a supported version!`
- For solving this you need to upgrade to the last version of requests with the command:
<pre>
pip uninstall urllib3
pip uninstall chardet
pip install requests
</pre>
- See more [here](https://stackoverflow.com/questions/50202238/python-pip-requestsdependencywarning-urllib3-1-9-1-or-chardet-2-3-0-doe)
---
#### Sending report failed, please check your configuration
- In this case could be that your token expired, in that case you need to generate a new apiToken on [svmon.eudat.eu](svmon.eudat.eu)
- Once you have your token you just need to do this:
<pre>
svmon --token yourtoken --dump
</pre>
---
#### You have no token configured, please go to svmon.eudat.eu webpage to create a token and configure it
- Same as previous step
----
#### ImportError: No module named dotenv
<pre>
pip install python-dotenv
</pre>
---
In case of problems please contact agustin.pane@kit.edu together with log of the error.
......@@ -3,8 +3,8 @@
from setuptools import setup
setup(
name='svmon_python_client',
version='2.2.7',
name='svmon-client',
version='2.3.1',
packages=['svmon_client'],
scripts=['svmon'],
package_data={'' : ['*.json','*.pem']},
......
......@@ -15,7 +15,7 @@ def main(argv):
try:
opts, args = getopt.getopt(argv,'h:t:S:THVpd:',['help','version','site=','test',\
'host=','tag=','print','dump','type=','list-service-type','show-config','send',\
'handle-server-path=','with-version=','component=','token='])
'handle-server-path=','svmon-app-path=','with-version=','component=','token='])
except getopt.GetoptError:
print (help_info())
sys.exit(0)
......@@ -29,6 +29,7 @@ def main(argv):
readfile =0
filename=""
handle_server_path=""
svmon_app_path=""
specified_components = []
no_specified_components = 0
specified_versions = []
......@@ -92,25 +93,26 @@ def main(argv):
if opt in ['--send']:
send=1
if opt in ['--svmon-app-path']:
svmon_app_path = arg
if opt in ['--handle-server-path']:
handle_server_path = arg
if opt in ['--token']:
token = arg
if no_specified_versions != no_specified_components:
print("The no of specified components does not equal to the no of specified versions")
save = 0
exit(-1)
if save == 1:
print("Saving configurations ....")
if ( handle_server_path ==""):
svmonreport.save_to_json()
if (handle_server_path !=""):
svmonreport.save_b2handle_config_to_json(handle_server_path=handle_server_path)
if (svmon_app_path !=""):
svmonreport.save_svmon_app_path(svmon_app_path=svmon_app_path)
if no_specified_components >0:
svmonreport.save_with_components_to_json(specified_components, specified_versions)
if token != "":
......@@ -166,6 +168,7 @@ def help_info():
--show-config Show the site,host, service type configurations.
--send Send the reports to SVMON server.
--handle-server-path Locate the directory where handle is installed when service type is "b2handle".
--svmon-app-path Locate the directory where svmon is installed when service type is "svmon".
--token Add API token to post report to svmon.eudat.eu
To get a token, one could go to the webpage (svmon.eudat.eu); in the profile dialog, then create.
......
{"service_type": "b2handle", "handle_server_path": "/home/jie/develop/handle-server", "host": "gitlab.eudat.eu", "site": "KIT"}
\ No newline at end of file
{"service_type": "b2handle", "handle_server_path": "/home/jie/develop/handle-server", "svmon_app_path": "/home/svmon-app", "host": "gitlab.eudat.eu", "site": "KIT"}
......@@ -75,7 +75,6 @@ class SVMONReport:
else:
self.service_names = services.get_service_name(self.service_type)
b2handle_config= {}
if self.service_type == 'b2handle' or self.service_type == "B2HANDLE":
cwd = os.path.dirname(services.__file__)
......@@ -93,10 +92,28 @@ class SVMONReport:
else:
print("No configuration for b2handle, please see svmon help")
exit(1)
if self.service_type == 'svmon' or self.service_type == "SVMON":
svmon_config= {}
cwd = os.path.dirname(services.__file__)
filename = cwd + "/config.json"
readok = os.access(filename, os.R_OK)
if readok == False:
print("You have no access to config file: config.json")
exit(-1)
with open(filename, 'r') as f:
load_dict= json.load(f)
if 'svmon_app_path' in load_dict:
if load_dict.get('svmon_app_path') != None and load_dict.get('svmon_app_path') != '':
svmon_config['svmon_app_path'] = load_dict.get('svmon_app_path')
self.tags = services.get_service_tag(self.service_type,svmon_config)
self.service_names = services.get_service_name(self.service_type)
else:
print("No configuration for svmon, please see svmon help")
exit(1)
else:
self.tags = services.get_service_tag(self.service_type)
def get_site(self):
return self.site
......@@ -140,7 +157,6 @@ class SVMONReport:
this.service_names.append(service_name)
this.tags.append(tag)
def print_report(self):
if len(self.service_names) > 0:
res=[]
......@@ -150,19 +166,17 @@ class SVMONReport:
if self.tags == None or len(self.tags) < 1:
print("no service component version is resolved at your host")
exit(1)
j=0
for i in range(len(self.service_names)):
if j < len(self.tags) and self.tags[j] != None and self.tags[j] != "" :
res.append(self.site+'\t'+self.host+'\t'+self.operating_system+'\t'+self.service_type+'\t'+self.service_names[i]+'\t'+self.tags[j]+'\n')
j = j +1
#print(self.service_names[i])
serviceVersionIterator=0
for serviceIterator in range(len(self.service_names)):
if serviceVersionIterator < len(self.tags) and self.tags[serviceVersionIterator] != None and self.tags[serviceVersionIterator] != "" :
res.append(self.site+'\t'+self.host+'\t'+self.operating_system+'\t'+self.service_type+'\t'+self.service_names[serviceIterator]+'\t'+self.tags[serviceVersionIterator]+'\n')
serviceVersionIterator = serviceVersionIterator +1
return res
else:
print("Zero service components")
return []
def jsonify(self):
if len(self.service_names) > 0:
res = {}
......@@ -195,7 +209,6 @@ class SVMONReport:
if handle_server_path != "":
res['handle_server_path'] = handle_server_path
cwd = os.path.dirname(services.__file__)
filename = cwd + "/config.json"
writeok = os.access(filename, os.W_OK)
......@@ -207,9 +220,37 @@ class SVMONReport:
print("Save json data failed")
exit(1)
else:
print("Saved B2HANLE configuration")
print("Saved B2HANDLE configuration")
def save_svmon_app_path(self,svmon_app_path=None):
if svmon_app_path == None :
print("No svmon_app_path configuration input")
exit(-1)
if svmon_app_path != None and isinstance(svmon_app_path,str) == False:
print("Svmon path should be a string")
exit(-1)
if svmon_app_path == "":
print("No input for svmon_app_path")
exit(-1)
res={}
res['site']= self.site
res['host'] = self.host
res['service_type'] = self.service_type
res['svmon_app_path'] = svmon_app_path
cwd = os.path.dirname(services.__file__)
filename = cwd + "/config.json"
writeok = os.access(filename, os.W_OK)
if writeok == False:
print("You have no access to write config file: config.json")
exit(-1)
success = json_operations.save_to_file(filename,res)
if success == False:
print("Save json data failed")
exit(1)
else:
print("Saved SVMON configuration")
def save_to_json(self):
if self.site != "" and self.site != None:
......@@ -217,7 +258,7 @@ class SVMONReport:
res['site']= self.site
res['host'] = self.host
res['service_type'] = self.service_type
if self.service_type == "b2hanle" or self.service_type == "B2HANDLE":
if self.service_type == "b2handle" or self.service_type == "B2HANDLE":
cwd = os.path.dirname(services.__file__)
filename = cwd + "/config.json"
with open(filename, 'r') as f:
......@@ -242,8 +283,6 @@ class SVMONReport:
print("No services to be jsonified")
exit(1)
def save_with_components_to_json(self,specified_components, specified_versions):
if specified_components== None or specified_versions == None and len(specified_versions) == 0 \
......@@ -275,9 +314,6 @@ class SVMONReport:
print("Successfully saved component data")
exit(1)
def set_pair(self,key,value):
if isinstance(key,str) == False:
print("Key should be a string")
......@@ -296,14 +332,12 @@ class SVMONReport:
elif key == "tags":
self.set_tags(value)
else :
print("such key is not supporteed: %s\n" %key)
print("such key is not supported: %s\n" %key)
exit(1)
def check_site(self):
return True
def check_host(self):
return True
......@@ -318,8 +352,6 @@ class SVMONReport:
print('Please use svmon --site <SITE> --host <HOST> --type <TYPE> --dump to change your configuration')
print('More details, please see svmon --help')
def send_report_to_svmon_server(self):
import requests as re
print('Sending report to svmon server...')
......@@ -403,7 +435,6 @@ class SVMONReport:
print('Report has been sent successfully.')
exit(0)
def save_token(self,token):
print('Saving token...')
if token == "":
......@@ -426,8 +457,6 @@ class SVMONReport:
exit(1)
if __name__=="__main__":
re=SVMONReport()
re.save_to_json()
......@@ -94,6 +94,15 @@ def get_service_tag(service_type,configs=None):
exit(-1)
elif service_type == "svmon":
if 'svmon_app_path' in configs:
print('Getting svmon version from config file')
tags={}
if configs.get('svmon_app_path') != None and configs.get('svmon_app_path') != '' :
svmonVersions = get_svmon_version(configs.get('svmon_app_path'))
tags['svmon'] = [svmonVersions[1], svmonVersions[3]]
_services['svmon']=[svmonVersions[0], svmonVersions[2]]
return tags['svmon']
print('Returning default versions for svmon')
return _tags['svmon']
elif service_type == "b2handle" or service_type == "B2HANDLE":
......@@ -161,6 +170,46 @@ def get_handle_server(handle_server_path):
tmp = tmp.split('version')
return tmp[1].lstrip()
# get svmon backend and frontend versions
def get_svmon_version(svmon_app_path):
config = remoteConfig.RemoteConfig().getConfig()
cwd = os.path.dirname(svmon_app_path + '/scripts/')
filename = cwd + "/getVersions.bash"
if os.path.exists(filename) == False:
print("Svmon path does not exists or missing getVersions.bash")
exit(-1)
if os.access(filename,os.R_OK) == False:
print("You have no access to read script file: getVersions.bash")
exit(-1)
svmonVersionExecResult = subprocess.Popen(filename, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
svmonVersionExecResult = svmonVersionExecResult.communicate()
if svmonVersionExecResult == None and len(svmonVersionExecResult) < 1:
print("Svmon path executable is incorrect")
exit(1)
# [,svmon-frontend, frontendVersion, svmon-backend, backendVersion,]
svmonVersionsArray = svmonVersionExecResult[0].decode('utf-8').split(',')
enableDebug = config["DEBUG_MODE"]
if(enableDebug):
print('SvmonVersionsArray: ')
print(svmonVersionsArray)
print('Length: ', len(svmonVersionsArray))
if len(svmonVersionsArray) < 5:
print("No versions were found for svmon")
exit(1)
frontendName = svmonVersionsArray[1].lstrip()
frontendVersion = svmonVersionsArray[2].lstrip()
backendName = svmonVersionsArray[3].lstrip()
backendVersion = svmonVersionsArray[4].lstrip()
return [frontendName, frontendVersion, backendName, backendVersion]
#get svmon client version