The Rorodata CLI¶
The primary interface to the rorodata platform is the command-line interface.
The command-line client can be installed using:
pip install roro-client
Make sure you have at least version 0.1.6 of the client.
$ roro version
roro, version 0.1.6
Once installed, make sure you log in to the platform using:
$ roro login
Email address: anand@rorodata.com
Password:
Login successful.
It is prompt for your email and password. If you don’t already have an account, please sign up at http://www.rorodata.com/.
Most of the commands work in the context of a project and they must be executed from the project directory. These commands look at the roro.yml file to find the project name.
Projects¶
The list of projects can be found using the roro projects command.
$ roro projects
hello-world
credit-risk
New project can be created using:
$ roro create my-new-project
Created project: my-new-project
The project name is unique across the platform and a project name once used by any user can not be used by anyone else.
Support for deleting a project is in progress and will be available in future versions.
Deploy¶
The roro deploy command is used to deploy a project. Deploy is the only way to send any changes in the project to the platform, including code changes, changes to roro.yml to add/delete more services/periodic tasks.
While deploy is just a single command, lot of things happen behind the scenes.
- all the files in the project directory are archived and send to the platform
- A docker image is created with the specified runtime as base image and all the dependencies in the
requirements.txtfile installed, if present - All the services specified in the
roro.ymlare (re)started and end points are created - The scheduled tasks are updated
The deploy command prints the summary of changes and end points for each service in the project.
$ roro deploy
Deploying project credit-risk. This may take a few moments ...
Building docker image... done.
Updating scheduled jobs... done.
Restarted one service.
default: https://credit-risk.rorocloud.io/
Deployed version 5 of credit-risk project.
Scripts & Notebooks¶
The roro run command is used to run any script, typically a python program in the rorodata platform.
$ roro run python train.py
Started new job 4fa27081
That starts a new job and that runs on the platform. You can look at the logs of the job using the roro logs command, which contains the all logs printed by the script.:
$ roro logs 4fa27081
starting the job
training decision tree model...
training complete.
the model is saved to /volumes/data/model.pkl
Please remember that it uses the code that is last deployed. If you have any changes to the code that you want to run, you need to deploy before running the script.
Notebooks can be run using the roro run:notebook command.
$ roro run:notebook
starting the job
Jupyter notebook is available at:
https://517832f3.rorocloud.io/?token=rorocloud
The jupyter notebook server can be stopped using:
roro stop 517832f3
It starts a new notebook in the project’s software environment created using the previous deploy and provides a URL endpoint to access it.
Please remember that the notebook server continues to run until it is stopped.
Processes & Logs¶
The list of processes currently running in a project can be found using the roro ps command.
$ roro ps
JOBID STATUS WHEN TIME INSTANCE TYPE CMD
-------- -------- ------------- ------- --------------- -------------------
c19f745b running 7 seconds ago 0:00:07 C1 python train.py
137f3d2a running 9 seconds ago 0:00:07 C1 [notebook]
A process can be stopped using roro stop command.
$ roro stop 137f3d2a
The logs of any process can be seen using the roro logs command.
$ roro logs c19f745b
started training
iteration 100 - accuracy 0.57
iteration 200 - accuracy 0.65
iteration 300 - accuracy 0.68
iteration 400 - accuracy 0.69
The roro ps command shows only the active processes. To see all processes ever run in the project, call with -a flag.
JOBID STATUS WHEN TIME INSTANCE TYPE CMD
-------- -------- ------------- ------- ------------- ---------------
c19f745b running 7 seconds ago 0:00:07 C1 python train.py
137f3d2a running 9 seconds ago 0:00:07 C1 [notebook]
18cb1ce2 success 1 day ago 0:00:01 C1 python task.py
d75e8553 success 1 day ago 0:00:01 C1 python task.py
f95b01a1 success 2 days ago 0:00:02 C1 python task.py
71fe89cc success 2 days ago 0:00:02 C1 python task.py
b46cbb8e success 3 days ago 0:00:02 C1 python task.py
dd75b3fb success 3 days ago 0:00:02 C1 python task.py
Volumes¶
The rorodata platform has built-in support for volumes for storing persistent data. By default, two volumes data and notebooks are created for every project when the project is created. The volumes used to store any input data, intermediate results, checkpoints and final results.
Volumes can also be used for storing machine learning models, but the model management system provided by the rorodata platform offers much better capabilities.
To roro volumes command can be used to list the volumes in a project.
$ roro volumes
data
notebooks
New volumes can be created using the roro volumes:add command.
$ roro volumes:add new-volume-name
Volume new-volume-name added to the project credit-risk
To list files in a volume:
$ roro volumes:ls notebooks
credit-risk.ipynb
Files can copied to and from a volume.
For example, to copy a local file to data volume:
$ roro cp dataset.csv data:dataset.csv
Or the other way:
$ roro cp data:dataset.csv dataset.csv
Config¶
The rorodata platform provides support for storing the project secrets like database urls, access and secret keys for various third-party services, etc.
The config variables are set in the environment of every process that is run in the project.
The roro config comamnd lists all the available config variables.
$ roro config
=== credit-risk Config Vars
DATABSE_URL: postgres://yxulQ5Ib9:QOJoFJZwv5FYIM0y@db1.example.com
One or more config variables can be added using the roro config:set command.
$ roro config:set X=1 Y=2
Updated config vars
$ roro config
=== credit-risk Config Vars
DATABSE_URL: postgres://yxulQ5Ib9:QOJoFJZwv5FYIM0y@db1.example.com
X: 1
Y: 2
The roro config:unset command is used to unset config vars.
$ roro config:unset X
Updated config vars
$ roro config
=== credit-risk Config Vars
DATABSE_URL: postgres://yxulQ5Ib9:QOJoFJZwv5FYIM0y@db1.example.com
Y: 2
Please remember that the services are not restarted after config:set or config:unset. They may have to be restarted using the roro deploy command to use the new configuration.