Systems¶
Once you are authorized to make calls to the various services, one of first things you may want to do is view storage and execution resources available to you or create your own. In Tapis a storage or execution resource is referred to as a system.
Overview¶
A Tapis system represents a server or collection of servers exposed through a single host name or IP address. Each system is associated with a specific tenant. A system can be used for the following purposes:
Running a job, including:
Staging files to a system in preparation for running a job.
Executing a job on a system.
Archiving files and data on a remote system after job execution.
Storing and retrieving files and data.
Each system is of a specific type (such as LINUX or S3) and owned by a specific user who has special
privileges for the system. The system definition also includes the user that is used to access the system,
referred to as effectiveUserId. This access user can be a specific user (such as a service account) or dynamically
specified as ${apiUserId}
. For the case of ${apiUserId}
, the username is extracted from the identity
associated with the request to the service.
Model¶
At a high level a system represents the following information:
- id
A short descriptive name for the system that is unique within the tenant.
- description
An optional more verbose description for the system.
- systemType - Type of system
LINUX, S3, IRODS or GLOBUS
- owner
A specific user set at system creation. By default this is the resolved value for ${apiUserId}, the user making the request to create the system.
- host - Host name, IP address or Globus ID
FQDN, IP address, Globus endpoint ID or Globus collection ID.
- enabled - Enabled flag
Indicates if system is currently considered active and available for use. By default this is true.
- effectiveUserId - Effective User
The username to use when accessing the system. A specific user (such as a service account) or the dynamic user
${apiUserId}
. By default this is${apiUserId}
.- defaultAuthnMethod - Default authentication method
How access authentication is handled by default. Authentication method can also be specified as part of a request. Supported methods: PASSWORD, PKI_KEYS, ACCESS_KEY, TOKEN.
- bucketName - Bucket name
For an S3 system this is the name of the bucket.
- rootDir - Effective root directory
Directory to be used when listing files or moving files to and from the system. For LINUX and IRODS this is required and must begin with
/
. For S3 this is optional and typically will not begin with/
. May not be updated. Contact support to request a change.- dtnSystemId - DTN system Id
An alternate system to use as a Data Transfer Node (DTN) during job execution. The execution system and the DTN system must have shared storage.
- dtnMountPoint - DTN mount point
Mount point (aka target) used when running the mount command on this system. During job execution this is the path on this system for files transferred to rootDir on dtnSystemId.
- dtnMountSourcePath - DTN mount source path
The path exported by dtnSystemId that matches the dtnMountPoint on this system. This will be relative to rootDir on dtnSystemId. Used during job execution.
- isDtn
Indicates if system will be used as a data transfer node (DTN). If this is true then rootDir is required, canExec must be false and following may not be specified: dtnSystemId, dtnMountSourcePath, dtnMountPoint and all job execution related attributes. By default this is false.
- canExec
Indicates if system can be used to execute jobs.
- canRunBatch
Indicates if system supports running jobs using a batch scheduler. By default this is false.
- enableCmdPrefix
Indicates if system allows a job submission request to specify a cmdPrefix. Since cmdPrefix is a free form command it is a security concern. By default this is false.
- Job related attributes
Various attributes related to job execution such as jobRuntimes, jobWorkingDir, batchScheduler, batchLogicalQueues
When creating a system the required attributes are: id, systemType, host, defaultAuthnMethod and canExec. Depending on the type of system and specific values for certain attributes there are other requirements.
Note that a system may be created as a storage-only resource (canExec=false) or as a system that can be used for both execution and storage (canExec=true).
Getting Started¶
Before going into further details about Systems, here we give some examples of how to create and view systems.
In the examples below we assume you are using the TACC tenant with a base URL of tacc.tapis.io
and that you have
authenticated using PySDK or obtained an authorization token and stored it in the environment variable JWT,
or perhaps both.
Creating a System¶
Create a local file named system_example.json
with json similar to the following:
{
"id":"tacc-sample-<userid>",
"description":"My storage system",
"host":"tapis-vm.tacc.utexas.edu",
"systemType":"LINUX",
"defaultAuthnMethod":"PKI_KEYS",
"effectiveUserId":"${apiUserId}",
"rootDir":"/",
"canExec": false
}
where <userid> is replaced with your username, and your host name is updated appropriately. Note that although credentials may be included in the definition we have not done so here. For security reasons, it is recommended that login credentials be updated using a separate API call as discussed below.
Using PySDK:
import json
from tapipy.tapis import Tapis
t = Tapis(base_url='https://tacc.tapis.io', username='<userid>', password='************')
with open('system_example.json', 'r') as openfile:
my_storage_system = json.load(openfile)
t.systems.createSystem(**my_storage_system)
Using CURL:
$ curl -X POST -H "content-type: application/json" -H "X-Tapis-Token: $JWT" https://tacc.tapis.io/v3/systems -d @system_example.json
Registering Credentials for a System¶
Now that you have registered a system you will need to register credentials so you can use Tapis to access the host. Various authentication methods can be used to access a system, such as PASSWORD, PKI_KEYS and TOKEN. Note that the TOKEN authentication method is for systems of type GLOBUS. Registering credentials for a GLOBUS type system is a special case that involves steps different from those described in this section. Please see the section below on Registering Credentials for a Globus System for more information.
Here we will cover registering PKI_KEYS (i.e. ssh keys) as an example.
Create a local file named cred_tmp.json
with json similar to the following:
{
"publicKey": "<ssh_public_key>",
"privateKey": "<ssh_private_key>"
}
where <ssh_public_key> and <ssh_private_key> are replaced with your keys. The keys must be encoded on a single line with embedded newline characters. You may find the following linux command useful in converting a multi-line private key into a single line:
cat $privateKeyFile | awk -v ORS='\\n' '1'
Using PySDK:
t.systems.createUserCredential(systemId='tacc-sample-<userid>', userName='<userid>', publicKey='<ssh_public_key>', privateKey='<ssh_private_key>'))
Using CURL:
$ curl -X POST -H "content-type: application/json" -H "X-Tapis-Token: $JWT" https://tacc.tapis.io/v3/systems/credential/tacc-sample-<userid>/user/<userid> -d @cred_tmp.json
An optional attribute loginUser may be included in the request body in order to map the Tapis user to a username to be used when accessing the system. If the login user is not provided then there is no mapping and the Tapis user is always used when accessing the system. When a loginUser is provided the json would be similar to the following:
{
"publicKey": "<ssh_public_key>",
"privateKey": "<ssh_private_key>",
"loginUser": "<linux_host_username>"
}
Note that credentials are stored in the Security Kernel. Only specific Tapis services are authorized to retrieve credentials.
Use of PKI_KEYS as credentials¶
When using an ssh keypair as credentials there are several important points to keep in mind. As discussed above, the public key and private key must be encoded on a single line. This can sometimes be challenging. For example, copying and pasting may convert newline characters in a way that is not compatible with processing in Tapis. You may find the following linux command useful in converting a multi-line private key into a single line:
cat $privateKeyFile | awk -v ORS='\\n' '1'
Also, Tapis does not currently support OPENSSH type keys. After generating the keypair, please inspect the first few
lines of the private key file and confirm that it is not of type OPENSSH. Typically, a valid private key file will
start with the line -----BEGIN RSA PRIVATE KEY-----
.
If your private key is of type OPENSSH please use a command similar to the following to generate your keypair:
ssh-keygen -t rsa -b 4096 -m PEM
When generating the keypair, do not use a passphrase. This can interfere with non-interactive use of the keypair.
Finally, please be aware that if the host has multi-factor authentication (MFA) enabled this may prevent Tapis from communicating with the host. Tapis does not currently support MFA.
When encountering problems here are some suggestions on what to check:
Public and private keys are each on one line in the json file. Newline characters in private key are properly encoded.
Keypair is not of type OPENSSH
Keypair does not have a passphrase
Public key has been added to the authorized_keys file for the target user. File ~/.ssh/authorized_keys
File ~/.ssh/authorized_keys has proper permissions.
MFA is not enabled for the target host.
If problems persist you can also attempt to manually validate the keypair using a command similar to this:
ssh -i /tmp/my_private_key testuser@myhost.com
where /tmp/my_private_key contains the original multi-line private key. If everything is set up correctly and the keypair is valid you should be logged into the host without being prompted for a password.
Registering Credentials for a Globus System¶
Registering credentials for a GLOBUS type system is a special case that involves steps different from those described in
the section above. For a GLOBUS type system, the user will need to use the TOKEN authentication method and generate
an accessToken
and refreshToken
using two special-purpose System service endpoints.
Please note that your Tapis site installation must have been configured by the site administrator to support Globus. Please see Globus_Config.
Obtain Globus Authorization Code¶
The first step in generating Globus credentials is for the user to call the systems authUrl credential endpoint to obtain a Globus authorization code.
Using CURL, the request would look something like this:
$curl -H "X-Tapis-Token: $JWT" https://dev.tapis.io/v3/systems/credential/globus/authUrl
The response should look similar to the following. Note that for brevity and readability, only the result portion of the response is shown, the response has been split into multiple lines and various IDs are not filled in:
{
"url": "https://auth.globus.org/v2/oauth2/authorize?client_id=<client_id>
&redirect_uri=https%3A%2F%2Fauth.globus.org%2Fv2%2Fweb%2Fauth-code
&scope=openid+profile+email+urn%3Aglobus%3Aauth%3Ascope%3Atransfer.api.globus.org%3Aall
&state=_default&response_type=code&code_challenge=<challenge_id>
&code_challenge_method=S256&access_type=offline",
"sessionId": "<session_id>"
}
The user should copy the url (as a single string, no line breaks) and make note of the session Id for later use. The user then visits the provided URL and is presented with a Globus logon page that will allow them to authenticate using one of thousands of supported identity providers, including through their existing organization using CILogon.
The user must use the following flow to obtain an authorization code:
Visit the provided URL and authenticate through Globus. After authentication, user is re-directed back to a Globus page showing the access being requested by Tapis.
Fill in a label for future reference and click Allow to authorize Tapis to access Globus on their behalf.
Copy the provided authorization code in preparation for the final step. Note that the code is valid for a short time (as of this writing it is valid for 10 minutes).
Exchange Authorization Code for Tokens¶
The final step is for the user to call the systems credential endpoint to exchange the authorization code and session ID for tokens which are stored by the Systems service in a credentials record.
Using CURL, the request would look something like this:
$curl -X POST -H "content-type: application/json" -H "X-Tapis-Token: $JWT"
https://dev.tapis.io/v3/systems/credential/<system>/user/<user>/globus/tokens/<authCode>/<sessionId>
The response should look similar to the following:
{
"result": null,
"status": "success",
"message": "SYSAPI_CRED_UPDATED Credential updated. ...",
"version": "1.3.1",
"commit": "619aa7ce",
"build": "2023-04-02T19:06:38Z",
"metadata": null
}
At this point the user will have registered credentials for a Tapis system that can be used as a source or destination for Globus operations.
Viewing Systems¶
Retrieving details for a system¶
To retrieve details for a specific system, such as the one above:
Note
See the section below on Selecting to find out how to control the amount of information returned.
Using PySDK:
t.systems.getSystem(systemId='tacc-sample-<userid>')
Using CURL:
$ curl -H "X-Tapis-Token: $JWT" https://tacc.tapis.io/v3/systems/tacc-sample-<userid>
The response should look similar to the following:
{
"result": {
"tenant": "dev",
"id": "tacc-sample-<userid>",
"description": "My storage system",
"systemType": "LINUX",
"owner": "<userid>",
"host": "tapis-vm.tacc.utexas.edu",
"enabled": true,
"effectiveUserId": "<userid>",
"defaultAuthnMethod": "PKI_KEYS",
"authnCredential": null,
"rootDir": "/",
"port": 22,
"useProxy": false,
"proxyHost": "",
"proxyPort": -1,
"dtnSystemId": null,
"dtnMountPoint": null,
"dtnMountSourcePath": null,
"isDtn": false,
"canExec": false,
"canRunBatch": false,
"enableCmdPrefix": false,
"jobRuntimes": [],
"jobWorkingDir": null,
"jobEnvVariables": [],
"jobMaxJobs": 2147483647,
"jobMaxJobsPerUser": 2147483647,
"batchScheduler": null,
"batchLogicalQueues": [],
"batchDefaultLogicalQueue": null,
"jobCapabilities": [],
"tags": [],
"notes": {},
"uuid": "f83606bf-7a1a-4ff0-9953-dd732cc07ac0",
"deleted": false,
"created": "2021-04-26T18:45:40.771Z",
"updated": "2021-04-26T18:45:40.771Z"
},
"status": "success",
"message": "TAPIS_FOUND System found: tacc-sample-<userid>",
"version": "0.0.1",
"metadata": null
}
Note that authnCredential is null. Only specific Tapis services are authorized to retrieve credentials.
Retrieving details for all systems¶
To see the list of systems that you own:
Using PySDK:
t.systems.getSystems()
Using CURL:
$ curl -H "X-Tapis-Token: $JWT" https://tacc.tapis.io/v3/systems?select=allAttributes
The response should contain a list of items similar to the single listing shown above.
Child Systems¶
Creating Child Systems¶
The ability to create child systems provides a way to easily clone and manage systems based on existing systems. Child systems allow a user to set only a few fields, and use all other values from an existing system. This reduces the difficulty in creating a child system, but also allows the child system to be updated when the parent is updated.
To create a child system, create a local file (for example child_system_example.json) with the following:
{
"id": "my-child-<userid>",
"effectiveUserId": "${apiUserId}",
"rootDir": "/home/<userid>"
}
Where <userid> is replaced with your username. Also ensure that the root directory path is correct. Now use the create child system REST endpoint to create the child system. Let’s assume that the new child system will be a child of a parent system called parent-system.
Using PySDK:
import json
from tapipy.tapis import Tapis
t = Tapis(base_url='https://tacc.tapis.io', username='<userid>', password='************')
with open('child_system_example.json', 'r') as openfile:
child_system = json.load(openfile)
t.systems.createChildSystem(parentId="parent-system", **child_system)
Using CURL:
$ curl -X POST -H "content-type: application/json" -H "X-Tapis-Token: $JWT" https://tacc.tapis.io/v3/systems/parent-system/createChildSystem -d @child_system_example.json
These fields are maintained independently for child systems:
Attribute |
Type |
Example |
Notes |
---|---|---|---|
id |
String |
ds1.storage.default |
|
owner |
String |
jdoe |
|
enabled |
boolean |
FALSE |
|
effectiveUserId |
String |
tg869834 |
|
rootDir |
String |
/home/${apiUserId} |
|
deleted |
boolean |
FALSE |
|
created |
Timestamp |
2020-06-19T15:10:43Z |
|
updated |
Timestamp |
2020-07-04T23:21:22Z |
|
During the creation of a child system, any of these fields may be specified except for created, updated and deleted. All other fields are taken from the parent system.
Updating a Child System¶
Updates are done just like any other system, however, only the following fields may be updated for a child system.
Attribute |
Type |
Example |
Notes |
---|---|---|---|
effectiveUserId |
String |
tg869834 |
|
Some other fields can be updated through special endpoints. For example deleted and enabled are updated through the endpoints for deleting, undeleting, enabling and disabling.
Child System Operations¶
Most operations other than update are the same for child systems as they are for parent systems. For more information see the appropriate section of the document for the operation.
Delete - see Deletion
Undelete - see Deletion
Enable - see “enabled” in System Attributes Table
Disable - see “enabled” in System Attributes Table
Unlinking a Child System from it’s Parent System¶
A child system may be unlinked from it’s parent. This is a permanent operation, and cannot be undone. This will make the child a standalone system with all of it’s current settings. When the unlink happens any fields that had previously been linked to the parent will be copied to the child, and it will be as if the child was created as in independent system with those values.
If the owner of the child system wants to unlink the child from it’s parent, the owner may use the unlinkFromParent endpoint.
Using PySDK:
import json
from tapipy.tapis import Tapis
t = Tapis(base_url='https://tacc.tapis.io', username='<userid>', password='************')
t.systems.unlinkFromParent(childSystemId="<child-system-id>")
Using CURL:
$ curl -X POST -H "content-type: application/json" -H "X-Tapis-Token: $JWT" https://tacc.tapis.io/v3/systems/<child-system-id>/unlinkFromParent
Replace <child-system-id> with the id of the child system.
The owner of a parent system can also decide to unlink child systems from the parent. In that case the parent system owner would use the unlinkChildren endpoint. The child systems to unlink may be specified in the request body. First create a json file (for example children_to_unlink.json):
{
"childSystemIds":
[
"<child-system-1-id>",
"<child-system-2-id>"
...
]
}
Using PySDK:
import json
from tapipy.tapis import Tapis
t = Tapis(base_url='https://tacc.tapis.io', username='<userid>', password='************')
with open('children_to_unlink.json', 'r') as openfile:
children_to_unlink = json.load(openfile)
t.systems.unlinkChildren(parentSystemId="<parent-system-id>", **children_to_unlink)
Using CURL:
$curl -X POST -H "content-type: application/json" -H "X-Tapis-Token: $JWT" https://tacc.tapis.io/v3/systems/<parent-system-id>/unlinkChildren -d @./children_to_unlink.json
Or all child systems using all=True (no json file required)
Using PySDK:
import json
from tapipy.tapis import Tapis
t = Tapis(base_url='https://tacc.tapis.io', username='<userid>', password='************')
t.systems.unlinkChildren(parentSystemId="<parent-system-id>", all=True)
Using CURL:
$ curl -X POST -H "content-type: application/json" -H "X-Tapis-Token: $JWT" "https://tacc.tapis.io/v3/systems/<parent-system-id>/unlinkChildren?all=true"
Minimal Definition and Restrictions¶
When creating a system the required attributes are: id, systemType, host, defaultAuthnMethod and canExec. Depending on the type of system and specific values for certain attributes there are other requirements. The restrictions are:
If systemType is S3 then bucketName is required, canExec and isDtn must be false.
If systemType is LINUX or IRODS then rootDir is required and must begin with
/
.If effectiveUserId is
${apiUserId}
(i.e. it is not static) then authnCredential may not be specified.If isDtn is true then rootDir is required, canExec must be false and following may not be specified: dtnSystemId, dtnMountSourcePath, dtnMountPoint, all job execution related attributes.
If canExec is true then jobWorkingDir is required and jobRuntimes must have at least one entry.
If canRunBatch is true then batchScheduler must be specified.
If canRunBatch is true then batchLogicalQueues must have at least one item.
If batchLogicalQueues has more than one item then batchLogicalDefaultQueue must be specified.
If batchLogicalQueues has exactly one item then batchLogicalDefaultQueue is set to that item.
Permissions¶
The permissions model allows for fine grained access control of Tapis systems.
At system creation time the owner is given full access to the system. Permissions for other users may be granted and revoked through the systems API. Please note that grants and revokes through this service only impact the default role for the user. A user may still have access through permissions in another role. So even after revoking permissions through this service, when permissions are retrieved the access may still be listed. This indicates access has been granted via another role.
Permissions are specified as either *
for all permissions or some combination of the
following specific permissions: ("READ","MODIFY","EXECUTE")
. Specifying permissions in all
lower case is also allowed. Having MODIFY
implies READ
.
Sharing¶
In addition to fine grained permissions support, Tapis also supports a higher level approach to granting access.
This approach is known simply as sharing. The sharing API allows you to share a system with a set of users
as well as share publicly with all users in a tenant. Sharing provides READ+EXECUTE
access.
When the system has a dynamic effectiveUserId, sharing also allows for MODIFY access to all paths for calls
made through the Files service.
Note that Tapis permissions and sharing are independent of native permissions enforced by the underlying system host.
The most common use case for sharing a system is to publicly share the system with all users in the tenant. This would allow any user to use the system for execution or storage when running an application.
For more information on sharing please see Resource Sharing
Authentication Credentials¶
At system creation time the authentication credentials may be specified if the effective
access user effectiveUserId is a specific user (such as a service account) and not
a dynamic user (i.e. not equal to ${apiUserId}
).
If the effective access user is dynamic (i.e. equal to ${apiUserId}
) then authentication credentials for any
user allowed to access the system must be registered in separate API calls. In this case the payload provided may
contain the optional attribute loginUser which will be used to map the Tapis user to a username to be used when
accessing the system. If the login user is not provided then there is no mapping and the Tapis user is always used
when accessing the system.
Note that the Systems service does not store credentials. Credentials are persisted by the Security Kernel service and only specific Tapis services are authorized to retrieve credentials.
By default any credentials provided for LINUX and S3 type systems are verified. The query parameter skipCredentialCheck=true may be used to bypass the initial verification of credentials.
Runtime¶
Runtime environment supported by the system that may be used to run applications, such as docker or singularity. Consists of the runtime type and version.
Logical Batch Queue¶
A queue that maps to a single HPC queue. Logical batch queues provide a uniform front end abstraction for an HPC queue. They also provide more features and flexibility than is typically provided by an HPC scheduler. Multiple logical queues may be defined for each HPC queue. If an HPC queue does not have a corresponding logical queue defined then a user will not be able use the Tapis system to directly submit a job via Tapis to that HPC queue.
Deletion¶
A system may be deleted and undeleted. Deletion means the system is marked as deleted and is no longer available for use. By default deleted systems will not be included in searches and operations on deleted systems will not be allowed. When listing systems the query parameter showDeleted may be used in order to include deleted systems in the results.
System Attributes Table¶
Attribute |
Type |
Example |
Notes |
---|---|---|---|
tenant |
String |
designsafe |
|
id |
String |
ds1.storage.default |
|
description |
String |
Default storage |
|
systemType |
enum |
LINUX |
|
owner |
String |
jdoe |
|
host |
String |
data.tacc.utexas.edu |
|
enabled |
boolean |
FALSE |
|
effectiveUserId |
String |
tg869834 |
|
defaultAuthnMethod |
enum |
PKI_KEYS |
|
authnCredential |
Credential |
|
|
bucketName |
String |
tapis-ds1-jdoe |
|
rootDir |
String |
/home/${apiUserId} |
|
port |
int |
22 |
|
useProxy |
boolean |
TRUE |
|
proxyHost |
String |
|
|
proxyPort |
int |
|
|
dtnSystemId |
String |
default.corral.dtn |
|
dtnMountPoint |
String |
/gpfs/corral3/repl |
|
dtnMountSourcePath |
String |
/gpfs/corral3/repl |
|
isDtn |
boolean |
FALSE |
|
canExec |
boolean |
|
|
canRunBatch |
boolean |
|
|
enableCmdPrefix |
boolean |
|
|
jobRuntimes |
[Runtime] |
|
|
jobWorkingDir |
String |
HOST_EVAL($SCRATCH) |
|
jobEnvVariables |
[KeyValuePair] |
|
|
jobMaxJobs |
int |
|
|
jobMaxJobsPerUser |
int |
|
|
batchScheduler |
String |
SLURM |
|
batchLogicalQueues |
[LogicalQueue] |
|
|
batchDefaultLogical Queue |
LogicalQueue |
|
|
tags |
[String] |
|
|
notes |
String |
“{}” |
|
uuid |
UUID |
|
|
deleted |
boolean |
FALSE |
|
created |
Timestamp |
2020-06-19T15:10:43Z |
|
updated |
Timestamp |
2020-07-04T23:21:22Z |
|
Credential Attributes Table¶
Attribute |
Type |
Example |
Notes |
---|---|---|---|
user |
String |
jsmith |
|
authnMethod |
String |
PKI_KEYS |
|
loginUser |
String |
|
|
password |
String |
|
|
privateKey |
String |
|
|
publicKey |
String |
|
|
accessKey |
String |
|
|
accessSecret |
String |
|
|
accessToken |
String |
|
|
refreshToken |
String |
|
KeyValuePair Attributes Table¶
Attribute |
Type |
Example |
Notes |
---|---|---|---|
key |
String |
“INPUT_FILE” |
|
value |
String |
“/tmp/file.input” |
|
description |
String |
|
|
inputMode |
enum |
REQUIRED |
|
notes |
String |
“{}” |
|
LogicalQueue Attributes Table¶
Attribute |
Type |
Example |
Notes |
---|---|---|---|
name |
String |
tapisNormal |
|
hpcQueueName |
String |
normal |
|
maxJobs |
int |
|
|
maxJobsPerUser |
int |
|
|
minNodeCount |
int |
|
|
maxNodeCount |
int |
|
|
minCoresPerNode |
int |
|
|
maxCoresPerNode |
int |
|
|
minMemoryMB |
int |
|
|
maxMemoryMB |
int |
|
|
minMinutes |
int |
|
|
maxMinutes |
int |
|
Searching¶
The service provides a way for users to search for systems based on a list of search conditions provided either as query parameters for a GET call or a list of conditions in a request body for a POST call to a dedicated search endpoint.
Search using GET¶
To search when using a GET request to the systems
endpoint a list of search conditions may be specified
using a query parameter named search
. Each search condition must be surrounded with parentheses, have three parts
separated by the character .
and be joined using the character ~
.
All conditions are combined using logical AND. The general form for specifying the query parameter is as follows:
?search=(<attribute_1>.<op_1>.<value_1>)~(<attribute_2>.<op_2>.<value_2>)~ ... ~(<attribute_N>.<op_N>.<value_N>)
Attribute names are given in the table above and may be specified using Camel Case or Snake Case.
Supported operators: eq
neq
gt
gte
lt
lte
in
nin
like
nlike
between
nbetween
Example CURL command to search for systems that have Test
in the id, are of type LINUX,
are using a port less than 1024
and have a default authentication method of either PKI_KEYS
or PASSWORD
:
$ curl -H "X-Tapis-Token: $JWT" https://tacc.tapis.io/v3/systems?search="(id.like.*Test*)~(system_type.eq.LINUX)~(port.lt.1024)~(DefaultAuthnMethod.in.PKI_KEYS,PASSWORD)"
Notes:
For the
like
andnlike
operators the wildcard character*
matches zero or more characters and!
matches exactly one character.For the
between
andnbetween
operators the value must be a two item comma separated list of unquoted values.If there is only one condition the surrounding parentheses are optional.
In a shell environment the character
&
separating query parameters must be escaped with a backslash.In a shell environment the query value must be surrounded by double quotes and the following characters must be escaped with a backslash in order to be properly interpreted by the shell:
"
\
`
Attribute names may be specified using Camel Case or Snake Case.
Following complex attributes not supported when searching:
authnCredential
jobRuntimes
jobEnvVariables
batchLogicalQueues
notes
Dedicated Search Endpoint¶
The service provides the dedicated search endpoint systems/search/systems
for specifying complex queries. Using a GET
request to this endpoint provides functionality similar to above but with a different syntax. For more complex
queries a POST request may be used with a request body specifying the search conditions using an SQL-like syntax.
Search using GET on Dedicated Endpoint¶
Sending a GET request to the search endpoint provides functionality very similar to that provided for the endpoint
systems
described above. A list of search conditions may be specified using a series of query parameters, one for each attribute.
All conditions are combined using logical AND. The general form for specifying the query parameters is as follows:
?<attribute_1>.<op_1>=<value_1>&<attribute_2>.<op_2>=<value_2>)& ... &<attribute_N>.<op_N>=<value_N>
Attribute names are given in the table above and may be specified using Camel Case or Snake Case.
Supported operators: eq
neq
gt
gte
lt
lte
in
nin
like
nlike
between
nbetween
Example CURL command to search for systems that have Test
in the name, are of type LINUX
,
are using a port less than 1024
and have a default authentication method of either PKI_KEYS
or PASSWORD
:
$ curl -H "X-Tapis-Token: $JWT" https://tacc.tapis.io/v3/systems/search/systems?name.like=*Test*\&enabled.eq=true\&system_type.eq=LINUX\&DefaultAuthnMethod.in=PKI_KEYS,PASSWORD
Notes:
For the
like
andnlike
operators the wildcard character*
matches zero or more characters and!
matches exactly one character.For the
between
andnbetween
operators the value must be a two item comma separated list of unquoted values.In a shell environment the character
&
separating query parameters must be escaped with a backslash.Attribute names may be specified using Camel Case or Snake Case.
Following complex attributes not supported when searching:
authnCredential
jobRuntimes
jobEnvVariables
batchLogicalQueues
tags
notes
Search using POST on Dedicated Endpoint¶
More complex search queries are supported when sending a POST request to the endpoint systems/search/systems
.
For these requests the request body must contain json with a top level property name of search
. The
search
property must contain an array of strings specifying the search criteria in
an SQL-like syntax. The array of strings are concatenated to form the full search query.
The full query must be in the form of an SQL-like WHERE
clause. Note that not all SQL features are supported.
For example, to search for systems that are owned by jdoe
and of type LINUX
or owned by
jsmith
and using a port less than 1024
create a local file named system_search.json
with following json:
{
"search":
[
"(owner = 'jdoe' AND system_type = 'LINUX') OR",
"(owner = 'jsmith' AND port < 1024)"
]
}
To execute the search use a CURL command similar to the following:
$ curl -X POST -H "content-type: application/json" -H "X-Tapis-Token: $JWT" https://tacc.tapis.io/v3/systems/search/systems -d @system_search.json
Notes:
String values must be surrounded by single quotes.
Values for BETWEEN must be surrounded by single quotes.
Search query parameters as described above may not be used in conjunction with a POST request.
SQL features not supported include:
IS NULL
andIS NOT NULL
Arithmetic operations
Unary operators
Specifying escape character for
LIKE
operator
Map of SQL operators to Tapis operators¶
Sql Operator |
Tapis Operator |
---|---|
= |
eq |
<> |
neq |
< |
lt |
<= |
lte |
> |
gt |
>= |
gte |
LIKE |
like |
NOT LIKE |
nlike |
BETWEEN |
between |
NOT BETWEEN |
nbetween |
IN |
in |
NOT IN |
nin |
Sort, Limit, Select and ListType¶
When a list of Systems is retrieved the service provides for sorting, filtering and limiting the results. By default, only resources owned by you will be included. The service provides a way for you to request that all resources accessible to you be included. This is determined by the query parameter listType.
When retrieving either a list of resources or a single resource the service also provides a way to select which fields (i.e. attributes) are included in the results. Sorting, limiting and attribute selection are supported using query parameters.
Selecting¶
When retrieving systems the fields (i.e. attributes) to be returned may be specified as a comma separated list using
a query parameter named select
. Attribute names may be given using Camel Case or Snake Case.
Notes:
Special select keywords are supported:
allAttributes
andsummaryAttributes
Summary attributes include:
id
,systemType
,owner
,host
,effectiveUserId
,defaultAuthnMethod
,canExec
By default all attributes are returned when retrieving a single resource via the endpoint systems/<system_id>.
By default summary attributes are returned when retrieving a list of systems.
Specifying nested attributes is not supported.
The attribute
id
is always returned.
For example, to return only the attributes host
and effectiveUserId
the
CURL command would look like this:
$ curl -H "X-Tapis-Token: $JWT" https://tacc.tapis.io/v3/systems?select=host,effectiveUserId
The response should look similar to the following:
{
"result": [
{
"id": "CSys_CltSrchGet_011",
"host": "hostCltSrchGet_011",
"effectiveUserId": "effUserCltSrchGet_011"
},
{
"id": "CSys_CltSrchGet_012",
"host": "hostCltSrchGet_012",
"effectiveUserId": "effUserCltSrchGet_012"
},
{
"id": "CSys_CltSrchGet_013",
"host": "hostCltSrchGet_013",
"effectiveUserId": "effUserCltSrchGet_013"
}
],
"status": "success",
"message": "TAPIS_FOUND Systems found: 12 systems",
"version": "1.0.0",
"metadata": {
"recordCount": 3,
"recordLimit": 100,
"recordsSkipped": 0,
"orderBy": null,
"startAfter": null,
"totalCount": -1
}
}
Sorting¶
The query parameter for sorting is named orderBy
and the value is the attribute name to sort on with an optional
sort direction. The general format is <attribute_name>(<dir>)
. The direction may be asc
for ascending or
desc
for descending. The default direction is ascending.
Examples:
orderBy=id
orderBy=id(asc)
orderBy=name(desc),created
orderBy=id(asc),created(desc)
Limiting¶
Additional query parameters may be used in order to limit the number and starting point for results. This is useful for implementing paging. The query parameters are:
limit
- Limit number of items returned. For example limit=10.
Use 0 or less for unlimited.
Default is 100.
skip
- Number of items to skip. For example skip=10.
May not be used with startAfter.
Default is 0.
startAfter
- Where to start when sorting. For example limit=10&orderBy=id(asc),created(desc)&startAfter=101
May not be used with
skip
.Must also specify
orderBy
.The value of
startAfter
applies to the majororderBy
field.Condition is context dependent. For ascending the condition is value >
startAfter
and for descending the condition is value <startAfter
.
When implementing paging it is recommend to always use orderBy
and when possible use limit+startAfter
rather
than limit+skip
. Sorting should always be included since returned results are not guaranteed to be in the same order
for each call. The combination of limit+startAfter
is preferred because limit+skip
is more likely to result in
inconsistent results as records are added and removed. Using limit+startAfter
works best when the attribute has a
natural sequential ordering such as when an attribute represents a timestamp or a sequential ID.
ListType¶
By default, you will only see the resources that you own. The query parameter listType allows you to see additional resources that are available to you.
Options:
- OWNED
Include only items owned by you (Default)
- SHARED_PUBLIC
Include only items shared publicly
- ALL
Include all items you are authorized to view.
Tapis Responses¶
For requests that return a list of resources the response result object will contain the list of resource records that match the user’s query and the response metadata object will contain information related to sorting and limiting.
The metadata object will contain the following information:
recordCount
- Actual number of records returned.
recordLimit
- The limit query parameter specified in the request. -1 if query parameter was not specified.
recordsSkipped
- The skip query parameter specified in the request. -1 if query parameter was not specified.
orderBy
- The orderBy query parameter specified in the request. Empty string if query parameter was not specified.
startAfter
- The startAfter query parameter specified in the request. Empty string if query parameter was not specified.
totalCount
- Total number of records that would have been returned without a limit query parameter being imposed. -1 if total count was not computed.
For performance reasons computation of totalCount
is only determined on demand. This is controlled by the boolean
query parameter computeTotal
. By default computeTotal
is false.
Example query and response:
Query:
$ curl -H "X-Tapis-Token: $JWT" https://tacc.tapis.io/v3/systems?limit=2&orderBy=id(desc)
Response:
{
"result": [
{
"id": "testMin0",
"systemType": "S3",
"owner": "testuser",
"host": "my.example.host",
"defaultAccessMethod": "ACCESS_KEY",
"canExec": false
},
{
"id": "MinSystem1c",
"systemType": "LINUX",
"owner": "testuser",
"defaultAccessMethod": "PASSWORD",
"host": "data.tacc.utexas.edu",
"canExec": true
}
],
"status": "success",
"message": "TAPIS_FOUND Systems found: 2 systems",
"version": "1.0.0",
"metadata": {
"recordCount": 2,
"recordLimit": 2,
"recordsSkipped": 0,
"orderBy": "id(desc)",
"startAfter": null,
"totalCount": -1
}