Meta

Meta V3 is a REST API Microservice for MongoDB which provides server-side Data, Identity and Access Management for Web and Mobile applications.

Why Meta V3

Meta V2 functionality was built on MongoDB version 1.0 technology. Although limited, it brought the basic functions of a document store to Agave/Tapis platform. Some projects decided to use basic key/value storage to add metadata to Jobs, Apps, Systems and Files. Other projects stored complex document data to associate Jobs and other entities to add a richer information layer to their portals. The limited search functionality and imposed document structure created impediments to using MongoDB as projects had envisioned. Meta V3 removes these impediments.

Meta V3 is built on top of MongoDB version 4.2 technology. The REST API opens up the full functionality of MongoDB as a document store. and delivers MongoDB as a service so that projects are free to create metadata and documents in a fashion that fits their needs. Some of the Meta V3 advantages over Meta V2 include:

• Any valid MongoDB document structure can be used.

• If a search runs in MongoDB CLI, it should run from the API.

• Aggregations are available.

• Database, collection and document creation can be managed by tenant administrator.

• Performance is many times faster.

Migration from Meta V2 to V3

Migration can simply be accomplished by creating a new database with one or more collections for your project. The tenant administrator can request the initial permissions setup. Once your collection(s) are in place move the result and associatedIds into your new document model.

Overview

Meta V3 is:

A Stateless Microservice. With Meta V3 projects can focus on building Angular or other frontend applications, because most of the server-side logic necessary to manage database operations, authentication / authorization and related APIs is automatically handled, without the need to write any server-side code except for the UX/UI.

For example, to insert data into MongoDB a developer has to just create client-side JSON documents and then execute POST operations via HTTP to Meta V3. Other functions of a modern MongoDB installation like flexible schema, geoJson and aggregation pipelines ease the development process.

Every tenant will have access to at least one database where they can store and manage json documents. Documents are the trailing end of a nested hierarchy of data that begins with a database that houses one or more collections. The collections house json documents the structure of which is left up to the administrators of the tenant database.

Permissions for access to databases, collections and documents must be predefined before accessing those resources. The definitions for access are defined within the Security Kernel API of Tapis V3 and must be added by a tenant or service administrator. See the Permissions section below for some examples of permissions definitions and access to resources in the Meta V3 API.

Getting Started

Create a document

We have a database named MyTstDB and a collection name MyCollection. To add a json document to MyCollection, we can do the following:

With CURL:

$ curl -v -X POST -H "Content-Type:application/json"  -H "X-Tapis-Token:$jwt" --data '{"name": "test document slt 7.21.2020-14:27","jimmyList": ["1","3"],"description": "new whatever",}'  $BASE_URL/v3/meta/MyTstDB/MyCollection?basic=true

The response will have an empty response body with a status code of 201 “Created” unless the “basic” url query parameter is set to true. Setting the “basic” parameter to true will give a Tapis Basic response along with the “_id” of the newly created document. A more detailed discussion of autogenerated ids and specified ids can be found in the “Create Document” section of “Document Resources”.

{
  "result": {
    "_id": "5f189316e37f7b5a692285f3"
  },
  "status": "201",
  "message": "Created",
  "version": "0.0.1"
}

List documents

Using our MyTstDb/MyCollection resources we can ask for a default list of documents in MongoDB default sorted order. The document we created earlier should be listed with a new “_id” field that was autogenerated by MongoDB.

With CURL:

$ curl -v -X GET -H "Content-Type:application/json"  -H "X-Tapis-Token:$jwt" $BASE_URL/v3/meta/MyTstDB/MyCollection

The response will be an array of json documents from MyCollection :

[
  {
    "_id": {
      "$oid": "5f189316e37f7b5a692285f3"
    },
    "name": "test document slt 7.21.2020-14:27",
    "jimmyList": [
      "1",
      "3"
    ],
    "description": "new whatever",
    "_etag": {
      "$oid": "5f189316296c81742a6a3e4c"
    }
  },
  {
    "_id": {
      "$oid": "5f1892ece37f7b5a692285e9"
    },
    "name": "test document slt 7.21.2020-14:25",
    "jimmyList": [
      "1",
      "3"
    ],
    "description": "new whatever",
    "_etag": {
      "$oid": "5f1892ec296c81742a6a3e4b"
    }
  }
]

Get a document

If we know the “_id” of a created document, we can ask for it directly.

With CURL:

$ curl -v -X GET -H "Content-Type:application/json"  -H "X-Tapis-Token:$jwt" $BASE_URL/v3/meta/MyTstDB/MyCollection/5f1892ece37f7b5a692285e9

The response will be a json document from MyCollection with the “_id” of 5f1892ece37f7b5a692285e9 :

{
  "_id": {
    "$oid": "5f1892ece37f7b5a692285e9"
  },
  "name": "test document slt 7.21.2020-14:25",
  "jimmyList": [
    "1",
    "3"
  ],
  "description": "new whatever",
  "_etag": {
    "$oid": "5f1892ec296c81742a6a3e4b"
  }
}

Find a document

We can pass a query parameter named “filter” and set the value to a json MongoDB query document. Let’s find a document by a specific “name”.

With CURL:

$ curl -v -X GET -H "Content-Type:application/json" -H "X-Tapis-Token:$jwt" --data-urlencode filter='{"name": "test document slt 7.21.2020-14:25"}' $BASE_URL/v3/meta/MyTstDB/MyCollection

The response will be an array of json documents from MyCollection :

[
  {
    "_id": {
      "$oid": "5f1892ece37f7b5a692285e9"
    },
    "name": "test document slt 7.21.2020-14:25",
    "jimmyList": [
      "1",
      "3"
    ],
    "description": "new whatever",
    "_etag": {
      "$oid": "5f1892ec296c81742a6a3e4b"
    }
  }
]

Resources

General resources

An unauthenticated Health check is in included in the Meta V3 API to let any user know the current condition of the service.

Health Check

An unauthenticated request for the health status of Meta V3 API.

With pySDK operation:

$ t.meta.healthCheck()

With CURL:

$ curl -v -X GET -H "Content-Type:application/json" $BASE_URL/v3/meta/

The response will be a Basic Tapis response on health:

{
  "result": "",
  "status": "200",
  "message": "OK",
  "version": "0.0.1"
}

Root resources

The Root resource space represents the root namespace for databases on the MongoDB host. All databases are located here. Requests to this space are limited to READ only for tenant administrators.

List DB Names

A request to the Root resource will list Database names found on the server. This request has been limited to those users with tenant administrative roles.

With pySDK operation:

$ t.meta.listDBNames()

With CURL:

$ curl -v -X GET -H "Content-Type:application/json"  -H "X-Tapis-Token:$jwt" $BASE_URL/v3/meta/

The response will a json list of database names:

[
    "StreamsDevDB",
    "v1airr"
]

Database resources

The Database resource is the top level for many tenant projects. The resource maps directly to a MongoDB named database in the database server. Case matters for matching the name of the database and must be specified when making requests for collections or documents. Currently

List Collection Names

This request will return a list of collection names from the specified database {db}. The permissions for access to the database are set prior to access.

With pySDK operation:

$ t.meta.listCollectionNames(db='')

With CURL:

$ curl -v -X GET -H "Content-Type:application/json"  -H "X-Tapis-Token:$jwt" -d '' $BASE_URL/v3/meta/{db}

Here is an example response:

[
  "streams_alerts_metadata",
  "streams_channel_metadata",
  "streams_instrument_index",
  "streams_project_metadata",
  "streams_templates_metadata",
  "tapisKapa-local"
]

Get DB Metadata

This request will return the metadata properties associated with the database. The core server generates an etag in the _properties collection for a database that is necessary for future deletion.

With pySDK operation:

$ t.meta.getDBMetadata(db='')

With CURL:

$ curl -v -X GET -H "Content-Type:application/json"  -H "X-Tapis-Token:$jwt" -d '' $BASE_URL/v3/meta/{db}/_meta

Here is an example response:

{
   "_id": "_meta",
   "_etag": { "$oid": "5ef6232b296c81742a6a3e02" }
}

Create DB

TODO: this implementation is not exposed. Creation of a database by tenant administrators is scheduled for inclusion in an administrative interface API in a future release.

This request will create a new named database in the MongoDB root space by a tenant or service administrator.

With pySDK operation:

$ t.meta.createDB(db='')

With CURL:

$ curl -v -X PUT -H "Content-Type:application/json"  -H "X-Tapis-Token:$jwt" -d '' $BASE_URL/v3/meta/{db}

Here is an example response:

Status: 200 ok

Delete a DB TODO: this implementation is not exposed. Deletion of a database by tenant administrators is scheduled for inclusion in an administrative interface API in a future release.

This request will delete a named database in the MongoDB root space by a tenant or service administrator.

With pySDK operation:

$ t.meta.deleteDB(db='')

With CURL:

$ curl -v -X DELETE -H "Content-Type:application/json"  -H "X-Tapis-Token:$jwt" -d '' $BASE_URL/v3/meta/{db}

Here is an example response:

Status: 204 Deleted DB

Collection Resources

The Collection resource allows requests for managing and querying json documents within a MongoDB collection.

Create a Collection

You can create a new collection of documents by specifying a collection name under a specific database. /v3/meta/{db}/{collection}

With pySDK operation:

$ t.meta.createCollection(db='',collection='')

With CURL:

$ curl -v -X PUT -H "Content-Type:application/json"  -H "X-Tapis-Token:$jwt" $BASE_URL/v3/meta/{db}/{collection}

Here is an example response:

Status: 201 Collection created. An Etag header value is returned for collection identification

List Documents

A default number of documents found in the collection are returned in an array of documents.

With pySDK operation:

$ t.meta.listDocuments(db='',collection='',filter='')

With CURL:

$ curl -v -X GET -H "Content-Type:application/json"  -H "X-Tapis-Token:$jwt" -d '' $BASE_URL/v3/meta/{db}/{collection}

The response will look like the following:

[
  {
    "_id": {
      "$oid": "5f1892ece37f7b5a692285e9"
    },
    "name": "test document slt 7.21.2020-14:25",
    "description": "new whatever",
    "_etag": {
      "$oid": "5f1892ec296c81742a6a3e4b"
    }
  },
  {
    "_id": {
      "$oid": "5f1892ece37f7b5a69228533"
    },
    "name": "test document slt 7.21.2020-14:25",
    "description": "new whatever",
    "_etag": {
      "$oid": "5f1892ec296c81742a6a3e444"
    }
  }
]

List Documents Large Query

A default number of documents found in the collection are returned in an array of documents.

With pySDK operation:

$ t.meta.submitLargeQuery(db='',collection='',page='',pagesize='',sort='',keys='',fileinput)

With CURL:

$ curl -v -X GET -H "Content-Type:application/json"  -H "X-Tapis-Token:$jwt" -d @FILENAME '' $BASE_URL/v3/meta/{db}/{collection}/_filter

The response will look like the following:

[
  {
    "_id": {
      "$oid": "5f1892ece37f7b5a692285e9"
    },
    "name": "test document slt 7.21.2020-14:25",
    "description": "new whatever",
    "_etag": {
      "$oid": "5f1892ec296c81742a6a3e4b"
    }
  },
  {
    "_id": {
      "$oid": "5f1892ece37f7b5a69228533"
    },
    "name": "test document slt 7.21.2020-14:25",
    "description": "new whatever",
    "_etag": {
      "$oid": "5f1892ec296c81742a6a3e444"
    }
  }
]

Delete a Collection

This administrative method is only available to tenant or meta administrators and requires an If-Match header parameter of the Etag for the collection. The Etag value, if not already known, can be retrieved from the “_meta” call for a collection.

With pySDK operation:

$ t.meta.deleteCollection(db='',collection='')

With CURL:

$ curl -v -X DELETE -H "Content-Type:application/json"  -H "X-Tapis-Token:$jwt" $BASE_URL/v3/meta/{db}/{collection}

Here is an example response:

Status: 204 Deleted named collection from the database

Get Collection Size

You can find the given size or number of documents in a given collection by calling “_size” on a collection.

With pySDK operation:

$ t.meta.getCollectionSize(db='',collection=' ')

With CURL:

$ curl -v -X GET -H "Content-Type:application/json"  -H "X-Tapis-Token:$jwt" $BASE_URL/v3/meta/{db}/{collection}/_size

Here is an example response:

Status: 200 ok

{
  "_size": 1
}

Get Collection Metadata

You can find the metadata properties of a given collection by calling “_meta” on a collection. This would include the Etag value for a collection that is needed for deletion.

With pySDK operation:

$ t.meta.getCollectionMetadata(db='',collection=' ')

With CURL:

$ curl -v -X GET -H "Content-Type:application/json"  -H "X-Tapis-Token:$jwt" $BASE_URL/v3/meta/{db}/{collection}/_meta

Here is an example response:

{
  "_id": "_meta",
  "_etag": {
    "$oid": "5f2b2b7a204ce7637579c85f"
  }
}

Document Resources

Document resources are json documents found in a collection. Reading, creating, deleting and updating documents along with batch processing make up the operations that can be applied to documents in a collection. There various ways to retrieve one or more documents from a collection, including using a filter query parameter and value in the form of a MongoDB query document. Batch addition of documents, as well as, batch updates based on queries is also allowed.

Create a Document

Creating a new document within a collection. Submitting a json document within the request body of a POST request will create a new document within the specified collection with a MongoDB autogenerated “_id”. Batch document addition is possible by POSTing an array of new documents with a request body for the specified collection. The rules for “_id” creation operates the same way on multiple documents as they do with a single document.

The default representation returned is an empty response body along with a 201 Http status code “Created”. However if an additional query parameter named “basic” is added with the value of “true” a basic Tapis response is returned along with the newly created “_id” of the document.

With pySDK operation:

$ t.meta.createDocument(db='',collection=' ',basic=' ',request_body=' ')

With CURL:

$ curl -v -X POST -H "Content-Type:application/json"  -H "X-Tapis-Token:$jwt" -d '{"docName":"test doc"}' $BASE_URL/v3/meta/{db}/{collection}

Here is an example response:

Status: 201 Document created

Multiple documents can be added to a collection by POSTing a json array of documents. The batch addition of documents only supports the default response.

With pySDK operation:

$ t.meta.createDocument(db='',collection=' ',basic=' ', request_body=' ')

With CURL:

$ curl -v -X POST -H "Content-Type:application/json"  -H "X-Tapis-Token:$jwt" -d '[{"docName":"test doc1"},{"docName":"test doc2"}]' $BASE_URL/v3/meta/{db}/{collection}

The response body will be empty:

Status: 201 Document created

Get a Document

Get a specific document by its “_id”.

With pySDK operation:

$ t.meta.getDocument(db='',collection=' ',docId=' ')

With CURL:

$ curl -v -X GET -H "Content-Type:application/json"  -H "X-Tapis-Token:$jwt"  $BASE_URL/v3/meta/{db}/{collection}/{document_id}

The response will be the standard json response:

Status: 200 Document returned
{
    "_id"}

Replace a Document

This call replaces an existing document identified by document id (“_id”), with the json supplied in the request body.

With pySDK operation:

$ t.meta.replaceDocument(db='',collection=' ',docId=' ')

With CURL:

$ curl -v -X PUT -H "Content-Type:application/json"  -H "X-Tapis-Token:$jwt" -d '{"docName":"test doc another one"}' $BASE_URL/v3/meta/{db}/{collection}/{document_id}

Here is an example response:

Status: 201 document upserted

Modify a Document

This call will replace a portion of a document identified by document id (“_id”) with the supplied json.

With pySDK operation:

$ t.meta.modifyDocument(db='',collection=' ',docId=' ')

With CURL:

$ curl -v -X PATCH -H "Content-Type:application/json"  -H "X-Tapis-Token:$jwt" -d '{"docName":"test changed"}' $BASE_URL/v3/meta/{db}/{collection}/{document_id}

Here is an example response:

Status: 201 document modified

Delete Document

Deleting a document with a specific document id (“_id”), removes it from the collection.

With pySDK operation:

$ t.meta.deleteDocument(db='',collection=' ',docId=' ')

With CURL:

$ curl -v -X DELETE -H "Content-Type:application/json"  -H "X-Tapis-Token:$jwt" -d '' $BASE_URL/v3/meta/{db}/{collection}/{document_id}

Here is an example response:

Status: 204 Document deleted

Index Resources

Indexes can help speed up queries of your collection and the API gives you the ability to define and manage your indexes. You can create an index for a collection, list indexes for a collection and delete an index. Indexes can’t be updated they must be deleted and recreated.

List Indexes

List the indexes defined for a collection.

With pySDK operation:

$ t.meta.listIndexes(db='database_name',collection='collection_name')

With CURL:

$ curl -v -X POST -H "Content-Type:application/json"  -H "X-Tapis-Token:$jwt"  $BASE_URL/v3/meta/{db}/{collection}/_indexes

Here is an example response:

[
  {
    "v":2,
    "key":{
      "id":1
    }
    "_id":"_id_"
  }
]

Create Index

Create a new Index with a new name. To create an index you have to specify the keys and the index options. Let’s create an unique, sparse index on property qty and name our index “qtyIndex”.

PUT /v3/meta/{db}/{collection}/_indexes/qtyIndex

{"keys": {"qty": 1},"ops": {"unique": true, "sparse": true }}

With pySDK operation:

$ t.meta.createIndex(db='jupyterhub_v3_metadata', collection='jupyterhub_dev', request_body={request_body}, indexName='test'

With CURL:

$ curl -v -X PUT -H "Content-Type:application/json"  -H "X-Tapis-Token:$jwt" -d '{ "keys":  <keys>, "ops": <options> }' $BASE_URL/v3/meta/{db}/{collection}/_indexes/{indexName}

Here is an example response:

Status: 201 Empty Response

Delete Index

Remove a named Index from the index list.

With pySDK operation:

$ t.meta.deleteIndex(db='jupyterhub_v3_metadata', collection='jupyterhub_dev', indexName='test1')

With CURL:

$ curl -v -X DELETE -H "Content-Type:application/json"  -H "X-Tapis-Token:$jwt" $BASE_URL/v3/meta/{db}/{collection}/_indexes/{indexName}

Here is an example response:

Status: 204 Deleted index from collection

Aggregation Resources

Aggregations operations process data records and return computed results. Aggregation operations group values from multiple documents together, and can perform a variety of operations on the grouped data to return a single result. Aggregations in the API are predefined and added to a collections properties. They may also be parameterized for use with multiple sets of inputs.

Create an Aggregation

Create an aggregation pipeline by adding the aggregation to the collection for future execution. The aggregation may have variables that are defined so that a future request may pass variable values for aggregation execution. See “Execute an Aggregation”.

{ "aggrs" : [
      { "stages" : [ { "$match" : { "name" : { "$var" : "n" } } },
            { "$group" : { "_id" : "$name",
                  "avg_age" : { "$avg" : "$age" }
                } }
          ],
        "type" : "pipeline",
        "uri" : "example-pipeline"
      }
    ]
}

Property	Mandatory	Description
type	yes	for aggregation pipeline operations is “pipeline”
uri	yes	specifies the URI when the operation is bound under the path /<db>/<collection>/_aggrs.
stages	yes	the MongoDB aggregation pipeline stages.

For more information refer to https://docs.mongodb.org/manual/core/aggregation-pipeline/

With pySDK operation:

$ t.meta  To Appear in Future Release

With CURL:

$ curl -v -X PUT -H "Content-Type:application/json"  -H "X-Tapis-Token:$jwt"
     -d '{ "aggrs" : [{ "stages" : [ { "$match" : { "name" : { "$var" : "n" } } },{ "$group" : { "_id" : "$name","avg_age" : { "$avg" : "$age" }} } ],
        "type" : "pipeline","uri" : "example-pipeline"}]}' $BASE_URL/v3/meta/{db}/{collection}

Here is an example response:

Status: 201 empty response body on created

Execute an Aggregation

TODO

With pySDK operation:

$ t.meta  To Appear in Future Release

With CURL:

$ curl -v -X POST -H "Content-Type:application/json"  -H "X-Tapis-Token:$jwt" -d '' $BASE_URL/v3/meta/

Here is an example response:

TODO

Delete an Aggregation

TODO

With pySDK operation:

$ t.meta To Appear in Future Release

With CURL:

$ curl -v -X POST -H "Content-Type:application/json"  -H "X-Tapis-Token:$jwt" -d '' $BASE_URL/v3/meta/

Here is an example response:

Status: 204 Empty response body on delete