API

The API connector imports metadata from an OpenAPI Specification (formerly Swagger). Both OpenAPI 2.0 (Swagger) and OpenAPI 3.x specifications are supported, in either JSON or YAML format. The connector can either retrieve the specification by providing a URL to a published file, or by uploading a specification file manually (use for testing or if the file is not available through a URL). The specification is parsed and validated with the swagger-parser library, and YAML input is read with js-yaml.

The API connector is a primary connector.

The API connector currently supports import of the following metadata:

• the API itself as a datasource
• endpoints (one per path + HTTP method combination)
• request parameters, request body, and responses
• fields (including nested object/array properties) with datatypes, descriptions, nullability and default values

It does not support import of:

• relationships
• lineage

Configuration

The default OpenAPI specification filename can be configured in the Settings -> Connectors -> API section. The filename specified there is the filename the connector looks for when a URL without a specified filename is provided, like “api.example.com”.

Concept Mapping

Because OpenAPI uses its own vocabulary, the connector maps OpenAPI concepts onto Katalogue’s metadata model as follows:

Katalogue concept	OpenAPI concept
datasource_name	info.title
datasource_description	info.description
datasource_hostname (labelled API URL in the GUI)	servers[0].url (base URL)
dataset_group_name	operation tags[0], or the first non-parameter path segment when no tag is present
dataset_group_description	the matching root-level tags[].description (only when the group came from a tag)
dataset_name (with dataset type api_endpoint)	a path + HTTP method combination (one endpoint)
child dataset_name, named parameters	path and query parameters
child dataset_name, named request body	requestBody
child dataset_name, named response (<statusCode>) (e.g. response (200))	each entry in responses
api_method	the HTTP method (get, post, …)
dataset_description	operation summary and description, concatenated.
dataset_technical_definition	the raw operation object
field	a schema property
datatype_name	property type
datatype_length	property maxLength
field_description	property description, with any enum values appended as Enums: a, b, c
is_nullable	membership in the schema’s required array (a required property is not nullable)
default_value	property default

Each endpoint becomes a top-level dataset whose child datasets are its parameters, request body, and one response (<statusCode>) per defined response code. Object and array properties nest structurally, so a property of an object becomes a child field of that property.

Note

All methods (GET, POST, PUT, PATCH, DELETE etc), responses (200, 401, 500 etc), request body and request parameters are imported by default. Add a datasource filter in the Connection to limit imports, see below.

Filtering

The full specification is fetched first, then datasource filters are applied within Katalogue before import. See Datasource Filters.

In addition to the standard filter fields, the API connector also supports filtering on api_method — the endpoint’s HTTP method, matched lowercased (e.g. get, post). A filter on api_method matches the endpoint dataset and its entire subtree (parameters, request body and responses), so it keeps (or excludes) the whole endpoint.

The example below imports only the success responses (2xx category) of GET endpoints:

{
  "datasource_filter": [
    {
      "concatenator": "and",
      "filters": [
        {
          "field": "api_method",
          "operator": "equal",
          "value": "get"
        },
        {
          "field": "dataset_name",
          "operator": "equal",
          "value": "response (2%"
        }
      ]
    }
  ]
}

When a row matches, its ancestors are always kept so it retains its place in the hierarchy. In the example above, the matched response (200) datasets cause their parent endpoint datasets to be kept as well.

The endpoint’s child datasets can be filtered on by dataset_name. They are named parameters, request body, and response (<statusCode>) — parameters and request body have fixed names, whereas response datasets carry the status code (response (200), response (404), …). So the same technique used above for response (200) can equally be used to filter on the parameters and request body child datasets.

Limitations

Dataset Statistics

Dataset statistics such as row count and disk size is not synced, as these concepts are not relevant for API datasources.

Relationships

This connector does not provide field relationships, as this concept is not relevant for API datasources.

Lineage

This connector does not provide dataset lineage, as this concept is not relevant for API datasources.