Warning

You're reading a development version (main) of the documentation. For the latest released version, please see 1.0.0.

5. Guidance for Data Providers¶

Note

This documented is primarily intended for technical and operations teams within organisations wishing to participate in Open Energy as Data Providers.

Note

The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “MAY”, and “OPTIONAL” in this document are to be interpreted as described in RFC 2119.

Data Provider Role and Responsibilities
Data API Requirements
Problem Resolution (Data Providers)
Change Management

5.1. Data Provider Role and Responsibilities ¶

Data Providers are Open Energy members which make data sets available to:

Data Consumers within the Open Energy ecosystem
- Applies to shared data sets
- Data APIs secured using Financial-grade API security (FAPI)
Any third party, in the case of open data
- Open data sets are made available through publicly accessible APIs
- Data are made available under an appropriate open license agreement

In this context, Data APIs can be anything from a simple link to a CSV or other file (for open data) to a complex query system returning custom output on demand.

To support this role, Data Providers have certain responsibilities:

5.1.1. Responsibility - Data classification ¶

Data Providers are responsible for classification of their data sets by data sensitivity class, and for ensuring that data are shared, or not shared, as appropriate.

Any data classed as OE-C (Closed) or OE-SP (Personal) MUST NOT be shared within the Open Energy ecosystem at present.
Data in OE-O, OE-SA, and OE-SB MAY be shared within the Open Energy ecosystem as described in the following sections

5.1.2. Responsibility - Access and licensing specification ¶

Data Providers are responsible for creating access rules for each dataset shared via Open Energy. For each data set, the Data Provider must create a set of one or more access rules as defined in Data Set Metadata. These rules specify the conditions under which access will be granted, along with the capabilities forming the license grant should those conditions be satisfied.

Data Providers MUST ensure that access conditions and capability grants are non-discriminatory and fair to Data Consumers
Data Providers SHOULD ensure that the validity time bounds of any published capability grants are sufficient for reasonable commercial operation of a data consumer

5.1.3. Responsibility - Data provision ¶

Data Providers are responsible for providing a data API for each data set, and for configuring and securing each API in accordance with the kinds of data it exposes (along with any other access conditions deemed necessary and appropriate by the Data Provider on a per-data-set basis)

Open data MUST be made available with no authentication or authorization requirements
Shared data MUST made available through a FAPI compliant resource server connected to the OEGS’s authorization service (see Common Security Requirements for more details)

5.1.3.1. Data formats ¶

Open Energy does not define or mandate individual data formats to be returned from data APIs - the diversity of kinds of information in the sector renders this impractical, and the additional overhead imposed on data providers would be undesirable.

With this said, we would encourage Data Providers to consider likely automated consumption of data when designing their data APIs - this implies using machine readable formats such as CSV, JSON and similar, with a preference for those formats compatible with existing software tools and libraries.

5.1.4. Responsibility - Data integrity and correctness ¶

Data Providers are responsible for the correctness of data returned by their published data APIs. The OEGS will provide a mechanism by which Data Consumers can report any data quality issues, and will convey any such reports to a nominated contact within the Data Provider.

5.1.5. Responsibility - Metadata provision ¶

Data Providers are responsible for creating, and publishing, metadata for each exposed data set. This provides visibility of each data set within the Open Energy Governance Service (OEGS) Registry.

The metadata file covers, for each provided data set, whether shared or open:

Semantic content
Access rules and licensing of the data set
Transport mechanism specification
Syntactic content

The content and format of the metadata file can be found at Data Set Metadata.

The Data Provider is responsible for the accuracy and completeness of this metadata.

5.1.6. Responsibility - API availability ¶

Data Providers are responsible for availability of their published data APIs. Availability will be monitored automatically by the OEGS, availability information will form part of the metadata for each data set record in the OEGS Registry.

The OEGS will provide a mechanism for Data Providers to announce scheduled downtime when planned, and to report unscheduled downtime when necessary.

5.1.7. Responsibility - API stability and change management ¶

Data Providers are responsible for managing any change to the data APIs such that disruption to Data Consumers is minimised. This is handled through:

Semantic versioning of all API methods
Publication of anticipated changes and retirement of API versions through the OEGS
Changeover periods where new and prior API versions are run in parallel

5.2. Data API Requirements ¶

Unlike open banking, open energy does not mandate particular APIs for data provision - it is expected that there will be a variety of mechanisms to expose data, with varying inputs (from a single data set ID to a complex query object) and varying kinds of output dependent on the information exposed.

With that said, there are certain properties that all data APIs must satisfy to interoperate successfully with other parties within the open energy ecosystem.

We refer to endpoints used to retrieve energy data as data endpoints, and others as infrastructure endpoints.

5.2.1. Date and time formats ¶

Whenever date or time quantities are accepted or returned from a data API, these values MUST conform to |RFC| 3339. This is referenced elsewhere in this document as date/time

5.2.2. Endpoint security ¶

5.2.2.1. Open data endpoints ¶

Data endpoints which provide access to open data (in class OE-O) MUST NOT require any form of authentication prior to access. This includes allowing access to entities which are not members of the open energy ecosystem.

5.2.2.2. Shared data endpoints ¶

Data endpoints which provide access to shared data (in classes OE-SA and OE-SB) MUST implement the subset of the resource server part of the FAPI specification used within Open Energy as described in Common Security Requirements, and authorize against the OEGS authorization service.

Protected data endpoints MAY use the token introspection mechanism provided by FAPI to gather additional information about the client making the request prior to any access decision.

5.2.3. Heartbeat and monitoring endpoint ¶

All data APIs SHOULD include a FAPI protected heartbeat infrastructure endpoint. This serves two purposes:

It allows the OEGS to monitor the liveness of the data API
It provides some level of verification that the resource server has been correctly configured

The heartbeat endpoint location is defined within the data set metadata file, if specified it MUST respond to GET requests from the OEGS monitoring system with a 200 status code.

If the heartbeat response contains a body, the body will be interpreted by the OEGS monitoring system as a JSON dictionary containing statistics for the period between the most recent two successful calls to the heartbeat endpoint (including the call to which this is a response). This response is RECOMMENDED as it provides oversight of API usage to the OEGS.

Legal keys, and the semantics of their associated values, are as follows:

Name	Content
`period_start`	date/time of the start of the period for which statistics are reported
`period_end`	date/time of the end of the period for which statistics are reported, this will typically be the date and time of the heartbeat request
`api_call_response_[CODE]_count`	integer number of requests to non-heartbeat endpoints within this data API which resulted in a response of type CODE. A distinct key:value pair is sent in the response for each distinct HTTP status code returned.

5.2.4. API documentation ¶

Data APIs MUST be described within the transport section of the data set metadata file.

5.2.5. Versioning ¶

All data APIs MUST include a version number in the path of each data endpoint. This version SHOULD use semantic versioning to differentiate between breaking and backwards compatible changes. This MUST be documented within the Open|API| transport section of the metadata file.

5.3. Problem Resolution (Data Providers)¶

5.3.1. Effective resolution of problems (Data Providers)¶

A Data Provider MUST create documentation to clearly outline the policies, processes and systems it has in place for problem resolution and its respective service level objectives. This framework should enable the effective resolution of Data Consumer issues and therefore cater for problems that relate specifically to a Data Consumer’s use of a Data Provider’s data APIs. In the event that a Data Consumer is unable to resolve an issue with a Data Provider, the issue MAY be flagged to the OEGS Dispute Resolution function for independent support.

5.3.2. Online support ¶

Data Providers SHOULD provide Frequently Asked Questions, which address areas that may be specific to Data Consumers such as technical advice or test facility guidance. They should also consider a means of identifying recurring questions or user-error issues so these can be collated into Frequently Asked Questions to support the early resolution of problems.

Problem resolution documentation, Frequently Asked Questions, contact details, opening times and out of hours support SHOULD be published and easily accessible in one collective area on the Data Provider’s website.

5.3.3. Ticket management process ¶

Data Providers MUST ensure they have a functioning ticket management system which enables them to respond to issues and problems raised within clearly defined service level targets. A successful problem resolution framework will enable the efficient identification and resolution of problems which specifically impact Data Consumers. The system for raising and reporting on tickets should be transparent in order to fully inform users and engender trust across the ecosystem.

5.3.4. Dispute resolution for access control and capability grant policies (Data Providers)¶

In cases where a Data Consumer believes that access conditions or capability grant rules are being applied unfairly, the OEGS will act as a mediating party.

5.3.5. OEGS support (Data Providers)¶

The OEGS Service Desk will provide participants with a supplementary ticket management process and supports Data Providers in communicating problems to ecosystem participants via the noticeboard.

5.4. Change Management ¶

This section outlines various change scenarios that may impact Data Consumers, and provides guidance for a Data Provider to consider when implementing a change and communicating to Data Consumers.

5.4.1. Downtime ¶

Planned downtime, by definition, is something that a Data Provider anticipates and therefore is able to give advance notice to Data Consumers. It is not generally possible to give advance notice of unplanned downtime, but Data Providers should give notice as soon as they are aware of the downtime.

When providing notifications, Data Providers SHOULD provide a specific time period, so Data Consumers are aware that the data API will be unavailable for that time, or upon a subsequent notification to confirm that the service has been reinstated sooner than anticipated.

OEGS Support Services can assist Data Providers with the dissemination of downtime information to the wider Open Energy ecosystem via its central noticeboard facility.

Planned downtime should be given at least five business days before the event. Apart from cancelling the planned downtime, no changes should be made to the planned downtime notification within the five business day period. Where practical, Data Providers should give advance notice via their own website, developer portal or OEGS of any planned downtime one calendar month in advance.

5.4.2. Licensing and access control changes ¶

Data Providers MUST provide advance notice of any changes to access control and capability grant policies. This is normally covered by the time-bounded nature of these grants, but Data Providers MAY also use any notification services provided by the OEGS to publish additional information about changes.

5.4.3. Data API changes ¶

Data Providers may periodically wish to upgrade and / or to deprecate versions of their data APIs. The following mitigation measures will reduce the impact on service providers of these changes.

5.4.3.1. Dual running and deprecation ¶

Data Providers SHOULD support a minimum of two non-compatible API versions in a production context, providing both versions were previously supported by the Data Provider, for a period of time long enough to ensure that Data Consumers have had sufficient time to successfully test the new version and migrate their applications and customers. OEGS recommends dual running for at least six months for a major version, and three months for a minor version. Where a Data Provider implements a data API for the first time, they will only need to support this one version to start with.

The deprecation of unsupported versions is at the Data Provider’s discretion – based on usage metrics. However, the OEGS may recommend that any specific version (major, minor, or patch) should be deprecated at any time, and this should be implemented within three months of notification by the OEGS. This is to cater for critical defects, especially those relating to security. In exceptional circumstances it may be agreed by OEGS that support for a specific version is terminated earlier.

Data Providers must not apply any measures to induce Data Consumers to adopt a new version of the APIs (e.g. rate limiting the older version while providing better performance on a newer version).