You're reading a development version (main) of the documentation. For the latest released version, please see 1.0.0.
5. Guidance for Data Providers¶
This documented is primarily intended for technical and operations teams within organisations wishing to participate in Open Energy as Data Providers.
The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “MAY”, and “OPTIONAL” in this document are to be interpreted as described in RFC 2119.
Data Providers are Open Energy members which make data sets available to:
Data Consumers within the Open Energy ecosystem
Any third party, in the case of open data
Open data sets are made available through publicly accessible APIs
Data are made available under an appropriate open license agreement
To support this role, Data Providers have certain responsibilities:
Data Providers are responsible for classification of their data sets by data sensitivity class, and for ensuring that data are shared, or not shared, as appropriate.
Data Providers are responsible for creating access rules for each dataset shared via Open Energy. For each data set, the Data Provider must create a set of one or more access rules as defined in Data Set Metadata. These rules specify the conditions under which access will be granted, along with the capabilities forming the license grant should those conditions be satisfied.
Data Providers are responsible for providing a data API for each data set, and for configuring and securing each API in accordance with the kinds of data it exposes (along with any other access conditions deemed necessary and appropriate by the Data Provider on a per-data-set basis)
Open data MUST be made available with no authentication or authorization requirements
Open Energy does not define or mandate individual data formats to be returned from data APIs - the diversity of kinds of information in the sector renders this impractical, and the additional overhead imposed on data providers would be undesirable.
With this said, we would encourage Data Providers to consider likely automated consumption of data when designing their data APIs - this implies using machine readable formats such as CSV, JSON and similar, with a preference for those formats compatible with existing software tools and libraries.
Data Providers are responsible for the correctness of data returned by their published data APIs. The OEGS will provide a mechanism by which Data Consumers can report any data quality issues, and will convey any such reports to a nominated contact within the Data Provider.
The metadata file covers, for each provided data set, whether shared or open:
Access rules and licensing of the data set
Transport mechanism specification
The content and format of the metadata file can be found at Data Set Metadata.
The Data Provider is responsible for the accuracy and completeness of this metadata.
Data Providers are responsible for availability of their published data APIs. Availability will be monitored automatically by the OEGS, availability information will form part of the metadata for each data set record in the OEGS Registry.
Unlike open banking, open energy does not mandate particular APIs for data provision - it is expected that there will be a variety of mechanisms to expose data, with varying inputs (from a single data set ID to a complex query object) and varying kinds of output dependent on the information exposed.
With that said, there are certain properties that all data APIs must satisfy to interoperate successfully with other parties within the open energy ecosystem.
We refer to endpoints used to retrieve energy data as data endpoints, and others as infrastructure endpoints.
It provides some level of verification that the resource server has been correctly configured
The heartbeat endpoint location is defined within the data set metadata file, if specified it MUST respond to
GET requests from the OEGS monitoring system with a
200 status code.
If the heartbeat response contains a body, the body will be interpreted by the OEGS monitoring system as a JSON dictionary containing statistics for the period between the most recent two successful calls to the heartbeat endpoint (including the call to which this is a response). This response is RECOMMENDED as it provides oversight of API usage to the OEGS.
Legal keys, and the semantics of their associated values, are as follows:
date/time of the start of the period for which statistics are reported
date/time of the end of the period for which statistics are reported, this will typically be the date and time of the heartbeat request
integer number of requests to non-heartbeat endpoints within this data API which resulted in a response of type CODE. A distinct key:value pair is sent in the response for each distinct HTTP status code returned.
Data APIs MUST be described within the transport section of the data set metadata file.
A Data Provider MUST create documentation to clearly outline the policies, processes and systems it has in place for problem resolution and its respective service level objectives. This framework should enable the effective resolution of Data Consumer issues and therefore cater for problems that relate specifically to a Data Consumer’s use of a Data Provider’s data APIs. In the event that a Data Consumer is unable to resolve an issue with a Data Provider, the issue MAY be flagged to the OEGS Dispute Resolution function for independent support.
Data Providers SHOULD provide Frequently Asked Questions, which address areas that may be specific to Data Consumers such as technical advice or test facility guidance. They should also consider a means of identifying recurring questions or user-error issues so these can be collated into Frequently Asked Questions to support the early resolution of problems.
Problem resolution documentation, Frequently Asked Questions, contact details, opening times and out of hours support SHOULD be published and easily accessible in one collective area on the Data Provider’s website.
Data Providers MUST ensure they have a functioning ticket management system which enables them to respond to issues and problems raised within clearly defined service level targets. A successful problem resolution framework will enable the efficient identification and resolution of problems which specifically impact Data Consumers. The system for raising and reporting on tickets should be transparent in order to fully inform users and engender trust across the ecosystem.
Planned downtime, by definition, is something that a Data Provider anticipates and therefore is able to give advance notice to Data Consumers. It is not generally possible to give advance notice of unplanned downtime, but Data Providers should give notice as soon as they are aware of the downtime.
When providing notifications, Data Providers SHOULD provide a specific time period, so Data Consumers are aware that the data API will be unavailable for that time, or upon a subsequent notification to confirm that the service has been reinstated sooner than anticipated.
Planned downtime should be given at least five business days before the event. Apart from cancelling the planned downtime, no changes should be made to the planned downtime notification within the five business day period. Where practical, Data Providers should give advance notice via their own website, developer portal or OEGS of any planned downtime one calendar month in advance.
Data Providers MUST provide advance notice of any changes to access control and capability grant policies. This is normally covered by the time-bounded nature of these grants, but Data Providers MAY also use any notification services provided by the OEGS to publish additional information about changes.
Data Providers SHOULD support a minimum of two non-compatible API versions in a production context, providing both versions were previously supported by the Data Provider, for a period of time long enough to ensure that Data Consumers have had sufficient time to successfully test the new version and migrate their applications and customers. OEGS recommends dual running for at least six months for a major version, and three months for a minor version. Where a Data Provider implements a data API for the first time, they will only need to support this one version to start with.
The deprecation of unsupported versions is at the Data Provider’s discretion – based on usage metrics. However, the OEGS may recommend that any specific version (major, minor, or patch) should be deprecated at any time, and this should be implemented within three months of notification by the OEGS. This is to cater for critical defects, especially those relating to security. In exceptional circumstances it may be agreed by OEGS that support for a specific version is terminated earlier.