-
Notifications
You must be signed in to change notification settings - Fork 31
Table Summary
This page summarise the different tables available through the nem-data tool. It gives both information on the contents of the tables, where the raw data can be found online and how you should expect the tables to behave when filtered by nem-data. The behaviour described here should be taken as the idealised behaviour, or how nem-data is intended to behave, if you observe something different congratulations! This is your opportunity to contribute, raise an issue and help make nem-data better.
Most tables available through nem-data are dynamic tables, this means they are tables that when queried will return different results depending on the start and end time provided i.e the contents of the tables change with respect to time.
Many of the dynamic tables are derived from tables in AEMO's Market Management System Data Model (MMSDM), the full summary of these tables is available in this pdf. The summary that follows is based partial on this document, but is also interpretive in nature as a precise definition of each table is not always provided. The names of the following sections aim to be as descriptive as possible, with the actual AEMO MMSDM name provided in parenthesis, note nem-data uses the MMSDM name.
The data contained in each table is briefly summarised by referring to columns they contain, for more detail on each column see the column summary.
Often the columns in these tables are a subset of the MMSDM table's columns, the full set can be accessed using lower level nem-data functions.
The following tables all contain the column SETTLEMENTDATE in the context of these tables this column defines the particular 5 min dispatch interval that a data record applies to, therefore when nem-data queries these tables only records when the SETTLEMENDATE timestamp is between the start (inclusive) and end time (exclusive) will be returned. THE EXCEPTION!!! Some tables do contain this column but do not conform to this generalisation. The table BIDPERDAY_D sets out the price bids for each generator, however there is only one record per day, rather than per 5 min interval, in this case nem-data will return all the bids that apply to the intervals between the start and end time rather filtering based on the actual values in the SETTLEMENTDATE column.
This table reports the results of the central dispatch process by 5 min interval and by dispatch unit.
Columns: SETTLEMENTDATE, DUID, INTERVENTION, DISPATCHMODE, AGCSTATUS, INITIALMW, TOTALCLEARED, RAMPDOWNRATE, RAMPUPRATE, LOWER5MIN, LOWER60SEC, LOWER6SEC, RAISE5MIN, RAISE60SEC, RAISE6SEC, LOWERREG, RAISEREG, SEMIDISPATCHCAP, AVAILABILITY.
This table reports the results of the central dispatch process by 5 min interval and by market region.
Columns: SETTLEMENTDATE, REGIONID, INTERVENTION, TOTALDEMAND, AVAILABLEGENERATION, AVAILABLELOAD, DEMANDFORECAST, DISPATCHABLEGENERATION, DISPATCHABLELOAD, NETINTERCHANGE, EXCESSGENERATION, LOWER5MINLOCALDISPATCH, LOWER60SECLOCALDISPATCH, LOWER6SECLOCALDISPATCH, RAISE5MINLOCALDISPATCH, RAISE60SECLOCALDISPATCH, RAISE6SECLOCALDISPATCH, LOWERREGLOCALDISPATCH, RAISEREGLOCALDISPATCH, INITIALSUPPLY, CLEAREDSUPPLY, TOTALINTERMITTENTGENERATION, DEMAND_AND_NONSCHEDGEN, UIGF, SEMISCHEDULE_CLEAREDMW, SEMISCHEDULE_COMPLIANCEMW.
This table reports the price of various market services by 5 min interval and by market region.
Columns: SETTLEMENTDATE, REGIONID, INTERVENTION, RRP, RAISE6SECRRP, RAISE60SECRRP, RAISE5MINRRP, RAISEREGRRP, LOWER6SECRRP, LOWER60SECRRP, LOWERREGRRP, PRICE_STATUS.
This table reports the scada values of scheduled, semi-scheduled and non-scheduled (where available) units where available.
Columns: SETTLEMENTDATE, DUID, SCADAVALUE.
This table reports generic constraints imposed on the dispatch process by 5 min interval, to the best of the authors knowledge it is the definitive information on which constraints were used in each dispatch interval. The definition of the constraints set out in this table can be found in the tables GENCONDATA, SPDREGIONCONSTRAINT, SPDCONNECTIONPOINTCONSTRAINT and SPDINTERCONNECTORCONSTRAINT. The mapping between tables is from CONSTRAINTID, GENCONID_EFFECTIVEDATE and GENCONID_VERSIONNO to GECONID, EFFECTIVEDATE and VERSIONNO.
Columns: SETTLEMENTDATE, RUNNO, CONSTRAINTID, INTERVENTION, RHS, MARGINALVALUE, VIOLATIONDEGREE, LASTCHANGED, GENCONID_EFFECTIVEDATE, GENCONID_VERSIONNO, LHS.
This table set out the capacity bids used in the central dispatch process. The records are by 5 min interval, by dispatch unit ID and by bid type. It also contains import constraining variables that generators submit along side there bids.
Columns: DUID, BANDAVAIL1, BANDAVAIL2, BANDAVAIL3, BANDAVAIL4, BANDAVAIL5,BANDAVAIL6, BANDAVAIL7, BANDAVAIL8, BANDAVAIL9, BANDAVAIL10, MAXAVAIL, RAMPUPRATE, RAMPDOWNRATE, BIDTYPE, SETTLEMENTDATE, ENABLEMENTMIN, ENABLEMENTMAX, LOWBREAKPOINT, HIGHBREAKPOINT, INTERVAL_DATETIME.
This table set out the price bids used in the central dispatch process. The records are by day, but apply to all dispatch intervals on that date.
Columns: SETTLEMENTDATE, DUID, BIDTYPE, OFFERDATE, VERSIONNO, PRICEBAND1, PRICEBAND2, PRICEBAND3, PRICEBAND4, PRICEBAND5, PRICEBAND6, PRICEBAND7, PRICEBAND8, PRICEBAND9, PRICEBAND10, T1, T2, T3, T4
The filtering by date for these tables occurs on the EFFECTIVEDATE column. The most recent effective dates applicable are returned, however if an effective date is between the start and end time, then multiple effective dates may be returned.
This table defines constraint matrix coefficients for regions, the absence of factor implies a zero coefficient, Note the table just defines constraints as they relate to regions, a full definition of any one constrain may require combining data from GENCONDATA, SPDREGIONCONSTRAINT, SPDINTERCONNECTORCONSTRAINT, SPDCONNECTIONPOINTCONSTRAINT and DISPATCHCONSTRAINT.
Columns: REGIONID, EFFECTIVEDATE, VERSIONNO, GENCONID, FACTOR, LASTCHANGED, BIDTYPE.
This table defines constraint matrix coefficients for connection points, the absence of factor implies a zero coefficient. Note the table just defines constraints as they relate to regions, a full definition of any one constrain may require combining data from GENCONDATA, SPDREGIONCONSTRAINT, SPDINTERCONNECTORCONSTRAINT, SPDCONNECTIONPOINTCONSTRAINT and DISPATCHCONSTRAINT.
Columns: CONNECTIONPOINTID, EFFECTIVEDATE, VERSIONNO, GENCONID, FACTOR, BIDTYPE, LASTCHANGED.
This table defines constraint matrix coefficients for interconnectors, the absence of factor implies a zero coefficient. Note the table just defines constraints as they relate to regions, a full definition of any one constrain may require combining data from GENCONDATA, SPDREGIONCONSTRAINT, SPDINTERCONNECTORCONSTRAINT, SPDCONNECTIONPOINTCONSTRAINT and DISPATCHCONSTRAINT.
Columns: INTERCONNECTORID, EFFECTIVEDATE, VERSIONNO, GENCONID, FACTOR, LASTCHANGED
This table defines general information about constraints applied to the central dispatch process. It maps to SPDREGIONCONSTRAINT, SPDCONNECTIONPOINTCONSTRAINTS and SPDINTERCONNECTORCONSTRAINT with the columns GECONID, EFFECTIVEDATE and VERSIONNO. Note the RHS values given in this table may not always be the ones used in dispatch, some RHS values are dynamically determined, for actual RHS values used in dispatch see the table DISPATCHCONSTRAINT.
Columns: GENCONID, EFFECTIVEDATE, VERSIONNO, CONSTRAINTTYPE, CONSTRAINTVALUE, DESCRIPTION, GENERICCONSTRAINTWEIGHT, LASTCHANGED, DISPATCH, PREDISPATCH, STPASA, MTPASA, LIMITTYPE, REASON
This table is used in the determination of recovery payments for regulation services, more information is available from AEMO here. This data maps to Elements_FCAS table by ELEMENTNUMBER column and the Variables_FCAS by VARAIBLENUMBER column. Note nem-data complies this table from current and historical data published by AEMO, however there is a 13 month gap in data availability between the current and historical data sets and this may result in nem-data failing to compile data from this table. If your local data cache is regularly update with AEMO's current data set then this would allow you to close the availability over a 13 month period.
Columns: TIMESTAMP, ELEMENTNUMBER, VARIABLENUMBER, VALUE, VALUEQUALITY
nem-data is an open source project and this means new tables can be added by anyone, these could be additional AEMO data sets, data sets from other sources or custom tables that represent a new synthesis of data from publicly available sources. This space is available for describing custom tables
While these tables may be periodically updated by AEMO, they do not contain a time based column and hence will not be filtered by any start and end time supplied. If you require the most up to date tables from AEMO delete or rename the corresponding tables from your local data cache and nem-data will download the most recent versions. Note the names provided in parenthesis below are nem-data names not AEMO names.
This table defines the element numbers used in the 4 second FCAS data. Note it appears AEMO has deleted the market names from this data set and no publicly available mapping of element numbers to market names appears to exist. More information is available from AEMO here
Columns: ELEMENTNUMBER, ELEMENTNAME, ELEMENTTYPE, NAME.
This table defines the variable numbers used in the 4 second FCAS data. More information is available from AEMO here
Columns: VARIABLENUMBER, VARIABLETYPE.
This table provides registration data for generators and loads in the NEM. It is derived from excel sheet provide by AEMO here. Note nem-data uses the Generators and Scheduled Loads tab, but drops duplicate entries for the same DUID values, this removes unit level information such as capacity, but enables the data set to more easily combined with other data sets.
Columns: Participant, Station Name, Region, Dispatch Type, Category, Classification, Fuel Source - Primary, Fuel Source - Descriptor, Technology Type - Primary, Technology Type - Descriptor, Aggregation, DUID.