Product Update – RealTime Tools

Posted on by William Griswold

Good Morning,

We are pleased to announce a new service catered towards your real-time desk’s need for continuously updated information. Ansergy scrapes everything in the public domain that is pertinent to WECC power. However, the volume of data scraped precluded guaranteeing within the hour updates to those sources which updated hourly.

Solution: Dedicate a server to scrape just real-time updates (hourly or less) and retool those scrapes to capture the data faster. We can now complete a cycle of all hourly or faster sources in around 35 minutes which assures you that your real-time desk will be seeing the most recent updates to all WECC fundamentals.

Real Time Sources

  1. EIA “Open Data” Items
    • Demand – 27 Balancing Authorities, hourly; updated hourly
    • Demand Forecasts – 27 Balancing Authorities, hourly, updated daily
    • Net Generation – 35 Balancing Authorities, hourly, updated daily
    • Net Interchange – 35 Balancing Authorities, hourly, updated daily
    • BA to BA Interchange – 160 lines, hourly, updated daily
  2. Actual Weather – Weather Underground
    • Six weather metrics, updated hourly
  3. Weather Forecast – National Weather Service
    • 27 weather stations; updated four times a day (temperature, precip, cloud cover, wind speed, humidity)
  4. BPA Generation
    • Wind, Hydro, and Generation; updated hourly
  5. BPA Transmission TTC; updated hourly
    • 20+ interfaces
  6. BPA Transmission, Flows, hourly, updated daily
    • AC & DC – hourly, updated daily
    • NI and Puget Area – every 5 minutes
  7. Cal-ISO L&R
    • Net generation, net import/export; updated hourly
  8. Cal-ISO Renewables
    • Wind and Solar; updated hourlyCal ISO Transmission; updated hourly
    • Actual flows, scheduled flows, TTC
  9. Cal-ISO Markets
    • By Balancing Authority, updated every 5 minutes (RT)
  10. Northwest River Forecast Center; updated hourly
    • 120 real-time stations, inflows and outflows
  11. Corps of Engineers – updated hourly
    • 16 major projects; inflows, outflows, flows for generation, and energy
  12. USGS; updated hourly
    • Outflows updated every 15 minutes

Every 35 minutes each of these sites is scraped, archives refreshed, and the Ansergy website is updated. We have begun the process of creating Bookmarks and Dashboards around this data, each organized by Hub.

Examples of Real-Time Reports

Below is a collection of reports built from our real-time service that will give you a taste of what can be done with the data. You can access real-time data from the following menu:

We’ve already added about 150 bookmarks to jump start your personal dashboards, here are a few examples:

Mid-C Summary (Dashboard)

In this report we merged the following real-time sources:

  • ISO HA prices – CAISO, prices updated every 5 minutes
  • Malin500 – CAISO,  flows updated hourly
  • NOB_ITC – CAISO, flows updated hourly
  • BPA Wind – BPA, wind energy updated hourly
  • BPA Hydro – BPA, hydro energy updated hourly
  • BPA Thermal – BPA, thermal generation, updated hourly

Dashboards are convenient for seeing lots of data on one screen, but you lose resolution; all of these reports can be viewed in full-screen mode:

Demand – Forecast versus Actual

The EIA, through its Open Data Project, provides actual hourly demand and each BA’s 24-hour forecast of today’s demand. The above chart plots both for the last two days plus today for Portland General Electric. We have annotated where PGE found itself long, or short, based upon its demand forecast and realized loads. A derivative report of this data is our Long|Short report:

PG&E Long Short (From Demand)

We define “Long” where Actual demand is Less than Forecasted demand. Since this data comes from the utility, we make the assumption that this is the same demand forecast used in the Next Day trading. Hence, if actual demand is greater than what was forecasted the previous day (and used in trading Next Day) the utility would be short (at least from the demand side.).

Much of this data is redundant to what Ansergy has been publishing for the last couple of years; the primary difference is our Real-Time reports will always include the most recent updates. The Demand and Long Short reports are available for 31 Balancing Authorities.

Real-Time Hydro

Real-Time hydro returns actual flows (QR) for all 33 WECC  basins used in our hydro energy forecasts. These can be viewed as stand-alone reports:

Pit River Discharge (CFS)

Streamflow data comes from one of three sources:

  • Corps of Engineers – the slowest of the three, but has the most detail as it distinguishes Total Outflows from Generation Outflows and includes reservation levels
  • USGS – most stations are updated every five minutes, and all are measured in outflow
  • River Forecast Center – updated every five minutes for most stations

You can also view a bundle of Stations in a dashboard:

Selected Northwest Hydro Projects

This dashboard shows how tables of data can be combined with charts. The source is provided in the table view:

Hydro Summary

Each of the four Real-Time Types (Demand, Hydro, Generation, and Transmission) has a similar summary table. A few of the summary highlights include:

  • Last Date & Hour – returns the most recent date: hour of data
  • Comparison Across Time – each summary returns the most recent hour, the previous 2 hours (H1 and H2), the previous two days for the same hour as current (D1 and D2) and same hour for the same day of the week from one and two weeks back.
  • Source – where we collected the data
  • Other – in hydro we include the capacity mapped to the basin for easy sorting.

ISO Data

The ISO provides a wealth of data; we’re confident you are already scraping all of this. Ansergy takes it one step further and merges many different scrapes into a single report.

SP15 Summary

There is a lot here, the breakdown follows:

  • Weather & Generation Summary – a quick recap of wind speed, cloud cover, temperatures and actual generation for SP15; this report is available for all 14 WECC hubs.
  • Actual Generation – we merged the ISO renewable data, the ISO L&R a report, and ISO transmission; in, his example we plot actual Solar, Wind, and Net Imports.
  • Demand Forecast for SCE – includes actual ISO demand plus the SP15 forecast
  • Path 26 Flows
  • Edison’s Long Short Report
  • Discharge on the Colorado River (Parker)

The idea is to put ALL fundamentals at your real-time desk’s finger tips and update every hour.

Transmission

Malin500

Each chart can be viewed in different time ranges; the above is the last seven days. We flagged those hours where actual flows on the AC exceeded the ISO’s real-time TTC (indicative of how brutally long the Northwest is at the moment).

Conclusions

There is a wealth of data in the RealTime reports, and we encourage all of our clients to explore. For those that completed their trial, and would like to have your real-time desk kick our “real-time tires” drop Garrett a note and we’ll re-activate the trial.

Bill and Garrett available to build custom bookmarks and dashboards to meet your team’s exact and unique needs.

Cheers,

Team Ansergy