Victoria won the hydrologic games, and they may not have cheated

As usual, the hydrologic games were held during the HWRS2018 conference dinner, and this time Victoria won.  Competitors threw a dart to select an initial loss and rainfall temporal pattern which were used as input to a RORB model to simulate a 1% AEP peak flow.  The team with the highest average peak was the winner.

Figure 1: Results of the hydrologic games (Source: A post by Ben Tate on LinkedIn)

Before the games started, I thought one of the small teams would win.  There were only two international competitors and six from Tasmania, but likely over 100 from Victoria.  Extreme results are more likely from small samples so I expected small teams to have both the highest and lowest scores with the Victorian team to be in the middle, near the true mean.

There is a nice discussion of the effect of sample size on variation in the wonderful book Thinking Fast and Slow by Daniel Kahneman.  See the start of Chapter 10 “The law of small numbers”.  Also, in Howard Wainer’s article The Most Dangerous Equation.  Wainer argues that misunderstanding of this effect has caused confusion and wasted effort. For example, the small-school movement was based on the idea that students performed better in smaller compared to larger schools.  In the US, grants from the Gates foundation were used to facilitate the conversion of large schools to several smaller schools.  However, evidence presented by Wainer shows the small-school effect is likely the result of the extra variation when only a small number of students are assessed.  Comparisons of many schools showed the smallest schools achieving both the highest and lowest scores.

Back to the hydrologic games.  How likely is it that the mean of a large sample, the Victorians, would be higher than all the means from a number of smaller samples?  I don’t know how many people participated in the hydrologic games, so let’s guess.  Ben says there were “over 150 players”; I’ve divided these up as shown in Table 1.

Table 1: Teams for the hydrologic games

 

Now, to simulate the game we draw random numbers from the normal distribution.  The Victorian score will be the mean of 100 random numbers, the NSW score will be mean of 15 random numbers and so on.  Victoria wins if its mean exceeds the mean of all the other teams.

Using this approach, the probably of Victoria winning is estimated to be 0.00830 (1 in 120), based on the mean of 100 sets of 100,000 simulations.  So its less likely than seeing a 1% AEP flood.  If a hydrologist attends 20 hydrologic games during their career, and this game was played every time, the probability that they would see Victoria win at least once is about 15%.

The upshot is that when Victoria won, and provided the games were fair, we witnessed a rare, but not impossible event.

Calculations are available as a gist.

 

 

Flow duration curves

Flow duration curves show the percentage of time in a flow record, that flow exceeds a particular value.   They are a long-standing tool of hydrological analysis and their construction and uses are thoroughly documented in two papers by Vogel and Fennessey (1994: 1995).    Flow duration curves can be based on daily, monthly or annual flows and constructed for periods of interest such as a whole year, seasons, high or low flow periods, or, for example, during a period of fish migration. They may use the whole period of record, or parts of the record, to analyse variability.

Basic flow duration curves are easy to create but more complicated versions are sometimes useful.  An example for the Broken River at Casey Weir (gauge 404216) is shown in Figure 1.  Here the annual flow duration curves are shown in grey for each year of record from 1973 to 2006.  The y-axis is log-transformed and the x-axis uses a probit scale. Wet years (1974, 1993) and a dry year (1976) are coloured, highlighting the substantial variation in annual flow.  In the driest year there was no flow for 50% of days.  The average flow duration curve for all years is shown in black.

Figure 1: Flow duration curves for the Broken River at Casey Weir (404216) (1973-2016)

Code to produce this figures is available as a gist.

References

Vogel, R. M. and N. Fennessey (1994) Flow-duration curves. I: New interpretation and confidence intervals. Journal of Water Resources Planning and Management-ASCE 120(4): 485-504. (link to abstract) (link to paper)

Vogel, R. M. and N. M. Fennessey (1995) Flow duration curves II: a review of applications in water resources planning. Water Resources Bulletin 31(6): 1029-1039. (link)

 

 

Comparing peaks at stream gauges

In hydrology, it is often useful to compare peak flow values at neighbouring streamflow gauges.  This is important for data checking and to see if it is feasible to use data from one gauge to infill missing values in another.  In reports and student projects, I’ve seen the comparison as a time-series of columns (Figure 1).  This figure shows annual peaks between 1980 and 1990 for two streams in northeast Victoria, near Wangaratta, Boggy Ck and Fifteen Mile Creek.

It is usually better to plot the data as paired values (Figure 2).  This allows a more direct comparison of the peaks and reveals the potential for establishing a useful relationship between flows at the two gauges.  On Figure 2, the value for 1981 is an outlier.  The flow is much larger and ratio of flow between the two gauges looks different to the other years.  A check of the flow record shows that the 1981 peak for Boggy Ck has quality code 104, “Records manually estimated”.  To increase confidence in the magnitude of this large peak, we could investigate further.  For example, by checking the high flow rating,  looking at the flow at other neighbouring gauges, and assessing the accuracy of the manual estimation method.

Figure 1: Comparison of annual peak flows as a time-series of columns

 

Figure 2: Comparison of annual peak flows as paired values (same data as shown in Figure 1)

Code to produce these graph is available as a gist.

Comparing rainfall measurements with a heat map

How to compare rainfall measured at two gauges?  One method I came across in a report was to make a table that showed the difference in totals for each month of each year in an overlapping period.  An example is shown in Table 1 below.   This provides some information but its not easy to see what’s going on.

Table 1.  Comparison of rainfall at two gauges

It is straightforward to turn a table of values like this, into a heat map (Figure 1), where its easy to see patterns.  The July column is mostly green (one gauge gets more rain in the austral winter) the January column is mostly purple (summers are drier at one site).

Figure 1.  Data from Table 1 as a heat map

If its important to look at the individual values, these can be overlaid on the coloured tiles in light grey so the patterns are not obscured.

Code to produce these figures is available as a gist.

 

Visualising Hydrologic Data

This is the companion webpage for my paper at the 2018 Hydrology and Water Resources Symposium, Visualising Hydrologic Data.

• Paper
• Presentation

How to make figures from the paper and presentation

• • Flood frequency plots
  • Line graphs
  • Slope graphs
  • Heat maps
  • Comparing peaks at neighbouring gauges
  • Graphical display of a water balance
  • Cut-and-Stack plots of a long flow record
  • Facet plots of flow data
  • Flow duration curves
  • Circular plots to show the time of year of hydrologic events
  • Plotting high and low flow spells
  • Tile plots (also see code as a gist)
  • Quality codes (here and here)
  • Season plots

Guides

• What not do do when visualising data

Colour

• Picking a colour scale
• Avoid rainbow colour scales

Graph catalogues (catalogs)

• R Graph Catalog
• A compendium of clean graphs in R 
• Top 50 ggplot2 visualizations – the master list (http://r-statistics.co/Top50-Ggplot2-Visualizations-MasterList-R-Code.html)
• The R graph gallery (r-graph-gallery.com/)
• The Python graph gallery
• Matlab plot gallery (eau.mathworks.com/products/matlab/plot-gallery.html)
• Excel charts gallery examples of 150 types of excel charts
• The data visualisation catalogue
• Chart Gallery (Google Charts) (developers.google.com/chart/interactive/docs/gallery)
• Gallery of data visualization examples

Software

• ggplot2 (a package for R)

Practitioners

• Mike Bostock (see his work here)
• Di Cook
• Stephanie Evergreen, Evergreen Data
• Hans Rosling (see his TED talk)
• Edward Tufte 
• Howard Wainer
• Hadley Wickham
• Nathan Yau, Flowing data

Some interesting papers and books

• Wainer, H., Friendly, M. and Millan P. (2015) Graphs ‘R Us’: A discussion of Anthony Unwins Graphical Data Analysis with R.  Journal of Educational and Behavioral Statistics 40(6):665-670.  https://doi.org/10.3102/1076998615606114 (copy at research gate).  This paper considers the relative merits of 16 graphics books.
• Weissgerber et al., (2015) Beyond bar and line graphs: time for a new data presentation paradigm. PloS Biol 13(4):e1002128 https://doi.org/10.1371/journal.pbio.1002128
• Nathan Yau’s books
• Edward Tufte’s books

 

Assessing the impact of blockage as part of flood modelling

Australian Rainfall and Runoff 2016 provides guidance on assessing the impact of blockage of culverts and bridges as part of flood modelling. Details are in Book 6 Chapter 6.

The need to assess blockage represents a change to hydraulic modelling practice.  In the past, industry-wide guidance was lacking and the effect of blockage was often not considered.    The new guidelines provide a standard procedure but, as yet, there is limited experience in their application.  Part of the process is to complete a blockage assessment form which is linked from ARR.  Unfortunately the link no longer works but the form is on the internet here.

The guidelines are being incorporated into hydraulic modelling software with a recent paper outlining the procedures that have been included in TUFLOW and an assessment of how they have performed in three case studies of recent flood modelling projects (Ollett, et al., 2017).

It is likely that future flood modelling briefs will require assessment of blockage so flood consultants will need to learn about, and be able to apply, the procedures and explain the significance of results to clients.  Some resources are listed blow.

Large floating debris collection. Chalmers St Wollongong after the Aug 1998 flood (Forbes, Rigby 1999) (Source: ARR Project 11 Stage 1 report, p. 14)

Form

Blockage Assessment Form

Tools

ARR2016 Blockage Tool (from Hydralinc Consulting Engineers)

References – articles

Ollett, P., Syme, B. and Ryan, P. (2017) Australian Rainfall and Runoff guidance on blockage of hydraulic structures: numerical implementation and three case studies.  Journal of Hydrology (NZ) 56(2) 109-122. (link)

Ollett, P. and Syme, B. (2016) ARR blockage: numerical implementation and three case studies.  37th Hydrology and Water Resources Symposium 2016: Water, Infrastructure and the Environment. Queenstown, NZ. pp. 346-359 (link)

Ribgy, E. and Weeks, W. (2015) Evolving an Australian design procedure for structure blockages.  36th Hydrology and Water Resources Symposium: The art and science of water. Hobart, Tas. pp. 154-161. (link)

Suitability of ARR guidelines as an alternative blockage policy for Wollongong. 36th Hydrology and Water Resources Symposium: The art and science of water. Hobart, Tas. pp. 370-377. (link)

References – ARR2016 project reports

Project 11 Draft Blockage Guidelines (2015)

Project 11 (Blockage of Hydraulic Structures) Stage 2 Report (2013) (link at arr-software.org)

Project 11 (Blockage of Hydraulic Structures) Stage 1 Report (2009) link at arr-software.org)

TQmean, a measure of the impact of urbanisation on flow

Urban development has a profound impact on flow and hydrologic indicators have been proposed to highlight the changes as suburbs spread over a catchment, increasing impermeable area.  A commonly used measure of impact is TQ_mean– the proportion of time that flow in each year is greater than mean flow for that year.  This decreases with urbanisation, and has been shown to be linked to ecological condition of a stream (Booth et al., 2004).
As an example, consider two neighbouring streams in eastern Melbourne: Brushy Creek which flows through the suburb of Croydon, with a catchment that is 28% impervious, and Olinda Creek with a catchment that is mainly forested and which is 5% impervious.
Often, the value of TQmean is calculated and then averaged for several years. Using this approach, The TQmean for Brushy Creek, is 0.21, with the less urbanised Olinda Creek, having a value of 0.37 (for the period 1988 to 2016). Using the relationships in Booth et al. (2004) this suggests Olinda Creek would be predicted to have ‘good’ biological condition, while Brushy Creek would be predicted to be ‘very poor’.

Calculating a single value of TQ_mean is instructive but the temporal distribution of the time of the year when flows exceed the mean is also altered by urbanisation. Using TQ_mean as the metric, urbanisation results in high flows occurring more often, but for shorter periods, that are dispersed throughout the year. Comparing the plots of Olinda Creek and Brushy Creek in  (figure), higher flows (flows above the mean), are clustered in the winter (June to August) for Olinda Creek. There is more winter runoff because the Olinda Creek catchment wets up owing to the higher rainfall, and reduced evaporation that occur seasonally in this area. For Brushy Creek, with the same climate, short bursts of high flow occur throughout the whole year. This can be attributed to runoff from impervious surfaces which will occur anytime there is rain.

Figure 1:  Periods of flow above and below the mean flow for (A) Olinda Ck and (B) Brushy Ck

A key issue revealed by this analysis is the changed seasonality of high flow which is a result of urbanisation. This is just one of the many changes in flow regime caused by urbanisation that leads to poor stream condition (Burns et al. 2014).

Code to plot these graphs is available as a gist.