Over the years, end users have expressed the general concern that data management isn’t worth their time. In all fairness, this misconception is understandable due to the inexpensive and continually decreasing cost of consumer-grade disk drives. Ultimately, firms should work towards changing that mindset because the true cost is far greater in an enterprise’s production environment. To identify the associated costs, I have prepared the following analysis of raw storage consumption, performance impacts, and resources needed to store data in a locally-hosted Microsoft Windows server environment.
Referencing Figure 1, a 1GB file not only consumes raw storage across multiple storage platforms (e.g. local storage, backup volumes, etc.), but at a higher quantity than in its original form. This is a direct result of high availability and fault tolerance achieved when using a Redundant Array of Independent Disks, better known as RAID.
One requirement for this essential feature with RAID level 5 and 6, as depicted in Figure 1, is the additional raw storage capacity needed to maintain parity across all drives. RAID parity is the mathematical method used to calculate used storage on each drive, which is striped (i.e. spread) across all other drives in the array. This allows a RAID volume to operate continuously and unimpeded if a single drive were to fail (or two drives with RAID 6), and protects from unrecoverable sector read errors. It’s worth noting there are other RAID levels designed to improve fault tolerance and/or performance, such as RAID 1, 10, 50, and 60. Each level has its own distinct advantages, however, no matter which RAID type is chosen, additional raw capacity is needed to support it.
In order to create and manage the RAID structure, a hardware controller or software utility is needed. Many different factors determine whether a software or hardware solution is appropriate for a firm’s RAID needs, but in either case, managing an array of disks requires resources. For a RAID hardware controller, these resources come in the form of a physical controller, with its own dedicated CPU, RAM, and specialized firmware designed to manage the disk array. Similarly, a software-based RAID needs the same resources, but instead places this burden on the server’s CPU and RAM. With either solution, computational resources are needed to operate these systems – the cost of each is directly driven by the size and number of disk drives in the array, which in turn is determined by the quantity of data stored.
The cost of storing a single 1GB file is further compounded by the price of enterprise-class drives that are used in servers and storage arrays. These drives, such as server-grade SATA or NL-SAS drives, are designed for 24/7 operation in a production environment and cost approximately 4 times more per GB than consumer-grade drives. SATA or NL-SAS are generally used when capacity is needed over performance, however, firms who require the highest-level of I/O performance must employ enterprise-class SSD or SCSI drives, which come with a substantially higher price per GB.
One might think, “Well, these costs just apply to firms hosting large files, small files don’t matter”. This is another misconception that many end users have. Although small files impact raw storage in a different manner (later explained), their biggest cost comes in the form of processing individual file records. New Technology File System, or simply NTFS, is the file system used by Windows operating systems on servers and workstations to store data on disk drives. This file system relies on a Master File Table (MFT), which is the heart of the NTFS volume structure. The MFT contains a record (its metadata) for every single file stored on the disk, consuming an additional 1KB of raw storage for each, and defines the file’s: file name, attributes, security descriptor, object ID, etc. Whenever an operation is performed on NTFS, each of those records must be processed. You have likely seen the impact this has on performance when executing various operations, such as copying a 1GB file vs. several smaller files that consume the same quantity of storage.
Small files also increase data fragmentation, which can reduce read/write speed due to additional seek times; albeit, this plays a greater factor with mechanical drives vs. solid state. Nevertheless, storing unnecessary small files can have a notable detriment on performance when carrying out the most basic operations.
Another way small files impact performance and drive cost is how they’re stored within the file system. When a disk drive is formatted, raw storage is broken into chunks called clusters, with each cluster representing a fixed size of bytes. When a file is stored on a disk drive it is allocated to as many clusters as it needs. To maintain optimal disk drive performance, cluster size should be increased as drive capacity increases – fewer clusters to seek, means less seek time. One consequence of larger cluster size is the impact it has when storing small files. For example, many drives today use a 4KB cluster size due to their high capacity – that means a 4KB file needs one cluster to store its data. Consequently, a 1 byte file also needs 4KB to store its data because 4KB is the smallest logical unit available for storage, leaving the remaining part of the cluster unused. This may seem negligible, but when you have a storage volume with millions of files that are smaller than the smallest cluster size, it consumes precious storage space; and don’t forget about the performance impact related to the MFT!
The cost of large and small files on a disk drive is worth noting, but another consequence comes in the form of indirect costs. This includes the labor and computational resource consumption associated with managing, indexing and searching, backing up, and other types of data processing, which puts an unnecessary burden on equipment and personnel.
In conclusion, the cost of storing unnecessary files, small or large, can’t be overstated. Firms who make a concerted effort to appropriately manage their data inevitably reduce their IT expenditures and associated management costs. End users may find it difficult to see the value of efficiently managing the data they create, but when the costs are aggregated across a firm’s entire IT systems, it’s apparent that data management really does matter.
Patterson, David; Gibson, Garth A.; Katz, Randy (1988). A Case for Redundant Arrays of Inexpensive Disks (RAID). SIGMOD Conferences. Retrieved 2018-06-20.
Chen, Peter; Lee, Edward; Gibson, Garth; Katz, Randy; Patterson, David (1994). "RAID: High-Performance, Reliable Secondary Storage". ACM Computing Surveys. 26: 145–185. Retrieved 2018-06-20.
Scott Lowe (2009-11-16). "How to protect yourself from RAID-related Unrecoverable Read Errors (UREs). Techrepublic". Retrieved 2018-06-21.
Intel Corporation (2017-10-02). “Defining RAID Volumes for Intel Rapid Storage Technology”. Retrieved 2018-06-10.
Code Idol (2010-09-17). “NTFS On-Disk Structure”. Retrieved 2018-03-02.
Microsoft Corporation (2009-10-08) “How NTFS Works”. Retrieved 2018-03-02.
As we discussed in our first blog post, there are many challenges facing the nuclear industry. One of the greatest is the current energy climate. There are many contributing factors to the general state of flux in energy production, which we would like to explore today. These challenges don’t just impact the nuclear industry, but also affect energy producers across generation types.
It may surprise you, but US energy consumption has effectively plateaued over the last 15 years. Below is a plot generated with the US Energy Information Administration Open Data Embedded Visualization Library. The EIA provides a wide range of information and data products covering energy production, stocks, demand, imports, exports, and prices; and prepares analyses and special reports on topics of current interest.
There are four sectors that are included when looking at total energy consumption. These include Residential, Commercial, Industrial, and Transportation, all of which are shown in the figure. As you can see, starting around the year 2000 the Total Energy Consumption has plateaued. The largest changes in trends have been experienced by the Industrial Sector, showinga significant decrease in consumption over that time. This is likely most attributable to a major focus on energy efficiency, which is improving consistently. There are still challenges, however, outlined in this US Department of Energy Report, which provides information on barriers to industrial energy efficiency.
The way energy is produced in the United States has changed dramatically over the last 15 years. Another plot from the EIA is provided showing the change in net generation for coal, natural gas, nuclear, hydroelectric and renewables. Each supply type is zeroed relative to its 2001 value for comparison.
It is obvious from this plot that while nuclear and hydroelectric production has remained relatively constant on an absolute basis, coal has suffered significantly while natural gas and renewables rise.
Solar Growth/Capacity Issues
Utility scale solar is the fastest growing renewable power generation source in the US on a percentage basis, as shown below. The figure shows the growth of various renewables as a percentage change from 2001.
The growth of solar, particularly in the Alden headquarters home state of Massachusetts, has been significant. Below is a plot from ISO New England showing the Projected Cumulative Growth in New England Solar Power. Starting in January of 2010, there was a minor amount of PV capacity in New England, however by 2025 they predict 3.27 Gigawatts of PV capacity.
Next week, we will continue this thread with a discussion of power prices and power storage, and how these effect the changing energy climate.
Attendees at the 2017 Alden Forum on Hydropower and Fish Passage
Based on presentations given by the various speakers, the primary takeaways from the forum include the following:
Cake consumed during a break at the 2017 Alden Forum, showing companies and agencies in attendance
The format and content of the forum was highly rated by the attendees and led to many in-depth and productive discussions. The setting appeared to be more conducive to open dialogue among all of the participants compared to typical relicensing meetings and agency consultations.
Planning for additional forums addressing other relevant topic areas related to fish passage and other environmental issues is underway by Alden staff, and may include hosting events in other regions of the U.S.
Searching for eels is an activity reserved for those who like to stay up late. Only under the cover of darkness is one able to have the best chance to find these nocturnal fish. Surveying for American eels, Anguilla rostrata, with lanterns at night or “shining” is a common method to document their presence in rivers across eastern North America. Juvenile American eels often congregate downstream of obstacles that block their upstream movement (eels are catadromous fish, meaning adults spawn in saltwater and the young move into freshwater to rear and mature before returning to the marine environment to complete the reproductive cycle). Hydroelectric dams are common impediments to upstream migrants along river courses. Finding optimal areas to establish passage routes for eels to move upstream is a primary reason to shine for eels.
Eels, being a fish, become more active as the water temperature rises. When the temperature is above 10 degrees C, which generally occurs from May to October in New England, eels of all ages and sizes become more active. The recently born glass eels, (so called because of their transparent appearance), are carried northward by Atlantic Ocean currents, floating along like crystalline feathers until tides pull them ashore where they begin a process of metamorphosis into the more recognizable elongated fish. These eels, now called elvers, grow and mature to the point where they are capable of swimming against the river current to seek inland freshwater and a chance to grow to appreciable sizes, (mature female eels in the St. Lawrence River can reach 1 meter in length and weigh up to 8 kilograms). Eels are long-lived fishes, some individuals are almost 30 years old before returning to the central Atlantic Ocean to complete their lifecycle and give life to the next generation.
Surrounded by complete darkness after nightfall during the warm spring or summer months, throw on a pair of waders, and the search for the secretive eel begins. A careful inspection of the area of interest such as a dam tailwater using flashlights or lanterns is best achieved from the shore or among the rocks and boulders of the riverbed. If the river is inaccessible for wading, a pair of binoculars can help view from a distance by aiming a strong spotlight at the suspected congregation area. Eels do not need a lot of flowing water to stimulate them to try and ascend. They can seek to move upstream through even the smallest trickle of water if that is all that is available, so be sure to pay attention in areas where these conditions exist. Look for rivulets of water flowing among boulder fields that offer a constant and uninterrupted path. They may try and lift themselves up with their tails through a plunging waterfall but they need substrate to support their long muscular bodies and push against it to get up, over and through the point of passage they seek. Eels even have the extraordinary ability to travel overland if the desire is strong enough to move past an obstruction. During warm humid nights, eels can be observed doing just this.
Figure 1: Migrating eels in a New England River
Just because its night-time and it’s the month of June doesn’t mean that eels will be on the move. Environmental conditions such as air and water temperature, precipitation, percent cloud cover, and lunar phase influence eel behavior. Variability in these conditions during the spring and summer months can cause eel activity levels to increase or decrease. Keeping track over a season the exact locations and environmental conditions that cause the greatest number of eels to congregate will provide the best information to decide where to establish an upstream passage route. There are many ways to provide upstream passage for eels (ladders and traps are common), but it is critical to know the optimal location to place these facilities. Eel shining is a proven way to establish these locations by observing the conditions that eels prefer at a site of interest. Alden advocates using night-time surveys as a low-tech yet effective method to shine the light on upstream eel behavior.
You might never know when one seemingly minor decision could change your life.
One summer weekend, just before entering my third year in the Civil & Environmental Engineering program at Tufts, I found myself on a whitewater kayaking class for beginners run by volunteer instructors with the Appalachian Mountain Club. A friend recruited me to join at the last minute; they needed more new “boaters” to reach their minimum capacity.
Some combination of perfect weather, good company, and new challenges that weekend got me hooked on the sport. The more time I spent on the river, the more folks I met who had degrees and careers related to hydrology or engineering. That would eventually include me, too – my love for this hobby & fluid dynamics led me to work here at Alden.
When I returned to school in the fall, I took my first fluid dynamics course. The coursework and the new hobby complemented each other – spending time in a boat made it easier for me to understand certain fluid mechanics topics.
One of those topics is the concept of a stagnation point: an obstruction in a flow field, like a rock or bridge abutment in a river, will cause the fluid to slow down to a velocity of zero at the object’s surface, resulting in high static pressure.
For a boater, the stagnation point is a dangerous place to be. You and your boat can get pinned on the upstream side of an obstruction, and if you can’t get free quickly, you could be injured or drown.
What makes the stagnation point so dangerous? Consider conservation of energy and the Bernoulli equation, a key concept in fluid dynamics:
The Bernoulli equation only applies to steady, incompressible, frictionless flow along a streamline. Under these conditions, total mechanical energy per unit mass of water (which is the sum of pressure (p/rho), kinetic (V2/2), and potential (gz) energies) is constant along a streamline. This concept can clearly explain why a kayaker doesn’t want to get stuck at the stagnation point.
Figure 1: Bridge Abutment Stagnation Point
Follow the light blue dotted streamline through the middle of the river in Figure 1. If we estimate negligible friction and minimal change in elevation as the streamline approaches the bridge abutment, Bernoulli’s equation says the energy from a high velocity in the river upstream of the abutment will change to a velocity of zero, along with high static pressure at the stagnation point.
For a boater, this means that if you float head on into an obstruction and become pinned, you will have a velocity of zero, and high water pressure will hold you in place against the obstruction. Yikes! It is best to avoid this situation by navigating around the obstruction, but boaters should always be prepared for the worst by staying up to date on swift water rescue techniques and carrying appropriate safety equipment. Thanks for the safety tip, Bernoulli!
Whitewater kayaking was also helpful in understanding my favorite dimensionless parameter: the Froude number! This parameter compares inertial to gravitational forces. It is also a ratio of the fluid’s velocity to the speed that a surface wave travels across the fluid (AKA wave celerity).
Froude numbers less than 1 indicate subcritical flow: the water is deep and moving relatively slowly. Froude numbers higher than 1 indicate supercritical flow: the water is shallow and moving quickly. At a Froude number of 1, we have critical flow: gravitational and inertial forces are equivalent. Critical flow can be achieved where the slope of the channel or river is zero, such as over the top of a weir. If you measure the depth (L) of water over the top of the weir as well as the width of the river along the weir, you can determine the flow area, flow velocity (using the Froude number), and ultimately the total flow rate of the river. The Froude number also helps us understand how waves form, and why kayakers are able to surf on them.
When conditions are just right in a river, standing waves and holes (shown in Figure 2) can form; they stay in the same place and don’t move with the flow of the water. Kayakers can have fun with these features by surfing on them, balancing on top of the wave or hole, and staying in the same location relative to the river bank in a fast-moving river.
Figure 2: Whitewater Hole & Wave1
In fluid dynamics, this phenomenon is called a hydraulic jump.
Hydraulic jumps suitable for surfing sometimes occur naturally in rivers, and can also be designed and installed where river flow and gradient (along with local stakeholders and regulatory agencies) permit. To create a hydraulic jump, a weir-like structure in a river (shown as a “play feature” in Figure 2) can be used to make the flow to transition from critical to supercritical (i.e., the water is shallow and moving quickly). An abrupt drop leading into the downstream pool creates a discontinuity along the river bed. The flow immediately becomes subcritical, getting deeper and slowing down. This abrupt change is mirrored at the river surface, shown in these pictures as a “seam” in a hole or as a “trough” in a wave, where the green water of the supercritical upstream flow meets the hydraulic jump.
Figure 3: Me, trying to throw a loop, in Tariffville CT. Photo credit goes to Andrew Nitchske.
The velocity of the subcritical flow is very low, and can in some cases reverse so the water is flowing upstream, which makes it possible to surf and do tricks in a wave or hole without floating downstream.
The flow is critical at the seam or trough, which means that the Froude number is equal to 1. The wave/hole will not move around too much since the flow velocity is equal to the wave celerity.
Kayak surfing (made possible by the Froude number) is my favorite thing to do!! Check it out:
Figure 4: Katelyn Green (yellow kayak) and me (blue kayak), surfing in Tariffville CT.
So… what’s the difference between a wave and a hole? Tune in next time for a post from my colleague Ben Mater!
Remember to use the comment field to share your favorite fluid dynamics hobbies with us.
 McLaughlin Whitewater Design Group, Ben Nielsen, April 15, 2014 Presentation “Recreational Whitewater: Keys to Successful Management”, available online at https://www.slideshare.net/rshimoda2014/nielsen-ben-rms-recwwworkshop2014submittededit (slide 38 of 66). McLaughlin is one of a small number of companies that create whitewater parks for surfing, and Alden has been lucky enough to collaborate with McLaughlin on occasion!
Part 1 of this series outlined how high concentrations of total dissolved gas (TDG) can occur downstream from high head dams when their spillways are open, and how this TDG can be harmful or even fatal to fish. Alden has been involved in several recent projects for which the objective was to reduce TDG downstream of high head dams. Alden performed the hydraulic and structural design of roughness elements that break up the high velocity jet of flow discharged from the spillway. TDG production is reduced by these roughness elements because they cause the jet to spread out and thereby reduce the plunge depth in the receiving water, which reduces TDG. The roughness elements work very well at reducing plunge depth, but they can cause cavitation, which can damage the spillway surface and the blocks themselves. The design and implementation of the roughness elements will be topic of another article. The present article focuses on reducing the potential for cavitation on the roughness elements.
Alden designed roughness elements have been installed on spillways at Cabinet Gorge and Boundary Dams. Cabinet Gorge Dam is shown in Figure 1. The first set of roughness elements installed at Cabinet Gorge Dam performed well at reducing TDG, but suffered cavitation damage (Figure 2). Cavitation can occur in high velocity flows on steep spillways, especially when roughness on the spillway surface causes flow separation. Air supply ramps are often used on spillways to lift the nappe from the spillway surface and supply air to the void underneath the nappe. Cavitation potential is reduced by introducing the air.
Figure 1. Cabinet Gorge Dam and 3D model for CFD (Dunlop, et al., 2016)
Figure 2. Cavitation Damage on Roughness Element (Paul, 2015)
Air ramps were installed upstream of the first row of roughness elements for one bay each at Cabinet Gorge and Boundary Dams to supply air and to lift the horseshoe vortices that form around the base of the blocks off of the spillway surface. The Cabinet Gorge air supply ramps are shown in Figure 3. The hydraulic design of the air ramp and the air supply ducts at Boundary and Cabinet Gorge Dams was performed by Alden and based on the Aerator Design Chapter (Chapter 5) of Dr. Hank Falvey’s “Cavitation in Chutes and Spillways – Engineering Monograph No. 42.” A conceptualization of the air ramp is shown in Figure 4. The flow over the ramp follows a trajectory influenced by: the velocity of the flow at the location of the ramp, the angle of the ramp, the angle of the spillway, and the air pressure in the air pocket underneath the nappe. The underside of the nappe entrains air which generates negative pressure in the air pocket underneath the nappe. Air is supplied to the air pocket by the air ramp, and the energy losses through the air ramp can be significant. The volume of air drawn through the ramp, the shape of the trajectory, the pressure in the air pocket underneath the nappe, and the energy losses of the air flow through the ramp are functions of each other, so there is no analytical solution to solve for the air flow rate. Alden developed an Excel VBA program to iteratively solve for the air flow rate for a given ramp geometry and spillway flow rate. The air ramp and the ducts supplying air to the ramp were designed by Alden to ensure that the high velocity air through the ramps would not cause sonic shocks and so that the unit flow rate of air on the spillway was approximately 10% of the unit flow rate of water, which has been shown to reduce the potential for cavitation (Falvey, 1990).
The air ramps installed at Cabinet Gorge and Boundary Dams successfully supplied air, and cavitation damage has not been observed since their installation.
Figure 3. Roughness Elements and Air Supply Ramp at Cabinet Gorge Dam (Dunlop, et al., 2016)
Figure 4. Conceptualization of Air Ramp (Based on Falvey, 1990)
Today’s entry comes from a guest blogger, Jim Walsh, President of Rennasonic, a small consulting firm specializing in turbine and pump performance testing and optimization of multi-unit hydroelectric power plants using ultra-sonic multi-path flow meters. Alden has partnered with Rennasonic for numerous turbine performance tests, providing supplemental flow measurements using dye dilution and current meter profiling.
As a hydropower electric power generating utility, how do I know when I should, or should not, invest in my equipment? The answer can be complicated due to many factors, including but not limited to: the current price of power, generating capacity, equipment age, and government regulations. To determine the performance of an installed hydropower turbine, the measurements of water flow, head, and power must be made within a reasonable amount of uncertainty. Generally speaking, flow is the most difficult parameter to measure in the field and, consequently, is the most expensive. The cost of measuring flow can seem unsurmountable for small hydropower owners, so the question becomes when testing expenditures yield a return on investment.
Field performance tests are often used only for large projects with Francis runners that have capacities greater than 50 or 60 MW. For these performance tests to be viable, a return on investment must be assessed, particularly because plants can have multiple units and the performance of the first replaced unit is often assumed to be representative of subsequent units. The author has been involved with test programs where the payback is perceived short, and the benefit cost relationship is easily greater than one. A recent project involved testing a 4 unit (50MW each) plant. This plant needed to have pre- and post-upgrade field tests on all 4 units for an energy analysis to support production tax credits (PTC) under the 2005 Energy Policy Act. The approximate cost of this test program, which occurred over a 4 year period, was $195,000. Some cost savings were available because the flow measurement was performed using previously installed acoustic transit time flow meters.
In order to qualify for PTC’s, models had to be created to predict the energy increase resulting from upgrades to the turbines. Inflows to the plant over an 8 year period were obtained and a computer model of plant operations was created. The model included the order in which each of unit would be dispatched. The historical inflows were used as inputs to the model, to determine the baseline energy production of the plant before any upgrades. Annual energy production from the model was compared to the Energy Information Administration (EIA) reported generation tables over the same period and the baseline model was found to be within 2% of the reported data for years when unit outages did not occur. Later, using newer characteristic curves obtained from field testing of the new units, a new dispatch table was created to represent the new plant model. Energy calculations were performed and compared to the baseline mode.
The results of the comparison yielded a 4.7% increase in annual energy production, on average. Using the current EIA average wholesale market rate of $45 per MWHR, annual expected gains in energy revenues can be calculated. The model projected increase was reduced to 2.7% to account for uncertainty in the model, yet the expected gain in revenue was still approximately $450,000 per year when all units are in service. The benefit cost ratio of this project is over 2.0 over a five year period, and the internal rate of return (IRR) calculates to 3% after 2 years and 26% after 5 years.
One can use these figures to determine that the same level of test effort applied to a lower capacity unit (25 MW x 4) can still be justified, because the benefit cost ratio will be 1.0 after 5 years and the IRR will be approximately ½ of the above example over the same time frame. In short, it makes sense to test all the units at plants having a combined capacity of 100 MW. One can also make the assumption that smaller plants having nameplate capacities of 50 MW and perhaps lower can justify the cost of field testing for at least one unit when transit time meters are installed.
Assuming transit time meters are not installed, the outlay increases by the cost of transducer fittings, electronics, transducers and the installation of the equipment. The sunk cost, or previously incurred unrecoverable cost, increases by slightly over $100,000. Under the same assumptions as above, the IRR becomes 15% after 5 years with a benefit cost ratio of 1.33. Even with the cost of the installation and equipment, it can make financial sense for similar capacity plants to initiate this program.
A view of the tailrace of a hydropower dam during performance measurement