White Paper – A Smart Electric Market Means a Smart Market Infrastructure

Written by: Keith Houghton, Tom McInally; Edited By: Gary Michor Screaming Power Inc.



At the turn of the millennium stakeholders within electricity supply chains around the world were facing deregulation, either in the process to achieve it, or in the process to resist it. Many jurisdictions established electricity markets, yet these frameworks were new to the industry. This pioneering of markets often lead to mistakes, some of which were quite major. Liberalization activity lasted for a decade or so, yet very few utilities at first (especially the vertically integrated ones) were interested in deregulation; they saw this as a threat to their business model – the regulators drove the process.


In the past decade, the talk in the industry is now related to Smart Grids. Unlike the deregulated electricity markets, Smart Grids are essentially a logical progression of technology as applied to the energy supply chain. While the roll-out costs are significant, the chances of major failures are less likely than was the case with the first electricity market roll-outs since there isn’t the same “newness”.

We are going to look at some of the features of Electricity Markets and Smart Grids. We will then make the case that to fully achieve a Smart Grid an Electricity Market is desirable. Along this journey, we will also look at the role of both the regulator and the utility in achieving the desired objectives.

Smart Grid Overview – What is it really?

Many in the industry and government are trying to define what the Smart Grid is with short “sound bite” descriptions. These short statements cannot adequately convey the level of detail needed to provide a clear understanding of the Smart Grid concept. The Smart Grid isn’t a thing, but rather a vision, and to be realized, that vision must be expressed from different perspectives: the values, its components, and the roadmap for achieving it.
It’s true to say that the Smart Grid is a generic label for the application of computer intelligence and networking abilities to an essentially dumb electricity supply system; an electrical supply system designed over 100 years ago. Smart Grid initiatives seek to improve efficiency, operations, maintenance and planning by ensuring that each component of the electric grid can both ‘talk’ and ‘listen’ to each other. The major theme of Smart Grid technology is integrated automation and there are many different Initiatives:


  • smart metering;
  • demand response;
  • distributed generation management;
  • storage management (electrical, thermal, other);
  • transmission management (Synchrophasors, WAMS);
  • power outage and restoration detection;
  • power quality management (Voltage Reduction, more use of D.C. MicroGrids);
  • preventive maintenance to improve the reliability, security and efficiency of the distribution grid (Advanced Asset management, Mobile Workforce Management);
  • distribution automation (61850, CCTV, Thermal Imaging, D.G support, dispatch of small plant);
  • Game changing technologies – Electric Vehicle Support, Self-healing networks; and

This list goes on . . . . . .
We have no time to explain the individual components in detail, but as can be appreciated, the transformation and commitment to a Smart Grid will require new investment by all its many stakeholders, but for a Smart Grid to work it needs to start with supply and delivery. Stakeholders in this part of the value chain expect significant value in return. Understanding how this value will be created is an important step in defining the vision given that expectations for the Smart Grid are high. Expectations will only be realized through advances described below:


  • It must be more economic. An economic grid that operates under the laws of supply and demand, resulting in fair prices and adequate supplies.
  • It must be more efficient. An efficient grid employing strategies that lead to cost control, minimal transmission / distribution losses, efficient power production, and optimal asset utilization while providing consumers options for managing their energy usage.
  • It must be more environmentally friendly. An environmentally friendly grid that reduces environmental impact thorough improvements in efficiency and enabling the integration of intermittent resources.
  • It must be more reliable. A reliable grid that provides power, when and where its users need it and of the quality they expect.
  • It must be more secure. A secure grid that withstands physical (including natural disasters) and cyber-attacks without suffering massive blackouts or exorbitant recovery costs.
  • It must be safer. A safe grid that does no harm to the public or to grid workers and is sensitive to users who depend on it as a life device (medical necessity).

The Smart Grid can only achieve its full potential if it provides for greater transparency and choice that Customers will need, supporting the transition to a sustainable future. Some utilities claim that they can achieve their own Smart Grid vision by deploying selective components (those components that give them a positive Return on Investment) and therefore have concentrated upon the deployment of technologies that are for the utilities’ benefit particularly automated meter reading (AMR) and advanced metering infrastructure (AMI). Technologies that would benefit the consumer can be deployed later on, once other technologies and processes are in place!


Traditionally, utilities have concentrated on optimizing their business around the reliable provision of energy, but tend to forget the consumers they supply. This cannot continue. New business models are needed to allow smart technologies to fulfil their true potential; this may only come with regulation (legislation). Changes in the energy production and distribution model will also be forced upon the utilities, as it breaks their traditional business model. For example, given the overwhelming amount of consumer and network data from Smart Grid even from just AMI, utilities will require new capabilities in data management, data sharing, customer relationship management and analytics, which these organizations simply don’t have today.


Without regulatory intervention to press for practices that directly benefit others in the value chain (not just the organization in which the technology is delivered), those that do not benefit will obstruct the process. This has been evident in some cases where Smart meters have been deployed – with the consumers bearing the cost but not seeing any benefit. Indeed a consumer can even have its consumption curtailed in times of tight supply breaching the incumbent’s SLA. The utility benefits in efficiency and cost saving but the consumers’ benefits are usually restricted.


However, we are now seeing a change from the initial projects of a few years ago. We are starting to see more consumer-centric technologies, whether by utility enlightenment (utility choice) or mandate, and this has produced positive results; in particular, those projects that gave consumers more choice or those that have focused on sustainability solutions.
We need to understand that Distribution utilities, as well as other organizations that receive a regulated rate of return, don’t like seeing a reduction in the amount of energy they deliver. They see self-generation or demand side management programs as a risk to their earnings, since the more they sell the more they earn. The fact is that in many cases, the assets are long lived (25+ years) and can be inflexible in their operating modes particularly for base generation or distribution. If smart technologies are introduced that fundamentally change the demand profile, the impact on these assets may be significant.

Problems with the Current Supply Model – Bulk Generation

In most current jurisdictions, bulk generation is the norm and electricity has some characteristics, which somewhat complicates its production and tradability differentiating it it from other commodities. The main issues are:


  • Electricity is not easily labelled, so identification of who produced what for whom needs some attention.
  • Electricity is not easily stored, so there needs to be a means of synchronizing production with consumption in real time.
  • The nature of electricity supply is that all (or almost all) consumers see the same quality of product delivered to their point of use (there is no high octane option for example).
  • The grid networks, which carry electricity, need additional (ancillary) services to make sure the quality of electricity delivered to end customers is adequate and supplies are secured. These services themselves can either consume energy or can affect the ability of some producers to supply energy, thereby having an influence on the markets.
  • The transportation of energy is not 100% efficient. Some energy is lost in the transportation process.
  • Electricity networks have been built for a finite capacity to transport energy (usually peak demand). They even have bottlenecks, which reduce the ability of electricity to flow between a producer and its customers.

Changes in the electricity supply chain should alleviate some of these legacy issues, particularly with respect to network capacity, storage and the reliance on bulk generation. A Smart Grid if implemented comprehensively will facilitate proper changes to the electricity supply chain, giving the system the ability to adapt.

Electricity Markets Overview – It works so why fix it

The objective of an electricity market is to provide a facility where there is normal commerce and competition in the exchange of electricity between producers, consumers, and traders. This “exchange” should flourish, in the same way as for many other commodities such as Oil and Grain.
To achieve the stated objective an electricity market needs to:


  • Provide visible price signals showing the value of electricity.
  • Ensure that physical trades struck are feasible.
  • Ensure that there is a reliable and high quality supply to consumers.
  • Provide security mechanisms for times of emergency.
  • Encourage investment in new generation and infrastructure when required.
  • Allow electricity to be freely traded between different organizations and companies.
         o To ensure the trade is fair and equitable.
         o To prevent freeloading.
         o To provide fair access to traders to move energy across transmission and distribution networks between producers and consumers.
  • Operate efficiently and cost effectively.

It is generally recognized that competition is possible only in the commodities that can be traded (electricity and ancillary services). That it is not really very practical when applied to the basic electricity infrastructure, as Transmission and Distribution networks which enable the transport of energy from one place to another, take part in the electric price equation.
As a result, the markets have produced various models, which encourage competition in electricity generation and supply. Markets have developed controls to ensure that the owners and operators of the grid networks can be adequately regulated and compensated in their role as service providers to the industry.
There exists today, two principal market models, each with its many variants in different parts of the world. The first is a centrally controlled marketplace taking the form of a power pool, and the second is a distributed environment known as a bilateral market. Each model and its variants have both good and bad points.

Power Pool – A Centrally Managed Grid

A Power Pool is a structure whereby a centralized marketplace acts as the counterparty to all physical exchanges of energy between producers and consumers. In most markets of this form the Power Pool is mandatory. This means that there is no direct way to exchange energy between producers and consumers without involving the Pool. In general, the picture of a power pool is one in which all generators offer their energy to the pool, at various prices for various volumes and with different prices at different times of day. In some jurisdictions the System Operator and the Market Operator is the same entity.

The price determination methodology generates the Pool Price for any given trading interval. Arriving at this Pool Price is something that differentiates many of the pool models, as does the way in which the adjustments are dealt with.

The original form of Power Pool was predominantly a day-ahead marketplace. The modern pool form is one which operates close to real time and which co-optimizes energy and reserves taking account of constraints on the transmission system. This new model leaves the job of committing (starting up) generation to the generation owner, while providing the feedback on optimum paths that electricity must take to the Transmission Owner through nodal pricing. A nodal price is calculated similarly to the way it is done in the cost based pool model, but adjusted wherever there is a constraint in the network.

The figure shows the main nodes in Taiwan with the black arrows showing the general flow of electricity along the transmission networks. This is a fairly simple system to model with only 11 nodes identified.


Bilateral Market – Freedom to Choose

A Bilateral Market is one in which producers, traders and consumers can freely contract with each other to buy and sell energy. The contracts negotiated between each are private between the companies and need not relate to any price other than the price of energy one is prepared to pay to the other.

Since these contracts are private, however, there needs to be some means of handling the situation when producers and consumers do not behave in the way as their contracts dictate. There needs also to be some way of making sure that whoever did not fulfill their contractual obligations is the one who pays the cost of any changes in production or consumption of energy introduced to balance the wires.

The approach taken to dealing with this is to have some form of balancing market. This is a marketplace, which buys regulating or balancing energy from generators who are willing to sell it. It then pays these for their services, recharging the costs to the producers and consumers who were out of balance on their contractual positions. For this to work, the balancing market needs to have some information about which participant was out of balance.

An overview of a bilateral market is shown in the diagram above. This shows how energy flowing into and out of the grid from producers to consumers, is being kept in balance by the regulating energy provided in the balancing market. The green lines show the flow of money from consumers to producers for their contracted consumption, and from both to the balancing market to cover the costs of their imbalance.


Interestingly enough, in a bilateral market there is no real indicator of the current price of energy. This is unusual in markets where the spot price of the commodity normally forms the basis for all forms of futures and derivatives trades. (The pool form, described prior, automatically provides this indication through the pool price.)
In order that some sort of price discovery is possible, bilateral markets have encouraged the development of power exchanges. Analogous to stock markets, these provide a forum for trade between participants, which makes prices visible to the marketplace. Although these markets are also to be encouraged in a power pool environment they are pretty much essential in a bilateral market, if the requirements of the market to provide indicators for investment in the future are to be met.


A Power Exchange

A power exchange is a financial marketplace, which allows energy to be traded between generators, consumers, retailers, and energy traders. Participants in the exchange will buy and sell energy (often many times) into the future. They will either net off all trades (i.e. buy the same amount as they sell when the market comes to clear) or may be left with some real obligation to produce or consume. Most traders will wish to make money based on the different prices they can buy and sell energy by buying long and short depending on their view on whether prices are going to go up or down in the future.

Summary of the Market Types

Electricity markets have many forms, but they are all based around two principle types: A Power Pool, and A Bilateral Market.


A Power Pool is a centrally coordinated market where all power is traded via the pool. A single pool price provides an indicator of the price of power and provides a foundation for the growth of the financial and futures markets. A Power Pool may be of two forms: Cost Based or Price Based. In a cost based pool, the prices are set independent of consumer influence. In a Price Based pool, prices are set independent of demand forecasts. Bilateral contracts in a Power Pool are of a Contract for Differences (CfD) form.


A Bilateral Market is an infrastructure, which allows energy to be traded freely and privately in bilateral agreements between buyers and sellers. It provides the infrastructure to make sure the trades are fulfilled, and to make sure that there are enough services and balancing energy to make sure the Power System remains secure and that there is always sufficient supply available to meet the required demand. A bilateral marketplace on its own provides no energy price discovery or visibility.


A Power Pool with Nodal Pricing is an infrastructure which adds visibility of the value of the transmission network, and is able to respond quickly to change. Operating near to real time it allows generators to modify their operation in accordance with the demands on the system in fairly short timeframes. It takes account of the physical network infrastructure and negates the need for Power System Operators to vary dispatch from the network schedules under normal operating conditions. This provides some assurances that the market is not biased by human decision, unless need be to assure security of supply.


All three market forms can benefit from financial markets providing a forum for future and derivative energy trading (the Power Exchanges). The power exchange can provide both medium and long term price discovery in the market. This can enable arbitrage between fuel and electricity, and, more importantly, can serve as a primary vehicle to encourage longer term investment in plants (generators, etc.).


Market Considerations for the Future

There are advantages and disadvantages to both the power pool and bilateral market forms. Much depends on the detail of the market rules and the way charges are levied.


A Power Pool provides good price discovery, the prices, however, may not be truly cost reflective. In a cost based pool, it can be possible for large generators to more or less set the market price they desire. They can forecast the demand just as well as the organization running the market, and can adjust their bids so as to make the price of the market as high or as low as they would like to see. If they do behave like this, however, the market has very clear visibility of who did what, such that this type of behaviour can be challenged by regulation.


A Bilateral Marketplace provides no price visibility in itself, but can do so if financial markets are encouraged. This type of market provides no default automatic customer for the generation company, so that they have to find their customers and woo them to buy their product. In principle this would seem ideal, as it is the way other commodities operate across the world. The main problems, however, come in the way in which imbalance charges are levied. These can put a lot of risk on the retailer or large customer, if they are not able to factor them into their energy purchase prices. Most markets have a form that allows this, the new Electricity Trading Arrangements (NETA) Market of the UK, however, is fairly unique in being one of the few that does not.


Generally, a power pool provides a faster feedback of price signals to encourage the development of new generation than does a bilateral market. This is due to the fact that a new entrant to a pool can guarantee to sell all their energy to the pool, even if they have not enough bilateral contract customer cover. It can be argued, however, that financial futures markets can provide the same incentives.


One of the biggest differences between a power pool and bilateral market comes when trading across multiple control regions and between different countries. In this environment, a bilateral market is the easier to accommodate. A pool form can cause problems with this, as a generator has uncertainty in whether they will be allowed to produce against their schedule in a particular region, although they may not really care about the market prices therein if they are simply transporting energy across it to sell into a third party area.
The more advanced Security Constrained Centrally Dispatched Pool model (otherwise referred to as LMP or Locational Marginal Pricing) with nodal pricing, by nature of its accommodating transmission network constraints is capable of operating across multiple jurisdictions. It also relieves much of the difference between market schedules and actual operation thereby providing more transparent price signals.


In a highly interconnected A.C. network, which crosses multiple jurisdictions, a Bilateral Market Form is becoming the accepted norm in Mainland Europe. This is fine in a developed market with the size and depth to support it, but in the developing world, the LMP pool approach is a better means of ensuring fairness and attracting investment, while keeping prices visible and competitive.


On an island or a loosely connected region, a pool form may well be the most appropriate. This is consistent with the markets of Australia, New Zealand, and Singapore.
A modern nodal priced power pool can support both highly interconnected and island systems.
Whatever the market form, the development of Power Exchanges that provide future and derivative trading is to be encouraged.
Of Note:


  • None of the market forms in existence today adequately provide for maintaining a secure electricity transmission and distribution network.
  • None of the market forms in existence today adequately provide for maintenance of a secure plant margin.

So What do Smart Grids have to do with an Electricity Market Anyways?


In almost all jurisdictions where an electricity market has been established, the regulator played and continues to play a very important role. The only glaring exception was New Zealand, where the market was designed to run without regulator intervention and this caused major problems. We have seen that privately owned utility companies frequently design projects to improve supply efficiency and profits, and do so in the regulatory economic framework under which they are operating. In many cases where no electricity market has been established, the reality is that the current utility regulatory framework dates back to a period when we were in need to build the wires infrastructure. Policy objectives were quite different back then and, therefore do not address the needs pertaining to supply chain efficiency. It really doesn’t matter where you look. There is a now an increasing need to balance emissions and create an ecosystem that can evolve with the changing world around us, including security of supply.


As it was in the early days when the regulator had to take a proactive stance to achieve the market objectives, today the regulator also needs to take a proactive stance to achieve the outcome that is best for the people and business that they represent.


There are many considerations which link electricity markets to a Smart Grid. It is true that some Smart Grid initiatives, particularly those that benefit the utility, can be undertaken in the absence of a market and some benefits can be derived, but to fully realize the benefits that consumers expect and deserve, there is a fundamental linkage between a market and the Smart Grid. This linkage is price discovery. Some of the objectives of a Smart Grid centre around energy efficiency and the ability for consumers to change their usage patterns, as well as opportunities to use innovative solutions such as electric vehicles and self-generation technologies. Experience shows is that consumers are more likely to change their consumption patterns if they’re incentivised by a cost saving. Therefore consumers need to know exactly what the price of electricity is at any given time during the day, so they can make informed decisions on what appliances to use at any given time.


We are not advocating here that consumers continually monitor the price of electricity and adjust their consumption manually. The financial savings do not justify this level of attention. The same result however can be achieved, if the process is fully automated all the way from the real time pricing signals down to the appliance that is consuming the power. Innovation around the product and service offerings allowing for greater automation and control can be expected for consumers who do not want to engage further with their energy consumption, creating significant opportunities for suppliers to use their insight in marketing products and services directly to businesses and domestic consumers.


The level of change that is needed requires innovation, further research and development and the ability to form new partnerships that will transform the networks. Once policy-makers have delivered the mandate to change, regulators need to rapidly impose the framework and industry structure to support the transition. No doubt utilities and others will resist some of the changes at first, since the current model is that the more electricity they sell, the more money they make. This model is diametrically opposed to the requirements of energy efficiency and new ideas must be implemented to make sure that the utility can still make a reasonable return in the future. One example was enacted in California, where a “decoupling” policy was introduced to ensure that utilities retain their expected earnings while energy efficiency programmes reduce energy sales. Under decoupling, utilities capital and operational costs are accurately evaluated and a reasonable rate of return is agreed with the regulator.


What Needs To Be Done And Where Are We Today?


Even in jurisdictions where there is no competitive electricity market, we still need to be able to offer some kind of price discovery. We can do this by modelling the short run marginal cost of plants dispatched at any given time to calculate an average price of electricity. We also need to break the cycle where utilities design their projects to solely satisfy the current regulatory objectives to their own benefit. All stakeholders need to work together to gather sufficient quantitative and qualitative data to help build the case for rewarding all stakeholders given the business case that Smart Grid promises, not just to the utility shareholders or the customer base, but also to society as a whole.


In those jurisdictions where markets have been established with the consequential changes to the value chain, additional regulatory challenges arise. In such cases, the incentives for generation, transmission, distribution and retail participants change significantly and are now reflective of the frameworks that were put in place to encourage competition and market efficiency. With the introduction of smart technologies, the investment requirements, risk and reward distribution and incentives also change. If an electricity market and a holistic Smart Grid are simultaneously established in a particular jurisdiction, the changes for utilities become frightening and resistance is inevitable. Therefore, the regulator should work closely with all stakeholders affected to ensure that no one is unduly disadvantaged and no one is unduly advantaged. Fortunately, time is on the regulators side, since a competitive market will take a year or so to establish, but a Smart Grid vision will not be achieved for many years or even decades.


The same principles apply to the allocation of risk and reward in relation to the Smart Grid investments themselves. The regulators should play close attention to how and where the risks are managed, and how risk and reward are passed on to the consumer throughout the process.


Smart Grid projects should be conducted to optimize a broader set of societal benefits – e.g. carbon savings, job creation, etc. Some leading utilities are embracing this and starting to measure this broader suite of outcomes. They are also working to understand the true breadth of the beneficiaries, and assign the value accordingly. Others should be encouraged to do the same.


Parts of the regulatory frameworks are still valid, but some parts are contradictory and actually discourage the changes that are needed to transition the network towards a Smart Grid. For instance, many regulatory frameworks still allow utility earnings to the based on the volume of electricity consumed, despite trying to achieve energy conservation. However, in liberalized competitive markets, transitions are already understood to some degree and regulators have the ability and scope to factor in those measures that can make the Smart Grid achievable; for example, in the cost of carbon and changes to how utilities can expect a return on their investment. Other important factors include the visualisation of clear pricing signals and the ability for consumers to receive time of use tariffs. None of these are possible in fully regulated markets and therefore the Smart Grid vision will always fall short, even though it may have intentions of being started with technologies such as smart meters. Remember smart meters are only good if they can be utilized and the information can be assessed and responded to.


In the End


The Smart Grid is the conduit that facilitates change in the electricity supply chain and enables efficiency through integration. IT integration with the energy network dictates that, because of the sheer scale and scope of sensor deployment, new data analytics need to be applied to multiple infrastructure layers, such as the water and waste networks, transport networks, etc.
Competitive retail markets are particularly important to achieving a Smart Grid vision, as they allow for consumers to switch between suppliers, which in turn requires a more integrated solution to share information. This integration will encourage multiple data and communications standards that will open closed platforms. This “openness” of supplier product and service offerings will provide opportunity for Utilities, thus allowing the true opportunity for simplified integration between many… a Smart Grid. We need to see suppliers driving towards providing hardware and service bundles, and users of these solutions requiring change, so the market as a whole can evolve.


The regulators need to foster a framework to ensure they engage consumers in driving behavioural change. Therefore, there will be a need for new business models that create a “win-win” experience for the consumer and the businesses that are operating in the market today. This is easier to do in a deregulated retail environment. In all cases, it will be important to collect and analyse long-term consumption and price sensitivity data to empower consumers. This, in turn, will push new business models for low-carbon technology adoption, such as electric vehicles and micro-generation, which in turn assists us all globally.


Creating a market value for demand is essential and perhaps in the more enlightened environments “negawatts” can be aggregated and traded if the right market mechanisms are put in place, but this is in the hands of the regulators, who will need the right technical advice on how to achieve an outcome that assists us all.


Undeniably, in some jurisdictions, there are legislative barriers to fully implement a Smart Grid to the benefit of all stakeholders, not just the utility. It is imperative that policy-makers and legislators collaborate with industry to understand what needs to be done for the benefit of society, not just in the short term, but for decades to come.



July 17th, 2014


About Screaming Power Inc.

Screaming Power is revolutionizing customer engagement by providing a mobile platform that connects the energy user, allowing for effective and secure two-way communications to educate, change behaviour and encourage sustainability. Our extensible Intellectual Property provides a low-cost, digital infrastructure for a self-sustaining Eco-System. Our Scream Utility & Scream Enterprise mobile solutions focus on reducing ‘cost-to-service’ for utilities while driving satisfaction and facilitating the delivery of innovation (e.g., connectivity to the IoTs). 

Skip to content