Cities and regions in the UK face ever-increasing economic, social and environmental challenges. They compete for investment in what is now a single global economy. Demographics are changing with more than 90% of the population now living in urban areas, and where the number of people aged over 65 will double to 19 million by 2050. The resources we consume are becoming more expensive, with cities especially vulnerable to disruptions in supply.

(Graphic of New York's ethnic diversity from Eric Fischer)

The concept of "Smarter systems" has captured the imagination of experts as an approach to turn these challenges into opportunities for more sustainable economic and social growth; particularly in cities, where most of us live and work. Smarter systems – in cities, transportation, government and industry –can analyse the vast amounts of data being generated around us to help make more informed decisions, operate more efficiently or even predict the future.

These systems enable city planners around the world to design urban environments that promote safety, community vitality and economic growth. They can bring real-time information together from city transportation, social media, emergency services and leisure facilities to better enable cities, such as Rio de Janeiro, to manage major public events. They can enable transport systems to better manage traffic flow and reduce congestion, as in Singapore. They can stimulate economic growth by enabling small businesses to better compete for business in collaboration with regional trading partners, in systems such as that operated by the University of Warwick.

Government policies such as Open Data, personal care budgets and open public services will dramatically increase the information available to citizens to help them take well-informed decisions. This information will be rich, complex and associated with caveats and conditions. Making it usable by the broad population is an immense challenge which will not be addressed by technology alone. Data needs not only to be made available, but understandable so that it can inform better decision-making.

Where does Smarter city data come from?

Raw data for Smarter systems is derived from three sources: the city's inhabitants, existing IT systems and readings from the physical environment.

Information from people has become more accessible with the continued spread of connected mobile devices, such as smartphones. Open Street Map, for example, provides a global mapping information service sourced from the activities of volunteers with portable satellite navigation devices. However, the quality and availability of crowd-sourced information depends on the availability and resources of volunteers, who cannot be held accountable for whether information is accurate, complete or up-to-date.

It is also important to understand data ownership and the associated privacy concerns. There is a difference between data freely and knowingly contributed by an individual for a specific purpose and information created as a side-effect of their activity – for example, the record of a person's movements created by the GPS sensor in their smartphone.

The Open Data movement, supported by central government, will dramatically increase the availability of data from public systems. For example, efforts are underway to make NHS healthcare data available, with appropriate security measures, to Life Sciences organisations to reinforce the UK's pre-eminent position in drug discovery research. However, the infrastructure required to make large volumes of data widely and rapidly available in a usable form will not be created for free. Until their cost is included in future government procurements – or until commercial systems of funding are created – then much data will likely only be made open on a more limited "best efforts" basis.

Furthermore, not all city data is held by public bodies. Many transportation and utility systems are owned and operated by the private sector, and it is not generally established what information they should make available, and how. Many Smarter city systems that use data from such sources are private partnerships rather than open systems.

Meanwhile, certain kinds of data are becoming far more accessible through the advancing ability of computer systems to understand human language. IBM's Watson computer demonstrated this recently by competing and winning against world champions in the American television quiz show, Jeopardy! Wellpoint is using this kind of technology to draw insight from medical information held in similar forms. Its aim is to better tackle diseases such as cancer by empowering physicians to rapidly evaluate potential diagnoses and explore the latest supporting medical evidence. Similar technology can draw insight from case notes in social care systems, as Medway Youth Trust is doing, or from the reports of engineers maintaining roads, sewers, and other city systems.

An early "mashup" application using open data from Chicago's police force

Information is also becoming more readily available from the physical environment. In Galway Bay, a network of underwater microphones is connected to a system that can identify and locate the sounds of dolphins and porpoises. Their location provides a dynamic indication of which parts of the Bay have the cleanest water. That information is made available to companies in the Bay to allow them to control their discharges of water; and to the fishing and leisure industries who are dependent on marine life. This Open Data approach is being used by cities across the world such as Dublin, Chicago and London as a resource for citizens and businesses.

Whilst advances in technology have lowered the cost of generating information from physical environments, challenges remain. From the perspective of a mobile telephone user, much of the UK has signal coverage. However, telephones are used one metre or more above ground level; at ground level, where many parts of our transport and utility infrastructures are located, coverage is much poorer. Additionally, mobile transmitters and receivers are relatively expensive and power-hungry. Cheaper, lower power technologies are needed to improve coverage, such as the "Weightless" standard being developed to use transmission bandwidth no-longer needed by analogue television.

Using and combining data appropriately

In order to make information from multiple sources available appropriately and usefully, several issues need to be tackled.

When computer systems are used to analyse information and take decisions, then the data formats and protocols used by those systems need to be matched. Information as simple as locations and dates may need to be converted between formats. At an engineering level, the protocols used to transmit data across cities using wired or wireless communications behave differently and require systems that integrate them.

The meaning of information from related sources also needs to be understood and adapted to context. Citizens who go shopping in wheelchairs need to know how to get between car-parks and shops with lifts, accessible public toilets and cash points. However, the computer systems of the organisations who own those facilities will encode the information separately, in ways that support their efficient management, not that support journey-planning between them.

The City of Portland in Oregon has gone further in a project to understand how information from systems across the city is related. They are now able to better predict the impact that key decisions will have on the entire city, years in advance.

Privacy and ownership of data may affect its subsequent use, often with terms and conditions in place for governing its access. Furthermore, safeguards are required to ensure that sensitive information cannot be inferred from a combination of sources. For example the location of a safe house or shelter being identified from building usage, building ownership and /or information concerning taxi journeys by the employees of particular council agencies.

The human dimension

Smarter systems will only succeed in improving cities if there is wide consumer engagement. To be of value, information will likely need to be timely and presented in a manner appropriate to consumer context. Individual behaviour will only change where personal value is derived as a result of new information being presented – a saving in time or money, or access to something of value to their family.

(Photo of traffic in Dhaka, Bangladesh, from Joisey Showa)

Many cities are experimenting with technologies that predict the future build up of traffic, by comparing real-time measurements to databases of past patterns of traffic flow. In Stockholm, this information is used by a road-use charging system that supports variable pricing. In California, commuters in a pilot project were given personalised predictions of their commuting time each day. Both systems encourage individuals to make choices based on new information.

Utility providers are exploring how information from smart meters can encourage water and energy users to change behaviour. A recent study in Dubuque, Iowa, showed that when householders were shown how their water usage compared to the average for their neighbours, they became better at conserving water – by fixing leaks, or using domestic appliances more efficiently. Skills across artistic and engineering disciplines are helping us understand how this type of information can be communicated more effectively. Many people will not want to study figures and charts on a smart meter or website; instead "ambient" information sources may be more effective – such as a glow-globe that changes colour from green to orange to red depending on household electricity use.

Systems that improve the sustainability of cities could also affect economic development. Lowering congestion through Smarter transportation schemes can improve productivity by reducing time lost by workers delayed by traffic. By making information and educational resources widely available, Smarter systems could improve access to opportunity across city communities. A city with vibrant communities of well-informed citizens may appear a more forward-looking and attractive place to live for educated professionals and, in turn, for businesses considering relocation. New York has improved its attractiveness since the 1970s by lowering the fear of crime. One of its tools is a "real-time crime centre" that brings information together from across the city in order to better react to crime and public order incidents. The system can even help to prevent crime by intelligently deploying police resources to the areas most likely to experience incidents based on past patterns of activity – on days with similar weather, transportation conditions or public events.

Success in delivering against these broader objectives is much more likely to be achieved where the cities themselves are more clearly accountable for them.

So where do we start?

Investments in Smarter systems often cut across organisations and budgets and many have objectives that are macro-economic, social and environmental, as well as financial. As such, they challenge existing accounting mechanisms. Whilst central government and the financial markets offer new investment solutions such as ethical funds, social impact bonds and city deals, so far these have not been used to fund the majority of Smarter solutions – many of which are supported by research programmes. The Technology Strategy Board's investment in areas such as "Future Cities" and the "Connected Digital Economy" will provide a tremendous boost, but there is much to be done to assist cities in using new investment sources to fund Smarter initiatives – or to develop sustainable commercial or social-enterprise business models to deliver them.

Although progress can be driven by strong leadership, the issues of governance and fragmented budgets will need to be overcome if we are to take full advantage of the benefits technology can bring.

We live in an era of major global challenges – well described in the recent "People and the Planet" report by the Royal Society. At the same time, we have access to powerful new technologies and ideas to address them, such as those proposed by the 100 Academics who contributed essays to the book "The New Optimists". When we focus those resources on cities, we focus on the structures in which we can have the greatest impact on the most people.

Already many forward-looking cities in the UK such as Sunderland and Birmingham are joining others around the world by investing in Smarter systems. If we can meet the technical, organisational and investment challenges, we will not only provide citizens, businesses and agencies with new choices and exciting opportunities; we'll also position the UK economy to succeed as the Information Revolution gathers pace.

(This post was first published as part of the "Growth Factory" report from the thinktank TLG Lab).