Why train departure information is not currently open data

Access to train departure information

Users of the London DataStore will have noticed that there is no train departure information available as open data, and the issue generates a lot of questions from developers. This blog post attempts to explain why this is so and to let you know what the London DataStore has being doing about this in the meantime.

The short answer to the question is that the Association of Train Operating Companies (ATOC), the only current player with a public-facing train departure information service and API, is a private sector body that does not release open data. ATOC’s National Rail Enquiries offers an API to its Live Departure Boards on a commercial basis and this is available to developers who meet their criteria and who are willing to pay for access. ATOC have granted some free licences to those not making any revenue from their service e.g. LiveTrains, so it may be worth applying to them if you fall into this category. There are also lots of commercial apps licensing data from ATOC, for example, myTrains for iPhone.

However, given that the UK taxpayer subsidises the rail industry to the tune of £5bn a year, there are many open data campaigners who would like get access to a genuinely free source of train departure information. If this applies to you, and you’d like to read the whole story, make a cup of tea, draw up a comfy chair, and read on for the full story.

Train departure information, simplified

Information on train departures in Britain is largely generated by Network Rail (the rail infrastructure owner) from a variety of signalling apparatus and train reporting services. This train departure information is then currently exclusively sub-licenced to ATOC for their National Rail Enquiries (NRE) service on phone and web for users outside the rail industry. NRE charges app developers or website owners for use of this data from Network Rail and imposes its own licence conditions under a code of practice approved by the Office of Rail Regulation (ORR). The net effect of this regime is that developers have to charge relatively high fees for apps using this data, up to £5 per app in the cases of UK Train times. The NRE licence conditions also prohibit developers from being critical of the rail industry or having an adverse effect on TOCs. Paragraph 2 of the licence says “Applications which in NRE’s reasonable opinion are of demonstrable benefit to passengers will be granted unless outweighed by a material adverse impact on TOCs (whether financially, strategically, operationally or in regards to their reputation or the reputation of the industry as a whole).” (Para 2).

Although there are regulators like the ORR and Passenger Focus that do valuable work, currently these bodies see the rail industry as solving the passenger information problem within the industry. Recently the ORR published a Passenger Information Consultation
(PDF)
that is open until 20th June for those that would like to put their own views across.

Many developers would like the opportunity to innovate with train departure data to create apps and web sites that provide alternative views on this data, for example, local public transport aggregation sites, novel visualisations, delay monitors or cheapest fare finders. Developers believe that these kind of services are a powerful voice for the consumer. So why can’t developers produce these free apps? It’s all down to the post-privatisation structure of the rail industry.

Who owns information about rail services in Britain?

The taxpayer does not own this information directly despite the public funding. Network Rail (not ‘National Rail’… that is an ATOC brand for train services) is a private company limited by guarantee that carries out publicly regulated tasks i.e. running the railway. It is a private company that can make profits… but since taxpayers are providing much of the income and also the financial guarantee, there is some public confusion about whether Network Rail belongs in the Public sector. So, for example, the National Audit Office thinks it is a public body. The Information Commissioner has ruled that Network Rail is covered by the Environmental Information Regulations for public bodies (PDF) as it has a public task. Network Rail is also licenced and regulated by a statutory body, the Office for Rail Regulation. And Network Rail gets much of its money from two public sources 1. the ‘Network grant’ (around £5bn annually from the taxpayer) and from 2. Train Operating Companies who get public subsidies of £450m per year (PDF)) (see the Network Rail company report and accounts, note 3). So, Network Rail is a private body carrying out a regulated public task. It could suddenly become a public body if it defaulted on its debts as its predecessor ‘Railtrack’ did in 2002).

As a private body Network Rail does not have to follow the rules of the public sector regarding transparency in data, despite its public task. The government does not exercise its right to appoint a Director of Network Rail. Network Rail does not have to answer Freedom of Information requests (PDF). It is being left out of the Protection of Freedoms Bill and the Public Data Corporation in legislation this year. Public input to Network Rail is via the 100 ‘Members’ drawn from the public and the rail industry who act as stakeholders in holding the Board of Directors to account, rather like governors at a school. Although the appointment of members is, in principle, independent, the Board of Network Rail “will not, in particular, appoint individuals whom it feels wish to pursue concerns or objectives which are inconsistent with the overall purpose of the company.” Ultimately, therefore, Network Rail is able to follow its commercial interests when deciding how to license train departure information collected with public support. Note that Network Rail is required by its licence from ORR to improve reliability and efficiency, but not transparency.

How the public becomes private

At present Network Rail exclusively licences train departure information outside the rail industry to the Association of Train Operating Companies (ATOC) for their National Train Enquiries (NRE) service based around a database called Darwin. NRE integrate several sources of data from Network Rail and some from train operating companies to produce the Live Departure Boards web site and data feeds for apps (developers can read more below on how this is done). This is a non-trivial task, but not impossible for others to replicate, as for example, Rockshore do for Network Rail. ATOC make charges for access to these data feeds and limit how they can be used with specific licence conditions. So, in effect, information that has been substantially funded by the public is now mixed with intellectual property from a fully private company ATOC, a situation that bodies such as the Open Rights Group have been concerned about in the public sector as a whole.

Going back in history, until February 2009 ATOC licensed train departure information under commercial terms to a very small number of organisations, mostly within the rail industry. Kizoom published the only smartphone app at that time, the free MyRailLite for iPhone. Then a dispute arose between ATOC and Kizoom, and ATOC withdrew Kizoom’s licence to use the train departure information. Kizoom complained to the ORR, who conducted an investigation (PDF) into whether ATOC had abused a dominant position under competition law. ORR decided that ATOC did have a dominant position in the supply of train departure information, but they “found no evidence that ATOC’s conduct in granting access to Darwin had prevented a new product from coming to market or hampered the emergence of new technology” in November 2009. When the free MyRailLite from Kizoom was taken off the market, it was immediately replaced by a £5 iPhone app from Agant which was marketed under the National Rail Enquiries brand.

After the Kizoom case, to regulate the dominant position of ATOC in this respect, the ORR asked ATOC to produce a Code of Practice on data licensing, and this was introduced in April 2010. Despite this Code, disputes over the licensing of train departure information are still occurring as independent developer Alex Hewson recently found out. He asked for a free licence and then published the text of the refusal, to find himself banned from getting a licence even if he paid, as he was publicly critical of ATOC and NRE, and they deemed this a prima facie breach of the Code.

The current situation… and where next?

What we see in the current situation is public funding going into the creation of train departure information at the level of Network Rail infrastructure and in the public subsidies to the Train Operating Companies. However, as two private bodies have responsibility for the public task of running the railways and communicating to passengers, this information is encumbered by private intellectual property rights. However, it would serve the transparency and accountability agenda if the raw feeds could be released as open data. The way forward might be to open access to Network Rail’s TDNet through their External Services Gateway (ESG), allowing independent developers and system integrators to add value to raw data and make apps to communicate with the rail traveller (see developer section below). This would allow developers the choice of paying ATOC for access to NRE or accessing raw data via TDNet and building their own services.

This issue needs urgent attention at the time that the Public Data Corporation is being designed and ORR are consulting on a Passenger Information Consultation. Train departure information is key national dataset and a way needs to be found to make it available to developers to produce apps for the public and accountability for the regulators. If we need an example of good practice to motivate this, we should look at the example of the non-profit national bus departures aggregator Traveline who have announced that after a small connection fee, access to their national Nextbuses API is now free for developers to use to create free apps (like UK TravelOptions) up to a negotiated hit limit. This has created a situation where train departures are charged for and bus departures are free, even though both industries have a public/ private structure.

In summary then, developers looking for access to open data on train departures must put their faith in Network Rail and the Office for Rail Regulation to enlarge the scope their vision for passenger information dissemination. Users of the London DataStore could play an important role in publishing passenger information, and we would like to see existing channels by which the rail industry publishes data internally (e.g. TD.Net through ESG) be opened up to external developers.

Even more details… for developers

Network Rail and ATOC have both developed sophisticated back office systems to handle train departure information. Comprehensive details are given in a report entitled ‘Integrated Passenger Information: Delivering the Rail “End to End” Journey’ by Aecom commissioned by Department for Transport Rail Group. The Stage 3 technical Annex (PDF) is 80 pages of detail about internal Network Rail systems for the serious geek.

In summary, train departure information from timetables (Train Service Database – TSDB) and train describer information are aggregated into an internal Network Rail system called Control Centre of the Future (CCF). Note that some sections of line do not have train describers and need lower level systems to provide train locations, so there is not complete uniformity across the network. Train delays are recorded into a system called TRUST to ensure that train or freight operating companies (TOCs/FOCs) get charged if they are the cause of any delays. Data from CCF and TRUST are available to users in the rail industry through the Network Rail messaging service known as TD.Net via Network Rail’s External Services Gateway (ESG). Train Operating Companies have their own Customer Information Systems that integrate data from their operations, notably a messaging service on day-to-day operations (e.g. cancellations) called Tyrell and information on the formation of trains from a rolling stock system called GEMINI. Any developer building train departure services would need to look at what they could build just with access to TD.net. It remains to be seen what information on cancellations and train formation TOC’s might make available to independent developers.

Network Rail is now investing in some new systems to improve this complex set of legacy technologies including GSM (R) for communication with trains and GPS for train positioning which will be integrated into its Intelligent Traffic Management (ITM) strategy. As the AECOM report points out “ITM could provide train location information to a far greater granularity, improving the accuracy of train running information for passenger information systems. This information could then be accessed by 3rd party systems.” (p13). As passengers already use smartphones with location services to report delays through crowd-sourcing services like @UKtrains using Twitter, in future passenger information from the industry will have to meet higher specifications to meet passenger expectations. This is a further driver to add to the expectations of openness for taxpayer funded public tasks.

20 thoughts on “Why train departure information is not currently open data

  1. Ben Smith

    I’m sad to announce that now the ‘dodge’ of using the BBC’s excellent TPEGml feed of travel disruption is no longer workable (as BBC Backstage project shuts down) I’ve had to shut-down @uktrains – I’ve written it up at http://uktra.in. The service used both ‘official’ sources and crowd-sourcing, but the volumes of crowd-sourced data weren’t enough to provide a useful (national) service on their own.

    Thanks for providing the detail in this post – it explains why (frustratingly) access to open data on this sort of stuff is so hard to come-by. And whilst I applaud ATOC (through National Rail Enquiries) moves into Twitter (and even free route-specific disruption alerts via DM now) a far better service could be provided by letting others do the innovation around how to deliver this data to consumers.

  2. Martin McDonald

    Excellent piece! Thanks so much for putting the full, complex story together.

  3. Bryce

    It is misleading to say that Network Rail is a “private company that can make profits”. It is 100% owned by the government, and thus NOT private by any reasonable definition.

    Whist any entity can make a surplus (aka profit), Network Rail is specifically a not-for-dividend entity, meaning that any surplus is retained and reinvested in itself, rather than paid out to its only shareholder – ie, us!

  4. Andy Mabbett

    I’m the ‘Roads and Transportation’ editor for the Make a Difference With Data Project: http://www.madwdata.org.uk/ and have featured this excellent post there. Your participation on the site, and the forthcoming on-line conference (date and time tbc), would be welcome – that’s an open invitation.

  5. Jonathan Raper Post author

    Thanks for linking to the post. Happy to help with madwdata any way I can.

  6. Jonathan Raper Post author

    Network Rail say on their own website “as a company limited by guarantee, we are a private company operating as a commercial business” (the link is in the post). Have a look at the Network Rail Report and Accounts on p16 where they give details of their 2010 profits of £400m, which as you say they retain to discharge their ongoing rail infrastructure management role. The reality is that Network Rail is private entity, publicly funded and regulated… but we cannot tell it what to do except through high level planning and long term funding settlements… especially as the government does not take up its option to appoint a Director.

  7. Alex

    I agree, train departure information needs to be open. Serve the country not the commercial interests of Network Rail. I can only see improvements and interesting visualisations and applications if it were made available. Thanks writing, it was very clear and balanced.

  8. Tim Johnson

    Great post! As a member of the team formerly known as Kizoom (now part of Trapeze Group) I have to say how encouraging it has been to work with Traveline on their very sensible NextBuses project. I despair at National Rail Enquiries’ continuing blindness to the inevitable trajectory of the industry and apparent disinterest in being enablers of innovation for passengers.

  9. Jonathan Raper Post author

    I have received an email from Chris Scoggins of NRE, attaching a Briefing on NRE publication of rail data. The whole communication is marked Confidential, and it would be a breach of common law of confidence to re-publish it without permission. I have therefore sought permission to publish: we will see if this is granted. Meanwhile I am free to publish my side of the correspondence: here is my reply…

    Chris,

    Thanks for your email and a copy of your Briefing. As your email footer requests that I treat your communication as confidential, can you tell me if you are intending to release it to a wider audience (if so, how?) or if you will allow me to circulate it further myself?

    Your Briefing makes some helpful points about NRE policies and the constraints it operates under. I think I have fairly conveyed these in my blog posting today. I am running a private company myself and I have been concerned to ensure that developers better understand the current commercial and regulatory environment in which train information is provided. I would welcome your acknowledgment that I am accurately reporting the current situation, and that your remarks about ‘misconceptions’ are not directed at me. You were provided with a copy of my blog posting in advance in case you wished to make observations, and I did not publish until I had received a reply from your Director of Corporate Affairs.

    You might also wish to reconsider some of the comments you end your briefing with when you make a comparison with TfL. The decision to open transport data was taken by the Mayor of London to increase accountability/ transparency and to encourage new startups to create new jobs in London around the data releases. All of the GLA data releases have been made on a marginal cost basis as the data feeds already exist to support internal business processes, and distribution is being undertaken by Microsoft. When you say “it seems illogical to us that all London taxpayers are picking up the cost of data services for people with high-end phones as well as boosting the profits of some developers”, I’m not sure you have grasped the objective of the Mayor’s policy. Since January 2010, or very little public expenditure, many new information services have been created as well as some new businesses that have created jobs and pay new taxes. Independent statistics have also been produced during industrial action on the tube. You can see the successful outcomes of this policy in the wide range of uses for the released data in the Inspirational Uses section of the London DataStore, which are accessible on web and a wide range of mobile devices.

    We would like to promote a debate about the provision of passenger information to ensure that the Office of Rail Regulation receives a wide range of views for its current consultation and it’s review of the Code of Practice governing NRE services. I think it would be productive for NRE to put across its position and hear the views of all parties interested in the provision of rail data through some appropriate forum. If you were willing to join in some event with balanced representation, we would welcome the opportunity to explore these points further.

    Best wishes

    Jonathan

  10. Pascal

    I had requested licencing cost for a free android app with no revenue (no ads or whatever) and was told “If you wish to proceed in applying for a licence you should be aware that there are minimum annual charges for maintaining the service (£1,000).”…
    So I gave up…

  11. Brianary

    As an outsider (in the US), I have to say that a public transportation service, which survives primarily based on access to scheduling data, hiring an agency which profits from restricting access to that data, sounds like a conflict of interest.

  12. Prashant Gandhi

    Reading between the lines on your response to Chris, it is quite clear that he doesn’t share your viewpoint about open data. Why would he ? His bio proudly reads that “he has created a valuable commercial asset and driven sales opportunities to £450m p.a.”

    Waiting to see how this develops further.

  13. Chris Puttick

    I think the bit for me that underlines just how far out of touch these guys are is the quoted line regarding “people with high-end phones”. This should be translated as “shiny phone looks better than my phone no fair” as high-end would imply expensive, exclusive; but a quick look at the price of Android phones on a retailer’s website gets me an entry level price of £100 SIM-free, and an entry contract level of £10/month. And then Android phones are out-selling everything else in the UK, so not so exclusive. Hardly high-end at all in fact.

    And just to make it clear. I’m a rail user and taxpayer. I know the data has to be gathered anyway, so the cost has already been incurred. I demand that it be made freely available without delay so I can make better use of my time. Come to think about it, I could write a letter to my MP (Witney and West Oxfordshire) about the matter, he has some influence over stuff I understand ;)

  14. Jim

    Separate consultation launched today on this by the ORR; different to the one listed above…

    ORR launches review of access to Real Time Train Information

    17 May 2011

    The Office of Rail Regulation (ORR) has today launched a review of third party access to Real Time Train Information (RTTI).

    In April 2010, the Association of Train Operating Companies through its subsidiary National Rail Enquiries published a code of practice for access to its RTTI database.

    ORR now wants to hear from all interested parties for their views on current arrangements – particularly those who have sought to gain access to RTTI since the code was published.

    The regulator wants to ensure that the process is transparent, open and encouraging effective market competition.

    For more information about ORR’s RTTI review and to find out how to contribute, visit: http://www.rail-reg.gov.uk/server/show/nav.2589

  15. Paul

    Thanks for this analysis.

    I have found NRE’s attitude frustrating, where they seem to put making money out of selling information ahead of making sure the travelling public have easy access to this information. A particular case is the journey planner in Google Maps which is easy to use and available on most smartphones and through desktop browsers, but at the moment can only do public transport planning for buses in a limited number of regions of the country (data supplied by The Traveline) but has no rail information at all.

    So passengers suffer with inferior information while presumably ATOC/NRE plays games trying to sell data.

  16. Alastair Dalby

    The train companies’ position is absolutely unsupportable – and am pleased that poacher turned gamekeeper, Nick Illsley, former MD of National Rail Enquiries, is now forcing this through from his current position at the Department for Transport. He must have knowledge and access to information that can force these profiteering monopolists to give us, the people, the information that is rightfully ours.

    Looking at the ORR’s investigation into NRE’s dominance, it is evident that the majority of data that forms the Darwin data feed for NRE is actually owned by Network Rail – which I believe is owned wholly by the government, despite being a nominally private company. Can’t we also ensure that Network Rail data is wholly publically owned?

  17. Mike_C

    Just for the record, things have moved on significantly so far as Network Rail train movement data is concerened. It has in fact been available to subscribers since 2007 but more recently has been formalised in a new Open Data portal hosted by Rockshire on behalf of Network Rail. This provised access to data feeds covering train describer movement data, TRUST movement data and a few other data classes. See https://datafeeds.networkrail.co.uk/ntrod/login for more details and to sign-up.

    Hope this helps.

Leave a Reply

Your email address will not be published.

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>