So look, it’s the morning after the night before and I’ve decided to take down the links to the JSON and RSS feeds.
In general don’t feel it ‘wrong’ for someone to link to them directly, after all they were all listed in a javascript file on the BBC News Website (or derived by changing easily guessable paths in the URL). That’s how I discovered them, it’s been years since I had access to the BBC News Webservers so there was no ‘insider knowledge’ of paths etc.
BUT yes I do admit that because I personally know the licensing position of this data it wasn’t something that I should have done.
And despite the fact that I warned on the blog post that the data wasn’t licensed for off-site use, I take the point that it was little irresponsible to encourage people to do it anyway. Feel free to ridicule me on your blog posts, mash-up photos of me with egg on face, etc etc.
I do, however, still maintain that if I hadn’t posted the urls, it’s highly likely that someone else would have as they were put into the public ether by the BBC on their site.
But please, kids, don’t try those urls at home. They’re bad for you.
(BTW I decided to take this down after a night’s sleep on it, the BBC didn’t ask me to take it down)
Ben, congratulations on doing something that will almost certainly lead to those feeds being pulled, and possibly never reinstated. If you had one ounce of sense, you’d realise that that data isn’t likely to have been licensed for external use, and by exposing how to use the data publically it just means the BBC will have to pull it – or risk the wrath of the Met Office. Which means that not only will no one be able to use that data, but the feature that uses it on the BBC site will probably have to be removed.
Did you even bother to talk to anyone at the BBC about this before annoucing, via your blog and Backstage? It’s not like you don’t know who to talk to. Or are you simply trying to scupper all the good work that people at the BBC have gone to to try and get more things there opened up?
/golfclap.
As for “by all means make use of this data”… Ben, it isn’t your fucking data to grant permission for. Grow up. This isn’t fucking Star Wars, and you aren’t the rebel alliance.
Hey Ian.
‘Isn’t likely?’ Oh, I know that it’s definitely not licensed for external use at all. I’ve been to enough meetings about it.
Yeah, or maybe the BBC will negociate with the Met Office to let this data be publicly available for non-commercial use – seeing as it’s a dependency on getting this data out?
Nope, cos I learnt from a wise sage that sometimes ‘it’s better to seek forgiveness than seek permission’.
There’s an irony in there somewhere seeing as that’s kinda what I did at the BBC. I’m not saying that gives me carte blanche to do what I like on this front, but my point is I think I’m well placed to know the issues thanks.
Nice
Look Ian, there’s many reasons why I did this. For a start, I’ve learned that often you don’t get a shift in policy without nibbling a little at the edges of the existing boundaries.
Secondly, if I didn’t publicise this then someone else would have — I’m not actually mentioning anything that’s not public information here.
I’ll agree you on your point that it’s not my data to grant permission for – and so me saying “by all means make use of this data” is indeed stoking the fire a little. But I’ll live with the controversey. I’m not going to loose sleep over it.
“Oh, I know that it’s definitely not licensed for external use at all. I’ve been to enough meetings about it.
In which case you’re even more of an idiot than I assumed. I’m sorry I underestimated you. Oh and by the way, saying that “you’ve been to enough meetings about it” rather undermines your position that you had no inside knowledge. I’m sure that your ex-employer will be interested to hear that.
“Yeah, or maybe the BBC will negociate with the Met Office to let this data be publicly available for non-commercial use – seeing as it’s a dependency on getting this data out?”
Or maybe – much more likely – the Met Office will laughingly threaten to go to their lawyers if the BBC doesn’t cough up twice as much money as it was previously going to. I can only assume that you’ve never had to negotiate any kind of licensing deal, because if you had, you’d know that this would be the likely course of action for the Met Office.
“I’m not saying that gives me carte blanche to do what I like on this front, but my point is I think I’m well placed to know the issues thanks.”
Actually, carte blanche to do what the hell you like is exactly what you’re assuming. You’ve taken an API that wasn’t public, with data that isn’t yours, and – like a 14 year old discovering his daddy’s porn stash – spread it around amongst all your friends.
And if you were well placed to know the issues, the fact that you have simply ignored them in favour of acting like a child shows that you’re more interested in getting geek-cred than acting in any way responsibly.
“For a start, I’ve learned that often you don’t get a shift in policy without nibbling a little at the edges of the existing boundaries.”
Well that “learning” just shows how disconnected with the real world of big companies and business. You do realise that you’ve basically put the BBC over a barrell in its negotiations with the Met Office, don’t you? Do you think for one second that the Met Office won’t use this to squeeze more money out of the BBC? And, in case you’ve forgotten, that BBC money is license fee money – thanks for wasting it.
“Secondly, if I didn’t publicise this then someone else would have — I’m not actually mentioning anything that’s not public information here.”
That’s a bullshit excuse Ben, of the kind that 12 year olds come up with. The fact is that no one else DID release the info – you did. Stop trying to say that the consequences of this aren’t your responsibility, because they are.
“But I’ll live with the controversey. I’m not going to loose sleep over it.”
Precisely. YOU don’t have to loose sleep over it, because you don’t have to deal with the shit. You can just walk away, while others clean up after you, like the spoiled little baby you’ve proven yourself to be.
Yes it does. I decided to include that information to confirm your point. I don’t think I’ve really let out any top secret information on that one (the kind of stuff I was alluding to that wouldn’t be appropriate would be urls of other APIs that aren’t public behind passwords, etc).
…which actually leads me onto your next point:
Er, it was public — its on a public webserver!
My father never had a porn stash — well not one that I ever found. Did yours? (sorry, but it’s just a really weird/amsuing analagy you’ve chosen)
Er, do you know that I have a history of doing this?? It’s expected of me. Check out http://bbcnews.benmetcalfe.com/ – which was probably responsible for me getting in on the backstage project in the first place. There’s also the BBC News Stats, My scraped BBC News AV RSS feeds, and lots more
Mate, I have a history of doing this stuff. In fact this BBC Weather feed business is nothing compared to scraping a BBC site and reproducing my own RSS feeds on this site. But I look at my track record of using unconventional ways of getting this data out (officially), and I seem to have a pretty good one.
The BBC now offers AV console RSS feeds, and like I said much of my old work led to me helping to launch backstage (which led to all sorts of successes).
What does me a dis-service, and what perhaps you Ian don’t understand/apprecaite/etc is my dotBen alter-ego which is edgy, rebellious and what I guess you descibe as ‘childlike’.
It’s how I release this stuff, with an admitidly bit of spin that is geek-friendly. After all it’s these guys that actually go on to mashup it up and produce the innovative results.
With my ‘Ben Metcalfe’ hat on I can do ‘business’, I can do ‘serious’ and I can tell you why this is a very positive exercise.
For a start this data is already available via other sources – such as Weather.com and even the US govenment weather site (I forget the URL off the top of my head). There are plenty of good reasons to argue that this data should be made available by the Met Office for non-commercial use (which is also funded by my tax, to counter your point about the BBC paying for any additional use out of it’s licence fee).
Met Office are missing out on all sorts of innovation opportunies and commercial up-sell opportunieis if people produce commercially exploitable mashups with such data.
Secondly in my peronal opinion the way in which the BBC is using this data (via JSON) is exactly how it should – it’s industry standard stuff. If that means that others can now access this data on non-BBC sites then maybe that shows that the Met Office is not up to date with the environment they are offering their data into?
And finally, anyone who REALLY wants to access BBC Weather can scrape it off the site anyway – http://farm.webc.srv-1.rs.webcoding.co.uk/feeds/weather.stuff.htm.
What you are seeing here is the conventional meeting the un-conventional. My means are unusual and maybe odd to you but they work and I feel comfortable with them. I’m not doing anything illegal and I’m confident that there will be a positive conclusion to this.
And finally…
No, the reason why no one else released this information is because the feature only just came out and I noticed it/poked around with it first. It would have come out anyway even if I didn’t list in here. Trust me.
Now, Ben, have you started modding comments? One of mine certainly seems to have disappeared…
I haven’t modded any comments… ???
Hey Ben,
I am with you….
The BBC’s lawyers have had a shit load of time to get this sorted as with all the other rights issues. I have no idea who Ian is, but his points sound like some old geezer who wants to keep the status quo and “lets all make loads of money”.
How about reversing the issue and beginning on Backstage with some “fake” weather feeds….. at the same time put up a page asking people for their “own” weather feeds, in REAL time, like look out of the window…… if anything comes of the use of the data then “we” can sell it all to the bloody Met office, it isn’t as though their data is always right anyway…… and as for those maps on TV, god they suck after 600 years of the same presentation. 🙂
Ben – my comment must have got lost on the intarnet somewhere. No worries – shit happens.
Richard: I have no idea who the fuck you are either, but you sound like some kind of 11 year old who’s been told to go to bed early. What I care about – and what Ben apparently doesn’t – is that he’s basically shafted his former colleagues, who have continued to work to get this data released. He’s done nothing to get it released sooner, and he’s probably set back the general cause of Backstage within the BBC. Oh, and what you might understand if you bothered to address the issues – instead of waving the “I’m a big rebel, look at ME ME ME” flag – is that this will cost the BBC money, and add more to the coffers of the Met Office. So well done!
Now Ben, what I said in the missing comment was this: you have, essentially, just back-stabbed a lot of people who’ve been working both before and after you left to get this data released. You’re increased the cost to the BBC for getting the data released.
As you say, you were involved heavily in discussing how to get this data out. That means you have no excuse for not knowing that your actions will have exactly the reverse consequences that you’re trying to achieve. Which leads me to conclude that, in fact, you don’t give a shit about the openness of the data: What you give a shit about is that you – sorry, “dotBen, edgy anti-hero who’s a little bit ooh, a little bit arrr” – should get the “kudos” of “unearthing” them.
So you put getting some kind of geek cred above both your wider reputation in the business world – and trust me, it WILL be damaged rather than boosted – and any kind of trust that your former colleagues had in you. You know – the people who will have to clean up the shit you’ve made. You shit on other people – and you don’t care. Nice one.
Aaah Ian, you are so gracious.
Time will tell if the “cause” of Backstage has been damaged, I am sure that the BBC is old enough to look after itself.
If you read, you would understand that this “new” feed of data was on a public server…. therefore, if that contravenes the BBC’s contract with the Met then the responsibility lies with the very people that you are talking about, or whichever other department they are in.
I congratulate Ben wholeheartedly for pushing the buttons to make this known. I would also assume that he has some knowledge of how much it may cost the Beeb, do you have any idea….? Thought not.
The official reply on Backstage is sufficient to cover any further points that you may have, assuming you have no personal issues that are really behind your comments.
One wonders how old you must be to be able to spell internet….. oh well, good to meet you, I’m off for a swim.
From Kim Plowright, on the BBC Backstage mailing list: “The availability of the data without that key system has the potential to really sour the BBC’s relationship with the Met Office. It may well make negotiating further data releases really hard, both internally and with other BBC partners; and might jeopardise the whole backstage project. It’s bigger than ‘just backstage’, too – it’s all of the BBC’s weather service that could be affected. So – and I’m sorry to have to ask you this – please could you refrain from using the feeds below.”
No offence to the people jumping to Ben’s defence but it’s pretty clear he’s in the wrong on this one.
Tom, it’s only one person coming to my defence. But I thank you for using the word ‘people’.
Yes, I’m just as clear as Kim (esp now I’ve added additional disclaimer) that this data is not licenced for use on non-BBC sites.
I don’t think this post or anything else I’ve done here jepodises the BBC’s relationship with the Met Office, or the entire backstage project.
If I pointed to something libelous on your blog Tom, would that be my fault?
Anyway Tom, you love to have a dig at me we you can. Surely this is just your attempt for this month? They’ll be something else next month I’m sure…
Hi Tom,
I understand what you are trying to say, and IF another person had posted the same email on Backstage I doubt whether I would have read the comments above and replied. That said, the word “potential” means that it “may” sour the relationship….. in which case one may ask, how much would it sour the Beeb’s relationship had the data been stolen without your knowledge and used for financial gain, and once again in the future the Beeb would, or may, have been held responsible? Anyone could have done that.
If you read the original email on Backstage then you will see that many references are made to the fact that it was probably unintentional….. it can therefore be argued that this has done the Weather dept of the BBC a favour.
So how long does it take to decide on a key system, if in fact that is what will happen? I have watched Backstage for over a year and many manageable ideas have been suspended due to contractual issues.
Personally I have no interest in those feeds, but tomorrow you can call up the guys that made it available on public RSS under that contract and tell them, as the responsible parties, that they need to make some changes, you might even get some merit.
It would seem that this is more about personal issues, and I personally do not know Ben, so I will not act as judge and jury with my comments, which is why they are not posted on Backstage.
Rightly or wrongly, in anyone’s opinion, it is hysterical to see this issue, at this point, take on such a personal tone. After all the BBC is trying to be innovative through Backstage, and at the same time the prehistoric corporation stifles that same FREE potential.
Ben, I’m afraid that I’m inclined to believe Kim on this one. Things are often put up on the public internet under specific terms of use that limit how people can use them. Given that the data is licensed from a third party, it seems pretty obvious to me that the Met Office would have issues with it being turned into an unofficial API. Exposing it as such therefore seems to me to be a clumsy and potentially irresponsible move. Are you legally liable? Probably not. Is it a bright or good thing to do? Probably not.
To Richard – you’re right that the BBC is a slow organisation that needs to change, but it does also have to be responsible. If it signs contracts with third parties, then it needs to stick to them. The way to get data out in public through the BBC is to get serious buy-in from the top of the organisation, backed with negotiation and cash money. If that’s not happening, then clearly that’s a problem, but it’s not going to be resolved by these tactics.
Woe.
Richard, if you’d read the official reply on Backstage, you’d see it’s completely in agreement with all the points I’ve made – without the comments about Ben’s fundamental fucking-up. The whole “it was on a public web server” argument is, of course, a red herring: Ben published the feeds, not the BBC, and knowing full-well that it would cause the BBC a whole mess of shit to clear up. He did it purely for egotistical motives, not for the greater good – because if he’d been concerned for the greater good, he would have talked to his contacts within the BBC before posting, found out how far away from going public it was, and THEN decided if he should publish.
He didn’t. And that, to me, indicates he’s more concerned about getting kudos from a handful of people who regard themselves as “rebels” than doing any good.
Ian: was the swearing really necessary?
FWIW I saw the new feature on news.bbc this morning, used Firefox’s Live HTTP Headers extension and figured out the JSON feed with a total of about 10 seconds work. It’s not like you can keep a public feed like this secret, but of course the T&C still apply.
Leon: Yes, frankly, the swearing was necessary. I swear when I’m angry, and this made me f-ing angry 🙂 As Ben. I think, now agrees, there was a difference between (say) you publishing that information, not knowing what the consequences would be, and Ben doing the same thing. Ben knows, from this experience at the BBC, what a big issue it could be – and hence shouldn’t have done what he did.
Ben, you’ve done the right thing. I hope you learn something positive.
At the end of the day, you can easily copy music, and that’s illegal. So is using these feeds.
Ben, get a proper job or failing that go back to your porn stash. You’ve clearly got too much time on your hands!
Rog (who posts from ‘IP: 132.185.144.15 , gatef.mh.bbc.co.uk’):
’tis true. I have far too much time on my hands. It’s 14:15 and I’ve yet to get out of bed (other than to get a drink/use the toilet).
I’ve taken the summer off. Actually I’m super busy arranging visas and some other topic secret shit.
But yes, I have to admit I am vastly under-stimulated at the moment.
Tom,
We obviously differ, there is not much that is truly innovative in any sphere that does not break the rules somehow, sometimes it works, sometimes it doesn’t….. that doesn’t mean it is worth a red mist.
A long time ago the BBC and the Met office were both publically funded government “ideas”. As such they have both been very dynamic and have shown incredible speed with which to change to suit, and lead, the new world in whichever era. When they had to, or when they wanted to.. Backstage is in fact a very positive reaction to the speed of change needed within the world of the net.
It is my opinion that most contracts are only changed, or re-negociated under pressure from third parties, or as a knee jerk reaction to someone else’s better ideas or financial clout, as you say.
Sometimes it is a necessity to be a leader instead of towing the line….. as such, in this case, the email on Backstage exposes a potential, contractually irresponsible move by the BBC. If the same information had been posted only here on this blog, then I would agree that you would have a point, but it was not.
There is one large problem with your way to get data out, the top of the organisation do not always know which data is valuable and which is not, and whether it is actually being used against the general view of how it should be or not.
That said, the BBC is an innovator itself, and should lead with the power that it has. Yes, be responsible, but to everyone…. especially the public and the customer. They are the ones who ultimately decide what is and isn’t valuable. So you have to test the waters.
I don’t believe that it is possible to cite a commercial argument for “buy-ins” and “cash money” yet jump when “commercial tactics” are used against the corporation.
Auntie has a totally unique position, but complacency has always been rife. Please try to look at the reverse, the commercial world if you will, and you will see that these “tactics” are used every day, in my experience last nights scenario was very tame and in fact “helpful” in many ways. The fact that you have commented and somewhat exposed some of the current thinking being a massive plus.
I am very interested to know who or what is the “top of the organisation” in your view?
Perhaps they can read the following clause in the Mets contract and ask the Backstage community to become a slightly more internal entity instead of a totally public one….
Met Office License and proprietary rights
“5.1. We grant to you a worldwide non-exclusive licence. Where not expressly specified you may view, print, copy for internal use, distribute internally and archive the Data subject to the limitation specified in clause 9.5”, where 9.5 says the data must be destroyed after five years.
Or does the BBC also want every positive with none of the negociation….
Backstage Terms….
“You must:
Include the following attribution that you use content from BBC Backstage: “supported by backstage.bbc.co.uk”
API Terms….
5. Indemnity
You hereby agree to indemnify, hold harmless and defend the BBC in respect of all damages, costs and expenses, including reasonable legal fees and litigation expenses, arising out of or as a result of any breach by You of the terms of this Licence or otherwise in connection with your use of the API’s. ”
Looks like the Beeb is pretty safe within its own license and will stand to fight another day.
I don’t dispute that various people may have felt as though they had a serious situation to deal with, but the truth today is that there will be many positive potentials as well. I hope you all have a great weekend.
Richard, god knows I know what the BBC is like. I worked there doing R&D work for a couple of years and regularly found myself pushing against the organisation in places, and trying to get them to move forward in places which they might have found uncomfortable. But fundamentally, you have to either work with/within an organisation or take personal responsibility for breaking the rules. If you break the rules and do something awesome, then people can be extremely forgiving. If you break the rules and alienate a bunch of people and cause the organisation trouble then you have to expect to get into trouble or have people jump on you. It sort of stays like that after the fact – if you break the rules and the BBC ends up in a much better position, all its stakeholders (the British tax-payer) will thank you for it in the end. If you put them in a harder position, then you take responsibility for your actions. I’m not saying you don’t push the boundaries occasionally, just if you do so and you fuck up then you can’t really hold anyone to blame but yourself. In my opinion, Ben’s quite rightly recognised this move as an error and has backed away from it and apologised.
The license thing you quote, by the way, is to stop someone taking the BBC to court because the data that the BBC provided failed or was inaccurate in some way – ie. it’s to stop people relying on it to keep their belongings, friends, children or themselves safe. I don’t really understand what your argument was around this stuff, but it certainly won’t act as a sop to data providers or protect the BBC from lawsuits.
Looks like you are doing the right thing, all things considered. This sets you apart from most.