Tuesday, 29 November 2016

Doctor doctor! Shame on your dissemination and peer review!


A reader asks:

"You really believe 'any' of what's in the DOE's press releases about this ‘contest’? How naive are you? You have a PhD in this field!!

This 'contest' was just a PR stunt! It actually violated so many scientific protocols and standards that none of it would EVER pass a peer review. It didn't even make any common sense (at least not in the 21st century). This was a 1980's, or high-school contest, not something worthy of serious researchers! Also even a cursory examination of its goals and rules, would have made it fail the smell test! Far worse, however, was that this 'contest of skill' did not even abide by any of 50 state laws governing contests, games and lotteries, because the (privately contracted) organizers couldn't be bothered with the time (and expense) of consulting with attorneys specializing in this area of law (much less IP attorneys)!

So what are you talking about, Alexandra, giving this nonsensical 'contest' any credibility at all, given that it clearly does [sic] deserve it? You should know better than this. Shame on you.

The stated goals and 'minimum requirement' for winning a prize in this competition, was the achievement of 'at least a doubling' of current 'state-of-the-art' (i.e. worldwide) conversion efficiency. In some cases, that's already pretty high. How could it be possible for extremely underfunded entries to exceed, by a factor of more than two, the performance capabilities of technologies that have been researched for over 30 years and have access to millions of dollars in research money?! Wouldn't that be super-embarrassing for the DOE and all other researchers in the world? Many of the peer reviewed efforts of the last could [sic] of decades have had access to supercomputers and the latest, and greatest, in hydrodynamics modeling software (not the cheap, and as yet invalidated, stuff the DOE used)!

If what they claim has now been achieved (and it could pass peer review), then what we have here is not just a 'significant technology advance', it is a real 'revolutionary technology advance' and the top prize winners would easily be able to get funding to commercialize it, within 2-3 years (and start making billions)! Did you ever consider these implications for all three of the prize winning technologies (since they all would have beaten the current state-of-the-art technologies, by at least a factor of two)? This represents a 'quantum leap' in performance/cost, doesn't it? Puts to shame all prior efforts in the field, doesn't it? That's what the DOE has now claimed, anyway!

Also how could the top prize winner have achieved a 'quintupling' of current conversion efficiencies as was claimed during the prize awards ceremony, when this would seem to be a physical impossibility (unless you can't have > 100% efficiency)?! The really scary thing about all of this is that none of these claims seem to have bothered any of the 'experts' involved, or the DOE - including the undersecretary of the DOE! Where do they find these people, anyway? Hopefully, Trump, (who I did not vote for) will start cleaning house over there and will fire most if [sic] these incompetent people, who have never before done anything in the wave-energy field (especially nothing involving commercialization of these technologies)! I have a dog that knows more about wave-energy conversion that some of these people! Again, shame on you Alexandra. Not sure what getting that PhD accomplished. Seems to have cost you your critical thinking skills!

Anyway, stay tuned! The truth may well shock you (and the rest of the world), when it comes out next year! I looks like the DOE won't be getting the kind of publicity they were hoping for when they decided to 'stage' this so-called 'contest'. "


Critical thinking skills

I’m sharing the message above because it contains some points worthy of discussion. Also, I can’t but help finding the suggestion that I am not critical enough rather funny. Here was me dreading the inevitable scowls and grumblings of disapproval about my recent article on the Wave Energy Prize. This article contains the seeds of a technical discussion which some might interpret as critical, and as such puts myself at the mercy of others’ cynical guesses about my motives. The claim about my critical thinking skills is an excellent example of my next point …

You can’t please everybody

The Wave Energy Prize didn’t suit everyone’s interests. Everyone wants something different, so it was inevitable that it would displease someone. Had the money been spent in a way that met your approval, I’m sure it would have annoyed another bunch of people elsewhere. You seem to dislike the element of publicity and social engagement in this competition. Perhaps I could offer a suggestion as to why other people may have pushed for this to be an important outcome?

There is considerable concern about the lack of social engagement in the sciences. Academia is generally thought of as being too ivory towered, to the extent that funding mechanisms now encourage ‘dissemination’ (the use of a word that sounds like a slightly seedy medical procedure is indicative of academia’s isolation). Many experts believe that the secrecy surrounding commercial wave energy development has hampered progress and knowledge sharing. It’s not clear what your preferred funding mechanisms are, but I imagine they require a larger budget than was available for the Wave Energy Prize (I’m giving you the benefit of the doubt by assuming you are not from the anti-renewables lobby). As we have seen from the recent election, public spending to combat climate change is an extremely politically divisive topic in the US at the moment. Larger budgets require the interest, understanding and support of the public. A competition that has a broad appeal, that is presented as a ‘high school contest’ (more like a reality-tv elimination contest if you ask me) is a very clever way of reaching that portion of the public who feel excluded from, or bored by, engineering.

Another area where opinion might differ is the relative importance of spending money on lawyers to check state rules on lotteries, as opposed to the underfunded teams in this competition.

Just a PR-stunt?

While I agree with you that this competition had a lot to do with PR, I disagree with the word ‘just’. Successful technologies have come out of such competitions in the past, and there is no reason why this shouldn’t be the case for this competition. You are right to question whether there was enough expert input for this competition to be viable. I know that there were some very capable experts involved in advising, setting up, and judging the competition. I presume there must be a good reason for the DOE or these experts to keep this information off the front page of the prize website, because providing an accessible list of the experts involved would increase the credibility of the prize, and this is in line with the goal of raising the profile of wave energy.

Peer review

As for peer review of the competition, I’m sure that there are many like myself who had a good, long, peer at the competition rules (I would certainly recommend a ‘peer’ over a ‘cursory examination’). My personal opinion is that they were written with considerable insight and pragmatism. Far from putting to shame prior efforts in the field, the competition put lessons learnt from the field at the heart of the assessment procedure. Ideally you’d want to assess the levelised cost of energy, but this metric can’t be calculated with any meaningful accuracy at such an early stage and with so few tests. The metrics they used instead take into account a lot of things that were missed by prior efforts in the field – in particular the peak loads on the structure and power take off mechanism, which dictate how strong (and expensive) your design needs to be. The robustness of the assessment procedure is the main reason that I believe this contest has more credibility than its populist-entertainment appearance belies.

I think it is useful to discuss technical aspects among peers – it helps us all get closer to the truth, even if that only means the truth that truth is perceived differently. I am slightly concerned that your last paragraph suggests you are in the process of sharing your truth (that the competition was flawed) with the wider public, without really engaging in a peer review of the technical aspects. Surely you do not think that Trump would react to this by diverting funding from things like the Wave Energy Prize to funding marine renewables in the form you would prefer to see? The only thing this would achieve is to undo all the good work that the Wave Energy Prize has done to raise public support for wave energy. While I shall give you the benefit of the doubt, I would be naive not to question whether this is in fact your motivation, and I shall indeed stay tuned. If you are genuinely motivated by the desire to give wave energy the chance it needs to prove itself, please continue with this discussion. Let’s get clear on the technical aspects.

Assessment criteria

Like yourself, I would be very surprised if the winner had quintupled current conversion efficiencies. This is not because conversion efficiencies can’t be higher than 100% (it depends on how you measure efficiency of course, but point absorbers with reactive control can capture over 100% in some conditions, and that is the reason terminology has moved away from ‘efficiency’ towards ‘relative capture width’.) My surprise would come from my experience of how much hard work is involved in bumping the relative capture width up by just 10%.

However, this is not the claim being made. I’m not sure what technology was used as the benchmark, but I wouldn’t be surprised if it was a politically safe benchmark – the conversion efficiency of a technology widely agreed to be unviable. There are probably existing technologies not in the competition which already exceed the benchmark’s score.

The main point is they were not judging by conversion efficiency. They are taking into account indications of the cost of the structure, including the mass, structural loads and moorings loads. Questions 12 and 24 of the FAQs describe the metrics in more detail. These are excellent goal posts and I’m not surprised that four of the teams did better than the baseline. My guess is that the baseline technology was developed with quite different goals incentivising the process – goals that (pragmatically) had more to do with attracting investment than improving the cost of energy.

Yet I don’t consider prior efforts in the field to be embarrassing, despite the time and money spent, despite the access to supercomputers and expensive hydrodynamics software. Unfortunately we had to go out there and put things into the water to find out where the cost drivers were. Surely what would be embarrassing is to continue to do things the same way, without success?


Image credit:
Screenshot from ISIS TROLL DUCKY video by Vide0m0nkey: https://www.youtube.com/watch?v=b0h1nDfN6eU

6 comments:

  1. Ha, ha, ha!!! Not sure how to interpret this! Are you laughing at me, or with me? Kind of important to specify. :-)

    ReplyDelete
  2. What can I say, maybe I'm just the 'Serpico' of science and government B.S.! He didn't have many 'friends' either.

    ReplyDelete
  3. "However, this is not the claim being made. I’m not sure what technology was used as the benchmark, but I wouldn’t be surprised if it was a politically safe benchmark – the conversion efficiency of a technology widely agreed to be unviable."

    What are you talking about?! It was ADVERTISED that the results WOULD BE 'at least a doubling' of current 'state-of-the-art' capabilities - or NO prizes would given out - NONE! That high bar 'intimidated' many would-be participants. Wouldn't that high bar have intimidated you? What does 'STATE-OF-THE-ART' mean to you?! I sure know what it means! It doesn't come with any qualifiers (except independent validation). It isn't geographically limited. It isn't 'linear PTO limited'. It isn't 'passive limited'. It isn't just what a handful of (who are well paid) agreeing to say what it is! Again, do you really have a PhD in this field? :-)

    ReplyDelete
  4. Alexandra, have you ever heard of a contest that required its competitors to 'at least double' the best prior performance in the world, as a 'prerequisite' for winning ANY prize? Wouldn't that be kind of intimidating to any sane person? Wouldn't that intimidate even a scientist ;-)? Wouldn't that be intimidating to Dr. LaBonte and her entourage of like-minded 'yes-people'? Wouldn't that discourage lots of potential competitors from ever entering such a contest? Wouldn't that discourage some of the most talented people out there?

    Would setting the bar that high be more or less likely to increase the odds of finding a (true) winner that could achieve at least a (true) 30% improvement in performance? Would you enter such a contest (if you weren't absolutely desperate, or just greedy and/or delusional)? I wouldn't.

    ReplyDelete
  5. If you put up $6.5M dollars in taxpayer money for a 'contest', you better the hell know what you are doing and how a 'contest' is legally defined and supposed to work, in the country you are doing it in! The people behind 'FanDuel' and 'DraftKings' are now finding this out the hard way! Maybe the DOE and its associates need to get a lesson in this, as well! There are plenty of lawyers out there that would be happy to give them a lesson they would not soon forget. And teach them something about humility, while they're at it!

    What's that, Alexandra? Scientists don't need to trouble themselves with the law (except those of physics)? Laws are just for 'regular people' that aren't interested in doubling wave-energy output?

    ReplyDelete
  6. With regards to your 'attempted rebuttals' ... not sure where to even start! Will have to come back to this at a later date. Some of my comments may not even be allowed to be made at this time, for reasons that will become apparent at a later date.

    ReplyDelete

Comments