The need for citizen science in the transition to a sustainable peer-to-peer-society

Futures Volume 91, August 2017, Pages 46-52 Original research article Author links open overlay panelDianaWildschut https://doi.org/10.1016/j.futures.2016.11.010 Get rights and content Under a Creative Commons license open access Highlights • Society is moving towards a peer-to-peer society. • Citizens become the new decision makers. • Scientists should produce knowledge for citizens. • Citizen science is needed to make the sustainable transition happen. Abstract Society is changing towards a peer-to-peer society that is characterised by a new way to produce things, ranging from software to food, to cities, to scientific knowledge. This requires a new role for science. Instead of focusing on knowledge production for NGO's, governments and business, scientists should become aware that the citizen will be the new decision-maker in a future peer-to-peer (p2p) society, and produce suitable and accessible knowledge, and work together with citizen scientists. Citizens have proven to be capable of asking their own research questions, setting up their own projects, educating themselves and managing complex projects. It is time for them to get taken seriously. There is still a large gap between citizens and academic scientists when it comes to knowledge sharing. Knowledge collected by citizens hardly ever gets used by academic scientists, and outcomes from scientific research are still for a large part hidden behind pay walls. If we want the sustainable transition to take place, we need to open up the barriers between science and the public. The concerns of citizens have to be taken seriously and their knowledge used and valued. 1. Introduction Society is amid a transition towards a peer-to-peer (p2p) society that is characterised by a new way of produce of everything, ranging from software to food, to cities, to scientific knowledge (Bauwens 2012). In a p2p society, networks of individuals, or peers, take over tasks that were previously in the hands of institutions. This requires a new role for science. Instead of focussing on knowledge production for NGO's, governments and business, scientists should become aware that the citizen might be the new decision-maker in a future p2p society, and produce knowledge that is suitable and accessible to them. In this essay, when I talk about citizens I mean non professionals who engage with their surroundings, which could be a city, a street, the environment, the democratic system etc. When I talk about citizen scientists I mean people who practise science outside of an academic or industrial environment. Citizen scientists may or may not have had any formal scientific training. My definition of a neoliberal system is a system with a free market, small government in a globalised world. Looking around us we see old structures fail. The effectiveness of NGOs is doubted (Edwards & Hulme, 1996) and they seem to be mainly after their own survival. They are stuck to the methods of the 80′s; they dress up as pigs or bees, or wear gas masks and show up in a small unimpressive group at company or government headquarters. Their causes are legitimate and the problems they put on the agenda are urgent, but their methods are completely unfit for the world we live in. The focus of neoliberal governments is no longer on the well being of their citizens, it has long shifted towards concern for corporate interests (Chomsky, 1999; von Werlhof, 2008). Long term interests, including science, suffer (Saltelli & Giampietro, 2017). Corporations have no legal duty to act in the public interest, and no tendency to do so if that conflicts with short term economic interests (Banerjee, 2008; Chomsky, 1999). The future of the planet could lie in the hands of small groups of citizens and individuals, who are building a new society next to the existing one by setting up grassroots sustainability initiatives that are interconnected by p2p networks. Already, science is providing technologies that give people the power to do their own research, using their smart phones, webcams and laptops (Bonney et al., 2014). But here is still a large gap between citizens and academic scientists when it comes to knowledge sharing. Knowledge collected by citizens hardly ever gets used by academic scientists, and outcomes from scientific research are still for a large part hidden behind pay walls. If we want the sustainable transition to take place, we need to open up the barriers between science and the public. The concerns of citizens have to be taken seriously and their knowledge used and valued (Irwin, 1995). In this essay I argue that citizens are ready to do sound science and to use and understand science produced by academics. They are ready to use science to jump into some of the gaps that governments, businesses and NGOs do not fill. 2. Citizen science In the early days of the scientific revolution, any interested person with time to spare could look at the stars, gather data and make predictions. They did not need equipment, nor a lot of mathematical skills. Scientists were often also farmers, artists or statesmen. They were generalists, rather than specialists. Since the second half of the 19th century, science has become a practise for specialists, using state-of-the-art equipment and mathematics, requiring many years of university studies. Science was no longer something anybody could engage in (Vermij, 2006). In the last decades, computers have become cheap and powerful. The internet enabled people to get access to data as well as training. The development of cheap programmable micro-controllers has enabled many lay men to develop their own electronics. The popularity of these micro-controllers can mainly be attributed to Arduino, a cheap, open source micro controller board with an online programming environment. Programming them is easy because of the large, community run database of examples, tutorials and code. The boards can be connected to a circuit, the schematics of which can be found online. The Arduino community maintains an excellent collection of knowledge for beginners as well as experts on www.arduino.cc. The availability of cheap sensors is the next problem to be solved. The use of sensors combined with ATMegas (the micro controllers used in Arduino boards) is very popular. People measure whatever they can think of in their environment. They typically log all their data and look for opportunities to compare their data with other people's. This creates a demand for online open data sharing platforms. Communities form, not only around the hardware, but also around research topics. Examples are spectralworkbench.org, where people share the spectra they take of different light sources. They use a spectrometer made of a piece of cardboard and a piece of a CD (Fig. 1) connected to a webcam. They upload their spectra to the database, which contains results from both home made spectrographs and professional ones. It is a mix of useful spectra and blurry images. The database gets filled mostly by curious hobbyists, and as it grows it becomes a valuable resource for determination of chemicals. On the website's section ‘learn’ you can see what use is being made of the data. Citizens use it for measuring food contamination, GMO’s in food (Critical Art Ensemble, 2003), nitrogen levels in agricultural soil, detection of air pollution etc. It empowers them to be critical of the information they get from official sources, like governments and manufacturers. Fig. 1 Download high-res image (912KB)Download full-size image Fig. 1. A home made cardboard spectrograph, using a piece of a CD as grating, connected to a webcam. A recent development is the Lab-on-a-Chip, a tiny laboratory that can be used for fluid analyses. The Lab-on-a-Chip is low cost and easy to use. The purpose is to make a lab that is so user friendly that anybody can use it without much knowledge. For medical purposes, it enables patients to do their own in-vitro diagnostics. It can also shorten the time needed for decision-making because the patient does not have to wait for an appointment if he already has the lab in his home (von Lode, 2005). A Lab-on-a-Chip is a small piece of glass with tiny fluid channels engraved in it. It is mounted very close to a semi-conductor sensor, connected to a micro-controller that either does the analysis or sends the data to an online analysis tool. The recent boom of Fablabs, open workshops for digital fabrication which originated at MIT (Gershenfeld, 2012), have enabled citizen scientists to engrave their own Lab-on-a-Chip systems, by providing access to laser-cutters and 3D printers. Using Arduinos and open-source software for analysis, they can now do complex chemical fluid analyses themselves. These new technologies are now available for everyone, provided they are not afraid of technology. New tools often start out as toys for wizzkids and it takes a while before people start making them accessible for a larger audience by making user friendly interfaces. This process is already taking place and many tools are becoming useful to people who are interested in using them, rather than getting them to work. Not only the data collected by citizens but also their knowledge, intelligence, creativity and social networks can be valuable for academic scientists. In many citizen science projects run by academic scientists, use is made only of the participant's time or computer power. Usually the participant is only used for data collection or classification. The participant's intelligence or creativity is rarely needed and the project only offers very limited opportunities for personal growth (Rotman et al., 2012). Sometimes there is some education available where the participant learns to recognise patterns, species or types of galaxies, but he or she is hardly ever invited to participate in the design of the research. Citizens have valuable knowledge that is often out of reach for university scientists (see also Pereira & Saltelli, 2017). They have local knowledge, which Warren (Warren, 1991) describes as “Knowledge that is unique to a given culture or society. […] It is the basis for local level decision making in agriculture, health care, food preparation, education, natural resource management, and a host of other activities in rural communities.” But of course, not only in rural communities is local knowledge important. Citizens are much more capable of participation than they get a change to show (Fischer, 2000). Many local governments now realise that they need the input of their citizens to govern their cities to the satisfaction of their citizens. Starting to work with critical citizens is not an easy step to take, but once citizens are involved in producing the knowledge for decision making, they are more likely to accept the outcomes and the resulting measures. In the Dutch city of Amersfoort, the local government needed data about the impacts of climate change on a very local scale, where only low resolution data was available. After visiting a conference about citizen science, they decided not to give the job to a consultancy, but to find a group of interested citizens who wanted to investigate climate change in their neighbourhoods. The group was asked to set up the research, decide on what indicators should be measured and how, who to cooperate with and how to disseminate their results. They have now started their project, of which the results can be followed on www.meetjestad.net (Meet je Stad, 2015). Academic scientists see the advantages of cooperating with citizens, not only because of the qualitative value of their local knowledge, but also because of their extensive social networks. Active citizens know what is going on in their community, and they can tell what stakeholders should be involved. So they have to be involved in the very beginning of a research project, otherwise it is too late to make use of this part of their knowledge. Twenty years ago, Alan Irwin (Irwin, 1995) argued for the coming together of the worlds of academic science and citizen science. Now both the need and the possibilities have grown. Citizen scientists have more tools at their disposal to create knowledge with, the infrastructure is there to share the knowledge, and we have some big problems at our hands for which we need all the help we can get. Cash et al. (2003) conclude “[…] that efforts to mobilize science and technology for sustainability are more likely to be effective when they manage boundaries between knowledge and action in ways that simultaneously enhance the salience, credibility and legitimacy of the information they produce”. Many citizens are ready for action or already active. They need scientific knowledge. 3. Open access For this essay, I planned to use only references to open access articles, just to prove a point. It turned out to be impossible, which is a better proof than I would have liked. To be able to write an essay like this a citizen scientist would have to email friends who work for universities, use the Twitter hashtag #icanhazpdf and write emails to authors, hoping for their article in reply. Some of the closed access articles, like this one, are about citizen science and some of those could be interesting to citizens. But citizens who struggle to do their own no-budget science projects are unlikely to be able to pay $41.95 for an article that may or may not be of use. In most cases, science about citizen science will not reach citizens doing science or citizens in need of scientific input. In the p2p-society it is agreed on that whoever uses the services of another, if possible gives something in return to the community it feeds on. So studying citizen science and not sharing the results would be to violate these unwritten rules. Copyrights and patents hinder the availability and development of knowledge. It is often thought that copyrights are the only way to protect authors, but there are other ways to protect the rights of authors and developers of open source products. Some institutions develop their own licenses for open source software (MIT Public License, 1988) or hardware (CERN Open Hardware License, 2011). These institutions deploy their knowledge of copyright and ownership to help people who want to add their data, ideas, designs and products to the public domain. Once there, it can be used and built on by anyone (depending on the license), but it can never be claimed, patented or taken out of the public domain. Different licenses have different conditions, giving the developer the choice of who can use his product and under what conditions. Some licenses allow commercial use of the product, others do not. Some demand attribution to the maker. Some demand that any offspring of the product is published under the same license as the original. Designing these licenses and making them available is a very welcome way for institutions to give something back to the commons. 4. Education Well educated citizens are a condition for a functioning democracy (Hiis Hauge & Barwell, 2017), and equally important for the p2p society. However, a university education in for instance the Netherlands is not easily accessible to every intelligent citizen. People who already have a degree pay enormous fees. Part-time studying is discouraged and for most fields not even available. This is in conflict with the ideal of ‘Life Long Learning’ that is propagated by educational institutions and it will not create broadly interested, well-informed citizens. Yet, we rely on them to make wise choices when it is time to vote (Westheimer & Kahne, 2000). It is fashionable for governments to want citizens to participate, but hardly any efforts are made to provide the necessary tools and information. But now, a growing part of the public are finding their way to new, more accessible and less structured forms of education. New methods for knowledge sharing are being invented, often using the internet. Some universities stream lectures. This can be a helpful addition to an education, but without exercises, it is not complete. Khanacademy.org is a website where you can educate yourself in any high school topic, up to a university level. It uses a clever algorithm that allows students to keep track of their progress. However, Khan Academy is very top-down; a person is lecturing and a student is consuming knowledge. OpenTOKO.org (http://web.archive.org/web/20140927205928/http://www.opentoko.org/) uses a different approach. A topic is offered online, and both experts and lay persons sign up, creating a knowledge sharing group that comes together for an afternoon. The dynamics vary a lot, sometimes a TOKO is planned far in advance, sometimes on short notice. Topics can be popular or obscure, there can be many people or only two or three. It often turns out that beginners have skills, knowledge or questions that the experts learn from. The experts get a deeper understanding of the topic, while the beginners get a valuable jump start. Sometimes no expert signs up and the participants figure it out together. OpenTOKO.org is waiting be improved. An automatic website will be created, enabling anybody to suggest a topic, sign up as an expert or lay person, offer a location and pick a date. When all conditions are met, an OpenTOKO automatically takes place. Entrance is free, everybody benefits by gaining knowledge. Topics range from learning a programming language, electronics, mathematics and statistics to knitting socks and growing vegetables, depending on what participants need. For scientists who want to include citizens in their audience, there is no need for scientists to simplify their results or their language. Citizens can educate themselves and get used to the terminology used in different areas of science. As a starting point of a search for information on almost any topic, Wikipedia can be used. From there, it is easy to find scientific publications, but again the citizen's search will often end at a publisher's pay-wall. 5. Towards a p2p society During the last decade, we have seen a trend towards sharing on a p2p basis: from one individual to another. In 1999, a teenager programmed Napster, which enabled anybody to share digital files (Carlsson & Gustavsson, 2001). Napster used a central server. Because of this, the record industry was able to shut Napster down with law suits. The next generations of music sharing tools were p2p networks, where users shared files directly with other individuals. These networks had no hierarchy and were hard to shut down. Brafman and Beckstrom (2006) compare these leaderless networks to a starfish and hierarchical organisations to a spider; if you cut off the spider's head, it will die. The starfish, however, has no head. Cut of a leg and a new leg will form. Cut the starfish in half, two starfish will form. The p2p network seems indestructible as long as it has the will to live. Bauwens and Lievens (2013) see the transition towards a peer to peer society as the inevitable next step in human development. They give an overview of the history of western society, beginning at the Roman times. In those days, around 80% of society were slaves. The Roman society was utterly dependent on those slaves, and had an ever growing need for more. To keep expanding the number of slaves, they had to keep extending their reign. Eventually this became more expensive than alternatives like sharecropping. They started freeing slaves, allowing them to use land in return for a share in the harvest. This situation developed into the feudal system, which lasted many centuries. In the 19th century, industrialisation caused workers to come to cities, buying their food instead of growing it. Workers got wages, and the time was right for capitalism. Also, people became specialised parts of big hierarcical structures. Now, automation has reduced the amount of human labour necessary to produce our society's basic needs. Political economist Skidelsky (2012) concludes: “The truth is that we cannot go on successfully automating our production without rethinking our attitudes toward consumption, work, leisure, and the distribution of income.” In the western society only a small percentage of our labour is used for society’s basic needs. Most people have jobs that can easily be missed (Graeber, 2013) Most of them know this, with often negative effects on their health and psychology. There is also a growing amount of unemployed citizens for whom the old system has failed. Their former employers have discarded of them, often after decades of dedicated work and loyalty from the employee’s side. A large part of the people that got fired have become too expensive because of their age. Employers tend to choose cheaper employees, rather than the more experienced ones. Here, money is preferred over quality and humanity. Some of the unemployed people choose to do things differently from now on, and start looking for a new system that is more respectful of people and that values real quality. Like the freed slaves, they are a growing group in our society who have the energy to start working in a new system. Bauwens and Lievens 22] call this new system ‘the p2p economy', which is based on exchange between individuals. The combination of available knowledge and a growing dissatisfaction with the way governments handle environmental problems, encourages people to get involved in experiments with small scale solutions for energy production, local food production, recycling of waste and democratic and social innovation. Some of these experiments fail, but others succeed and render stable, small, local alternatives that can be an inspiration to others. Not everything can easily be done locally. Specialised health care is best done in dedicated hospitals. Justice is best handled by experts, although the system of jury law is a bit more p2p than law systems with professional judges only, and is in many countries considered fair. Production processes that are said to be more efficient on a large scale, may not be if the real costs and efforts of transport, infrastructure, resources and waste are included in the calculation. These so called “externalities” are often not payed for by multinationals (Mansfield, 2011; Scherhorn, 2005). 6. When systems clash In many cases the p2p society can coexist with the capitalist society, as can the citizen scientist coexist with the academic scientist. There can be conflicts of a financial nature, like in the case of the record companies versus the p2p networks for music distribution, where both sides try to maximise their profit. The more interesting situations, however, occur when both viewpoints are valid, but incompatible. The p2p society is based on trust, earned by previous work. If you contribute to the community, you gain respect and trust. In the capitalist society however, trust comes with credentials. If you want a job as a software engineer, you need to have diplomas and certificates to prove you can write code. In the p2p society you prove you can write code by writing code. The code is open source so users can check if it works and experts can check if it is well written and contains no malware. When the two worlds meet, they are often skeptical about each other’s approach. Some individuals see the flaws and benefits of both systems, and a software engineer may get a job without a diploma, if he happens to run into the right employee manager. The connection between the two worlds comes from those who recognise the possibilities to cooperate and are in a position where they have the freedom not to stick to the protocol. In any case, the p2p way of assessing the quality or reliability of a peer will take more effort than assessing the quality or reliability of, for instance, an academic scientist. Some research into the person will be required, through personal contact or by looking at the peer’s reputation and work. For newcomers it is a completely different situation. But what about other differences between citizen scientists and academic scientists? To do real science, you have to work according to many rules, and citizens can just do whatever they want. What about quality and ethics? This seems to be the concern of many academic scientists. Citizens’ motivations and the quality of their work are suspect (Show, 2015). Even projects set up and run by academic scientists that use data collected by volunteers have trouble getting published (Bonney et al., 2014). It is true that citizen science does not necessarily have a fixed set of rules for ethics or methods. Even though there are many examples of networks that share a code of ethics or list of methods [29], like some diy-biolabs (Diybio.org, 2016) and makerspaces (Fablab.nl), there are plenty of people who never think of it nor discuss it and are simply not interrested. But in the same way any programmer can check open source computer code, any academic scientist can check if a citizen science project is well done, by the academic scientist’s standards. In my opinion this does not mean that the academic standards should be the ones we should measure citizen science by, but we could if we wanted to. In some cases, the university standards are less strict than the ones used in citizen science. For instance, in regular science it is not common to open up all data, while in citizen science it is. In this aspect citizens science is much more reproducible and fraud is more easily detected. Citizen scientists’ experiments are often repeated and improved on. Citizen scientists have no publication pressure (Saltelli & Giampietro, 2017, discuss the desperation to publish or perish as a key ingredient of the crisis in science), they publish when they think they have produced something interesting. They open their work for peer review right from the first attempts to the final conclusions. Citizen scientists are also more likely to publish negative results. They are not at all embarrassed of their errors. Publications of positive results often also include a section about previous attempts that failed, and a discussion about why they failed. These negative results are in my opinion as valuable as positive results, yet in regular science there is a strong tendency to only publish positive results, which makes academic science incomplete (Dwan et al., 2008). Citizen scientists are often suspected of having biases, hidden motives or agendas. We know by now that professional scientists have biases too. There are even types of biases that citizen scientists are less likely to have, like biases that are related to prestige or funding, and the bias towards positive results. But the fact that academic scientists do not necessarily perform any better than citizen scientists, does not mean we should ignore possible problems with citizen scientists’ quality. In order to enable more cooperation between citizen scientists and academic scientists we should insist on full transparency on possible conflicts of interest, for both citizen scientists and academic scientists. In academic science there is a healthy discussion going on about quality, biasses and conflicts of interest. In citizen science the subject is hardly ever raised, which is a shame since some biasses are less likely to occur when their owner is aware of them. Quality is often described as being ‘fit for function”. Sometimes we may need accurate data from expensive sensors. In this case academic science could deliver a higher quality. But at other times we may need a high resolution, and here large groups of citizens are more likely to achieve high quality. Knowledge is only fit for function if it is open to those that need it. It is therefore important that knowledge related to climate change and other global problems is completely open. I think it would be good to have a conversation between academics and citizen scientists about what quality results are useful both for other citizens, local policy makers and academic scientists, depending on the function. If we want to share data between academic scientists and citizen scientists, or between citizen scientists and decision makers, what barriers are there and how can we overcome them? Should we find a set of criteria for quality (including accessibility) that is acceptable for academic scientists and manageable for citizens? 7. The commons Most of the issues mentioned in the above paragraphs lead us to the concept of the commons. The commons includes everything that belongs to everybody. It includes our atmosphere, our planet's ecosystem, our culture and our knowledge. Most of the private wealth in the world exists because goods or services that belong to the commons can often be taken without payment, and there is no obligation to give something back to the commons. This applies to businesses that pollute or take resources, commercial entertainment companies that take folk stories from the public domain and protect them with copyrights, companies that patent genes that are for the largest part natural, as well as to scientific research that takes input from the commons, with results that are not given back to the commons (Barnes, 2006). Climate change, pollution and the depletion of resources are everybody's problem. These problems are as much part of the commons as the solutions are. Solving them does not have to be a private undertaking. By sharing the knowledge between academic scientists and citizens, we can work on these problems together. 8. Discussion and conclusion Citizen scientists are not angels, sent from above to save science. Nor have they come to destroy academic institutions and take over. They have their own agenda's, flaws and biasses, as do academic scientists. More awareness of these biasses can improve the quality of their work. Any discussion about quality should be respectful and open, with consideration of the limitations of citizen science and its practitioners. We know that citizens don't always take the right decisions when they vote, do not vote or buy products. So why should we count on them to solve societies' problems? We should not wait anybody, but we need to engage all the help we can get, and if people are already intrinsically motivated, we should not marginalise them. Even though the groups that use citizen science to solve problems that they feel are not being addressed by institutions are often still small, the interest they receive from others and the availability of user friendly tools and infrastructure can result in more and larger groups, and a stronger network of individuals. What we see now is the beginning of a trend of independent thinkers and do-ers. They might one day develop into institutions of their own, but they may also stay loose groups of individuals. The same goes for the transition towards a p2p society. This transition is already taking place. It does not need a revolution but grows inside the current system. It accellerates in areas where the current system fails. It needs those impulses, and if the current system suddenly solves the problems, or if the p2p system turns out not to, this trend may stop. For now, it is growing. Citizens are eager to contribute to the solution of problems that the neoliberal system fails to solve, like environmental problems and democratic representation (Publiclab.org). Citizens are increasingly sharing in decision-making as our society moves towards a p2p model (Public Laboratory, 2016). Citizens have proven to be capable of asking their own research questions, setting up their own projects, educating themselves and managing complex projects. It is time for them to get taken seriously. Scientists already contribute to the empowerment of citizens by developing for instance new technologies and licenses, but those are offspring of scientific knowledge. It is also necessary to share the knowledge itself. If scientists want their results to be useful, they should make them accessible to citizens. The knowledge created by citizens can be very valuable for academic scientists. However, the act of giving something back to the community has to be embraced by academic scientists who engage in citizen science projects. We are not so far away from getting rid of everything that stands in the way of a collaborative production of useful knowledge. It takes some adjustments from both sides to bring citizen science and academic science together, so we can join efforts in making the sustainable transition happen. References Banerjee, 2008 S.B. Banerjee Corporate social responsibility: The good, the bad and the ugly Critical Sociology, 34 (1) (2008), pp. 51-79 Barnes, 2006 P. Barnes Capitalism 3.0: A guide to reclaiming the commons Berrett-Koehler Publishers, San Francisco (2006) Bauwens and Lievens, 2013 M. Bauwens, J. Lievens De Wereld Redden: Met peer-to-peer naar een postkapitalistische samenleving Houtekiet, Antwerpen (2013) Bauwens, 2012 M. Bauwens Blueprint for P2P society: The partner state & ethical economy (2012) http://www.shareable.net/blog/a-blueprint-for-p2p-institutions-the-partner-state-and-the-ethical-economy-0 Bonney et al., 2014 R. Bonney, et al. Next steps for citizen science Science, 343 (28 March 2014) (2014), pp. 1436-1437 Brafman and Beckstrom, 2006 O. Brafman, R.A. Beckstrom The starfish and the spider Penguin Group USA, New York (2006) CERN Open Hardware License, 2011 CERN Open Hardware License, (2011). http://home.web.cern.ch/about/updates/2013/09/cern-releases-new-version-open-hardware-licence. (Accessed 16 August 2015). Carlsson and Gustavsson, 2001 B. Carlsson, R. Gustavsson The rise and fall of napster-an evolutionary approach Active media technology, Springer, Berlin Heidelberg (2001), pp. 347-354 Cash et al., 2003 D.W. Cash, et al. Knowledge systems for sustainable development PNAS, 100 (14) (2003), pp. 8087-8091 [July 8, 2003] Chomsky, 1999 N. Chomsky Profit over people: Neoliberalism and global order Seven Stories Press, New York (1999) Critical Art Ensemble, 2003 Critical Art Ensemble Free range grain (2003) [Accessed 02 April 2014] http://www.critical-art.net/FRG.html Diybio.org, 2016 Diybio. org Code of ethics for diybio labs in Europe (2016) [Accessed 31 August 2016] https://diybio.org/codes/draft-diybio-code-of-ethics-from-european-congress/ Dwan et al., 2008 K. Dwan, et al. Systematic review of the empirical evidence of study publication bias and outcome reporting bias Plos One (2008) Edwards and Hulme, 1996 M. Edwards, D. Hulme Non-governmental organisations: Performance and accountability beyond the magic bullet Earthscan publications LTD, London (1996), pp. 3-16 Fablab.nl, 2016 Fablab. nl Rules for use of Fablabs in Belgium, Luxembourg and the Netherlands (2016) [Accessed 31 August 2016] http://fablab.nl/wat-is-een-fablab/fabcharter/ Fischer, 2000 F. Fischer Citizens, experts, and the environment: The politics of local knowledge Duke University Press, London (2000) Gershenfeld, 2012 N. Gershenfeld How to make almost anything: The digital fabrication revolution Foreign Affairs 91 43, New York (2012) Graeber, 2013 D. Graeber On the phenomenon of bullshit jobs Strike Magazine Summer, 2 (2013), pp. 201310-2013411 Hiis Hauge and Barwell, 2017 K. Hiis Hauge, R. Barwell Post-normal science and mathematics education in uncertain times: educating future citizens for extended peer communities Futures, 91 (2017), pp. 25-34 Irwin, 1995 A. Irwin Citizen science: A study of people Expertise and Sustainable Development, Routledge (1995) MIT, 1988 MIT GNU public license v2.0 (1988) [Accessed 09 April 2014] http://web.mit.edu/drela/Public/web/gpl.txt Mansfield, 2011 B. Mansfield Modern indistrial fishing and the crisis of overfishing R. Peet, P. Robbins, M. Watts (Eds.), Global political ecology, Routledge, London (2011), pp. 92-94 Meet je Stad, 2015 Meet je Stad Guidelines for data collection by the flora observation group of Meet je Stad (2015) [Accessed 31 August 2016] http://meetjestad.net/flora/?info Pereira and Saltelli, 2017 A. Pereira, A. Saltelli Post-normal Institutional Identities Quality Assurance, Reflexivity and Ethos of Care Futures, 91 (2017), pp. 53-61 Public Laboratory, 2016 Public Laboratory, https://publiclab.org, (Accessed 18 October 2016). Rotman et al., 2012 Rotman, et al. Dynamic changes in motivation in collaborative citizen-science projects Proceedings of the ACM 2012 conference on computer supported cooperative work, ACM (2012), pp. 217-226 Saltelli and Giampietro, 2017 A. Saltelli, M. Giampietro What is wrong with evidence based policy, and how can it be improved? Futures, 91 (2017), pp. 62-71 ArticlePDF (425KB) Scherhorn, 2005 G. Scherhorn Sustainability, consumer sovereignty, and the concept of the market, consumers, policy and the environment K. Grunert, J. Thøgersen (Eds.), Consumers, policy and the environment, Springer (2005), pp. 302-303 Show, 2015 H. Show Rise of the citizen scientist Nature, 524 (7565) (2015), p. 265 [18 August 2015] Skidelsky, 2012 R. Skidelsky Return to capitalism ‘red in tooth and claw’ spells economic madness The guardian (2012) (2012) [Accessed 05 April 2014] http://www.theguardian.com/business/economics-blog/2012/jun/21/capitalism-red-tooth-claw-keynes?commentpage=2&fb=native#post-area Vermij, 2006 R. Vermij Een kleine geschiedenis van de wetenschap Uitgeverij Nieuwezijds, Amsterdam (2006) Warren, 1991 D.M. Warren Using indigenous knowledge in agricultural development World Bank Discussion Papers, 127 (1991) (1991), p. 1 Westheimer and Kahne, 2000 J. Westheimer, J. Kahne Education for action: Preparing youth for participatory democracy School Field, 11 (1) (2000), pp. 21-40 von Lode, 2005 P. von Lode Point-of-care immunotesting: Approaching the analytical performance of central laboratory methods Clinical Biochemistry, 38 (2005), pp. 591-606 ArticlePDF (272KB) von Werlhof, 2008 C. von Werlhof The globalization of neoliberalism, its consequences, and some of its basic alternatives Capitalism Nature Socialism, 19 (3) (2008), pp. 94-117 © 2017 The Author. Published by Elsevier Ltd.