Skip to content Skip to navigation

LinkedResearch - LR: A Suggested Platform to Make Research Exponential

Tomorrow's Research

Message Number: 
1733

We suggest there is a place to think exponentially with regards to research. In order to explain our argument and to illustrate how LR can foster exponential research, we lay out an imaginative (but plausible) case study.

Folks:

The posting below, longer than most, looks at a new approach for carrying out research through what the authors call a LinkedResearch (LR) platform.  It is by Professor Orit Hazzan, Technion - Israel Institute of Technology, et all., and reprinted with permission.  For further information please contact Orit Hazzan oritha@ed.technion.ac.il

Regards,

Rick Reis 

reis@stanford.edu

UP NEXT: Shut up and Write

 

Tomorrow’s Research

---------- 4,690 words ----------

LinkedResearch - LR:

A Suggested Platform to Make Research Exponential

 

Authors

Avi Salmon, Intel, Israel

Avital Binah-Pollak, Technion - Israel Institute of Technology 

Yael Dubinsky, StepAhead, Israel 

Tzahi Harari, Effect-Tiv, Israel

Tamir Hazan, Technion - Israel Institute of Technology

Orit Hazzan, Technion - Israel Institute of Technology 

Amit Livnat, JustDo, Inc.

Ronit Lis-Hacohen, Technion - Israel Institute of Technology 

Koby Mike, Technion - Israel Institute of Technology

Abstract

Universities have two main facets - research and education. This document proposes a complementary model for carrying out research that fits the current technological, economic and social changes taking place in our era. The model is based on concepts borrowed from exponential for-profit organizations and implements an exponential approach for carrying out research in a more influential way, that increases the impact of research and harnesses global research resources. It is suggested that such a model may increase the amount (not only by number of papers as is common today, but also by number of people involved), quality and impact of academic research. Our paper suggests a technological platform that implements and drives these ideas.

1. Introduction

In the academia, it is very common to collaborate with peers from other disciplines and universities, as well as to cooperate with the industry and the crowd in different ways and collaboration formats. In many cases, this type of collaboration is limited by resources. In recent years, a growing number of scholars take part in collaborative research. Social networks have been a major contributor to these collaborations (Niu et al., 2010). Several terms are being used to define projects with many contributors/members (as opposed to participants), for example: “Crowd science”, “citizen science”, “networked science”, and “massively-collaborative science” (Young, 2010; Nielsen, 2011; Wiggins and Crowston, 2011). These projects are usually characterized by two main features: Participation in a project is open to a wide base of potential contributors, and intermediate inputs such as data or problem solving algorithms are made openly available (Franzoni and Sauermann, 2014). While these projects have clear benefits on the project level and on the contributor level, they also face challenges such as leadership, integration of contributions, and sustaining contributor involvement. One particularly notable concern is that projects that are initiated by nonprofessional scientists may not follow the scientific method, calling in question the quality of research output (ibid).

Different platforms aim at organizing and collecting information related to the different aspects of research, for example: ResearchGate,Academia.edu,  Kaggle, Userlove, Google, Code Ocean, Amazon Mechanical Turk (MTurk), Crowd research and others. ResearchGate and Academia.edu are primarily social networking sites and the two most popular (Ovadia, 2014). Both ResearchGate and Academia.edu allow users to post public questions to the community and to group users by institution, allowing users to view colleagues. In both platforms, users can follow the network activity of other users and see things such as articles, answered and asked questions, and, in the case of ResearchGate, endorsements from other users. ResearchGate also has an area for intra-institutional collaboration on projects that is used for commenting and file sharing.Both research platforms aim at measuring scientific reputation.Academia.edutracks various metrics, showing users the number of times their profile hasbeen viewed, the number of times documents have been viewed, and eventhe searches that led people to their profile (Ovadia, 2014). These analytics are new and could be considered part of the altmetrics movement that tracks nontraditional bibliographic metrics (Galligan and Dyas-Correi, 2013). 

ResearchGate includes an algorithm that calculates the scientific reputation using bibliometrics and altmetrics(Galligan and Dyas-Correi, 2013). ResearchGate’s indicator was empirically examined against Research Excellence Framework (REF) and Quacquarelli Symonds (QS) World University Rankings, and was found to be an effective indicator for measuring individual researcher performance (Yu et al., 2016). A recent study claims that high ResearchGate scores are built primarily from activity related to asking and answering questions in the site, and that it seems impossible to get a high ResearchGate score solely through posting publications (Orduna-Malea et al., 2017). 

In this paper, we emphasize the exponential potential of academic research by proposing a global platform - LinkedResearch - LR - that provides all services related to research for all stakeholders. This platform merges several conceptual approaches of existing tools (described above) and conveys the idea of Research without Borders, thus exposes all stakeholders - in addition to universities and researchers, research participants, funding organizations, R&D divisions in corporates, etc. - to various research services. It encourages a research process that involves sharing, collaborating, think tanks and offers tools for feedback and research management. Using this platform, each stakeholder can perform his or her role in the research on a larger, and even, exponential, scale, with limited administrative and technical barriers and with more options and opportunities. By connecting stakeholders with other relevant stakeholders and all the needed resources, research will increase its impact, in terms of practice and values, and may become more efficient, faster, better, easier and cheaper. As stated above, exponential expression is often reflected in the number of people who are actively involved in research, and consequently the research quality and impact. Our paper describes how LR, which enables easy participation for both research-based organizations and individuals, can foster exponential research, in terms of the budget invested in research, the number of participants and their engagement, the used research equipment, research knowledge distribution and more.

2. Exponential Turn

Google, TED, Uber, Airbnb, and Waze are all familiar organizations which many of us use on daily basis. These organizations are framed as“Exponential organizations”(Ismail, Malone and van Geest, 2014). According to Ismail, Malone and van Geest (2014: 18) Exponential organizations (ExOs) are organizations for which: “Impact (or output) is disproportionally large — at least 10x larger — compared to their peers because of the use of new organizational techniques that leverage accelerating technologies.” In their book, Ismail, Malone and van Geest describe exponential organizations as characterized by ten attributes and a massive transformative purpose (MTP). The ten attributes contribute and foster the organizations’ exponential growth and they are derived from its massive transformative purpose.

The ten characteristics of ExOs are divided into internal and external ones: IDEAS (Interfaces, Dashboards, Experimentation, Autonomy, and Social technologies) is the acronym for five internal characteristics of ExOs and SCALE (Staff on demand, Community and Crowd, Algorithm, Leveraged assets, Engagement) which is the acronym for five external characteristics of ExOs. These internal and external characteristics of ExOs correspond to the right- and left-brain characteristics. 

3. Exponential Research

We suggest there is a place to think exponentially with regards to research. In order to explain our argument and to illustrate how LR can foster exponential research, we lay out an imaginative (but plausible) case study:

Dr. James, a doctor at a rural hospital who volunteers in Africa, had an amazing idea for a new way to cure brain cancer. She published a new research project on LR - the just launched new platform developed for exponential research that provides all services related to research for all stakeholders. LR gave her a list of all the brain cancer scientists sorted by relevance, and a list of other scientists with relevant research results and experience. Dr. James found Prof. Chen from China who was in the middle of research that can provide the theoretical basis for Dr. James’ research. Dr. James saw that Prof. Chen had a low score of availability and a high score of knowledge. She then decided to video chat with Dr. Chen thru LR. However, Dr. Chen was not willing to join the project.  

Dr. James’s current networking and familiarity with funding resources did not help her win the grant for her research so she decided to launch crowdfunding event on LR. She posted a video on the LR platform and it went viral. The project received 3.2 million US$ in just two weeks. Eventually, seven universities and 12,745 people contributed. Dr. James published a post looking for research collaborators. She received 890 potential research collaborators from all over the world. One of them, Judy, who constantly views the Internet Channel of LR, and loves it, recently joined an LR e-conference about medical research in her city.

Dr. James decided to team up with 18 researchers. They communicated through the platform.  Dr. James was the research project manager and used the platform to lead all the other stakeholders, teams and resources. Each partner could see the project’s current status, and what is needed to be done and by whom. Dr. James’s team read all the feedback that research project followers wrote and found out few very important ideas they didn't think about.

Dr. James needed a special experiment that she couldn't do, neither in her country nor in Africa. She searched the platform and found 34 places worldwide where the team could send the data to be analyzed. They chose three laboratories that got the highest score in quality on LR and were the least expensive and known for meeting short deadlines. After the experiment was carried out, the members analyzed the data together, based on one of the most current codes stored in LR, and agreed about the next steps.

After 18 months, they found that if you mix two drugs, commonly used for stomach ache, with certain chemical, you can cure 73% of the brain cancer patients. The team produced a video and posted it on the “breakthrough section” of LR. It went viral. In 12 days, it got 8.5 million views, 1.4 million likes and 4,298 comments. TV channels, from around the world, talked about this breakthrough. Dr. James and her team raised additional 17.4 million US$ in LR for their next project – manufacturing the drug - to enable patients to buy it for less than 10$ a month.

As a result of the research’s success, Judy, one of the research collaborators, participated in dozens of other studies and is now ranked as a gold member. She received the highest score for participants in her region. She loves it and it serves her as an additional side job.

n what follows, we use the ten attributes of ExOs offered by Ismail, Malone and van Geest (2014) in order to explain how we believe they should be implemented in research. As the MTP - Massive Transformative Purpose of LR, we propose “research without borders.” 

4. The LR Platform

IDEAS

Interface  

An interface is a bidirectional (BD)[E1] communication channel between stakeholders; in our case, the researchers. Interfaces offered by LR are essential for scaling up research and for accessing required resources (see Leveraged Assets below in the SCALE characteristics). AsGitHubis the de-facto standard for a software repository for any project, a similar research hub is required. LR will support an efficient model to communicate and publish information regarding research topics, research questions, request for resources (availability and cost) or partnerships, as well as availability of resources. LR and its interfaces and machine learning algorithms will help researchers all over the world to easily join resources and build partnerships in order to create exponential research. Learning algorithms will be applied in order to support smart search, automated matching proposals and push notifications.

Dashboard  

A variety of BD)platforms provide user interface UI[E2] tools for such dashboards. Integration to underline databases and control processes will be implemented. Such dashboards will provide information regarding costs, resource availability, community activity, publications, information and funds flow, ratings etc. With the appropriate dashboards, information from the platform will be easily available on demand. For example, The Researchers Dashboard will provide discovery tools for number of collaborators, funds raised so far, physical resources, etc. A dashboard will provide access to a list of research that is carried out, yet unpublished, and will allow researchers to fine tune their research goals (prevent duplication), identify collaborators, get preliminary results etc.

The Operation Dashboard will provide the UI for the operations team to view the LR state, resources utilization, trends, needs etc. This dashboard will monitor and present equipment utilization over time, will allow identifying and alerting resources that are in demand or about to be exhausted and be proactive for such scenarios.

The Success Dashboard will post continuously the success criteria as determined by:

  1. Quantity: The number of new cooperative relationships created between stakeholders outside and inside traditional research-based organizations (academia, R&D departments, etc). 
  2. Quality: The ranking of the community of the shared research. 

Experimentation

Experimentation is one of the basic elements of research, and as such, it should be highly accessible and encouraged. At the same time, due to limited available resources, in many cases researchers skip the preliminary research - informal, fast and dirty - that hints whether it is worth investing additional resources in a full and comprehensive research plan, and invest many resources in preparing a research proposal that eventually cannot be carried out.  As soon as data from any stage of the research is posted, e.g., byCode Ocean, it is possible to view it as a preliminary stage and continue exploring the research theme in different directions based on these preliminary results. Such an approach requires openness, honesty and ethical standards that only a transparent and equal platform, such as LR, can provide.  

This characteristic of ExOs supports exponential research on three dimensions: Time, scope, and reliability/validity. Time is accelerated due to the option to use available resources for a preliminary and small-scale research prior to the decision whether to initiate a full research program, submit a research proposal, wait for the reviews and hopefully, get funds. Scope is accelerated due to the option to further explore only research paths which have potential. Validity increases since the pressure to publish, which fosters the tendency to partially checked results, is reduced. This characteristic can be supported by the Leveraged Assets and the Autonomy characteristics of LR which will be explained below.

Autonomy  

LR will provide researchers and research stakeholders the opportunity to act autonomously in different levels:

1)     Open research opportunities to a wide range of stakeholders:

a)     Researchers will be exposed to other researchers who are usually not part of their regular peers and networking. This type of cooperation can become more dominant in interdisciplinary research (e.g.,MIT Media Lab).

b)    Researchers and research collaborators will be exposed to an exponential amount of research and will have the opportunity to approach each other easily and reach out over their familiar space.

2)    Provide researchers a wide picture of available resources:  HR, funding, partners, and opportunities that are not limited or dependent on specific resources of a university.

            3)     Expand research collaboration, often an outcome of personal connections or part of formal collaboration between organizations/universities. LR provides the researcher the opportunity to choose partners, resources and ideas.

The exponential resources will not only allow physical and ideological autonomy to each researcher and stakeholder, it may also influence their way/mode of thinking and operation. Exponential research can occur in an atmosphere of no boundaries, neither physical, political, or cultural. This also means that one can and should (in LR) act autonomously – and vice versa – An autonomous state of operation will contribute to exponential research. When a critical mass of researchers and stakeholders act autonomously, the exponential effect will be created. Implementation of autonomy in research may cause conflict between the organization and the researcher who carries out the research. It means that LR should be “Autonomy by design” – it should be built in a way that enables any single researcher/entity to act with full rights on the platform, independent of formal affiliation to a university/organization - that is, without borders. Every eligible partner will have the complete set of rights and duties as any other partner.

Social MediaMethods

A research process involves sharing, collaborating, thinking together, receiving feedback, and managing a project. By using LR everyone can do all these activities on a large scale, without borders and with more options and opportunities. LR uses social media methods for making research faster, better, and cheaper, by:

●     Connecting researchers with other relevant people (e.g., LinkedIn)

●     Connecting researchers with needed resources and research space (e.g., Booking.com)

●     Providing social ranking and feedback for resources and stakeholders.

●     Enabling data sharing (e.g., Facebook), public stages for idea sharing (e.g., TED) and Crowdfunding (e.g., Kickstarter)

●     Offering individuals to volunteer as research subjects (e.g.,Fiverr)

●     Managing research as a shared project (Jiralike).

SCALE

Staff on Demand

Research requires qualified professionals (researchers, reviewers, and participants) to perform the research efficiently. Sometimes, other professionals, such as an expert developer or a highly-qualified mechanical engineer, are needed as part of the research team. In the past decade, the volume and quality of freelancers have dramatically increased (Pofeldt, 2017). Implementing the exponential attribute of Staff on Demand, LR can increase the agility of the research process and promote the quality of the research in a relatively short period of time. For example, hiring professionals from multidisciplinary fields may complement the qualities of the team members. Another example is research that strives for medical data but faces difficulties in receiving it from medical centers. LR will enable hiring of participants more easily for the sake of research, keeping rigorous rules of ethics and privacy.

We suggest a module that supports Staff on Demandin a dual manner: On the one hand, people who wish to join a research project can promote themselves, specifying their experience, interest and expertise. On the other hand, research coordinators can search, identify and approach these individuals in order to hire them for the required service and for the required period of time.Fiverrmarketplace of freelancers is an example of such a model In the case of research, these freelancers can be researchers, reviewers, mentors, data analysts, participants and more. It will also enable experienced practitioners to utilize their skill and professional experience for longer periods of time, while changing roles in different research projects. 

Community and Crowd

The academic world has a limited number of people who can participate in research activities as research participants. As a result, the wisdom of the crowd is limited. When we consider an exponential growth of research, we wish to increase the scale of number of people involved (the wisdom of the crowd). The Community and Crowd attribute of LR may enlarge the scale of people involved in research without increasing costs:

  1. “Ideas Market”: The wisdom of the community may be harnessed to initiate, rate, and promote research proposals. LR will enable posting research ideas and then, commenting and ranking those ideas. The community can supply a rich flow of ideas but, in addition, can implement a “crowd review”, making sure that high quality and relevant research is carried out. Such a crowd review may clarify the research ideas, remove ideas, merge or consolidate ideas and create derivative ideas.
  2. Communities of Interest: LR will support communities of practice and interest by providing forums for specific topics and will provide news and updates on studies related to this forum. The community will have access to research data, and can provide its own data analysis, using machine learning methods and modern data science in the model of Kaggle.
  3. Undergraduate and graduate student participation: Participation in research can be part of students’ commitment. Students’ involvement will enrich their knowledge of their studies and help researchers, not necessarily in the same institute, in conducting research. When these students become part of the community, they can start creating partnerships, collaborating and expanding their network from an early stage of their professional development. In the future, this mode of working becomes their habit of work.
  4. Crowd investment: In the model of Kickstarter, LR enables people all over to give micro donations to a study for any credit.

Algorithms

Research environments deal with a vast amount of data. LR offers an algorithm which systematically collects and stores the relevant data, activating an AI engine that uses different algorithms to enhance and automate research processes. The ability to collect data from multiple sources in the IOT (Internet of Things) era, legally and ethically, increases the challenge of collecting the suitable data.

Considering this challenge, we illustrate, for example, the need in the healthcare industry that is certainly undergoing a major evolution. Medical devices and wearable cloths provide a huge amount of personal data that was not available till recent years, and is highly valuable for different research works especially when implementing data algorithms and providing medical as well as other types of insights. This data can be used by the Algorithms exponential attribute of LR. Machine and deep learning algorithms can promote, for example, smart searches, smart connections between different studies, and resource sharing.

Leveraged Assets 

Any research needs many and a variety of resources, such as, qualified human resources, research tools, equipment, research space, computing power, data, patents / IP, and funds, which are, in many cases, in short supply. Accordingly, one of the challenges researchers are facing is finding existing resources, information, collaborators, etc., However, LR should easily be able to identify resources utilization/load, lack of resources, etc.

Getting ownership or access to required resources is a major challenge for many researchers. At the same time, many overlaps exist and resource allocation is not always efficient. Thus, once resources are available and are not used at a specific moment, they can be leveraged - offered to other research projects and utilized more efficiently. Leveraged assets can utilize available resources, enable more efficient budget usage, open new research topics, and enrich people’s skills and knowledge.

Think about a researcher who has just finished using his or her budget at the middle of the research process. LR informs him or her about another research group that has the needed resources. Since the researcher trusts the crowd ranking of the equipment of this group, provided by the members of LR (in the framework of Community and Crowds), the researcher decides to borrow this equipment and thus, could complete the research on time. 

LR will enable each member of the community to offer his or her resources (either human resources, physical equipment or computational resources), to watch what resources are available and offered by other research groups at each moment, to share resources, to lease resources, and as a by-product, sometimes even create new research-based partnerships or increase income resources. Further, since ranking is a key principle for mutual trust, the incentive to keep the research resources and assets updated and of high quality increases.

Engagement

Academic studies are well perceived in the organic academic environment and, in fact, no real need exist to engage different stakeholders to cooperate. However, if we want to increase the capacity of participants exponentially - the number of participants, researchers, reviewers and so on - we need to set activities that increase the exposure of academic research and make it available and attractive to an audience that is not naturally exposed to it today.

Exposing research activities to an increasing number of people, as well as creating a platform that will allow those individual to interact with academic researchers and with each other, will enable exponential expansion in the number of ideas, research projects, participants in research, and donations and resources.

LR will enable arranging public events and gamifications to increase the engagement of all relevant stakeholders. Examples of such activities could be:

  1. Public events, being arranged by the community, in the format like TEDx, that will focus on interesting research, studies and idea marketing. These events will be self-organized with the help of LR.
  2. Research TV channel. This will be a live Web site with recent, best, interesting activities, whose schedule is ranked in a way that it will always be interesting to browse and get the most fascinating stories. In addition, a public TV show on the national TV, centralizing and bringing top stories to the public, will increase the awareness to research.
  3. Public festivals, in the style of e-conferences. This will be enabled and arranged by LR which will host open events, self-organized by the active community with interesting talks on research projects.
  4. Gamification parameters. Each LR member, either a researcher, a research participant, professor, donor, etc., will be able to define his or her profile in LR with characteristics such as: Credits (points for activities), Titles (beginner, master…), and to be defined by Scoreboards (“You are the 3ed ranked reviewer in your region”). Participants will also be able to participate in competitions, open challenges about data sets and will receive prizes.  

The Unique Features of the LR Platform

Specifically, LR lets research processes to be:

1)    Transparent and open to the public: The community can initiate research, participate in research, rate and promote ideas and share knowledge. Not only will the quality of research be improved, but also the ethic norms of research will be alleviated as a result of such a transparent and open research process.

2)    Networked: Many resources (such as laboratories, researchers, subjects, data, knowledge) will be public and shareable.

3)    Self-organized for funding: LR enables calls for research and it also supports mechanisms for asking for grants and funds.

4)    Self-regulated: A due diligence process is embedded in the system for quality.

5)    Based on big data: Its future development will be based on recognized patterns of research.

6)    Forced by rules that keep the process rigorous, professional and ethical.

7)   Effectively managed as a project.

Risks, Regulations and Challenges with LR

LR characteristics allow stakeholders to perform research on a larger, and even, exponential scale without the “traditional” administrative or technical barriers. We are aware that these characteristics might expose the stakeholders to a new format of barriers such as risks and ethical issues that require a great deal of responsible discussion. Below are several examples:

1)    Competition: Phenomenon such as network effects might have influence on university research and ranking and cause interference of traditional research.

2)   Breakthrough entrepreneurs who were the founders of two-sided networks, are people that understood that being an ExO may require bending laws and regulations. For example, GetTaxiinitiated their activity in Israel, even though in Israel, one needs a specific license to be allowed to drive a taxi. Yet, GetTaxi drivers do that without any license. Are they breaking the law? The social and cultural benefits are obvious.

3)   Privacy regulation: Different regulations in different countries, which regulation should be applied?

4)   Fair and ethical use of the research finding. 

5)   IP rights

6)   Social responsibility with regards to harmful use of research findings, personal data that was collected in the research.

7)   LR as an ExO platform may become a monopoly platform. There might be harmful consequences on traditional research in universities and on the commercialization of research.

8)   Cultural gaps.

9)   The more exponential LR will become, the more self-organized it will be, and the more difficult to be controlled from a moral human perspective.  Therefore, a parallel system of regulation and control should be structured.

Profit Model

Today, research organizations have diversified financial models. Unlike for-profit organizations, most universities are nonprofit. Both types of organizations have several funding resources such as: paying customers, governmental, industrial and private funds, income from patents and so on. All those profit models can fit the LR platform. One might expect an exponential growth in profit as well as growth in research quantity and quality.

Similar to other exponential services, the basic profit model, paying users, is applicable for this service as well. The platform might enable free access to some of the services, while paying for others. For example, searching for a research topic could be free while promoting a research topic could be a paid service. More complex profit models exist as well: The platform might charge a commission for the funding that is passed through its network and might have legal and financial rights in research that came to life thanks to the platform’s connection. Another possible model is the non-profit model of scientific funding. A governmental fund, for example, might fund the platform itself, as a means to complement its current research funding.

4. Conclusions

We believe that such a transparent and open research process will not only increase the quality of research, but also have a positive impact on ethic [E11] norms of research. As an open interface, with no boundaries, it might be challenging to move from the physical territory of traditional university experience and practice to the LR platform. We believe that a group of social scientists, philosophers, psychologists, computer scientists, engineers, lawyers, etc, will be able to bring as much perspective as possible.

References

Franzoni, Chiara and Henry Sauermann. 2014. “Crowd Science: The Organization of Scientific Research in Open Collaborative Projects.” Research Policy43: 1-20. 

Galligan, Finbar, and Sharon Dyas-Correia. 2013. “Altmetrics: Rethinking the Way We Measure.” Serials Review39(1): 56-61.

Ismail, S., Malone, M. S. and van Geest, Y. 2014. Exponential Organizations: Why New Organizations Are Ten Times Better, Faster, and Cheaper Than Yours (And What to Do about It). Diversion Books.

Nielsen, M. 2011. Reinventing Discovery: The New Era of Networked Science. Princeton: Princeton University Press. 

Niu, X., Hemminger, B.M., Lown, C., Adams, S., Brown, C., Level, A., McLure, M., Powers, A., Tennant, M.R. and Cataldo, T. 2010. "National Study of Information Seeking Behavior of Academic Researchers in the United States.” Journal of the American Society for Information Science and Technology61(5): 869-890. 

Orduna-Malea, Enrique et al. 2017. “Do ResearchGate Scores Create Ghost Academic Reputations?” Scientometrics112(1): 443-460.

Ovadia, Steven. 2014. “ResearchGate and Academia.edu: Academic Social Networks.” Behavioral & Social Sciences Librarian33(3): 165-169.

Pofeldt E. 2017.  “Are We Ready for a Workforce That Is 50% Freelance?” Forbes. Accessed May 10, 2019.  https://www.forbes.com/sites/elainepofeldt/2017/10/17/are-we-ready-for-a-workforce-that-is-50-freelance/#551eb3813f82

Wiggins, A. and K. Crowston. 2011. “From Conservation to Crowdsourcing: A Typology of Citizen Science.” Paper Presented at the 44th Hawaii International Conference on Systems Sciences (HICSS).

Young, J. 2010. “Crowd Science Reaches New Heights.” The Chronicle of Higher Education. Accessed May 10, 2019. https://www.chronicle.com/article/The-Rise-of-Crowd-Science/65707/.  

Yu, Min-Chun, et al. 2016. “ResearchGate: An Effective Altmetric Indicator for Active Researchers?” Computers in Human Behavior55: 1001-1006.