By the end of this course, you should be able to:
- 1. List the key concepts involved in developing Open Source Software (OSS).
- 2. Understand how OSS evolved into today's dominant position.
- 3. Explain the benefits of using OSS as compared to using proprietary products, and know how to dispel the FUD that arises.
- 4. Familiarize yourself with open source projects that form the foundations of today's word-wide technology infrastructure.
- 5. Work with projects that can help you get a solid footing in OSS methods and communities, such as the TODO Group and OpenChain Project.
- 6. Behave properly while engaging in OSS projects, to develop the best code and build on the strengths of other contributors.
- 7. Understand OSS licensing issues, such as types (restrictive vs permissive), and explain how to choose the right one.
- 8. Make use of projects such as FOSSology and SPDX to foster compliance.
- 9. Have a sense of best practices for long term project sustainability, including how to respect and encourage diversity.
- 10. Know about GitHub and other hosting providers and advanced interfaces such as Gerrit.
Jerry Cooperstein, PhD has been working with Linux since 1994, developing and delivering training in both the kernel and user space. During a two decade career in nuclear astrophysics, he developed state-of-the-art simulation software on many kinds of supercomputers and taught at both the undergraduate and graduate level. Jerry joined The Linux Foundation in 2009. He is currently a Senior Content Manager for The Linux Foundation.
-
1. Audience: software developers who want to understand the rules of the road
-
2. Knowledge/Skills: none prior required
-
3. Software Environment
- distribution-flexible
- three main Linux distribution families focused:
-
- Debian/Ubuntu
-
- Red Hat/Fedora
-
- openSUSE/SUSE
-
-
4. Lab Environment
- any operating system
One great way to interact with peers taking this course is via the Class Forum. The forum can be used in the following ways:
- To introduce yourself to other peers taking this course
- To discuss concepts, tools and technologies presented in this course, or related to the topics discussed in the course materials
- To ask questions or report issues with labs or course content
- To provide solutions/suggestions to peer learners
- To review issues that were already raised, including possible solutions - the forum can be a great learning resource as well, and you may find that your questions have already been - answered in previous posts
- To share resources and ideas related to Linux system administration
Please note that all announcements regarding course updates are posted in the forum, so we highly recommend that you periodically check it out.
If you have questions regarding your course enrollment, you can reach out to us via our Customer Support system. You will be required to login with your LF Account, which will help us to quickly locate your account and respond to your request. This will also allow you to track your support request through to resolution, and create an ongoing record of your support requests.
The Linux Foundation Training & Certification Customer Support system also offers enhanced functionality, such as:
- Knowledge Base Articles - to help you find a quick response to your commonly asked questions
- Service Request Forms - asking the right questions so that you can get the right answers.
This course is entirely self-paced; there is no fixed schedule for going through the material. You can go through the course at your own pace, and you will always be returned to exactly where you left off when you come back to start a new session. However, we still suggest you avoid long breaks in between periods of work, as learning will be faster and content retention improved.
You have unlimited access to this course for 12 months from the date you registered, even after you have completed the course.
The chapters in the course have been designed to build on one another. It is probably best to work through them in sequence; if you skip or only skim some chapters quickly, you may find there are topics being discussed you have not been exposed to yet. But this is all self-paced, and you can always go back, so you can thread your own path through the material.
The labs and knowledge check questions are not graded.
We would also like to emphasize that you will not be required to take a final exam to complete this course.
In all The Linux Foundation courses (in any format) we put a heavy emphasis on learning by doing. In live, instructor-led classes, we almost always aim for a 50/50 balance between lecture and discussion, and working on laboratory exercises, or homeworks, that either perform the tasks just described in the class, or try more ambitious variations. Instructors help students figure out how to do things, debug their code and scripts, etc., during these lab sessions. Because this course is self-paced, without a live instructor, it will be up to you to control your time budget and make sure you take enough time to do labs.
📦 For your convenience, you can also access a single file containing all the lab exercises (LFD102 Labs) by navigating to the Table of Contents located on the left-hand side of the screen and selecting Resources > Files.
You will find lab exercises located throughout the course.
At the end of each chapter, you will also find a series of knowledge check questions. These questions, just like the labs, were designed with one main goal in mind: to help you better comprehend the course content and reinforce what you have learned.
In order to make it easier to distinguish the various types of content in the course, we use the color coding and formats below:
Dark blue: text typed at the command line
Green: Output
Black: file content
Brown: File/Directory names
Light blue: Hyperlink
You may also notice text boxes with different icons. The purpose of each is clarified below:
📦 Additional course documentation
💡 Additional information
🚩 Important information
🔍 Think, explore
📖 External resources
Copyright 2019-2023, The Linux Foundation. All rights reserved.
Linux Foundation Projects include:
- Linux
- Kubernetes
- Node.js
- ONAP
- PyTorch
- RISC-V
- SPDX
- OpenChain
Over 85,000 open source technologists and leaders worldwide gather at Linux Foundation events annually to share ideas, learn and collaborate. Linux Foundation events are the meeting place of choice for open source maintainers, developers, architects, infrastructure managers, and sysadmins and technologists leading open source program offices, and other critical leadership functions.
These events are the best place to gain visibility within the open source community quickly and advance open source development work by forming connections with the people evaluating and creating the next generation of technology. They provide a forum to share and gain knowledge, help organizations identify software trends early to inform future technology investments, connect employers with talent, and showcase technologies and services to influential open source professionals, media, and analysts around the globe.
The Linux Foundation Training & Certification Group works with expert instructors and experienced open source developers to create training courses for every level of experience, from complete newbies to veteran developers, as well as certification exams which demonstrate your skills to potential employers in a trusted verifiable way.
To get more information about specific courses and certification exams offered by the Linux Foundation, including technical requirements and other logistics, visit the Linux Foundation Training and Certification website.
Congratulations! You've Completed
Course Introduction Up Next:
Open Source Software (OSS)
- watch video
By the end of this chapter, you should be able to:
- Define Open Source Software (OSS) and distinguish between different uses of the word free.
- Explain the differences between proprietary (closed source) and OSS.
- Discuss the pragmatic and idealist philosophical strains that inhabit the OSS world, and how they conflict, as well as merge.
- Sketch out the history of OSS as it evolved from the early days of the computer age to its current, widespread, even dominant use.
- Explain and distinguish between different governance models, including Company-Led, Benevolent Dictatorship, and Governing Board.
Sharing of software has gone on since the beginnings of the computer age. In fact, not sharing software was the exception, and not the rule. The concepts of Open Source Software (OSS) long predate the use of the term.
There are many models for governance in OSS projects. Each of these models is frequently seen and has its reasons for existence. There is no one method that fits all projects. Contributing to OSS requires learning standard methods and etiquette to do it properly.
In English, the word free has two meanings:
- Free as in free speech, freedom to distribute
- Free as in no cost, or, as is often said, as in "free beer".
Use of the term OSS is partly meant to avoid this confusion, but also to better define what is meant.
We are not going to try and give a detailed definition, but just point out some salient facts which we will discuss in more detail.
Use of OSS means source code is made available with a license which provides rights to examine, modify and redistribute, without restriction on the user's identity or purpose.
🚩 There is a multiplicity of licensing methods, falling into the two general classifications:
- Permissive (such as BSD-licensed software)
- Restrictive (such as GPL-licensed software)
Proprietary software had historically been the only real model used by commercial projects until the recent rise of OSS.
Only the owners of such software projects have full legal access to the source code involved, although they may grant inspection rights to trusted partners who have signed non-disclosure agreements (NDAs). For this reason, this model is also called closed source.
The owners of proprietary software projects may or may not be the authors of the code; usually, the authors have been working under contracts giving up their individual rights. Indeed, many such programs are no longer even owned by their originators, but have been passed on through sales over time.
To use proprietary software, end users must accept a license restricting their rights, often through a confusing click-through box presented after a very long and dense description during install.
Such licenses generally restrict the user’s rights of re-distribution, indemnify the product from damages due to either malfunction or misuse, and prohibit trying to reconstruct source code or use inside another product. They may sometimes restrict how a product is used, but whether such restrictions are legal will vary with details and legal jurisdictions, including country.
🚩 Price is not the point! The difference with OSS and proprietary models has nothing to do with price. One can charge as much as one wants for an open product and one can provide a proprietary product for free. The license differences have to do with redistribution, modification, reuse of code, etc.
Since its beginning, OSS has had two different philosophical strains: idealistic and pragmatic.
-
Idealistic:
- Here, free means as in freedom, not beer. There is a profound belief that all software should be open for ideological and ethical reasons, not just technological ones.
-
Pragmatic:
- Here, the primary considerations are technical ones, including faster and better development involving more contributors and review, easier debugging, etc.
Note that the more ideological perspective has strong technical imperatives as well, and in many cases, the objectives of both streams coincide. For example:
- Should the software powering a life-saving medical device (such as a pacemaker, or an insulin pump) be secret? Do we not have the right to know what is controlling such devices? How do we know they are not vulnerable to external attacks that can kill us?
- Should the software powering voting machines be closed? How can we be sure we can respect the integrity of tabulated results? This is particularly true when experiment after experiment has shown closed source voting machines to be incredibly easily hacked.
Unfortunately, the conflicts between these two attitudes towards open source has often been acrimonious and destructive. It is unlikely to ever end; the pragmatic approach has most of the economic resources behind it due to corporate buy-in, and the more philosophical camp will always have determined adherents.
Open Source Software (OSS) has a long history, although the actual term only dates back to 1998.
Next, we give a timeline for some of the major developments in the world of OSS. We do not mean to be exhaustive or insult anyone by leaving out important steps, as any list is inherently subjective.
Major OSS Developments
The 1950s and Earlier
Software arose from researchers, both academic and corporate
Distributed openly and cooperatively
Source always distributed, binaries less often
Software not seen as separate commodity
Software bundled for free with hardware
Licensing was sloppy
The 1960s
Major aspects of computer science and software rapidly developed both in academia (MIT, UC Berkeley) and industrial research labs (Bell Labs, Xerox)
1968: ARPANET emerges, eventually leads to Internet, essential for developers and researchers to communicate, share, collaborate
1969: UNIX born at Bell Labs (AT&T) and given for free to universities and research centers
1969: IBM forced by US government to break software and hardware apart and sell/distribute separately, due to unfair business practices.
The 1970s
1976: emacs released by Richard Stallman and Guy Steele. Versions by other authors proliferate, including proprietary ones. GNU emacs not released until 1985
1978: First version of TeX released by Donald Knuth, an open source typesetting system often used for publishing journal articles and books, still in widespread use, usually in the LaTeX version.
The 1980s
1980: Usenet begins as the ancestor of user forums and the World Wide Web
1982: GNU project announced by Richard Stallman
1984: X Window System released out of MIT, with X11 protocol released in 1987 (now run by X.Org)
1985: Free Software Foundation (FSF) founded by Richard Stallman
1987: gcc released (now known as Gnu Compiler Collection)
1987: Perl released by Larry Wall.
The 1990s
1991: Linux begun by Linus Torvalds
1992: Python released by Guido Van Rossum
1992: 386BSD released
1992: Samba developed by Andrew Tridgell in Australia
1993: Debian first released by Ian Murdock and still survives as the largest non-commercial Linux distribution
1993: Red Hat is founded; while earlier commercial Linux distributions already existed, Red Hat was the first company built on open source to become very large
1993: NetBSD released as a fork from 386BSD
1993: FreeBSD released
1993: Wine released to run Windows applications on Linux
1994: MySQL development begins in Sweden, first release in 1995
1995: PHP, GIMP, and Ruby released
1996: Apache web server released
1996: KDE released
1997: GNOME released
1998: Netscape open sources its browser, which will later become Firefox
1999: OpenOffice released (eventually forks into LibreOffice).
The 2000s
2000: LLVM compiler project begun at UI-Urbana-Champaign
2002: Blender released as an open source project
2003: Firefox released
2004: Ubuntu releases its first version, which Canonical builds on top of Debian
2005: Git released by Linus Torvalds
2007: Android released, based on Linux kernel; first devices on the market in 2008
2008: Chromium released by Google; basis of Google Chrome.
The 2010s
In this decade, OSS becomes ubiquitous, dominating the smartphone market, supercomputing, worldwide networking infrastructure, etc.
There is no shortage of governance models used by OSS projects. They differ according to the degrees of centralization, strong influence by one or a few organizational entities, and existence of democratically-inspired decision mechanisms and leadership selection.
The choice of model and how well it is executed have profound influences on the quality of a project, as well as how quickly it is adopted and evolves, improves and succeeds. There is no one-size-fits-all model and each has its advocates and detractors. Furthermore, different projects have different inherent needs, which may be better satisfied by one particular model.
OSS Governance Models
Company-Led (mostly, a closed process)
Development is strongly led by one corporate or organizational interest
One entity controls software design and releases
May or may not solicit contributions, patches, suggestions, etc.
Internal discussions and controversies may not be aired very much
It is not definitively known what will be in the next software release
Upon release, all software is completely in the open
Examples: Google Android, Red Hat Enterprise Linux
Benevolent Dictatorship (strong leadership)
One individual has overriding influence in every decision
Project's quality depends on that of the dictator
Can avoid endless discussions and lead to quicker pace
As project grows, success depends critically on the dictator's ability to:
- Handle many contributors
- Use a sane, scalable version control system
- Appoint and work with subsystem maintainers
The dictator's role may be social and political, not structural (forks can occur at any time)
Maintainers write less and less code as projects mature
Example: Linux kernel
Governing Board (tighter control by smaller groups)
A body (group) carries out discussions on open mailing lists
Decisions about design and release dates are made collectively
Decisions about who can contribute, and how patches and new software are accepted, are made by the governing body
There is much variation in governing structures, rules of organization, degree of consensus required, etc.
Tends to release less frequent, but hopefully well-debugged versions
Examples: FreeBSD, Debian.
- free in OSS refers to freedom, not cost
- Android by Google is controlled by one entity but still OSS
- OSS predates Linux
- The programming language used is independent of its status as OSS.
- shared knowledge is better than closed knowledge especially with critical software
Question 2.1
What are the two main types of Open Source Software licenses? Select all answers that apply.
A. Permissive B. Proprietary C. Free of charge D. Restrictive E. Educational use
Question 2.2
What is the difference between proprietary software and OSS?
A. Proprietary software can charge and OSS must be free B. Proprietary software does not expose its source and OSS does C. Proprietary software requires a license and OSS does not D. Proprietary software pays its developers and OSS does not
Question 2.3
The software for medical devices:
A. Must be open source in most countries, as you have a right to know about your health issues and dangers, etc. B. Can be a complete mystery as manufacturers are under no obligation to reveal their code
Question 2.4
Linux began:
A. In 1998, when IBM made a major investment B. In 1991 with an Internet post by a student in Finland C. When UNIX was first born, about 1969 D. When SCO ported Xenix to the Intel 8036 processor in 1983
Question 2.5
Which method best describes the way Android is developed?
A. Company-Led B. Benevolent Dictatorship C. Governing Board
Question 2.6
Which method best describes the way FreeBSD is developed?
A. Company-Led B. Benevolent Dictatorship C. Governing Board
Question 2.7
Which method best describes the way Linux kernel is developed?
A. Company-Led B. Benevolent Dictatorship C. Governing Board
RTR
- video watched
By the end of this chapter, you should be able to:
- Explain the advantages of collaborative development.
- Explain how to obtain flexibility that can give users more options, more possibilities of adaptation, and lower costs while avoiding vendor lock-in.
- Explain how the work of external developers, as well as incorporating suggestions and fixes, can help businesses bring better products to market faster.
- Show how OSS can provide new marketing strategies since customers know what they are getting is part of an entire ecosystem. - Understand how both quality and security of code is enhanced by development in the open.
- Enumerate the advantages OSS brings to educational institutions at all levels.
- Explain how OSS gives developers greater visibility and helps them build careers.
Collaborative development enables software projects to build better software.
When progress is shared, not everyone has to solve the same problems and make the same mistakes. Thus, progress can be made much faster and costs can be reduced.
Having more eyeballs viewing code and more groups testing leads to stronger and more secure code, as well.
It is often hard for competitors to get used to the idea of sharing, and grasping that the benefits can be greater than the costs. But experience has proved this to be true over and over again.
Competitors can compete on user-facing interfaces (e.g. internal plumbing that everyone needs) so that end users still see plenty of product differentiation and have varying experiences.
There are advantages of having freely available source code:
- Code published openly tends to be cleaner because:
- It is embarrassing to show ugly, sloppy code
- Coding standards and styles tend to be cleaner and more consistent on community projects
- More people have to understand and work on the code
- There are more eyeballs examining code and looking for security weaknesses before they are discovered by bad actors
- There is more input in original design to avoid bad ideas
- No security through obscurity
- No just trust me
- Potentially faster bug repair.
- Flexibility
- mix and match software
- save money on buying or leasing software
- avoid vendor lock-in, maintaining choice
- look under the hood ("trust, but verify")
- more fun!
-
Collaborative Development
- lowers total cost of development
- speeds up time to market
- work is submitted to wider community for critique, suggestions and contributions
- upstreaming reduces future costs for new products that reuse code
- uses well-delineated APIs
-
Marketing
- customers know what they are getting - have confidence in quality as there are no secrets
- product is seen as part of a large ecosystem of related products
- more flexible, possibly modular construction
- Adoption by larger community can help build customers' confidence about the products durability and stability
-
Childhood Education (Public Systems)
- large amount of available teaching resources at little or no cost
- wide range of areas available for using, operating and system administration, and programming
- students do not become locked into vendor products
- school systems do not have to pay for expensive software, even at a discount
- generally lower hardware costs, and easier to use old hardware
- students learn the skills they need in the workforce
- unleashes student creativity, and is more fun!
-
University (all the advantages of earlier education and more)
- students can study and work on the internals of operating systems, applications and libraries, and system administration utilities
- students are ready to enter the workforce where they are most needed
- good habits are developed, including how to work with the open source community
- student work is easy for prospective employers to evaluate, sentence, publicly accessible
-
Developers
- no need to invent (reinvent anything)
- helps to make good, early decisions on product design
- more eyeballs on code can fix bugs faster
- suggestions and contributions are provided by large group of developers
- Great for finding the next job
- code is readily available for evaluation
- can demonstrate how well you work and play with others
- can show how good you are at mentoring and maintaining projects and subprojects
- know you are not alone!
Lab 3.1. Using OSS Methods to Produce Better Products
Exercise 3.1: How OSS Methods Can Produce a Better Product
Enumerate a few reasons how adopting OSS methods can improve and accelerate a product’s success.
Solution 3.1
- Expanding the pool of collaborators brings in more talent, more ideas.
- Competing groups become collaborators and do not have to keep reinventing the wheel. By standardizing on one implementation and building upon it, progress can be faster.
- More eyeballs on the code leads to more bug and security hole discovery as well as faster fixes.
- More eyeballs on the code leads to improved and cleaner style. Furthermore, having style standards throughout a project leads to increased clarity and lessens the path to involvement by new contributors.
- Getting recognition from a wider audience motivates people.
- Having a wider variety of use cases induces a project to have more flexibility and also helps unearth problems that might be rarer and harder to identify with a smaller audience.
Chapter 3 Quiz
Question 3.1
From a business perspective, use of OSS (Select all answers that apply):
Correct Answer
A. Lengthens time to market, but is worth it because of improved product B. Enables use of ingredients from other sources and speeds development Your Answer: Correct C. Makes marketing more difficult because it is hard to differentiate products D. Makes marketing easier, as some ingredients are already well-known and trusted Your Answer: Correct
Question 3.2
For school systems at any level, use of OSS (Select all answers that apply):
Correct Answer
A. Should be avoided, as students become confused by choices B. Can lower costs by letting low-cost or free software be used Your Answer: Correct C. Can lower costs by letting older hardware be used Your Answer: Correct D. May cause problems with vendor lock-in
Question 3.3
OSS is (Select all answers that apply):
Correct Answer
A. Insecure, as bad actors can easily see the code and hack it B. Secure, because many developers can easily see the code, look for problems, and mitigate problems when they are discovered Your Answer: Correct C. Causing security problems, because developers come from different organizations and companies and do not understand each other's code D. Insecure, because no one is in charge of security E. Secure or insecure, depending on the quality and priorities of the project maintainers, but at least users can judge this by open discussion and code inspection Your Answer: Correct
QUIZ RESULTS YOU GOT 3 OUT OF 3 CORRECT: 100%
3 3 3 0
TOTAL (3) ANSWERED 3 CORRECT 3 INCORRECT 0
RTR
OSS ecosystems dominate almost all of the major ingredients in the world’s technology infrastructure. In this section, we will just survey a few of the most widely used and/or recognizable ones. We do not mean to slight those not specifically mentioned here, as a full list would be extremely long and actually impossible to accurately compile.
By the end of this chapter, you should be able to enumerate and briefly discuss successful OSS projects, including:
- Linux kernel
- Git
- Apache
- Python, Perl and other computer languages
- TeX and LaTeX
- GNU: gcc, gdb, etc.
- X and desktop managers
- OpenStack, Kubernetes and other projects.
The Linux kernel has been an open source project since its inception in 1991. It is the basis of almost all of the world’s computing infrastructure, from the most powerful supercomputers to the largest number of mobile devices, based on Android, built on a Linux kernel.
The Linux kernel has become ubiquitous in an enormous range of devices and embedded products, from personal fitness devices such as FitBit, to almost all DVR boxes and televisions, to many automotive systems, to in-flight entertainment systems, to medical devices; the list is infinite.
The kernel development community is large and mature. While still headed by its founder, Linus Torvalds, there is a well-formed organizational structure of subsystem maintainers and mailing lists, contributor guidelines and methods, many world-wide conferences every year where developers can get to know each other and work in person, etc.
The pace of the kernel development is rather astounding, and a new version is put out every 10-12 weeks; take a look at The Linux Kernel Archives to see the latest information. Literally thousands of developers (most of whom work for hardware and software companies, as well as for industrial collaborative consortia) are actively engaged in advancing the new releases.
The Linux kernel is released under GPL Version 2; there are some sections that have dual licensing, with other open source licenses. For more details, read GNU General Public License, Version 2.
Git is a distributed version control system that is used worldwide for an astounding number of collaborative products.
Git was created by Linus Torvalds in 2005 to handle the increasingly difficult task of coordinating and consolidating the work of thousands of contributors to the Linux kernel. Many others have continued to develop Git, which has Junio Hamano as its long time maintainer.
While there have been version control systems in existence as long as people have worked together, Git is unparalleled in its ability to handle widely dispersed individuals over the Internet.
It is also the basis of GitHub, which hosts more than one hundred million of open source projects repositories; GitLab, another easily available host, handles quite a few projects as well. Most of these OSS projects actually fail, but having a freely available (and zero cost in most instances) home for these projects has enabled easy startup and a straightforward path to scaling to larger communities.
Like many other tools (such as compilers and debuggers), Git can be used in non-open source projects as well; this in no way contaminates using it for OSS work.
Work on the Apache HTTP Server began in 1995. Today, it is the most widely used web server, with roughly one third of the market share. Another open source project, nginx, has almost as many users. It operates under the umbrella of the Apache Software Foundation (ASF), while the server can be found at the Apache HTTP Server Project website. It is hard to overestimate the importance of Apache; there are many related projects that permeate the entire international technical space. Apache is released under the Apache Software License, which has also been adopted by many other projects. It is more permissive than the GPL, as we will discuss later.
Perl, Python, Ruby, GNU Compiler Collection (GNU), LLVM compiler infrastructure project, Rust
Each of these has somewhat different licensing situations. However, the license does not extend to programs or libraries, or any other product developed using the language. In quite a few cases, the language standards may be set by some kind of standards body, but there can be multiple implementations, some OSS, some not, such as the case with C++. Or, the language can be more closed (such as C# or Java) in terms of accepting contributions from beyond a single entity, such as a corporation or committee.
TeX is really a typesetting system rather than a word processor program, and was developed from the outset to work on any kind of hardware imaginable. LaTeX, short for Lamport TeX, is a document preparation system.
TeX was invented by Donald Knuth and first released in 1978. Knuth is an extremely influential computer scientist widely recognized for his wide-ranging contributions, in particular to algorithm analysis and development. TeX is really a typesetting system rather than a word processor program, and was developed from the outset to work on any kind of hardware imaginable.
LaTeX, short for Lamport TeX, is a document preparation system widely used especially in academia for communication and publication of scientific documents in many fields , such as mathematics, computer science, engineering, physics, linguistics, philosophy, etc. It was created by Leslie Lamport in 1983. LaTeX is intended to provide a high-level, descriptive markup language that leverages the power of TeX in an easier way for writers. Essentially, TeX handles the layout side, while LaTeX handles the content side for document processing.
Most users of TeX use higher level versions, such as LaTeX, which are more user-friendly, and, in turn may have multiple implementations; the most popular LaTeX version is texlive.
Note that input is in plain text, rather than the WYSIWG scenario you have in popular office suites, and mark up tags and macros control the final appearance of the output.
Given its provenance, TeX was first widely adopted by mathematicians and a specialized version, amstex was used, and later AMS-LaTeX. To date, there is simply no other system which can render mathematical formulae and equations as well at TeX. From there, TeX and then LaTeX rapidly spread through academia, where it is still the dominant form in many science disciplines, and is also used in many other places. There are specialized versions or additions to make LaTeX work with a wide variety of languages, and there are also extensions for embedding it in other applications and computing languages and vice versa, as well as WYSIWYG versions that some find easier to use. And all has always been under an open source license.
The GNU Project has provided many essential ingredients for virtually all modern computer technologies, under various versions of the GPL (General Public License). It was started in 1983 by Richard Stallman. Its name stands for Gnu’s Not Unix. It is very closely related to the Free Software Foundation (FSF).
GNU Projects
-
gcc
- The GNU Compiler Collection (GCC) is a compiler system produced by the GNU Project supporting various programming languages. It's a compiler for C, C++, Objective C and other high level languages.
-
gdb
- The GNU Debugger is a portable debugger that runs on many Unix-like systems and works for many programming languages, including Ada, C, C++, Objective-C, Free Pascal, Fortran, Go and partially others.
-
glibc
- The GNU C Library, commonly known as glibc, is the GNU Project's implementation of the C standard library. It is the most fundamental and widely used program library. Many other libraries are closely related to it.
-
bash
- GNU Bash or Bash is a Unix shell and command language used widely as the default login shell for most Linux distributions. It is the most common command line shell.
-
coreutils
- The GNU Core Utilities or coreutils is a package of GNU software containing reimplementations for many of the basic tools, such as cat, ls, and rm, which are used on Unix-like operating systems.
GNU tools such as gcc and gdb were essential to the development of Linux from its earliest stages. The adoption of the GPL was absolutely essential to the success of the Linux kernel and the rise of operating system distributions based on it.
There are a number of ingredients that are used to instantiate the graphical desktop interface seen on any Linux laptop or workstation, including:
-
X Window System
- The underlying software that handles basic screen, input device and other point operations. A newer and more secure alternative, Wayland, is moving into its current role.
-
GNOME, KDE, XFCE
- Desktop Manager frameworks that control the operation of graphical interfaces, drag and drop between them, appearance of the desktop, etc.
All of these are distributed and developed under commonly used open source licenses.
There are many other desktop manager alternatives used in specialized distributions and embedded devices.
There are many other large scale (as well as small) collaborative projects that are based on open source software. A few critical examples:
- OpenStack: the most widely deployed open source cloud software in the world
- Kubernetes: also known as K8s, is an open-source system for automating deployment, scaling, and management of containerized applications.
- ONAP: Open Network Automation Platform
- Hyperledger: The open source, global ecosystem for enterprise-grade blockchain technologies that are at the core of critical developments and implementations around the world
- Node.js: Node.js® is an open-source, cross-platform JavaScript runtime environment.
- Xen: THE MISSION OF THE XEN PROJECT IS TO ADVANCE VIRTUALIZATION TECHNOLOGY ACROSS A WIDE RANGE OF COMMERCIAL AND OPEN-SOURCE DOMAINS. BY PROVIDING A POWERFUL AND VERSATILE HYPERVISOR, THE PROJECT AIMS TO ENABLE INNOVATION, SCALABILITY, SAFETY, AND SECURITY IN VIRTUALIZATION SOLUTIONS.
- Core Infrastructure Initiative: The Core Infrastructure Initiative (CII) was an effort to improve the security of open-source software.
- Automotive Grade Linux: Automotive Grade Linux is a collaborative open source project that is bringing together automakers, suppliers and technology companies to accelerate the development and adoption of a fully open software stack for the connected car. With Linux at its core, AGL is developing an open platform from the ground up that can serve as the de facto industry standard to enable rapid development of new features and technologies.
- Node.js
- PostgreSQL
Solution 4.1
- Android: Used in the majority of smart phones and other mobile devices world wide, built on top of the Linux kernel
- Apache Web Server (httpd): About half of all web servers are running Apache and its related products.
- Social Media: Virtually all social media platforms (Facebook, Twitter, Instagram etc.) are using OSScomponents throughout.
- Search Engines: GOOGLE and other search engines have vast data farms running Linux.
- Weather Forecasting: Essentially 100 percent of the world’s supercomputers run Linux, and forecasting the weather is one of the most intensive uses.
- Personal Fitness Devices: IncludingFitBit.
- DVRs: Almost all set top boxes and video recorders run Linux
- Medical Devices: A large number of medical devices used routinely every day are running Embedded Linux.
All these projects make extensive use of OSS tooling, such as gcc, make, glibc and languages such as Python, Perl and Ruby.
QUIZ RESULTS YOU GOT 3 OUT OF 3 CORRECT: 100%
Question 4.1
The git version control system arose from needs of which project?
A. Apache B. The Linux kernel Your Answer: Correct C. Kubernetes D. Python E. GNU
Question 4.2
Which of the following are OSS projects used to construct a graphical interface? Select all answers that apply.
A. X Window System Your Answer: Correct B. Microsoft Windows C. GNOME Your Answer: Correct D. Wayland Your Answer: Correct E. KDE Your Answer: Correct
Question 4.3
Which of the following are projects emanating from the Free Software Foundation? Select all answers that apply.
A. gcc Your Answer: Correct B. bash Your Answer: Correct C. gdb Your Answer: Correct D. Apache E. glibc Your Answer: Correct
- video introduction
By the end of this chapter, you should be able to:
- Understand the importance of constructing a coherent open source strategy for your organization.
- Explain the strategic considerations involved.
- Discuss the factors involved in designing an effective open source program office.
- Sketch out the ingredients used in the design.
- Plan the implementation of the strategic policy.
No engineer would construct a bridge or highway without a design and specifications, and no military would enter a war without planning.
However, OSS is entered into many environments without clear conscious strategies. There are many cases where engineers and system administrators began to adopt it simply because it was better and/or easier to fix, and even hoped that those above them in the food chain would not notice, e.g. there are many examples of a Linux print or email server being snuck into the office so the IT staff would need to come in less on weekends to fix breakage more common on other operating systems.
Using administrative tools without careful consideration of licensing requirements is less likely to run into potential problems than using code during development. Sloppiness, however, can lead to a host of problems if anything is distributed outside the shop, even if there are no modifications made, depending on the particular licensing involved.
Nowadays, use of OSS has become ubiquitous and even shops that were virulently opposed to its use are now heavily invested in it, with the best known example being Microsoft. Thus, a well-thought-out and implemented strategy for OSS use and adoption is not a luxury, it is a basic building block essential for success.
Key considerations involved in developing a strategy for integrating open source into an organizational and business framework have been well summarized in the article Open Strategizing: Key Considerations for an Open Source Strategy by Anat Richter.
The three main sequential steps listed by Anat Richter:
-
Establishing an Open Source Office
- This cannot be an afterthought. Someone has to be in charge and aware of all considerations.
-
Formulating an Open Source Policy
- This will depend on the current situation, needs, and future plans.
-
Implementing an Open Source Policy
- This can be tricky, but good planning helps a lot.
The open source program office serves as a center for functions, including:
- Choosing OSS code to be used
- Facilitating adoption of the code and popularizing its availability and usage
- Keeping track of usage
- Auditing compliance considerations
- Making sure proper information and training is available to all employees, as needed
One of the first tasks is to determine the placement of such an office within the organization. It should be relatively high in the organizational chart. Such an office may reside in the research and development command structure, or it may be placed under the legal department. There is no one answer for this choice of placement, but whomever is responsible must be sensitive to both the developer and legal needs. Otherwise, impedance mismatches will only grow and cause trouble down the road.
Perhaps the first step in open source policy construction is to ascertain how much of your product is already using open source components and how large an achievable goal can be. It is highly common for successful projects to be built of more than half OSS, and fractions of the code base that are OSS as high as 90 percent are not unusual. When done properly, it may even be just the upper crust of a product, the customer facing user interface that contains private code, and even that interface may be built entirely on OSS components. The value added can be in the integration of the components rather than the components themselves.
Some key considerations:
- How much of the product is desired to be OSS? Are there current closed source components which could be replaced by open source ones, perhaps both improving the product and its forward maintenance?
- Which license(s) are a good fit to your needs and which licenses can you use in various OSS components you integrate, such as libraries, APIs, etc.? Are you best off with permissive licenses (like BSD or Apache) or more restrictive ones (like GPL)? Can you mix them in one or more released products?
- Do you have a strong enough understanding (and legal help) to set up procedures for using both proprietary and open source code in a way which protects both types?
- How will you review the code to make sure things are being done properly, that inappropriately licensed material does not find its way into your code, both the open and closed parts? Will you use automated license compliance tools, such as FOSSology, SPDX, or FOSS Bar Code Tracker.
A good starting point would be to check the Linux Foundation's Open Compliance Program. Security considerations should be built in from the beginning. There will be OSS components that are generally assumed to be of high quality in this regard, with strong and prompt action when security problems are uncovered. Someone will be directly responsible for auditing and approving the secure nature of all incorporated OSS. Disclosure requirements mean that you have to be able to report to any interested client or user information about all OSS components, including information such as the original upstream source, the version, licensing considerations and dependencies. This is the same information you want about any components you include, i.e. you may be someone else’s upstream.
A policy is worthless without both a structure for ensuring implementation, and having a dedicated staff tasked with carrying it out. Training will need to be developed (or outsourced), targeted to more than one staff element, including:
- Developers
- Procurement
- Legal
- Security
Community participation and outreach can also be critical. Who are designated spokespeople? Who monitors developer participation and contributions? Specific procedures will have to be developed for (often automated) code inspection of any incorporated elements for license compliance, etc. Clear assignment of responsibility is required in all areas.
Question 5.1
Establishing an open source office and implementation plans:
Correct Answer: B
A. Should be done once your organization has some good OSS experience under its belt. Otherwise, a lot of work may be wasted and adoption of OSS delayed while time is wasted debating plans that may never be needed and issues that will not be of any concern.
B. Should be done from the outset, before bad practices and sloppy code usage gets baked in and is hard to fix later, and to give a guide for developers and planners on how to incorporate OSS early in their engineering work.
Your Answer: Correct
QUIZ RESULTS YOU GOT 1 OUT OF 1 CORRECT: 100%
- watched video
By the end of this chapter, you should be able to:
- Explain the mission of the TODO Group.
- Understand the importance of spreading success stories in using OSS to new venues and organizations.
- List reasons why codifying goals and practices makes OSS adoption easier and faster.
The TODO Group was founded in 2014 with the express mission of fostering the adoption and spread of open source software in the corporate business community. Founding members included: Box, Dropbox, Facebook, GitHub, Google, HPE, Khan Academy, Microsoft, Square, Stripe, Twitter, Walmart and Yahoo. A full list of members is available at the project's website.
Some of the TODO Group Members Retrieved from the TODO Group website
TODO is a open community of practitioners who aims to create and share knowledge about Open Source Program Offices or similar Open Source initiatives. TODO Group community is open to all and is formed by its Community participants (including OSPO Associates) and General Members. TODO Group Community is open to all. Usually, practitioners coming from an organization with an OSPO or an established strategy and policies around Open Source / FOSS efforts within its organization support TODO by becoming General Members. Our General Members include: TODO Group Member Products/Projects (96). The TODO Group became a Linux Foundation collaborative project in March 2016.
Every industry today uses and relies on OSS. It is virtually impossible to build a successful business model that fails to incorporate open source components at multiple levels. However, there still exist gaps in taking advantage of the full capabilities available, and in organizational comprehension of the essential concepts and necessary practical methods and considerations. Many corporations and other organizations have faced the task of appreciating and incorporating OSS methods, and the members of the TODO Group have united to share their experiences so as to help each other surmount the challenges involved. Besides easing individual members to accomplish their goals effectively, sharing helps develop a fertile ecosystem that enables organizations to work cooperatively and speak the same language, helping to develop partnerships and make deals, etc.
🚩 Note that most members of the TODO Group do not have OSS as their central mission (as do companies such as commercial Linux distributions like Red Hat, SUSE or GitHub), but they employ OSS in the essential parts of their products as well as their infrastructure.
The TODO Group focuses on:
- Sharing best practices for running large open source programs.
- Codifying goals for successfully managed projects.
- Spreading knowledge about helpful tooling and instrumentation.
- Listing and describing member projects that are successful examples.
Every organization has its own unique set of needs, challenges and practices. Sharing allows for faster and better establishment of enabling open source strategies.
The TODO Group and the Linux Foundation have made available a series of guides that publicize best practices gained from the experience of companies that use and develop OSS.
- How to create an open source program office
- Measuring your open source program's success
- Tools for managing open source programs
- Using open source code
- Participating in open source communities
- Recruiting open source developers
- Starting an open source project
- Open source reading list
- Improve your open source development impact
- Shutting down an open source project
- Building leadership in an open source community
- Setting an open source strategy
The TODO Group has also published a series of case studies by a diverse group of companies, that describe how they adopted and manage open source into their corporate DNA:
- Autodesk
- Capital One
- Comcast
- Dropbox
- Meta
- Microsoft
- Oath
- Porsche
- Red Hat
- Rit
- Salesforce
- SAP
- Uber
The best way to learn about how companies have successfully incorporated OSS into their DNA is to read some the case studies we listed previously. You will notice the companies range from those built on and around OSS (like Red Hat) to those in other industries, such as Capital One in the financial sector, Comcast in telecommunications and entertainment, and general business products, such as SAP and Salesforce. Pick whatever interests you and learn about those experiences.
Question 6.1
The TODO Group:
Correct Answer
A. Operates under the auspices of the Free Software Foundation (FSF) and GNU B. Is a project under the umbrella of the Linux Foundation Your Answer: Correct C. Is run by the OpenStack Foundation D. Is a joint project of the UK's MI6 and Russia's FSB, and consults with the CSA and NSA as needed
THE TODO GROUP Knowledge Check QUIZ RESULTS YOU GOT 1 OUT OF 1 CORRECT: 100%
1 1 1 0 TOTAL (1) ANSWERED CORRECT INCORRECT
- watch video
- OpenChain is a Linux Foundation project
- certification process
By the end of this chapter, you should be able to:
- Explain the role of the OpenChain Project in elucidating and sharing the ingredients in successful open source compliance programs.
- Discuss the requirements that must be satisfied to receive the OpenChain Conforming certification and the right to display the logo.
- List the compliance artifacts that are incorporated in these requirements.
The OpenChain Project is tasked with identifying and sharing the components of high quality open source compliance programs. It is a Linux Foundation project that was born in 2013 out of the recognition that many organizations had adopted open source software components, and had developed license compliance programs. While there was considerable overlap in methods and quality, there were insufficient methods of establishing trust in the quality and consistency of software being shared and exchanged.
The OpenChain Project established an open set of requirements for effective management of free and open source software (FOSS). If an organization meets this specification of requirements, it can be certified as OpenChain Conforming and receive the right to display the logo indicating that it is fully compliant. The most current version of the Specification can be accessed here.
Please note that the distributed ledger technology Openchain is not related to this project.
OpenChain Project Members Retrieved from the OpenChain Project Members web page (404)
There are six main categories required for a compliance program:
- Know Your Free and Open Source (FOSS) Responsibilities [i.e., “Policy and Training”]
- Assign Responsibility for Achieving Compliance
- Deliver FOSS Content Documentation and Artifacts
- Review and approve FOSS content
- Understand FOSS Community Engagement
- Certify Adherence to OpenChain Requirements
It is important to note that the OpenChain Specification does not indicate compliance with any particular open source license; it describes the level of trust that an organization has built a program that establishes license compliance.
Such a program will have compliance artifacts, including:
- Source code
- Build scripts
- License copies
- Attribution notices
- Modification notices
- SPDX data
- Any other materials licenses may require in a deliverable.
OpenChain Conformance can be established online through a self-certification process. It is as simple as answering a series of questions. A step by step video showing the process can be viewed on YouTube: OpenChain Self-Certification.
There is also detailed documentation available on how to achieve conformance, including guides for best practices for compliance, engineering and reuse, as well as checklists for compliance and conformance. There are also tooling overviews guidance for establishing proper workflows.
Successful certification lets an entity use the OpenChain logo.
OpenChain Conformance Logos
Below is a figure showing an example enterprise process flow:
End-to-End Compliance Management: Enterprise Process Example
Exercise 7.1: OpenChain Self-Certification
As noted, an organization can undergo a self-certification process at https://www.openchainproject.org/conformance.Watch the video at https://youtu.be/EAUAr9S8Ipk to get a sense of how to earn the use of the OpenChain Conformance logo.
Question 7.1
The OpenChain Project:
Correct Answer
A. Operates under the auspices of the Free Software Foundation (FSF) and GNU B. Is a project under the umbrella of the Linux Foundation Your Answer: Correct C. Is run by the OpenStack Foundation D. Is a joint project of the UK's MI6 and Russia's FSB, and consults with the CSA and NSA as needed
THE OPENCHAIN PROJECT Knowledge Check QUIZ RESULTS YOU GOT 1 OUT OF 1 CORRECT: 100%
1 1 1 0 TOTAL (1) ANSWERED CORRECT INCORRECT
- watched video
By the end of this chapter, you should be able to:
- Know what resources are available to ease entrance into an OSS project community.
- Start off by studying how the community works and deciding what it is you want to contribute and why.
- Identify maintainers and understand their methods and proper workflow.
- Learn how to get constructive input for your contribution and how to work in the open.
- Develop best strategies for how often to contribute and in what size pieces.
- Work on the human interaction skills necessary to work with other members of the community, especially those who may be difficult.
- Strive to develop long-term relationships and really join the community.
Before making contributions to any OSS project, you should investigate it, understand its workflow and style, and identify the scope and nature of your work: 🔍
- Identify how the project communicates; start by reading the mailing list for the project. Also, study any existing archives.
- If there is an Internet Relay Chat (IRC) network, join it.
- Understand how contributions are submitted. Is it through mailing lists? Email? A revision control system, such as git or subversion? A good way to lessen your standing on the project - is to be sloppy or ignorant of proper procedures, including even trivial sounding matters like using an email client that does not mess up white space.
- Study previous history. Has your idea been considered before and rejected? Or, if generally seen as worthwhile, is there already an individual or group working on the idea? If so, you - should join them in many cases, not start over.
- Does the project offer veteran contributors as mentors?
Rather than beginning by submitting code, it is often far better to offer your services for testing, finding bugs, and similar work. Many open source projects complain that this is their weakest link. Make sure you are competent at whatever programming or scripting language the project uses; developers will be impatient in correcting your code on that level. Find the right balance between asking for suggestions and review early in the process, instead of waiting too long and dumping a lot of perhaps poorly conceived work on people at once. No one enjoys contributors who pontificate all the time without contributing code, and no one likes massive patch sets that cannot be absorbed easily. Finally, please be polite, respectful, avoid obscenities, flaming and trolling. While some communities tolerate it, it should not be encouraged, as it affects the quality of work and it is just wrong.
When contributing to OSS projects, you must keep in mind various things, to ensure you abide by the rules of the project and its community.
It is very important for any open source project to document and understand where all contributed code is coming from. This is important for multiple reasons, including:
- Knowing who the expert(s) are on the contributions; when there are questions or modifications, generally these people need to either answer them or do the work, or at least review it before acceptance.
- Knowing the legal provenance of the code; if someone either inadvertently or intentionally uses code that violates the license it may have used in another context this can be a big problem. In this case it is important to know who to blame.
Two common methods for handling these issues are the use of either a Contributor License Agreement (CLA) or a Developer Certificate of Origin (DCO). There can be arguments about which is more appropriate for a given project. However, a number of well-established projects have migrated from CLA to DCO as their operative integrity documentation. With a CLA (the older method) contributors must sign an agreement which can vary from project to project. For example, they may or may not grant copyright and/or patent licenses. The DCO was introduced by the Linux Foundation in 2004 and is simpler and of lighter weight. Every git commit must be signed off with this approach. With either approach, the servers hosting the software repositories (such as GitHub or GitLab) can use checks to ensure all contributions either have a CLA in place or are signed off as required by a DCO. Generally speaking a CLA is a one-time operation, whereas a DCO requires a sign-off with each contribution, but this is trivial (i.e., adding the -s option to each commit operation). A detailed summary of the use of DCOs is given here. When you use the -s option, each commit will have a message embedded such as:
Signed-off-by: Some Developer <[email protected]>
Unless you are starting a new project, there will already be a community established ethos, and formal or informal leadership structure. 🔍
Questions to ask include:
- Why does this project exist and why was it started?
- Has it diverged far from its original purpose?
- Is the contributing community large or small? Continuously active, or only sporadically so?
- Does it have a collective or singular governance structure?
- What kind of license does it adopt, and is there a contributor license agreement (CLA) you have to agree to?
Keep in mind the majority of OSS projects never really take off; there are millions of them on GitHub. However, all projects have to start somewhere, and most start rather small.
Most contributors get involved in open source projects because:
- There is a bug/problem that interferes with their use of the project
- They want to add additional functionality to the project.
Sometimes, you are just interested in something cool and want to learn about it. Some projects have a "janitorial" list of things to do, relatively straightforward tasks that can help you get your feet wet. Beginning contributors who go on to become important to a project rarely start with patches that do not affect functionality, such as those that just tidy up code appearance. [So make a significant contribution from the start.]
There will always be maintainers whose job is to carefully review code submissions. Having a good relationship with relevant maintainers is absolutely crucial. There may be one maintainer for the whole project (perhaps the benevolent dictator), but, for projects with many pieces and large communities, there will be subsystem maintainers as well. A good example is the Linux kernel community. The job of a maintainer is difficult. They have to:
- Understand and review all submissions.
- Make sure they add something other than complication and do not introduce bugs.
- Make sure they do not conflict with other patches.
- Do all this in a timely fashion; it is very frustrating for contributors if maintainers appear not to be working on review.
- Do everything respectfully.
Be respectful and be patient. If you treat a maintainer as either someone who works for you, or as a difficult boss, it will not be appreciated. Offer to review other contributors' submissions and be helpful. Offer to debug new features and bug fixes; this is often one of the most neglected areas in OSS development.
As a new member of the community, you probably do not know the history. Your bright new idea might actually be an old one. If so, perhaps your idea has been discussed and rejected. Or, perhaps there was work on it and it petered out for a variety of reasons. This does not mean your idea is a bad one; maybe it could not have been done in the past, but is feasible now, as software and hardware have evolved. Propose your ideas to the community before you go very far with them; take advantage of institutional memory. Get suggestions and input. Never just introduce an idea and suggest someone else do it; this is often a sure road to oblivion ("Show me the code!"). Just say you are asking for opinions before you go too far, to see if people think it is a worthwhile and feasible path. In doing this, you will get more eyeballs from the beginning, and it can only help. If you are uncomfortable having other people look at your work often, OSS is not for you!
Code contributions are best digested in small unit-sized bites. Overloading maintainers and mailing lists with large code dumps is a good way to make sure your code is not reviewed. Besides just being a lot for reviewers to consume, it becomes difficult to process the individual parts; perhaps most of the changes are good, but some require modification. It is hard to back out if it is in one large patch. So, breaking up your work into a series of sequential patches is often the best way to go. For example, you may have a separate patch for each file you are modifying. It is possible your functionality will not work until all patches are included. That's fine. Wings and feet were not fully functional as they began to evolve.
Like any Internet-based community, OSS projects tend to attract some rather interesting characters. Unfortunately, some of them can be quite irritating, even offensive. Leave your ego at the door. Take a deep breath if someone is nasty to you and do not answer in kind. Respect other opinions. Maybe your approach is not the best one. Learn when to yield if you want to get code upstream. When making criticisms or engaging in any discussion thread, be polite and respectful. If someone is going over the line, rely on community members with long history to try and calm things down and resolve issues. Do not feed the trolls. Occasionally, contributors may be purged from a project. This is actually a pretty rare occurrence. The main reason is maintainer inactivity, or being unpleasant. While in a good project every effort is made to keep things energetic but mellow, there can be problems with behavior involving discrimination and offensive conduct with respect to:
- Race
- Sex
- Sexual preference
- Religion
- National origin
- etc.
In addition, expressing (strongly) political or religious sentiments and opinions is generally out of place. People from rather far sides of the spectrum often work together successfully.
Most projects do not appreciate drive-by contributors: those who provide some enhancement or fix, but then disappear. Besides the initial contribution, there often needs to be a commitment to long-term maintenance. Otherwise, bit rot will ensue for all but trivial contributions. For this reason, one sometimes sees good code rejected unless someone steps forward and volunteers long-term maintenance. Thus, think about the future when doing submissions.
One good way to develop (good) long-term relationships in the community is to offer to help in areas not directly related to your main interests. It is helpful to be viewed as a good citizen. This is particularly true if you are contributing as part of a commercial organization; community members will see you as being a real member of the community, and not just someone exploiting it.
Exercise 8.1: You have submitted a patch to a subproject maintainer and have received no response for a few days. You should:
- Quietly resend the patch every day until there is an acknowledgment of receipt, acceptance, or rejection. You can even use an automated email bot to do this.
- Since the maintainer is ignoring you, you should send the patch set to the entire project, through a mailing list if that is an appropriate channel. Make sure to tell people you are doing this because the maintainer is not doing their job andy ou are routing around them.
- Send a follow up inquiry referring to the patch to the maintainer, possibly resubmitting. Make sure you know the person’s normal workflow and response times to see if you are being singled out. See if you can ascertain if the maintainer seems to be active but ignoring you, or is inactive in general.
Solution 8.1
Obviously the last answer is most appropriate. You need to be patient and respectful. When you achieve the exalted status of being a maintainer you will learn how stressful, time consuming and often thankless a task it is. Maintainers are squeezed by contributors from below, and by upstream leaders and reviewers from above.As usual there is not one exact protocol to follow. But be a good citizen and appreciate the work people do and next time you have a change set, your treatment can only be better.
Exercise 8.2: Is it Better to submit Small or Large Patches?
Suppose you are making a rather large set of changes to a project. Is it better to:
- Submit a sequential series of bite-size patches that are easier to digest, even if there is not much functionality until all are accepted.
- Wait until the work is essentially complete and submit one or two large change sets.
Solution 8.2
Usually small and incremental patch sets are preferred. This has the advantage that they are easier to understand, and if something is not quite right, it is preferable to catch it early and nip it in the bud. Larger patch sets tend to overwhelm reviewers and maintainers. Sometimes they are just refused out of hand for that reason.There is no hard and fast rule here and some projects might like to see the complete set. Sometimes it is very hard to test anything until the work is complete, as the intermediate evolutionary steps are just that. However, smaller patch sets are more often the preferred method.
Chapter 8 Quiz
Question 8.1
When first getting involved in an OSS project, you should (Select all answers that apply):
Incorrect Answer
A. Start contributing code immediately and learn the workflow through doing B. Start by helping test and report results Correct Answer C. Consider what it is you would like to work on, either because it is needed by you, or it just interests you Your Answer: Correct D. Start by posting messages on the appropriate mailing lists suggesting things you would like to see, pointing out the weak aspects of the project, etc.
Question 8.2
When working on an OSS project, the best strategy is to:
Correct Answer
A. Save up changes until you can submit multiple changes in one large patch, rather than bother maintainers often B. Submit changes one at a time in a sequential manner, even if the full change will not do much until the entire patch series is incorporated
Question 8.3
When there is someone in the project community that gets abusive or just difficult to deal with, you should (Select all answers that apply):
Incorrect Answer
A. Immediately respond in kind and slap them down B. Let established community members moderate the person Your Answer: Correct C. Actually read what they are saying; even if phrased in a nasty manner, the points might need addressing, and you can respond to them calmly Correct Answer D. Find out what their demographic characteristics are (race, gender, nationality, etc.) and use that to belittle them
QUIZ RESULTS YOU GOT 1 OUT OF 3 CORRECT: 33%
3 3 1 2
- watch video
By the end of this chapter, you should be able to:
- Define Continuous Integration (CI) and explain its potential benefits.
- Explain Continuous Delivery (CD) and Continuous Deployment, and how these three concepts (CI, CD and Continuous Deployment) are interrelated and form different stages of one coherent process.
- Weigh the costs and benefits in some detail of using Continuous Integration methods; both sides of the balance sheet can be quite appreciable.
- Investigate some of the pre-packaged tools that can be used to ease Continuous Integration.
- See how the Linux Kernel Continuous Integration Project is a great example of a large scale use of these techniques.
- Understand the role of the Continuous Delivery Foundation as a neutral gathering point for work in the CI/CD world.
Once upon a time, most software was written by a relatively small group of developers, often working in the same location and in frequent contact. Coordination and division of responsibilities was done in a straightforward manner. Revision control systems were developed long ago to accommodate more than one contributor working on a project. Usually, there is a central repository which stores the authoritative copy of a project, with one or more developers possessing the ability to make changes and then check them in. Things get more complicated when there are many developers working in many different locations on a project with many subsystems. The Linux kernel was the first really huge distributed development project, and its creator, Linus Torvalds, invented the git system for rationalizing distributed development. We will discuss revision control systems, especially git, in greater detail later. However, a revision control system does not guarantee that what a diverse group of contributors is doing together actually works, or that one set of new code or bug fixes does not conflict with another. That can only be done by testing.
🔍 While testing, the following factors need to be taken into consideration:
- Can overlapping sets of changes be applied simultaneously, or do they conflict (a good revision control system such as git can handle most of this work, but it still often requires human intervention).
- When all changes are applied, does the project even compile? For example, one patch set might remove a header file that another one needs. This does not get picked up by the revision control system.
- Does it work on all possible targets? That might mean different hardware (say x86 vs. ARM) or different operating systems (say Linux vs. Solaris or Windows) or different library, language, or utility versions.
- What does working mean? Are there non-trivial test suites that can exercise a representative workload enough to give confidence things are fine?
Continuous Integration techniques ensure that testing is so frequent that any problems cannot persist for long and that distributed developers stay on the same page. Projects absorb changes rapidly and in real time (usually multiple times per day) and run automated tests to make sure things are in harmony.
So far, we have been a little sloppy, and conflated what might be delineated as three separate steps, or stages.
-
Continuous Integration (CI) -Changes are to be merged into the main branch as often as possible. Automated builds are run on as many variations of software and hardware as possible; conflicts are resolved as soon as they arise.
-
Continuous Delivery (CD)
- The release process is automated and projects are ready to be delivered to consumers of the build. Thorough testing is done on all relevant platforms.
-
Continuous Deployment
- The product is released to customers, once again in an automated fashion.
The time gap between these steps is meant to be as close to zero as possible. In a perfect world, developer changes can reach end user customers the same day or even in minutes. These terms can be defined somewhat differently; for example, Continuous Integration can be considered to include both delivery and deployment.
-
Costs
- Changes have to be merged very often, probably at least once a day, putting a possible strain on developers.
- The repository must be monitored by a continuous integration server, which runs scripted automation tests every time contributions are made. Staff has to be allocated to do this.
- Scripts and other tools have to be run to perform automated tests, report their results, and take appropriate actions. It can be a lot of work to prepare this infrastructure.
-
Benefits
- Developers do not go down the wrong path and compound fixable mistakes or get in each other's way. In the end, this saves time.
- The build steps are fully automated, all the work has been done upfront.
- Regressions (bugs which break the working product) may be minimized. Releases should have fewer bugs.
Setting up a continuous integration pipeline is not trivial, and can take quite a bit of experience and effort to get it right. But "an ounce of prevention is worth a pound of cure". There are many existing tools and services that can help make work less daunting.
There are many well-developed Continuous Integration software tools. For their summary, please take a look at the Top Continuous Integration Tools: 51 Tools to Streamline Your Development Process, Boost Quality, and Enhance Accuracy article by Stackify.
Some of the main tools are:
- Jenkins
- Travis CI
- TeamCity
- GoCD
- GitLab CI
- Bamboo
- Codeship
- CircleCI
Jenkins is the most widely used. Some of these products are free in cost, others are not.
The Linux kernel development community employs a very robust continuous integration project called kernelci. While it was originally initiated by Linaro, it is now an independent Linux Foundation project. Its mission is to help ensure long-term quality control, stability and facilitate maintenance of the Linux kernel. It works with:
- The kernel developer community, giving them a tool to use in their daily workflow as needed.
- Hardware manufacturers to help them stay close to kernel changes and development.
- Helping downstream developers by performing solid upstream testing.
Detailed reports are continuously updated in real time.
The launching of the Continuous Delivery Foundation (CDF) was announced by the Linux Foundation in March 2019. It is designed to be a vendor-neutral home for coalescence of significant projects in the CI/CD universe. By establishing and documenting best practices, working out guidelines and making training available, the goal is to evangelize and spread CI/CD and DevOps practices and improve product release processes.
- Jenkins is an open source CI/CD system.
- Jenkins X Is an open source CI/CD solution for cloud applications on Kubernetes.
- Spinnaker is an open source multi–cloud continuous delivery platform.
- Tekton is Is an open source software specification for CI/CD components.
The Technical Oversight Committee (TOC) has an open governance model. Founding members include: Alauda, Alibaba, Anchore, Armory.io, Atos, Autodesk, Capital One, CircleCI, CloudBees, DeployHub, GitLab,Google, HSBC, Huawei, IBM, JFrog, Netflix, Puppet, Rancher, Red Hat, SAP, Snyk and SumoLogic.
Question 9.1
What is the correct order for the Continuous Delivery, Continuous Deployment and Continuous Integration steps?
Correct Answer
A. Continuous Delivery, Continuous Deployment and Continuous Integration B. Continuous Delivery, Continuous Integration, and Continuous Deployment C. Continuous Integration, Continuous Delivery, and Continuous Deployment Your Answer: Correct D. Continuous Deployment, Continuous Delivery, and Continuous Integration
Question 9.2
Continuous Integration (Select all answers that apply):
Correct Answer
A. Can put a strain on developers who have to submit changes constantly Your Answer: Correct B. Can be costly to set up, as it requires a server and staff and training developers to use properly Your Answer: Correct C. Can speed development, as wrong paths can be discovered before they are pursued for a long time Your Answer: Correct D. Speeds up the development process, as builds and tests are fully automated Your Answer: Correct E. Minimizes regression bugs since thorough testing is done at every stage before release (i.e. continuously) Your Answer: Correct F. None of the above
Question 9.3
There are many tooling suites used for Continuous Integration. The most widely used one is:
Correct Answer
A. Travis CI B. Bamboo C. Jenkins Your Answer: Correct D. Emacs E. Eclipse
QUIZ RESULTS YOU GOT 3 OUT OF 3 CORRECT: 100%
There are many ways to work with open source software. In particular, what kind of license is adopted will have strong effects on:
- How the project is developed
- How the project is deployed.
There are two broad classes of licenses:
- Restrictive, such as the GPL in all of its forms
- Permissive, such as the BSD and Apache Foundation Licenses.
There is a lot of misinformation spread about open source software and licenses, which can make organizations reluctant to use and develop for it. We will discuss the main kinds of this false information. Let's get started!
By the end of this chapter, you should be able to:
- Understand the difference between restrictive and permissive OSS licenses.
- Explain the main features of restrictive licenses, such as the various GPL versions.
- Explain the main features of permissive licenses, such as BSD and Apache.
- Discuss what considerations are to be weighed in determining the correct license.
- Debunk the main Fear, Uncertainty and Doubt (FUD) attacks on OSS licensing.
- Explain the relationship between software licenses and patents and the role of the Open Invention Network (OIN).
There is an almost infinite variety of available open source licenses (unfortunately, perhaps). Which one should be chosen depends on both needs and philosophy.
Restrictive Licenses Demand the software remains open and place strong limitations on any attempt to make proprietary closed products. Changes to the code must be made available to future recipients. A prominent example is the GPL (General Public License), a copyleft license.
Permissive Licenses Do not require modifications and enhancements to be generally available. Prominent examples include the BSD (Berkeley Software Distribution) and Apache licenses.
We will discuss some of the major licenses available and how to work with more than one license at a time. Because we are not lawyers, please investigate for yourself what is most appropriate for your project. For a list of available licenses and their attributes, see the Comparison of Free and Open-source Software Licenses article on Wikipedia.
The term FUD is shorthand for Fear, Uncertainty and Doubt (or Elmer Fudd). Surprisingly, first usage goes back as far as the 1920s; use of the acronymic form seems to date from the 1970s. Broadly speaking, it means disseminating misinformation to influence recipients to avoid certain strategies, products or classes of products by appealing to fear. Microsoft was widely accused of spreading FUD about Linux in the 1990s. However, in present day, Microsoft has stopped doing so and is actually employing OSS widely.
As applied to OSS, FUD statements include:
-
OSS is a virus. If you include it in your product, all your source must be made available to everyone.
- This is simply not true. You have to be careful about respecting licenses, but many prominent companies have learned how to combine open and closed software in their offerings. And, there are companies and organizations dedicated to helping ensure this is done properly.
-
OSS infringes on software patents, and the related claim that it forces you to grant patent rights to others.
- This is simply not true. Once again, proper legal analysis is required.
-
OSS products leave nowhere to turn when they break, or to get technical help.
- Many open source products are supported by serious, long-living companies (such as Red Hat Enterprise Linux), as well as smaller organizations. In addition, there is a lot of freely available help online, and there are many consultants that can be hired. In fact, there is more competition available for such help than there would be with vendor lock-in on a product.
-
OSS requires a lot of legal help to avoid the above pitfalls, and is thus very expensive.
- Even proprietary software requires significant legal analysis to properly avoid copyright and patent infringement, etc. OSS is no different and not more expensive. Also, having all the software being available in source form expedites the auditing process. Companies will indeed require interaction with lawyers, either on staff or external, to make sure they do not violate copyrights and licenses. There are many kinds of licenses and you have to be careful. But once an organization develops proper reasonable procedures, it is just a standard part of any project. Part of this is to train the developers to understand the dos and don'ts of working with OSS.
A software patent gives exclusionary rights to material such as a computer program, library, interfaces or even techniques and algorithms. The earliest software patents appear to have been granted in the early 1960s. Like all patents, software patents must be filed for in each nation (or trading block such as the European Union) in which coverage is desired. This makes it an expensive and time-consuming project to do thoroughly. Exactly what a patent can or cannot cover varies from jurisdiction to jurisdiction, e.g. in the United States this means exclusion of “abstract ideas”, although what is or is not covered can always be fought about. Many people feel there should be no software patents at all, that sufficient protections for intellectual property already exist under copyright and trademark laws. However, since it is highly unlikely software patents will be abolished, developers and organizations have to learn to deal with them properly. More often than not, software patents have been used defensively, with corporations cross-licensing each other’s work to avoid litigation. However, there are many well-known cases of expensive legal battles as well.
The Open Invention Network (OIN) was created as a global patent pool; companies and other entities which join the OIN enter in a mutual non-aggression agreement within the OSS-based ecosystem. OIN members agree in return for not suing each other over patent issues, they gain access to each other’s patent portfolio free of charge. While the OIN has existed since 2005 it has been growing rapidly, to over 3000 members in 2019. Major members include: Google, IBM, NEC, Philips, Red Hat, Sony, SUSE, Toyota, and Microsoft. Microsoft’s joining of OIN in October 2018 was a major event, opening up over 60,000 patents for use by OIN members and, not surprisingly, leading to a spike in new memberships. For many, this represented proof of Microsoft’s determination to be a good citizen in the OSS world (of course for others, it fed conspiracy theories...).
The intersection of software licenses and patents is rather complicated, and different OSS licenses have different properties in this regard. For example, here is a partial table of properties about whether or not a license conveys explicit patent rights from contributors to users.
🚩 Please note that sometimes, projects will use a license that does not provide an express grant of patents, but then add a grant of patents in a separate file. An example of this can be seen with the use of a LICENSE or PATENTS file in the source code.
Choosing the best license is a very important decision and requires careful consideration. Switching to a different license later, during the life of the project, can be rather difficult, if not impossible, especially when there are many contributors with rights.
🔍 An excellent website by OSS Watch, the United Kingdom's advisory service for issues related to free software and open source software, can help you pick an appropriate license by interactively asking you a series of questions. Briefly, these questions involve:
- Should the search include only very popular and widely used licenses?
- Should the licenses be restrictive or permissive? Depending on choice, there are more detailed questions.
- Jurisdiction? Might not be important.
- Patent policy on your patents (if relevant)?
- Patent retaliation clauses?
- Giving credit to original authors? And, if so, in what form should such attribution appear?
- Must source be given, out even if the application is used only on a server?
- Can authors' names be used to promote a product or service?
Based on your answers to these eight (or more) questions, the above website will rank available licenses according to how they correlate with your answers.
📖 There are several resources you can use to help you choose the most appropriate license for your project:
- Open Source Initiative - Open Source Licenses by Category
- Sponsored by GitHub - Choose an Open Source License - this site walks you through the properties you must consider, helping you decide what license makes sense.
- TLDRLegal - Browse Software Licenses and Summaries - this site tags software licenses with key properties to help you navigate easier and better understand the existing licenses.
- GNU - Various Licenses and Comments About Them - this site provides a description of copyleft licenses and comments about them.
Projects sometimes take contributions from sources utilizing different OSS licenses. Mixing and matching can be a complicated process with restrictive licenses generally dominating over permissive ones when combined. In addition, certain sections of code can contain additional provisions and it can require analysis to determine what additional rights may have been granted to certain particular parties. Individual projects often have specific guidelines as to what combinations are in tune with the project. Some examples include:
🚩 You should always check with your legal counsel before contributing to a project under a different license than the project uses. Thinking about this early can save a lot of pain later!
- Go to http://oss-watch.ac.uk/apps/licdiff/ and interactively specify the parameters needed to choose the right type ofOSSlicense.
- Try a number of variations.
Summary This is your selection:
- Popular and widely used: Yes
- Licence type: Permissive
- Jurisdiction: No
- Grants patent rights: No
- Patent retaliation clause: -
- Specifies enhanced attribution: No
- Addresses privacy loophole: No
- Includes 'no promotion' feature: No
Question 10.1
What are some considerations that go into OSS license selection (Select all answers that apply)?
Correct Answer
A. How much you want to charge for buying software based on the project B. How the code will be used (i.e. good or evil) C. Policy on patents Your Answer: Correct D. Should all modifications be public (restrictive vs permissive) Your Answer: Correct
Question 10.2
What does FUD stand for?
Correct Answer
A. Fear, Uncertainty and Doubt Your Answer: Correct B. Elmer Fud(d) C. Free Unsupported Development D. Failed, Unbelievable and Damaged E. Fitbit, USB and Dirty
Question 10.3
Which of the following is a copyleft license?
Correct Answer
A. GPL Your Answer: Correct B. BSD
QUIZ RESULTS YOU GOT 3 OUT OF 3 CORRECT: 100%
- watch video
By the end of this chapter, you should be able to:
- Explain the role of tools that can examine entire software projects and evaluate whether they have dealt with OSS correctly.
- Appreciate the important work of FOSSology as a comprehensive tool for measuring and documenting license compliance.
- Enumerate the tasks that FOSSology can accomplish using associated applications, including Nomos and Monk.
- Explain how SPDX standardizes, as well as simplifies methods for sharing open source project data.
- List the data SPDX tools can communicate in a software bill of material, including copyrights, licenses and security information.
- Appreciate the role of the CHAOSS project in developing tools for evaluating the health of open source ecosystems.
- Grasp that there are other compliance projects gaining traction, such as Quartermaster and Tern.
FOSSology is a system for measuring and documenting open source license compliance. It is also a toolkit for running various kinds of scans from the command line, as well as using a graphical, generally web-based interface. The first version of the FOSSology project was founded and hosted by Hewlett-Packard and appeared in 2007; it became a hosted project under the Linux Foundation in 2015. FOSSology is part of the Automated Compliance Tooling (ACT) project. In short, this tool can scan either individual files, or entire source archives, and identify licenses, copyrights etc., and document the results, and help identify where information is lacking or is poorly documented.
The presented figure is just a small subset of the major organizations that utilize FOSSology.
Fundamentally, FOSSology is a scanner that can work on individual files as well as entire projects, performing operations on every file in the source archive. There are several types of scans that can be done, each of which has an associated scanner application.
-
Nomos
-
Nomos searches for licenses using regular expression matching, combining with context identification, i.e., does a certain phrase occur in the vicinity of another certain phrase? Nomos will return strings such as GPL or MIT, but generally not give the specific version of a license. It can also detect an unknown license and report that.
-
Monk
- Monk also does text-based searching and can report back on license variations, such as GPL-2 and GPL-3+, etc. It cannot recognize unknown licenses, as it works off of a stored table in the server. Monk is complementary to Nomos.
-
Copyrights
- Copyrights are investigated by searching for “copyright” and “(C)” in the text. Less explicit assignments such as phrases like “written by” are also evaluated. Users can examine the reported results and decide on the correct answers (including no explicit copyright).
-
Export Control Codes (ECC)
- Export Control Codes (ECC) are also determined by regular expression searching (this is also called export control and customs codes).
The FOSSology interface is designed to let the user evaluate all the results found, and correct or investigate further. If software is well designed with respect to explicit statements of copyrights, licenses, authorship, export controls, etc., then the tools will have a much easier time. Thus, software design that considers these issues from the start will breeze through a FOSSology scan.
The Software Package Data Exchange (SPDX) open standard provides a common format for sharing data about software copyrights, licenses, security information, etc. This open standard is used to communicate software bill of material information. By its very nature, this avoids repetition of work and streamlines compliance. The SPDX workgroup (hosted by the Linux Foundation) is responsible for maintaining the specification. Founded in 2010, this group has representatives from more than 20 organizations who have agreed to unite around a common standard. A copy of the latest (version 2.3 as of 20231106) can be found on the project's website.
Please take a look at the diagram to learn what the contents of an SPDX document are. SPDX v2.1 document components
A project will include an SPDX file that consists of a series of tags and values, separated by a colon and space (there are other possible formats but this is the simplest one). Here is an example lifted from David Wheeler's excellent SPDX Tutorial:
```sh
SPDXVersion: SPDX-2.1
DataLicense: CC0-1.0
PackageName: FooPackage
Originator: David A. Wheeler
PackageHomePage: https://github.com/david-a-whe...
PackageLicenseDeclared: MIT```
```
The full list of possible tags can be found here. [This is different than the broken link in the lesson. I googled this link and put it here.]
The list of possible licenses has “short form”, such as Apache-2.0, or GPL-2.0-only or GPL-3.0-or-later. These are convenient for inclusion in source files, without a rather long list of lines describing the license.
For example, source files in the Linux kernel will have at the top lines like:
```sh
// SPDX-License-Identifier: GPL-2.0
/*
* linux/kernel/sys.c
*
* Copyright (C) 1991, 1992 Linus Torvalds
*/
#include <linux/export.h>
#include <linux/mm.h>
```
The Community Health Analytics Open Source Software (CHAOSS) project was launched in 2017 under the umbrella of the Linux Foundation. The purpose is to develop tools and metrics for evaluation of the health of open source projects and ecosystems, with a view to increasing sustainability over time. This is important for stakeholders making good, informed decisions about how to be involved. The goals are:
- Establish excellent metrics for measuring community activity, contributions, and health
- Produce software tooling or doing the analysis
- Build project health reports.
FOSSology and SPDX are not the only tooling projects designed to evaluate and ensure open source license compliance. One particular grouping of tools was announced in November 2018 by the Linux Foundation. The name of the new consolidated project is Automated Compliance Tooling. At present (April 2019) it has four constituent projects:
- FOSSology runs license, copyright and export control scans, etc. We have discussed this in some brief detail.
- SPDX tools construct and communicate software bills of material. We have also discussed this in some brief detail.
- Quartermaster (QMSTR) is integrated into the product build system itself, rather than be a separate “final” step. Thus, it is DevOps-oriented and it facilitates Continuous Integration and Development methods. QMSTR was contributed by its initiator, Endocode.
- TERN inspects package metadata in container images, and elucidates the bill of materials. With the growing adoption of containerized deployments, demystifying their contents and examining their compliance become of increased importance. TERN was created and contributed by VMware.
The word "automated" in the ACT name is important. Having a smooth workflow with compliance evaluation built into the work flow end-to-end can only lead to speedier and more confident development and deployment, and integration with CI/CD methods. ACT welcomes additional tooling projects and offers a stable, neutral foundation for them with the support of the Linux Foundation.
Exercise 11.1: Experimenting with FOSSology
The FOSSology project offers easy to understand options for learning the basics of how to use the toolsThe easiest way is to use the project’s online testing facility. To do this:
- 1. Point your browser to https://fossology.osuosl.org/repo/2.
- 2. Login with username=
fossyand password=fossy. - 3. You can choose from a number of examples already enumerated; most of these are archives in zipped or tarball form. To keep things simple, choose at first a simple file.
- 4. In the
select action box, click onCopyright/Email/Urlin the drop down menu. You see the copyright, license and url information highlighted in appropriate colors. - 5. Try clicking on some of the other fields in the top ribbon banner to see more useful information.
Two possible variations are:
- 1. Look at one of the archives on the test site and learn how to browse through its files.
- 2. Even better, upload one of your files or source archives and analyze its contents.
FOSSology Using: End-to-end workflow
- Everything in this course can be another course.
Exercise 11.2: Running FOSSology in a container locally
Extra Credit
Doing this requires some familiarity with running docker containers. If you already have this knowledge the exercise is easy. Otherwise you can follow the instructions to install docker for your system to do so, and then follow the instructions below from the FOSSology project.
Question 11.1
Which organizations offer comprehensive open source license compliance measuring tools, which can often be run in an automated fashion (Select all answers that apply)?
Incorrect Answer - it's both FOSSology and SPDX Tools
A. The GNU Project B. FOSSology Your Answer: Correct C. OpenStack Foundation D. SPDX Tools Correct Answer E. Apache Software Foundation F. Apple
QUIZ RESULTS YOU GOT 0 OUT OF 1 CORRECT: 0%
How an open source software project is led has important consequences for its growth, longevity and capability. Good leadership empowers contributors to do their best work. It also leads to a more efficient and rapid development process. Building trust is important, as is making sure experienced project members can mentor less experienced ones and train subsystem maintainers.
By the end of this chapter, you should be able to:
- Explain how good leadership can empower contributors to do their best work
- Analyze the elements that contribute to the success or failure of open source projects
As we have discussed, there are many ways an open source project can be organized, ranging from very tight control, to an almost anarchistic ecosystem. Assuming that there is enough structure that some entity, either an individual or a committee, has final control, there is still a wide choice of operating methods. The object of this section is to point out the advantages of leading a project, as opposed to just controlling it. Capable leadership empowers participants to contribute higher quality and more imaginative work.
A good leader listens. Good ideas can originate from many different contributors, even recent entrants into a project. Even though leadership paradigms such as BDFL (Benevolent Dictator for Life) are popular, note the use of the word “benevolent”. The “dictator” may make the final decision (generally more efficiently than by a majority vote or consensus of a committee) but that does not alleviate them from listening and giving proper consideration to debates in the community and even occasionally including changes and including contributions they personally have problems with. If the controllers of a project take and do not give back by mentoring and moderating, they are limiting what a project can accomplish.
Depending on the size of the project, there may be only one level of leadership, or there may be more, including subsystem maintainers. It takes thought, preparation, and experience to be a good maintainer; one has to handle contributors efficiently, know when to send patches back for revision, send them up the chain with or without revision, or discard them with proper explanation. This will require training, or more precisely mentoring, from those who have done it successfully before. This is a crucial role and nothing screws up an open source project more than when contributors feel they are not getting sufficient attention and help. Another level of mentoring involves new members of a project. If they can attach themselves to more experienced contributors, who may or may not be maintainers, they can learn the ropes with less pain. They can learn how to produce patches in the proper form, how to phrase questions and proposals, and acquire realistic expectations of how quickly their work might be considered and whether it has a chance of success. Proper mentoring empowers people, leads to a more efficient workflow, and contributes to long term vitality and growth of a project.
Without trust, an open source project cannot function:
- Contributors must trust that their submissions will be treated with respect and given reasonably prompt consideration.
- Project leaders must trust that subsystem maintainers are qualified and capable of doing their job. And if they are not living up to their responsibilities, they can be tutored on doing a better job or be asked to transition out of the role.
Trust must be earned and must go beyond the benefit of the doubt. As a project matures, reputations are earned and new members should be aware of the history.
The vast majority of OSS projects do not succeed. The reasons are varied, but include:
- Insufficient interest.
- Competition from more established projects that duplicate the intended functionality, even if the new project seems to have features the older one does not.
- Poor leadership. This can mean a lack of attention and prompt response, as well as antisocial behavior or autocratic rule, etc.
- Not enough developers.
- Insufficient funding. Serious projects often require developers who have employers that see the work as relevant to their mission at best, or at worst, do not disparage contributing as either a time waster or potential intellectual property leak.
Insufficient or uninformed attention to licensing issues that alienate contributors who are wary of either giving their code to others without adequate protections or compensation. It is not surprising that most projects fail as the barrier to entry is very low and projects are very easy to start. The low survival rate should not be seen as a bad thing; most (but not all) successful open source projects start small and it can be quite difficult to predict which ones will survive and thrive.
Exercise 12.1: Enabling New Contributors
Enabling new contributors to join a project is essential to both longevity and vitality. It is the normal life cycle that many project members eventually leave a project for many reasons ranging from not enough time or lost interest, to disagreeing with changes in goals or governance issues or personal conflicts. New blood brings in both more labor or fresh ideas and tools.
What are some of the methods that can be used to ease the glide path of new project members and get them to the point where they both feel comfortable with the project and can make useful contributions?
Notes 12.1
- mentorship
- open discussions
- avoid abuse
- lead by example
Question 12.1
The abbreviation for BDFL stands for:
Correct Answer
A. Better Delivery from Licensing B. Benevolent Dictator for Life Your Answer: Correct C. Big Data for Linux D. Bad Drama from Litterers
Question 12.2
The job of a mentor includes (Select all answers that apply):
Correct Answer
A. Training new project contributors in how to submit their work successfully in the right form Your Answer: Correct B. Knowing when to eject someone from a project because their work is not of high enough quality C. Teaching contributors how to handle criticism, as well as how to give constructive criticism Your Answer: Correct D. Finding ways to empower people to give their maximum contribution Your Answer: Correct
Question 12.3
Which statement is true?
Correct Answer
A. A project can start without a clear idea of what license to follow, and it can adopt it later, as needed B. A project should have a clear license to begin with. It may possibly make a change later if it really needs to, but that is often non-trivial if there are quite a few contributors Your Answer: Correct
QUIZ RESULTS YOU GOT 3 OUT OF 3 CORRECT: 100%
By their very nature, open source projects are able to draw on a wide variety and dispersion of contributors. This only strengthens projects and speeds their pace of development. Besides the fact that respecting and encouraging diversity is the right thing to do, it is also beneficial both to a project and to the individuals who work on it.
By the end of this chapter, you should be able to:
- Discuss the importance of encouraging and respecting diversity in open source projects.
- Illustrate collaboration best practices in an open source project.
The word open that appears in OSS might be taken to indicate a friendly, welcoming environment. Unfortunately, unless a proper atmosphere permeates a project, this may be a false promise. Diversity in an OSS project can mean many different things:
- Race and national origin
- Sex and gender identity
- National and geographical/regional characteristics, including both language and cultural differences
- Religious beliefs and political views
- Acceptance of different opinions and methods about how the project should take shape and develop in the future.
While it is just the right thing to do to accept contributors and reviewers from divergent backgrounds, diversity also leads to a better project due to unleashing more sources of new ideas, approaches, and contribution.
It is no secret that software engineering, system administration, and IT in general has been dominated by men. We have all seen conference group pictures where it is hard, if not impossible, to pick out the women (if any) in the photographs. Furthermore, incidents of unwelcome behavior, including harassment, have long been a problem at some conferences. For this reason, all conferences organized by the Linux Foundation event staff (and other responsible conference organizers) have very stringent policies regarding avoiding any improper behavior and dealing with it when it unfortunately rears its head. Besides the fact that discrimination is wrong, it also shrinks the potential pool of contributors and users. It can only improve a project to let its reach expand to the full extent of its potential community. It has to be a hard and fast rule that any misogynistic and/or homophobic statements in mailing lists, discussion groups and chats, or in the comments that accompany patches, should be promptly beaten down and criticized for what they are. Conference speakers should also be aware of what they are projecting. This includes avoiding marketing with sexual pictures, jokes and innuendos. Furthermore, one should try to use sex-neutral terms and pronouns, etc. Obviously, there are gradations, and one does not have to tie oneself up in knots trying to be sensitive. If someone finds a community has behavior that they find demeaning or just unwelcoming, they won’t contribute. Everyone loses.
There are a number of diversity issues that involve variation of race, national origin, geographical region, and language. Discrimination based on race and national origin is obviously wrong, but is rarely explicit. However, there are more subtle issues. For example, language and geographical differences can lead to misunderstandings and insults. While it is true that the vast majority of open source projects are primarily done in English, and thus developers tend to have at least a working familiarity with it, that is quite different than everyone being on the same page. Even among English speakers, words can have quite different meanings and usage in various countries, such as the United States, the UK, Ireland, Canada, Australia, etc. Without giving examples, it is not hard to come up with cases where a word has a much stronger meaning in one region than another, or is obscene in only some locations, etc. Obviously, we cannot avoid all such problems, but we can choose our words with sufficient care to avoid being offensive, as well as to be clear. This includes the use of idioms and metaphors which will seem mysterious to non-native speakers unless properly explained. Another aspect is tone. In some cultures, cursing is rather routine and coarse and not offensive in the way it is in others. Especially important is how criticism is offered. To some groups, direct criticism and saying they are wrong is seen as hostile and poorly received. Adopting the proper criticism methods is a two way street; those used to a very direct approach should continue to do so, but should avoid unnecessary nastiness; those not used to such an approach need to develop a thicker skin and pay attention to the content as much as possible, not the wrapping it comes in. Depending on the project, quite a bit of work can be done on localization. This includes providing translated documentation, as well as using standard methods to provide help and error messages in multiple languages. This kind of internationalization also can provide an initial glide path for individuals to contribute to a project while learning its internals and history. Once again, the goal is to promote healthy and rapid development from as large a pool of contributors and users as possible, not just to behave nicely because it is good manners.
Discussion of religion is generally inappropriate and has nothing to do with either technical or governance aspects of an open source project. While overt denigration of anyone’s beliefs is obviously rare and inappropriate, so is proselytizing in favor of any religion. The same is true with political opinions extraneous to the project, such as national or international affairs. However, this can be harder to avoid than issues of religion. For one thing, OSS is built on the concept of freely available intellectual product. Once freedom is in the air, it becomes hard to rule all political discussion as out of order, especially if there are actions in repressive countries that can be seen as attacking fellow open source contributors, users and advocates. Once a discussion with political aspects begins to veer away from direct bearing on the project (a divergence usually quite visible to the naked eye), it should move off of mailing lists and other project discussion venues into other channels. And at all times, one should not dissuade anyone from participating in a project because their political or religious views are different from the main currents followed by project members and the leadership; once again, this limits the potential pool of contributors and users and hurts a project.
Collaboration involves:
- Dividing up work among more than one group or individual, to be merged later.
- Reviewing the work done primarily by one contributing force and either accepting as is, sending back for revisions, or changing and sending further along.
- Competition between differing approaches to adding new facilities or solving problems.
OSS projects can run into trouble when these kinds of discussions get ugly. Besides the usual advice about being civil and respectful, it is also very important to keep in mind that different people do things in different ways. This can mean accepting someone else’s contribution that has the same effect as yours, even if you think yours is better; if you do not prevail in the discussions of which approach to merge, you may just have to accept your less desired outcome. Or, you can put on your thinking cap and think more of things your approach can take care of the other approach cannot, and add increased functionality and try again. What is preferred this year may change next year; do be prepared to reinvigorate a rejected submission - if things change going forward. We discuss in other places how to handle things when they get out of hand. Here we are just saying that with more debate and more opinions, often a better project can be built.
Question 13.1
The majority of OSS projects carry out their discussions in English. Therefore:
Correct Answer
A. Language misunderstandings are unlikely to occur B. Misunderstandings will only occur between native and non-native English speakers C. It is fine to often use unexplained idioms and cultural references D. One should always speak as clearly as possible, and while not being dull, think about the audience and what they will be familiar with Your Answer: Correct
Question 13.2
OSS projects should explicitly encourage diversity with respect to (Select all answers that apply):
Correct Answer
A. National origin and language Your Answer: Correct B. Sex and gender identity Your Answer: Correct C. Height and weight D. Race Your Answer: Correct
Question 13.3
You propose a new way of doing something, or adding a new feature. Another contributor points out this idea was rejected previously. You should (Select all answers that apply):
Correct Answer
A. Study the previous discussion and see whether it applies, including whether or not differences between your ideas and the old ones were not seen Your Answer: Correct B. Ask whether or not technical abilities have evolved to the point where something that was not feasible can be done today Your Answer: Correct C. Ask whether the need for this feature has grown; the earlier discussion may have been premature Correct Answer D. Forget about it. No one will pay attention to you if you go forward if the other contributor has more history and trust than you do
QUIZ RESULTS YOU GOT 3 OUT OF 3 CORRECT: 100%
When a distributed development project uses Git, a repository must be available somewhere on the Internet. GitHub has played that role for literally millions of shared source projects and sub-projects, as have other repository hosting servers.
By the end of this chapter, you should be able to:
- Explain how Git enhances collaboration in open source projects.
Git began as an offshoot of the Linux kernel development community, initially created by Linus Torvalds himself. However, it was quickly realized it could be used for any project that had similar needs:
- A large group of contributors
- A widely dispersed community of contributors
- A very open development method with frequent releases.
Use of Git grew explosively after the founding of GitHub in 2008, three years after the creation of Git. In 2018, GitHub was acquired by Microsoft. In early 2020, GitHub had over 40 million users and 100 million repositories! Before GitHub projects needed to have their own servers to host repositories and have a good amount of knowledge to setup, administer and secure and protect the integrity of repositories. It is now pretty easy to get a project rolling quickly. There are other sites that offer similar services, including:
- GitLab
- GitKraken
- Launchpad
We will concentrate on GitHub, as it is the largest and it is exclusively focused on Git as its source control system.
Only selected collaborators can see the repository, or clone it (make a local copy), or download its contents in a variety of forms. The owner must specifically authorize each collaborator.
Anyone given the proper link, such as https://github.com/an-account/a-repo, can copy, clone or fork the repository, or download its contents. However, unless the owner authorizes them as a collaborator, one does not have permission to upload or make modifications.
Public repositories are free of charge on GitHub. Originally, all private ones cost money, but since Microsoft’s acquisition of GitHub, this policy has changed, and many users can easily use the free access versions. More complex needs will incur charges depending on the scale or the number of projects and subprojects that require separate repositories, the number of collaborators and how many institutions they represent and other criteria.
However, by definition, OSS projects are almost always hosted on public repositories if they are using Git.
💡 GitLab also has both private and public repositories with the same essential considerations in force as for GitHub.
Originally, Git had only a command line interface. Working this way is quite comfortable to long-time Linux and UNIX users. While there were and are graphical interfaces to browse repositories, these were essentially read-only, they did not permit making changes or performing many other essential operations. GitHub and similar sites, such as GitLab, offer a full GUI (Graphical User Interface) that is instantly more familiar to many users and makes collaboration on projects that much easier. Furthermore, the GUI is operating system agnostic; i.e. whether you are on a Windows, MacOS, Solaris, Linux, etc. system, you have the same capabilities. It is easy to ignore the GUI when working with GitHub and just interact with it over the Internet, totally from the command line. Experienced users find this more convenient so there are no limitations involved.
Question 14.1
Select the correct statement. GitHub:
Correct Answer
A. Is maintained by The Linux Foundation B. Is a private company acquired by Microsoft in 2018 Your Answer: Correct C. Is maintained by the GNU project
Question 14.2
When using a GitHub public repository:
Correct Answer
A. Anyone on the internet can download the data as well as upload information and make modifications B. Anyone on the Internet can download the data, but only authorized collaborators can upload information and modifications Your Answer: Correct
Question 14.3
To use Git, you:
Correct Answer
A. Must be using Linux B. Be on any operating system and version has Git installed (almost all do) Your Answer: Correct
QUIZ RESULTS YOU GOT 3 OUT OF 3 CORRECT: 100%
While Git is very powerful, and graphical interfaces such as GitHub may be very useful, more complicated workflows can benefit from the use of tools such as Gerrit.
Describe git workflow with Gerrit.
Git has well-established methods of workflow. It is pretty flexible and projects can choose quite different approaches, but it is always based on a cycle of:
- Make changes in the code, probably in a development branch.
- Commit those changes to the branch; there may be one or more changes (patches) per commit.
- Publish those changes through a push or a pull request.
- The changes will be reviewed and merged if necessary or sent back for further work.
This method works well if you have a basic pyramid view of things where each subsystem has a maintainer managing their piece of the work, and that manager has ultimate authority over the changes to that subsystem before passing them up the pyramid for ultimate review and merging. But what if you want to have more dispersed review? That’s where Gerrit comes in.
Gerrit is built completely on Git, but it adds another layer of code review before changes are committed to the authoritative master repository. This review is likely to be done by multiple contributors rather than just one powerful maintainer. While such a workflow is not exactly new (most projects have multiple reviewers with some structure for who makes the ultimate decisions) the Gerrit architecture is designed to formalize this procedure. This works best when there is one change per commit, rather than a block of them, as it makes it easier to review and modify/reject/accept each one on its own merits. To learn more about Gerrit read its Documentation.
From the preceding diagram, you can see that Gerrit introduces a reviewing layer that lies between the contributors and the upstream repository.
- Contributors submit their work (one change per submission is best) to the reviewing layer.
- Contributors pull the latest upstream changes from the upstream layer.
- Reviewers are the ones who submit work to the upstream layer.
The reviewers evaluate pending changes and discuss them. According to project governing procedures they can grant approval and submit upstream, or they can reject or request modifications. Gerrit also records comments about each pending request and preserve them in a record which can be consulted at any time to provide documentation about why and how modifications were made.
Question 15.1
Gerrit is:
Correct Answer
A. A more advanced replacement for git B. An enhanced method of using git for more complex projects Your Answer: Correct
Question 15.2
Gerrit works best when:
Incorrect Answer
A. There is one change per commit Correct Answer B. There are many changes to commit Your Answer: Incorrect
Question 15.3
With Gerrit, contributors submit their work to:
Correct Answer
A. The reviewing layer, rather than fully upstream Your Answer: Correct B. The upstream and then reviewers pick from there
QUIZ RESULTS YOU GOT 2 OUT OF 3 CORRECT: 67%
20231114
I feel like a dog chasing my tail vis-a-vis finding the right entry point into making an open source software contribution.