Skip to content

Commit

Permalink
Merge pull request #3 from TeamNewPipe/master
Browse files Browse the repository at this point in the history
Repo update
  • Loading branch information
snappyapple632 authored Aug 4, 2019
2 parents c479e74 + da41b6d commit 3e81f09
Show file tree
Hide file tree
Showing 6 changed files with 143 additions and 19 deletions.
7 changes: 3 additions & 4 deletions docs/00_Prepare_everything.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,8 @@
# Before You Start

These documents will guide you through the process of creating your own Extractor
service of which will enable NewPipe to access additional streaming services, such as the currently supported YouTube and SoundCloud.
The whole documentation consists of this page, which explains the general concept of the NewPipeExtractor, as well as our
[Jdoc](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/) setup.
These documents will guide you through the process of understanding or creating your own Extractor
service of which will enable NewPipe to access additional streaming services, such as the currently supported YouTube, SoundCloud and MediaCCC.
The whole documentation consists of this page and [Jdoc](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/) setup, which explains the general concept of the NewPipeExtractor.

__IMPORTANT!!!__ This is likely to be the worst documentation you have ever read, so do not hesitate to
[report](https://github.com/teamnewpipe/documentation/issues) if
Expand Down
73 changes: 58 additions & 15 deletions docs/01_Concept_of_the_extractor.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,19 +57,20 @@ class MyExtractor extends FutureExtractor {

Information can be represented as a list. In NewPipe, a list is represented by a
[InfoItemsCollector](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/InfoItemsCollector.html).
A InfoItemCollector will collect and assemble a list of [InfoItem](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/InfoItem.html).
For each item that should be extracted, a new Extractor must be created, and given to the InfoItemCollector via [commit()](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/InfoItemsCollector.html#commit-E-).
A InfoItemsCollector will collect and assemble a list of [InfoItem](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/InfoItem.html).
For each item that should be extracted, a new Extractor must be created, and given to the InfoItemsCollector via [commit()](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/InfoItemsCollector.html#commit-E-).

![InfoItemsCollector_objectdiagram.svg](img/InfoItemsCollector_objectdiagram.svg)

If you are implementing a list for your service you need to extend InfoItem containing the extracted information
and implement an [InfoItemExtractor](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/Extractor.html),
that will return the data of one InfoItem.
If you are implementing a list in your service you need to implement an [InfoItemExtractor](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/Extractor.html),
that will be able to retreve data for one and only one InfoItem. This extractor will then be _comitted_ to the __InfoItemsCollector__ that can collect the type of InfoItems you want to generate.

A common implementation would look like this:
```
private MyInfoItemCollector collectInfoItemsFromElement(Element e) {
MyInfoItemCollector collector = new MyInfoItemCollector(getServiceId());
private SomeInfoItemCollector collectInfoItemsFromElement(Element e) {
// See *Some* as something like Stream or Channel
// e.g. StreamInfoItemsCollector, and ChannelInfoItemsCollector are provided by NP
SomeInfoItemCollector collector = new SomeInfoItemCollector(getServiceId());
for(final Element li : element.children()) {
collector.commit(new InfoItemExtractor() {
Expand All @@ -90,20 +91,21 @@ private MyInfoItemCollector collectInfoItemsFromElement(Element e) {
```

## InfoItems Encapsulated in Pages
## ListExtractor

When a streaming site shows a list of items, it usually offers some additional information about that list like its title, a thumbnail,
There is more to know about lists:

1. When a streaming site shows a list of items, it usually offers some additional information about that list like its title, a thumbnail,
and its creator. Such info can be called __list header__.

When a website shows a long list of items it usually does not load the whole list, but only a part of it. In order to get more items you may have to click on a next page button, or scroll down.
2. When a website shows a long list of items it usually does not load the whole list, but only a part of it. In order to get more items you may have to click on a next page button, or scroll down.

This is why a list in NewPipe lists are chopped down into smaller lists called [InfoItemsPage](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/ListExtractor.InfoItemsPage.html)s. Each page has its own URL, and needs to be extracted separately.
Both of these Problems are fixed by the [ListExtractor](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/ListExtractor.html) which takes care about extracting additional metadata about the liast,
and by chopping down lists into several pages, so called [InfoItemsPage](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/ListExtractor.InfoItemsPage.html)s.
Each page has its own URL, and needs to be extracted separately.

Additional metadata about the list and extracting multiple pages can be handled by a
[ListExtractor](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/ListExtractor.html),
and its [ListExtractor.InfoItemsPage](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/ListExtractor.InfoItemsPage.html).

For extracting list header information it behaves like a regular extractor. For handling `InfoItemsPages` it adds methods
For extracting list header information a `ListExtractor` behaves like a regular extractor. For handling `InfoItemsPages` it adds methods
such as:

- [getInitialPage()](https://teamnewpipe.github.io/NewPipeExtractor/javadoc/org/schabi/newpipe/extractor/ListExtractor.html#getInitialPage--)
Expand All @@ -117,5 +119,46 @@ such as:
The reason why the first page is handled special is because many Websites such as YouTube will load the first page of
items like a regular web page, but all the others as an AJAX request.

An InfoItemsPage itself has two constructors which take these parameters:
- The __InfoitemsCollector__ of the list that the page should represent
- A __nextPageUrl__ which represents the url of the following page (may be null if not page follows).
- Optionally __errors__ which is a list of Exceptions that may have happened during extracton.

Here is a simplified reference implementation of a list extractor that only extracts pages, but not metadata:

```
class MyListExtractor extends ListExtractor {
...
private Document document;
...
public InfoItemsPage<SomeInfoItem> getPage(pageUrl)
throws ExtractionException {
SomeInfoItemCollector collector = new SomeInfoItemCollector(getServiceId());
document = myFunctionToGetThePageHTMLWhatever(pageUrl);
//remember this part from the simple list extraction
for(final Element li : document.children()) {
collector.commit(new InfoItemExtractor() {
@Override
public String getName() throws ParsingException {
...
}
@Override
public String getUrl() throws ParsingException {
...
}
...
}
return new InfoItemsPage<SomeInfoItem>(collector, myFunctionToGetTheNextPageUrl(document));
}
public InfoItemsPage<SomeInfoItem> getInitialPage() {
//document here got initialzied by the fetch() function.
return getPage(getTheCurrentPageUrl(document));
}
...
}
```
78 changes: 78 additions & 0 deletions docs/07_maintainers_view.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
# Maintainers View

So I want to document some of the views i have when maintaining NewPipe.


### Keep it Streamlined
NewPipe is a Player for online videos on a smart phone, by means it is used
for entertainment reason. This means it does not have to be some professional
application, and it does not have to be complicated to be used.
However NewPipe might not focus on the casual user completely as there are
many features that are a bit more "tecki" and may require some knowledge about
technology, however all in all NewPipe should be easy to use, even for not teck
guys.

1. NewPipe does not have to be a air plane cockpit: __Don't add to much special
features__. If people want to do professionally things with Videos they
might use professional tools.
2. __Design the UI so it does make sense to the user__. Try to make it comply with
[material design guidelines](https://material.io/design/guidelines-overview/).
3. __Don't add to much features__: Think about the Betamax vs. VHS phenomena
or the Unix principle of having one program designed for one specific task:
If you add to much functionality you add complexity and this is not appealing
to the user. Focus on what NewPipe should be, and make it be only that.

### Bugfixes

![kde_in_a_nutshell](img/kde_in_a_nutshell.jpg)]

*Disclaimer: This is a meme maybe in real live it is different. Pleas no shit storm.*

__Always go for Bugfixes__, as the best application with the best features
does not help much if it is broken, or annoying to use. Now if a program
is in an early stage it is quite understandable that many things brake. This
is one reason why NewPipe still has no 1 in the beginning of its version
number.
However by now NewPipe is in a stage where there should be a strong focus on
stability.

1. If there are multiple Pull requests open, check the ones with the bugfixes first.
2. Do not add to much features every version, as every feature will inevitable
introduce more bugs. It is quite ok, if PRs stay open for a while (not to long though).
3. If there are bugs that are stale, or open for a while bump them from time
to time, so devs know that there is still something left to fix.
4. Never accept bugs. From my perception the community does not like to fix bugs, this is why you as a maintainer should
especially focus on perusing bugs.


### Features

Well features are also something that can cause a headache. You should always see adding features critical and question
whether that features does make sense, is useful and would actually be an advantage for the app. You should not blindly
say yes to features even if they are small, however you should also not directly say no as well. Think about it, may
be even for days before deciding whether you want to accept a feature or not. If you are not sure, try it, look into the
code, speak with the developer, and then make a decision and justify it. The criteria whether to add a feature or not
should be:

- Is the features just requested by one or two people or was the feature requested by multiple people?
- Is the code of the feature written well?
- Is it a quick and hacky solution and could a proper solution be implemented later on?
- Does the amount of code justify the outcome?

Maybe people will send a pull request that will add a frequently requested feature, but is implemented in a hacky way,
than don't add it, as you might get into trouble with that solution later on. Either through problems of extending the
feature, by introducing to much bugs or simply by braking the architecture or the philosophy of NewPipe. If so don't add it.

### PRs

If a PR contains one or more features/bugs be curious. The more stuff a PR changes the longer it will take to be added.
Also there might be things you are ok with, but then there are other parts that are not ok with and because of these you
can't merge it. This is why you should insist to make the dev chop down the PR into multiple smaller PRs if it's possible.

### Community

When you talk to the community stay friendly and respectful, and make sure a friendly and respectful tone will stay.
When you have a bad day just don't go to GitHub (an advice from my experience ;D ).



Binary file added docs/img/kde_in_a_nutshell.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
3 changes: 3 additions & 0 deletions theme/css/theme_child.css
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
.wy-nav-top i {
line-height: 50px;
}
1 change: 1 addition & 0 deletions theme/main.html
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@

<link rel="stylesheet" href="{{ base_url }}/css/theme.css" type="text/css" />
<link rel="stylesheet" href="{{ base_url }}/css/theme_extra.css" type="text/css" />
<link rel="stylesheet" href="{{ base_url }}/css/theme_child.css" type="text/css" />
<!-- local code syntax highlighting -->
<link rel="stylesheet" href="{{ base_url }}/css/github.min.css" type="text/css" />
<link rel="stylesheet" href="{{ base_url }}/css/highlight.css" type="text/css" />
Expand Down

0 comments on commit 3e81f09

Please sign in to comment.