Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[REVIEW, DO NOT MERGE] WebScience Migration Review #96

Open
wants to merge 138 commits into
base: ws-migration-review
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 111 commits
Commits
Show all changes
138 commits
Select commit Hold shift + click to select a range
1755a44
Improvements to the Messaging module
jonathanmayer Dec 12, 2020
d39c385
Adding new PageManager module and content script
jonathanmayer Dec 12, 2020
2f1ba38
Removing window type logic from PageManager
jonathanmayer Dec 28, 2020
990cdf3
Removing current page notification logic from PageManager
jonathanmayer Dec 28, 2020
7064fb6
Removing window and tab privacy state from PageManager
jonathanmayer Dec 28, 2020
ee5d0f3
Cleaning up PageManager initialization
jonathanmayer Dec 28, 2020
aae0159
Removing tab state cache from PageManager
jonathanmayer Dec 28, 2020
2ab115a
Cleaning up window state cache in PageManager
jonathanmayer Dec 29, 2020
97d5d11
Removing background script page attention events from PageManager
jonathanmayer Dec 29, 2020
0b9cda8
Adding an Events module for event listener management
jonathanmayer Dec 29, 2020
fd06a7d
Reimplementing page visit start and stop events in PageManager
jonathanmayer Dec 29, 2020
42f00a0
Documentation updates in PageManager
jonathanmayer Dec 29, 2020
b5d2d58
Fix for initial page attention state in PageManager
jonathanmayer Dec 29, 2020
d6982e7
PageManager improvements
jonathanmayer Dec 30, 2020
67d0cb5
Refactoring PageNavigation and removing PageDepth
jonathanmayer Dec 30, 2020
d953aa9
Documentation improvements in PageManager
jonathanmayer Dec 31, 2020
9e966c1
Improvements to scroll depth tracking in PageNavigation
jonathanmayer Dec 31, 2020
7c1752e
Improved event syntax in Events
jonathanmayer Dec 31, 2020
8e56ee2
Refactoring LinkExposure from PageEvents to PageManager
jonathanmayer Dec 31, 2020
5e959ca
Removing unused PageEvents import in SocialMediaLinkSharing
jonathanmayer Dec 31, 2020
c38e65b
Refactoring Scheduling to use Events
jonathanmayer Dec 31, 2020
ab32579
Removing unused domain category label in LinkExposure
jonathanmayer Dec 31, 2020
c2f0378
Updating README
jonathanmayer Dec 31, 2020
f5c2b23
Improvements in LinkExposure
jonathanmayer Dec 31, 2020
fda8185
Removing deprecated PageEvents
jonathanmayer Jan 1, 2021
34464d2
Adding match pattern support in Matching
jonathanmayer Jan 2, 2021
28e3374
Removing PageEvents import
jonathanmayer Jan 2, 2021
2122489
Bug fix in Matching
jonathanmayer Jan 2, 2021
b07391a
Updating PageNavigation to use match patterns
jonathanmayer Jan 2, 2021
61c32f2
Improvements to PageManager
jonathanmayer Jan 2, 2021
da1b659
Updating LinkExposure to use match patterns
jonathanmayer Jan 2, 2021
7fe6f22
LinkExposure rewrite
jonathanmayer Jan 4, 2021
92f1ab7
Improvements for link shorteners and AMP links
jonathanmayer Jan 4, 2021
a9e95e8
Improvements for event types and documentation
jonathanmayer Jan 8, 2021
698fbd8
Migrating PageNavigation to the event model
jonathanmayer Jan 8, 2021
9888872
Updating events in the PageManager content script
jonathanmayer Jan 8, 2021
6622350
Adding documentation in the PageManager content script
jonathanmayer Jan 8, 2021
f02c7ad
Bumping the manifest minimum version
jonathanmayer Jan 14, 2021
3c5cb62
add eslint config from rally core addon repo
akohlbre Jan 22, 2021
7a43069
add back incognito permission for tracking attn changes
akohlbre Jan 22, 2021
76194dd
Merge branch 'master' into PageManager
akohlbre Jan 22, 2021
035efa8
Passing isHistoryChange to the background page
jonathanmayer Jan 26, 2021
a4164f1
Bug fixes in the PageManager and LinkExposure content scripts
jonathanmayer Jan 26, 2021
6995fc7
Debugging output cleanup in PageManager content script
jonathanmayer Jan 28, 2021
25e1d94
Bug fix in the PageManager content script
jonathanmayer Jan 29, 2021
518bb0d
Merge branch 'PageManager' of github.com:citp/news-disinformation-stu…
akohlbre Feb 2, 2021
2d86a8e
Add the Rally template files
Dexterp37 Feb 8, 2021
1066290
Tweak the Rally build config
Dexterp37 Feb 8, 2021
5b4a92e
Use the bundled file in the addon
Dexterp37 Feb 8, 2021
5217162
Merge pull request #76 from Dexterp37/rallyjs
akohlbre Feb 10, 2021
845618c
Merge branch 'PageManager' of github.com:citp/news-disinformation-stu…
akohlbre Feb 10, 2021
8f7a224
partial conversion to event model, many small fixes from linter
akohlbre Feb 10, 2021
426d526
re-integrate modules, more events, temporary fix for LE lagginess
akohlbre Feb 19, 2021
6fc6b24
switch to dexie, linting fixes, use UTC, report timezone, redo analys…
akohlbre Feb 21, 2021
0befb82
page classification revamp
akohlbre Feb 22, 2021
48b79b8
small bug fix for new page class
akohlbre Feb 22, 2021
b63a232
.gitignore, JSDoc, and Makefile cleanup
jonathanmayer Feb 22, 2021
75c35b9
Updating VS Code build configuration
jonathanmayer Feb 22, 2021
dfd7c1c
Matching improvements
jonathanmayer Feb 23, 2021
bcea03e
Slightly more correct relative URL handling in LinkExposure
jonathanmayer Feb 24, 2021
e37aa43
Remove experimental regular expression optimization
jonathanmayer Feb 24, 2021
490bc68
Cleanup for URL normalization
jonathanmayer Feb 24, 2021
b62ad91
Matching regular expression support for parameters and fragment ident…
jonathanmayer Feb 24, 2021
4fd2d21
Matching reorganization, documentation, and cleanup
jonathanmayer Feb 24, 2021
db6390a
Adding optimized match pattern class in Matching
jonathanmayer Feb 25, 2021
f6b7bc4
add page classification to SMLS and PN, fix LE timestamp
akohlbre Feb 26, 2021
2866ef3
Merge branch 'url-matching' of github.com:citp/news-disinformation-st…
akohlbre Feb 26, 2021
a2ba899
Add a CircleCI configuration
Dexterp37 Feb 26, 2021
ca54c2e
Add a basic integration test
Dexterp37 Feb 26, 2021
422c0a5
temp push
akohlbre Feb 26, 2021
74f50bb
clean up link res hsts fix
akohlbre Feb 26, 2021
644ae5b
Matching tests and bug fixes
jonathanmayer Feb 27, 2021
c110f36
rally launch functionality
akohlbre Mar 1, 2021
dc48c07
add t.co in addition to twitter to catch shim visits
akohlbre Mar 1, 2021
88518c9
cleanup and linting fixes
akohlbre Mar 1, 2021
6bcf1a8
temporary revert for political classifier weights
akohlbre Mar 1, 2021
8b2c5aa
trim permissions
akohlbre Mar 1, 2021
f7dc202
turn on daily aggregation runs, bump version to 2.0
akohlbre Mar 1, 2021
8c1f1cb
list of permssions requested and rough usage
akohlbre Mar 1, 2021
0b71d6f
remove unused files
akohlbre Mar 1, 2021
d882de7
ignore Readability since it's a copy-paste
akohlbre Mar 1, 2021
8f9bc0d
Fixing bug in political classifier model data
benjaminhkaiser Mar 2, 2021
55a296b
Merge pull request #78 from Dexterp37/add_ci
Dexterp37 Mar 3, 2021
bade86a
syntax for export
akohlbre Mar 3, 2021
a7eb059
remove eslint rules for deleted files
akohlbre Mar 3, 2021
32b5b30
remove references to old consent notice
akohlbre Mar 8, 2021
27e7095
change the popup contents when the user rejects the survey
akohlbre Mar 8, 2021
4838e62
remove Consent module (handled by Rally)
akohlbre Mar 8, 2021
ecd4f41
update storage for new model
akohlbre Mar 8, 2021
685870a
remove old StorageManager module
akohlbre Mar 8, 2021
e6f7358
remove old schemas, update data documentation
akohlbre Mar 12, 2021
5b40928
Manifest updates #90 #91
jonathanmayer Mar 26, 2021
07fb965
Updating Rally study public key #88
jonathanmayer Mar 26, 2021
1f80799
run debug code with __ENABLE_DEVELOPER_MODE__ flag
akohlbre Mar 26, 2021
25df79b
submit ping through Rally instead of telemetry api
akohlbre Mar 27, 2021
98d95e9
add first draft of QA instructions
akohlbre Mar 27, 2021
cb04e86
fix integration test, Nightly UI changed slightly
rhelmer Apr 5, 2021
6ba056f
Merge pull request #95 from rhelmer/fix-integration
rhelmer Apr 6, 2021
0f3b08e
begin migration, add page classification worker as blob
akohlbre Apr 22, 2021
efbda2d
update all old code, begin adding pageTransitions measurement
akohlbre May 21, 2021
5aa3c12
add transitions data, move measurement coordination to worker
akohlbre May 24, 2021
2e7ac50
documentation fixes
akohlbre May 24, 2021
32090cd
add current websci version
akohlbre May 24, 2021
492e680
fix watching build
akohlbre May 26, 2021
0fea672
remove old code
akohlbre May 26, 2021
01a4b76
documentation updates
akohlbre May 28, 2021
3024e01
link sharing interface improvements
akohlbre May 28, 2021
32f026e
add web sci updates
akohlbre May 28, 2021
d258681
updated schema
akohlbre May 28, 2021
ac46d5a
remove vendored websci
akohlbre May 28, 2021
98e3997
remove copy of websci library
akohlbre May 28, 2021
d9cbc74
remove unused code, image, and popup files
akohlbre Jun 2, 2021
b575494
report attn for untracked visits, report attn and scroll as floats
akohlbre Jun 2, 2021
76f6728
remove files that no longer exist
akohlbre Jun 2, 2021
28ab60e
remove unused permission
akohlbre Jun 2, 2021
6339ab2
update version and descriptions
akohlbre Jun 2, 2021
8a94b28
update module names and clarify data used
akohlbre Jun 2, 2021
a9827a2
use rollup config from template study
akohlbre Jun 2, 2021
81024e2
documentation updates
akohlbre Jun 3, 2021
9987904
add explanations for tracked sites
akohlbre Jun 3, 2021
1c80e95
Use web-ext config from template study.
akohlbre Jun 3, 2021
d314b61
fix Icon pattern
akohlbre Jun 3, 2021
0bb4a71
fix typos, clarify explanations
akohlbre Jun 3, 2021
781d64e
remove unused, add web-sci, update Rally and web-ext
akohlbre Jun 3, 2021
3c6cee2
update schema docs to remove old fields and add changed names
akohlbre Jun 3, 2021
2a9c97a
upgrade Rally, fix docs, small fixes
akohlbre Jun 3, 2021
69b5805
remove unused
akohlbre Jun 3, 2021
6ffab17
remove old field
akohlbre Jun 3, 2021
8fe4a2f
report history presence, report untracked exposures on source urls
akohlbre Jun 4, 2021
709a7e8
only count history/pageNav visits before share events, make history/p…
akohlbre Jun 4, 2021
500ccdc
fix ping type
akohlbre Jun 4, 2021
2f7add2
Adding WebScience Rollup plugin
jonathanmayer Jun 7, 2021
39c239e
Updating name and WebScience version in package.json
jonathanmayer Jun 7, 2021
e9d9a38
Update package-lock.json
jonathanmayer Jun 7, 2021
f66f4fd
Renaming to match Rally study template
jonathanmayer Jun 7, 2021
03e61e2
Updating lint options
jonathanmayer Jun 7, 2021
00115e7
Update web-ext-config.js
jonathanmayer Jun 7, 2021
da2aabb
Merge branch 'ws-migration' of github.com:citp/news-disinformation-st…
akohlbre Jun 7, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 38 additions & 0 deletions .circleci/config.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@

---
version: 2.1

jobs:
Linting:
docker:
- image: cimg/node:14.13.1
steps:
- checkout
- run:
name: Install the dependencies
command: npm install
- run:
name: Build
command: npm run build
- run:
name: Run linting
command: npm run lint

Firefox integration tests:
docker:
- image: cimg/node:14.13.1
steps:
- checkout
- run:
name: Install the dependencies
command: npm install && sudo add-apt-repository ppa:ubuntu-mozilla-daily/ppa && sudo apt update && sudo apt install firefox-trunk
- run:
name: Run Selenium tests
command: export PATH=.:$PATH && npm run test-integration

workflows:
version: 2
ci:
jobs:
- Linting
- Firefox integration tests
Empty file added .empty.js
Empty file.
3 changes: 3 additions & 0 deletions .eslintignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
dist/*
WebScience/Utilities/Readability.js
WebScience/Utilities/content-scripts/Readability.js
akohlbre marked this conversation as resolved.
Show resolved Hide resolved
56 changes: 56 additions & 0 deletions .eslintrc.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
/* This Source Code Form is subject to the terms of the Mozilla Public
* License, v. 2.0. If a copy of the MPL was not distributed with this
* file, You can obtain one at http://mozilla.org/MPL/2.0/. */

module.exports = {
env: {
browser: true,
es6: true,
node: true,
webextensions: true,
},
extends: [
"eslint:recommended",
"plugin:import/warnings",
"plugin:node/recommended",
],
overrides: [
{
files: "tests/**",
env: {
mocha: true,
},
extends: [
"plugin:mocha/recommended",
],
},
],
globals: {
ChromeUtils: false,
ExtensionAPI: false,
__ENABLE_DEVELOPER_MODE__: false,
},
parserOptions: {
ecmaVersion: 2021,
sourceType: "module",
},
plugins: [
"import",
"node",
"mocha"
],
root: true,
rules: {
"node/no-deprecated-api": "error",
"node/no-extraneous-require": "off",
"node/no-missing-import": "off",
"node/no-unpublished-import": "off",
"node/no-unpublished-require": "off",
"node/no-unsupported-features/es-syntax": "off",

"no-multi-spaces": "error",
"no-unused-vars": [ "error", { vars: "all", args: "none", ignoreRestSiblings: false } ],
"no-var": "warn",
"prefer-const": "warn",
},
};
10 changes: 9 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,8 @@
.LSOverride

## Icon must end with two \r
Icon
Icon
akohlbre marked this conversation as resolved.
Show resolved Hide resolved


## Thumbnails
._*
Expand Down Expand Up @@ -37,3 +38,10 @@ Temporary Items

# JSDoc
docs

# Generated study files
dist
web-ext-artifacts/

# npm
node_modules/
33 changes: 21 additions & 12 deletions .vscode/launch.json
Original file line number Diff line number Diff line change
Expand Up @@ -2,23 +2,32 @@
"version": "0.2.0",
"configurations": [
{
"type": "firefox",
"name": "Build Study",
"type": "node",
"request": "launch",
"reAttach": true,
"name": "Launch add-on",
"addonPath": "${workspaceFolder}",
"log": {
"fileName": "${workspaceFolder}/log.txt",
"fileLevel": {
"default": "Debug"
}
}
"runtimeArgs": [
"run-script",
"build"
],
"runtimeExecutable": "npm"
},

{
"name": "Build Study (Developer Mode)",
"type": "node",
"request": "launch",
"runtimeArgs": [
"run-script",
"dev"
],
"runtimeExecutable": "npm"
},

// Note: automatic extension reloading is not reliable... after the first reload, the debugger appears
// to remain connected, but the Visual Studio debug console disconnects and the Firefox debug console
// shows errors
{
"name": "Launch WebExtension (With Automatic Extension Reloading)",
"name": "Launch Study (Automatic Reloading)",
"type": "firefox",
// To use the privileged API, you'll need to run a developer version of Firefox.
// Set the environment variable FFDEVLOC to the folder containing a dev version,
Expand Down Expand Up @@ -50,7 +59,7 @@
},
// Note: it's possible to manually reload the extension inside Firefox from about:debugging
{
"name": "Launch WebExtension (Without Automatic Extension Reloading)",
"name": "Launch Study",
"type": "firefox",
// To use the privileged API, you'll need to run a developer version of Firefox.
// Set the environment variable FFDEVLOC to the folder containing a dev version,
Expand Down
34 changes: 0 additions & 34 deletions Makefile

This file was deleted.

39 changes: 0 additions & 39 deletions WebScience/.gitignore

This file was deleted.

65 changes: 0 additions & 65 deletions WebScience/.vscode/launch.json

This file was deleted.

Loading