add prompt functionality (in progress) #62

rabpitpie · 2024-03-07T00:42:21Z

This adds (incomplete) functionality to scrape data from /collections/{collection name}/prompts/{prompt id}
eg. /collections/test_prompt_meme_2024/prompts/3566161
I mostly copied the existing code, I hope it worked out!

Includes:

prompt posted date
prompt id
summary as HTML
collection name (url name)
collection display title (title displayed to visitors)
ratings
prompt author
fandoms
warnings, characters, relationships and additional tags
claims(1)
categories

Does not (yet) include:

is prompt filled?
get prompt title (more difficult that I expected!)
get prompt fills. I created an interface for this but did not get to start working on it.

Issues

(1) We've had discussion about revising the interface prompt > claims.
I think this page cannot be accessed in closed prompt memes when the visitor is not part of the prompt meme
I don't know how to access this page for exchanges, and can't be certain it exists for that challenge type. (for prompt memes, the prompt id can be found on the /collections/{collection name}/requests page, on the prompt's claim button's form action url.)

…d better tests

…wn prompt claimers. Squished bugs this created

…0 claims

essential-randomness

Thank you so much for the excellent work, please let me know if any of the comments are unclear.

essential-randomness · 2024-03-16T23:48:23Z

package.json

@@ -34,6 +34,7 @@
  "scripts": {
    "build": "rm -rf dist/ && tsup src/index.ts src/urls.ts --format esm,cjs --dts",
    "test": "NODE_OPTIONS=--experimental-vm-modules jest --no-cache .test.ts$ --verbose=true",
+    "test-prompts": "NODE_OPTIONS=--experimental-vm-modules jest --no-cache prompts.test.ts$ --verbose=true",


Please revert this change, we'll add documentation on how to do it without needing to add a new script.

essential-randomness · 2024-03-16T23:50:21Z

src/collections/prompts/index.ts

+      relationships: getPromptRelationships(promptPage),
+      additional: getPromptAdditionalTags(promptPage)
+    },
+    //TODO:


Fill or delete this TODO

essential-randomness · 2024-03-16T23:57:20Z

src/urls.ts

+
+
+export const getPromptDetailsFromUrl = ({
+  // defining the input structure, eg.


This might work better as a single function-level comment. You could write it as:

/** * Extracts promptId and collectionName from the URL of a collection. * * For example, given the url "https://archiveofourown.org/collections/mo_dao_zu_shi_kink_meme_2020/prompts/2644428" * this function will return: * - promptId: 2644428 * - collectionName: mo_dao_zu_shi_kink_meme_2020 **/ export const getPromptDetailsFromUrl = ({ url, }: { url: string; }): { promptId: string; collectionName: string } => {

essential-randomness · 2024-03-17T00:01:17Z

src/urls.ts

+  promptId: string;
+  collectionName: string
+} => {
+  const promptUrlMatch = url.match(/prompts\/(\d+)/);


You might want to merge these in a single regex test, something like:

const collectionUrlMatch = url.match(/collections\/(\w+)/prompts\/(\d+)/); if (!collectionUrlMatch) { throw new Error("Invalid prompt URL"); } return { promptId: collectionUrlMatch[2], collectionName: collectionUrlMatch[1] };

Double-check that I'm extracting the groups correctly, cause I always forget the syntax.

essential-randomness · 2024-03-17T00:02:48Z

src/urls.ts

+  promptId: string,
+  collectionName:string
+})=>{
+  let workUrl = `https://archiveofourown.org`;


You can do this with a single return statement like

return `https://archiveofourown.org/collections/${collectionName}/prompts/${promptId}`

essential-randomness · 2024-03-17T00:28:57Z

tests/prompts.test.ts

+
+  describe("Fetches prompt information", () => {
+
+    test("Fetches prompt published date", async () => {


"published date and summary"

essential-randomness · 2024-03-17T00:30:43Z

tests/prompts.test.ts

+      });
+    });
+
+      test("Fetches Prompt Categores (single)", async () => {


s/categores/categories

essential-randomness · 2024-03-17T00:30:50Z

tests/prompts.test.ts

+    });  
+
+
+    //WIP PROMPTS


Remove the comment

essential-randomness · 2024-03-17T00:32:34Z

tests/prompts.test.ts

+
+  });
+
+


Suggestion: i think this file is getting long so it might be worth considering splitting it in multiple files

essential-randomness · 2024-03-17T00:34:00Z

tests/prompts.test.ts

+          collectionName: "mo_dao_zu_shi_kink_meme_2020",
+          id: "1909048",
+          filled: true,
+          // fills: [


If these are commented out for a reason, write a comment about why, if not either uncomment or delete.

essential-randomness · 2024-03-17T00:39:40Z

package.json

@@ -34,6 +34,7 @@
  "scripts": {
    "build": "rm -rf dist/ && tsup src/index.ts src/urls.ts --format esm,cjs --dts",
    "test": "NODE_OPTIONS=--experimental-vm-modules jest --no-cache .test.ts$ --verbose=true",
+    "test-prompts": "NODE_OPTIONS=--experimental-vm-modules jest --no-cache prompts.test.ts$ --verbose=true",


I believe the way to do this should be:

npm run test -- prompts.test.ts$

rabpitpie added 30 commits March 2, 2024 16:05

initial commit

b8367f4

added Prompt interface type

003a992

fixed link-breaking typo in README

e2139f8

added reminder to notes

f6df295

added cross-env to include NODE=ENV compatibility on my windows machine

ee7f8be

duplicated works into new prompts folder

0936f4c

created prompt test file. implemented get Prompt Id from url

23e81cd

get Prompt Details gets collection name (not display title) from url

e387772

Prompt gets url from collection name, prompt id

0d62c25

deleted extraneous examples and comments

c1ffadc

untested: created scraper that returns the updated date of a prompt

1ef926a

tested: prompt scraper returns updated date of a prompt

81fba93

gets prompt summaries + removed console logs

12ba058

extracted collection display name from prompt page

3bc27cb

fetching rating lists for prompts

09d7f78

ran tests for fetching ratings

5f0a1d8

created tests for prompt authors + added mocks for this purpose

96bec3a

implemented get prompt authors

1addc20

testing upstream/fork relationship

5fedc19

created test for get tags

c17f71b

added tags functionality; updatedAt became postedAt; discovered I nee…

f4c64c4

…d better tests

Prompt entities now describe the whole prompt

1077f86

Prompt entities now describe the whole prompt

052ef2c

Created test+added mocks for prompts with multiple fills and anon/kno…

b11ec64

…wn prompt claimers. Squished bugs this created

restructured Prompt claims, rewrote and passed tests, added test for …

e9c5ebf

…0 claims

(untested) created tests for categories

478f87c

added test for prompt categories + multiple fills

0cba4c3

get categories passes tests

42294d1

fetches prompt id and collectionName

ae8edb5

updated get tags for multiple fills

4189922

removed cross-env package

a65b1d5

essential-randomness requested changes Mar 17, 2024

View reviewed changes

essential-randomness reviewed Mar 17, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

add prompt functionality (in progress) #62

add prompt functionality (in progress) #62

Uh oh!

rabpitpie commented Mar 7, 2024 •

edited

Loading

Uh oh!

essential-randomness left a comment

Uh oh!

essential-randomness Mar 16, 2024

Uh oh!

essential-randomness Mar 16, 2024

Uh oh!

essential-randomness Mar 16, 2024

Uh oh!

essential-randomness Mar 17, 2024

Uh oh!

essential-randomness Mar 17, 2024

Uh oh!

essential-randomness Mar 17, 2024

Uh oh!

essential-randomness Mar 17, 2024

Uh oh!

essential-randomness Mar 17, 2024

Uh oh!

essential-randomness Mar 17, 2024

Uh oh!

essential-randomness Mar 17, 2024

Uh oh!

essential-randomness Mar 17, 2024 •

edited

Loading

Uh oh!

Uh oh!



		export const getPromptDetailsFromUrl = ({
		// defining the input structure, eg.


		describe("Fetches prompt information", () => {

		test("Fetches prompt published date", async () => {

add prompt functionality (in progress) #62

Are you sure you want to change the base?

add prompt functionality (in progress) #62

Uh oh!

Conversation

rabpitpie commented Mar 7, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

essential-randomness left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

essential-randomness Mar 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

rabpitpie commented Mar 7, 2024 •

edited

Loading

essential-randomness Mar 17, 2024 •

edited

Loading