Skip to content

Commit 00faf69

Browse files
morgan3dmorgan3d
committed
loadmanager
1 parent bd25e72 commit 00faf69

File tree

3 files changed

+352
-0
lines changed

3 files changed

+352
-0
lines changed

‎README.md

+1
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,7 @@ JavaScript
4848
- **jsmaze** - Recursive perfect maze generator, with wrapping and imperfect options. 2018.
4949
- **jsvectorify** - Vector operations and operator overloading. 2018.
5050
- **jsmenu** - Lightweight library for dropdown menus. 2018.
51+
- **jsloadmanager** - Caching and post-processing for recursive resource loading via XMLHttpRequest/fetch. 2018.
5152

5253

5354
Java

‎jsloadmanager/LoadManager.js

+297
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,297 @@
1+
/**
2+
\file LoadManager.js
3+
4+
Load manager for asynchronous resource loading requests in
5+
complicated loading trees, with post processing of resources and
6+
caching. This is useful for loading files that recursively trigger
7+
the loading of other files without having to manage promises or lots
8+
of callbacks explicitly.
9+
10+
The routines are:
11+
12+
- dataManager = new LoadManager()
13+
- dataManager.fetch()
14+
- dataManager.end()
15+
16+
----------------------------------------------------
17+
18+
This implementation uses XMLHttpRequest internally. There is a newer
19+
browser API for making HTTP requests called Fetch
20+
(https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API/Using_Fetch)
21+
that uses Promises; it is more elegant in some ways but also more
22+
complicated because it relies on more language features. Since the
23+
current implementation is working fine, I don't intend to port to
24+
the Fetch API until some critical feature of (such as the explicit
25+
headers or credentialing) it is required.
26+
27+
----------------------------------------------------
28+
29+
Open Source under the BSD-2-Clause License:
30+
31+
Copyright 2018 Morgan McGuire, https://casual-effects.com
32+
33+
Redistribution and use in source and binary forms, with or without
34+
modification, are permitted provided that the following conditions
35+
are met:
36+
37+
1. Redistributions of source code must retain the above copyright
38+
notice, this list of conditions and the following disclaimer.
39+
40+
2. Redistributions in binary form must reproduce the above copyright
41+
notice, this list of conditions and the following disclaimer in the
42+
documentation and/or other materials provided with the distribution.
43+
44+
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
45+
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
46+
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
47+
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
48+
HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
49+
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
50+
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
51+
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
52+
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
53+
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
54+
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
55+
*/
56+
57+
// We don't support a native 'xml' fetch because XMLHttpRequest.responseXML
58+
// doesn't work within web workers.
59+
60+
/** Begin a series of fetches. Options:
61+
62+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
63+
{
64+
callback : function () {...},
65+
errorCallback : function () {...},
66+
67+
// If true, parse JSON locally so that better error messages
68+
// can be provided than via server validation.
69+
parseJSONLocally : false,
70+
71+
// If true, append '?' to each URL when fetching to force
72+
// it to be reloaded from the server, or at least validated.
73+
// Defaults to false.
74+
forceReload : false
75+
}
76+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
77+
78+
Invoke \a callback when all fetch() calls have been processed,
79+
or \a errorCallback when the first fails. */
80+
function LoadManager(options) {
81+
82+
// Map from url to:
83+
// {
84+
// raw : result of the fetch
85+
// status : 'loading', 'success', 'failure'
86+
// post : Map from post-processing functions (and null) to:
87+
// {
88+
// value: value
89+
// callbacks: array of callbacks to invoke when data arrives and is post-processed
90+
// errorCallbacks: etc.
91+
// }
92+
// }
93+
94+
this.resource = new Map();
95+
this.crossOrigin = 'anonymous';
96+
this.pendingRequests = 0;
97+
98+
// 'accepting requests', 'loading', 'complete', 'failure'
99+
this.status = 'accepting requests';
100+
101+
this.forceReload = options.forceReload || false;
102+
103+
// Invoke when pendingRequests hits zero if status is not 'failure'
104+
this.callback = options.callback;
105+
this.errorCallback = options.errorCallback;
106+
}
107+
108+
109+
/**
110+
Invoke \a callback on the contents of \a url after the specified
111+
post-processing function, or \a errorCallback on failure. Results
112+
are cached both before and *after* post-processing, so that there
113+
is no duplication of the HTTP request, the post processing, or any
114+
of the storage.
115+
116+
Set \a postProcessing to null to get the raw result in the callback.
117+
118+
\param url String, mandatory
119+
\param type Type of data at the URL: 'text', 'json', 'arraybuffer', or 'image' (uses Image)
120+
\param postProcess(rawData, url) function
121+
\param callback(data, rawData, url, postProcess)
122+
\param errorCallback(reason, url) optional. Invoked on failure if some other load has not already failed.
123+
124+
You will receive a failure if the post process fails.
125+
*/
126+
LoadManager.prototype.fetch = function (url, type, postProcess, callback, errorCallback) {
127+
console.assert(typeof type === 'string', 'type must be a string');
128+
console.assert((typeof postProcess === 'function') || !postProcess,
129+
'postProcess must be a function, null, or undefined');
130+
131+
if (this.status === 'failure') { return; }
132+
const LM = this;
133+
134+
console.assert(this.status !== 'complete',
135+
'Cannot call LoadManager.fetch() after LoadManager.end()');
136+
137+
++this.pendingRequests;
138+
let rawEntry = this.resource.get(url);
139+
if (! rawEntry) {
140+
rawEntry = {
141+
url: url,
142+
raw: undefined,
143+
status: 'loading',
144+
failureMessage: undefined,
145+
post: new Map()
146+
};
147+
this.resource.set(url, rawEntry);
148+
149+
function onLoadSuccess() {
150+
if (LM.status === 'failure') { return; }
151+
rawEntry.status = 'success';
152+
// Run all post processing and callbacks
153+
for (let [p, v] of rawEntry.post) {
154+
v.value = p ? p(rawEntry.raw, rawEntry.url) : rawEntry.raw;
155+
156+
for (let c of v.callbackArray) {
157+
// Note that callbacks may increase LM.pendingRequests
158+
if (c) { c(v.value, rawEntry.raw, rawEntry.url, p); }
159+
}
160+
}
161+
162+
--LM.pendingRequests;
163+
if ((LM.pendingRequests === 0) && (LM.status === 'loading')) {
164+
LM.status = 'complete';
165+
// Throw away all of the data
166+
LM.resource = null;
167+
if (LM.callback) { LM.callback(); }
168+
}
169+
}
170+
171+
function onLoadFailure() {
172+
if (LM.status === 'failure') { return; }
173+
rawEntry.status = 'failure';
174+
rawEntry.failureMessage = '';
175+
176+
// Run all failure callbacks
177+
for (let [p, v] of rawEntry.post) {
178+
for (let c of v.errorCallbackArray) {
179+
if (c) { c(rawEntry.failureMessage, rawEntry.url); }
180+
}
181+
}
182+
LM.status = 'failure';
183+
LM.resource = null;
184+
if (LM.errorCallback) { LM.errorCallback(); }
185+
}
186+
187+
// Fire off the asynchronous request
188+
if (type === 'image') {
189+
// Allow loading from other domains and reading the pixels (CORS).
190+
// Works only if the other site allows it; which github does and
191+
// for which we can use a proxy and an XMLHttpRequest as an
192+
// annoying workaround if necessary.
193+
const image = new Image();
194+
rawEntry.raw = image;
195+
if (LM.crossOrigin) {
196+
image.crossOrigin = LM.crossOrigin;
197+
}
198+
image.onload = onLoadSuccess;
199+
image.onerror = onLoadFailure;
200+
image.src = url + (LM.forceReload ? '?' : '');
201+
} else {
202+
const xhr = new XMLHttpRequest();
203+
204+
// Force a check for the latest file using a query string
205+
xhr.open('GET', url + (LM.forceReload ? '?' : ''), true);
206+
if (LM.parseJSONLocally && (type === 'json')) {
207+
xhr.responseType = 'text';
208+
} else {
209+
xhr.responseType = type;
210+
}
211+
212+
xhr.onload = function() {
213+
const status = xhr.status;
214+
if (status === 200) {
215+
if (xhr.response) {
216+
if (LM.parseJSONLocally && (type === 'json')) {
217+
// now parse
218+
try {
219+
rawEntry.raw = JSON.parse(xhr.response);
220+
onLoadSuccess();
221+
} catch (e) {
222+
rawEntry.failureMessage = "JSON parse error: " + e;
223+
onLoadFailure();
224+
}
225+
} else {
226+
rawEntry.raw = xhr.response;
227+
onLoadSuccess();
228+
}
229+
} else {
230+
rawEntry.failureMessage = "File was in the incorrect format";
231+
onLoadFailure();
232+
}
233+
} else {
234+
rawEntry.failureMessage = "Server returned error " + status;
235+
onLoadFailure();
236+
}
237+
};
238+
xhr.send();
239+
} // if XMLHttp
240+
}
241+
242+
let postEntry = rawEntry.post.get(postProcess);
243+
if (! postEntry) {
244+
// This is the first use of this post processing
245+
if (rawEntry.status === 'success') {
246+
// Run the post processing right now
247+
postEntry = {
248+
value: postProcess ? postProcess(rawEntry.raw, rawEntry.url) : rawEntry.raw,
249+
};
250+
if (callback) { callback(postEntry.value, rawEntry.raw, url, postProcess); }
251+
} else if (rawEntry.status === 'failure') {
252+
if (errorCallback) { errorCallback(rawEntry.failureMessage, url); }
253+
} else {
254+
// Schedule the callback
255+
postEntry = {
256+
value: null,
257+
callbackArray: [callback],
258+
errorCallbackArray: [errorCallback]
259+
};
260+
}
261+
rawEntry.post.set(postProcess, postEntry);
262+
} else if (rawEntry.status === 'success') {
263+
// Run the callback now
264+
if (callback) { callback(postEntry.value, rawEntry.raw, url, postProcess); }
265+
} else if (rawEntry.status === 'failure') {
266+
// Run the callback now
267+
if (errorCallback) { errorCallback(rawEntry.failureMessage, url); }
268+
} else {
269+
// Schedule the callback
270+
postProcess.callbackArray.push(callback);
271+
postProcess.errorCallbackArray.push(errorCallback);
272+
}
273+
}
274+
275+
276+
/**
277+
Call after the last fetch.
278+
*/
279+
LoadManager.prototype.end = function () {
280+
if (this.status !== 'failure') {
281+
console.assert(this.status === 'accepting requests');
282+
283+
if (this.pendingRequests === 0) {
284+
// Immediately invoke the callback
285+
this.status = 'complete';
286+
// Throw away all of the data
287+
this.resource = null;
288+
if (this.callback) { this.callback(); }
289+
} else {
290+
// Tell the loading callbacks that they
291+
// should invoke the completion callback.
292+
this.status = 'loading';
293+
}
294+
}
295+
}
296+
297+

‎jsloadmanager/readme.md

+54
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
Load manager for asynchronous resource loading requests in
2+
complicated loading trees, with post processing of resources and
3+
caching.
4+
5+
This is useful for loading files that recursively trigger the loading
6+
of other files without having to manage promises or lots of callbacks
7+
explicitly. I originally used it for implementing project loading in a
8+
web integrated development environment.
9+
10+
The routines are:
11+
12+
- dataManager = new LoadManager()
13+
- dataManager.fetch()
14+
- dataManager.end()
15+
16+
----------------------------------------------------
17+
18+
This implementation uses XMLHttpRequest internally. There is a newer
19+
browser API for making HTTP requests called Fetch
20+
(https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API/Using_Fetch)
21+
that uses Promises; it is more elegant in some ways but also more
22+
complicated because it relies on more language features. Since the
23+
current implementation is working fine, I don't intend to port to
24+
the Fetch API until some critical feature of (such as the explicit
25+
headers or credentialing) it is required.
26+
27+
----------------------------------------------------
28+
29+
Open Source under the BSD-2-Clause License:
30+
31+
Copyright 2018 Morgan McGuire, https://casual-effects.com
32+
33+
Redistribution and use in source and binary forms, with or without
34+
modification, are permitted provided that the following conditions
35+
are met:
36+
37+
1. Redistributions of source code must retain the above copyright
38+
notice, this list of conditions and the following disclaimer.
39+
40+
2. Redistributions in binary form must reproduce the above copyright
41+
notice, this list of conditions and the following disclaimer in the
42+
documentation and/or other materials provided with the distribution.
43+
44+
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
45+
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
46+
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
47+
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
48+
HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
49+
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
50+
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
51+
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
52+
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
53+
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
54+
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

0 commit comments

Comments
 (0)