In a few words, µFunds works by fetching an HTML page from the chosen data source, parsing the asked data in it and just returning it back to you. More concretely, the performed steps are the following:
option (asset attribute), id (asset identifier) and source (website where fetch data from).source is explicitly given by the user, then µFunds plans to fetch a the asset detail page for that id. Otherwise, µFunds plans to fetch a Morningstar page in generic mode (morningstar).morningstar-aumorningstarmorningstar-atmorningstarmorningstar-bemorningstarmorningstar-dkmorningstarmorningstar-fimorningstarmorningstar-frmorningstarmorningstar-demorningstarmorningstarmorningstarMUTF_IN: + tickermorningstarmorningstar-iemorningstarmorningstar-itmorningstarmorningstar-nomorningstarmorningstar-aumorningstarmorningstar-ptmorningstarmorningstarmorningstarmorningstarmorningstar-zamorningstarmorningstar-esmorningstarmorningstarquefondosmorningstarmorningstar-chmorningstarmorningstar-gbmorningstarError: Asset identifier is empty.
Please check that you introduced an asset identifier (string) as the second argument of the muFunds function.
Data source is required as third argument. Please see www.mufunds.com/usage.html
-Please check that you introduced a valid data source as the third argument of the muFunds function. Check the Usage and examples documentation page to learn about the available options.
Note: The data source argument is required from June 24, 2022, since country-agnostic Morningstar reports from quotes.morningstar.com (used when no source was explicitly defined) stopped working on June 23, 2022.
Error: Source is not compatible. Please check the documentation for the compatibility list.
Please check the Usage and examples and Compatibility list documentation pages to verify that the data source is compatible with µFunds. Remember that, if you want to obtain data from Google Finance, you should use the Google Finance function included in Google Sheets instead.
@@ -81,10 +76,12 @@Error: Wrong combination of asset identifier and source. Please check the accepted ones at the documentation.
Please check the Compatibility list documentation pages to verify that the asset is compatible with the data source you have chosen. This error is due to an exception when fetching the data page, for example a Not Found (404) code.
+If you were using a generic Morningstar (morningstar) call, consider using a country-specific page (morningstar-**).
Error: (A certain option) is not available for this asset and source. Please try another data source.
Some data sources do not offer all the data options that are possible to call via µFunds. In this case, consider using an alternative data source for your same asset, as sometimes more than one source is compatible at the same time.
+If you were using a generic Morningstar (morningstar) call, consider using a country-specific page (morningstar-**).
Error: Morningstar ID search failed. Try using the asset's Morningstar ID or another compatible data source.
diff --git a/docs/usage.html b/docs/usage.html index ba26b48..43291e6 100644 --- a/docs/usage.html +++ b/docs/usage.html @@ -60,18 +60,9 @@
- Important: Country-agnostic Morningstar reports from quotes.morningstar.com stopped working on June 23, 2022. Therefore, from June 24, 2022, µFunds requires a data source as the third argument of the function call.
-
- You should now call µFunds with =muFunds(option, id, "morningstar-**"), where ** represents the 2-letter code of a country where your asset is available for sale. In most cases, you can just use the country where you live. Please check the list of available countries and codes in the Data sources section below.
-
µFunds makes you able to use the muFunds function in any cell within your spreadsheet. This function will return the asked value based on three arguments, so you can call it as
=muFunds(option, id, source) (if you use a period as decimal separator)
µFunds makes you able to use the muFunds function in any cell within your spreadsheet. This function will return the asked value based on three arguments, so you can call it as
=muFunds(option, id, source) (if you use a period as decimal separator)
=muFunds(option; id; source) (if you use a comma as decimal separator)
where option is a valid returned data option, id is an asset identifier (e.g. ISIN, ticker or Morningstar ID) and source is an identifier for the data source where you want to get the data from.
This function call will return the latest information available from the chosen source. Most of these sources don't offer historical values, which makes comparisons and backtesting difficult. If interested, you can consider saving a history of daily data to your own spreadsheet using a Google Apps Script function and a time-driven trigger (link to guide).
@@ -129,7 +120,13 @@id)source)Although µFunds was originally designed to import data from Morningstar, now it is compatible with other information sources that potentially offer more updated or reliable data in certain cases. This is performed, as of now, via HTML parsing.
+If no source is defined, then the asked information will be loaded from Morningstar in generic mode.
Morningstar generic mode (morningstar)
This source relies on Morningstar screener pages, that happen to be the easiest way to get information from an asset, regardless of the used identifier, as they work with ISINs, mutual fund tickers, ETF tickers, Morningstar IDs, etc.
+Morningstar screener info is not a user-friendly interface and therefore loading times are faster. Furthermore, no country needs to be chosen and no translation from standard identifiers to Morningstar IDs is required. On the other hand, expenses data is usually not available and sometimes data is provided in a different currency of the asset's, is provided incorrectly or is not provided at all. In these cases, Morningstar country-specific mode should be used.
+Morningstar country pages (morningstar-**)
µFunds visits Morningstar snapshot pages and simulates the human behavior when checking for the data manually in a Morningstar website for a certain country. All attributes are up-to-date and reliable, including expense ratios, as the information is the one provided directly to Morningstar's visitors.
diff --git a/main.js b/main.js index 84ef9c8..cee4e6a 100644 --- a/main.js +++ b/main.js @@ -20,8 +20,8 @@ function muFunds(option, id, source) { throw new Error( "Asset identifier is empty." ); } - if (!source) { - throw new Error( "Data source is required as third argument. Please see www.mufunds.com/usage.html" ); + if (!source || source === 'morningstar') { + return loadFromMorningstarScreener(option, id); } // Input already validated diff --git a/sources/morningstar-screener.js b/sources/morningstar-screener.js new file mode 100644 index 0000000..da98452 --- /dev/null +++ b/sources/morningstar-screener.js @@ -0,0 +1,210 @@ +function loadFromMorningstarScreener(option, id, attempts = 0) { + const cache = CacheService.getScriptCache(); + + let url = 'https://lt.morningstar.com/api/rest.svc/klr5zyak8x/security/screener'; + + const params = { + outputType: 'json', + page: 1, + pageSize: 1, + securityDataPoints: [ + 'CategoryName', + 'ClosePrice', + 'ClosePriceDate', + 'ExpenseRatio', + 'Name', + 'PriceCurrency', + 'GBRReturnD1', + 'SecId', + 'Universe', + ], + term: id, + universeIds: [ + 'FOEUR$$ALL', // Europe funds + 'ETEUR$$ALL', // Europe ETFs + + 'FOBEL$$ALL', // Belgium funds + 'ETEXG$XBRU', // Belgium ETFs + 'FOBEL$$PEN', // Belgium pension plans + + 'FODNK$$ALL', // Denmark funds + 'ETEXG$XCSE', // Denmark ETFs + + 'FODEU$$ALL', // Germany funds + 'ETEXG$$XFRA', // Germany ETFs (Frankfurt) + 'ETEXG$$XETR', // Germany ETFs (Xetra) + + 'FOESP$$ALL', // Spain funds + 'ETEXG$XMAD', // Spain ETFs + 'FOESP$$PEN', // Spain pension plans + + 'FOFRA$$ALL', // France funds + 'ETEXG$XPAR', // France ETFs + 'FOFRA$$FXP', // France pension plans (FCPI/FIP) + + 'FOIRL$$ALL', // Ireland funds + 'ETEXG$XDUB', // Ireland ETFs + + 'FOITA$$ALL', // Italy funds + 'ETEXG$XMIL', // Italy ETFs + 'FOITA$$PEN', // Italy pension plans + + 'FONLD$$ALL', // Netherlands funds + 'ETEXG$XAMS', // Netherlands ETFs + + 'FONOR$$ALL', // Norway funds + 'ETEXG$XOSE', // Norway ETFs + + 'FOAUT$$ALL', // Austria funds + 'ETEXG$XWBO', // Austria ETFs + + 'FOPRT$$ALL', // Portugal funds + 'ETEXG$XLIS', // Portugal ETFs + + 'FOCHE$$ALL', // Switzerland funds + 'ETEXG$XSWX', // Switzerland ETFs + + 'FOFIN$$ALL', // Finland funds + 'ETEXG$XHEL', // Finland ETFs + + 'FOSWE$$ALL_5498', // Sweden funds + 'ETEXG$XSTO', // Sweden ETFs + + 'FOGBR$$ALL', // UK funds + 'ETEXG$XLON', // UK ETFs + // 'CEEXG$XLON', // UK investment trusts + // 'SAGBR$$LSA', // UK pension funds 1 + // 'SAGBR$$PSA', // UK pension funds 2 + + 'FOIND$$ALL', // India funds + 'FCIND$$ALL', // India close-ended funds + + 'FOCAN$$ALL', // Canada funds + 'ETCAN$$FFE', // Canada ETFs + 'E0CAN$$FST', // Canada stocks + + 'FOBRA$$ALL', // Brazil funds + + 'FOUSA$$ALL', // US funds + 'ETUSA$$FFE', // US ETFs + + 'FOSGP$$ALL', // Singapore funds + 'ETEXG$XSES', // Singapore ETFs + + 'FOIDN$$ALL', // Indonesia funds + + 'FOCHL$$ALL', // Chile funds + 'ETEXG$XSGO', // Chile ETFs + + 'FOMEX$$ALL', // Mexico funds + 'ETEXGRXMEX', // Mexico ETFs 1 + 'ETEXGIXMEX', // Mexico ETFs 2 + + 'FOAUS$$ALL', // Australia investment trusts + 'FOAUS$$SAF', // Australia superannuation funds + 'FOAUS$$PAF', // Australia pension and annuities funds + 'FOAUS$$IBF', // Australia investment bonds + + 'FONZL$$ALL', // New Zealand investment trusts + 'FONZL$$SAF', // New Zealand superannuation funds + 'FONZL$$IBF', // New Zealand investment bonds + + 'FOHKG$$ALL', // Hong Kong funds + 'FOHKG$$PEN', // Hong Kong pension plans + 'ETEXG$XHKG', // Hong Kong ETFs + + 'FOMYS$$ALL', // Malaysia funds + 'ETEXG$XKLS', // Malaysia ETFs + + 'FOTHA$$ALL', // Thailand funds + 'ETEXG$XBKK', // Thailand ETFs + + 'FOTWN$$ALL', // Taiwan funds + 'ETEXG$XTAI', // Taiwan ETFs + + 'FOISR$$ALL', // Israel funds + 'ETEXG$XTAE', // Israel ETFs + + // 'FOALL$$ALL', // Fallback + + // 'E0WWE$$ALL', // All stocks + ], + version: 1, + }; + + url += Object.keys(params) + .reduce((url, key, position) => { + const joiner = position === 0 ? '?' : '&'; + + let value = params[key]; + + if (Array.isArray(value)) { + value = value.join(encodeURIComponent('|')); + } else { + value = encodeURIComponent(value); + } + + return url + joiner + key + '=' + value; + }, ''); + + if (option === 'url') { + return url; + } + + // const fetch = UrlFetchApp.fetch(url, { muteHttpExceptions: true }); + const fetch = UrlFetchApp.fetch(url, { muteHttpExceptions: true }); + + if (fetch.getResponseCode() == 403) { + const cached = cache.get(`mslt-${option}-${id}`); + + if (cached !== null) { + if (['nav', 'change', 'expenses'].includes(option) && !isNaN(cached)) { + return parseFloat(cached); + } else if (cached !== '-') { + return cached; + } + + throw new Error('Selected option is not available for the given asset.'); + } + + if (attempts >= 5) { + throw new Error('Morningstar error. Please delete the cell, wait 3 seconds and press Ctrl+Z to reload data.'); + } + + Utilities.sleep(1000); + + return loadFromMorningstarScreener(option, id, attempts+1); + } + + const json = JSON.parse(fetch.getContentText()); + + if (json.total < 1) { + throw new Error('Search failed. Please use a valid unique identifier for the asset.'); + } + + const values = { + nav: json.rows[0].ClosePrice ?? '-', + change: json.rows[0].GBRReturnD1 ? json.rows[0].GBRReturnD1 / 100 : '-', + date: json.rows[0].ClosePriceDate ?? '-', + currency: json.rows[0].PriceCurrency ?? '-', + expenses: json.rows[0].ExpenseRatio ? json.rows[0].ExpenseRatio / 100 : '-', + category: json.rows[0].CategoryName ?? '-', + } + + cache.put(`mslt-nav-${id}`, values.nav, 12 * 60 * 60); + cache.put(`mslt-change-${id}`, values.change, 12 * 60 * 60); + cache.put(`mslt-date-${id}`, values.date, 12 * 60 * 60); + cache.put(`mslt-currency-${id}`, values.currency, 12 * 60 * 60); + cache.put(`mslt-expenses-${id}`, values.expenses, 12 * 60 * 60); + cache.put(`mslt-category-${id}`, values.category, 12 * 60 * 60); + + if (option in values) { + if (values[option] !== '-') { + return values[option]; + } + + throw new Error('Selected option is not available for the given asset.') + } + + throw new Error('Unknown option'); +}