Image2Measurement Library

[mikepsinn]

Image2Measurement Library: Develop Server-Side Next.js API and Framework-Agnostic Client Library for UPC Barcode Scanning and Measurement

We aim to develop a server-side Next.js API that will process images into measurements, handling product labels, UPCs, nutrition facts, ingredient lists, or images of foods. We'll also create a client-side library, compatible with any JavaScript framework, that communicates with this API. Both components will be developed within our Nx monorepo, using TypeScript.

Here's where the scanning is currently done:
https://github.com/curedao/decentralized-fda/blob/75423cb44f6f2a3b43720e2b6e7ff95bced06c0f/apps/dfda-1/public/app/public/js/controllers/searchPageCtrl.js#LL63C14-L63C25

Here are the UPC usages for reference:
https://github.com/search?q=repo%3Acuredao%2Fdecentralized-fda%20upc&type=code

User Story

As a user, I want to use my device's camera to take a picture of a product (label, UPC, nutrition facts, ingredients list, or food image), and have the app convert this image into an array of measurements for me.

Steps

1. User activates the camera button.
2. User takes a picture.
3. The client-side library securely uploads the image to a decentralized storage platform (e.g., NFT.storage) and a URL is generated.
4. The server-side Next.js API fetches the image using the URL.
5. The API attempts to scan the barcode from the image.
6. If successful, it looks up the product via a UPC API.
7. The API creates a measurement using the obtained variable name.
8. If not found, the API attempts to OCR the image.
9. The API converts the ingredients list, nutrition facts label, and product name to measurements if possible.
10. If no useful data is found, the API tries to identify the food with an image identification AI model.
11. The API returns an array of measurements to the client library. If no useful data can be extracted, the API returns an error message.

Measurement Objects

Measurements should be an array with at least the following properties:

• combinationOperation – string – Way to aggregate measurements over time. SUM should be used for things like minutes of exercise. If you use MEAN for exercise, then a person might exercise more minutes in one day but add separate measurements that were smaller. So when we are doing correlational analysis, we would think that the person exercised less that day even though they exercised more. Conversely, we must use MEAN for things such as ratings which cannot be SUMMED.
• startAt – required – The current time in UTC ISO 8601 YYYY-MM-DDThh:mm:ss date time format. Time zone should be UTC and not local.
• value – double – required – Measurement value. Default to 1 serving for foods, x %RDA in one serving for nutrition facts, maybe 1 for yes with yes/no unit, unless we can figure out a more precise unit from the measurement
• unitName – string – required – Unit of measurement like serving, %RDA, yes/no
• variableCategoryName – string – Ex: Treatments, Foods, or Nutrients. See options here https://curedao.readme.io/reference/getvariablecategories
• variableName – string – required – Name of the variable for which we are creating the measurement records
• upc – string – UPC or other barcode scan result
• note – string – Optional additional information

Server-side Next.js API Features

1. Barcode scanning
2. OCR processing
3. Product lookup via UPC API
4. Conversion of text to measurements
5. Food identification using AI

Client-side Library Features

1. Image capture and upload
2. Interaction with the server-side API
3. Handling API responses
4. Compatibility with any JavaScript framework

Developer Registration Page

We will include a developer registration page on the Next.js site. This page will generate API keys for the client-side libraries to use when interacting with the server-side API.

Relevant Libraries and APIs

1. Barcode Scanning: QuaggaJS
2. OCR Processing: Tesseract.js
3. UPC API: UPCitemdb
4. Image Recognition AI: Google Vision AI

https://opendata.stackexchange.com/questions/17714/food-ingredient-api-database

https://www.logmeal.es/api

https://www.marktechpost.com/2023/07/10/meet-minigpt-4-an-open-source-ai-model-that-performs-complex-vision-language-tasks-like-gpt-4/

Demo Implementation

We'll create an Img2Measurements demo application in the apps folder within the monorepo. This application will demonstrate the usage of the client-side library and be built using a cross-platform framework like React Native.

Testing

We'll write unit tests for all functionality in the client-side library and the server-side API. Cypress UI tests should be implemented in the demo app.

Automatic Deployments

We'll set up GitHub Actions for automatic deployments. The server-side API will be deployed to Vercel and the client-side library will be published to npm.

GitHub Actions for Testing

Before merging any feature branches to the develop branch, we'll require that all tests are run via GitHub Actions and that all tests pass successfully.

Library Publishing

The client-side library should be included in the libs folder of our Nx monorepo.

Documentation

We'll include comprehensive documentation in the README.md and also publish it on a site built using Mintlify. Documentation should be accessible and follow web content accessibility guidelines (WCAG).

Additional Clarifications

1. The library should be developed using TypeScript. This will ensure type safety and compatibility with our Nx monorepo.
2. Data privacy and security are of utmost importance. All image uploads and data handling should be done securely, with user consent where necessary.
3. The system should be designed with scalability in mind. It should be capable of handling a high number of requests and provide a response quickly.
4. Error handling should be robust. The system should be able to gracefully handle errors, and communicate them effectively to the user.

---

We want to be able to include the barcode in any app with a script tag that has

• a trigger function for the variable search box popup
• a barcode button to trigger the camera scanner
• the search or UPC matches to be passed to the provided callback function or displayed
• the selected variable to be passed to an onVariableSelected callback function