Flowise on Hugging Face. Persistent storage issues.

[SleepingSmel]

Hi everyone! Well, Hugging Face's server for Flowise is fantastic, it offers more free space and RAM than other paid options. However, there's no documentation on maintaining persistent storage across restarts. Even with the option to purchase a paid storage tier on HF, it doesn't work. Achieving this would make Flowise more accessible by preventing data loss during server updates. Thanks by advance!

[bitsnaps]

There are many Spaces on huggingface which you can try, the following code support persisting storage on data directory:

FROM node:18-alpine
USER root

# Arguments that can be passed at build time
ARG FLOWISE_PATH=/usr/local/lib/node_modules/flowise
ARG BASE_PATH=/data/.flowise
ARG DATABASE_PATH=$BASE_PATH
ARG APIKEY_PATH=$BASE_PATH
ARG SECRETKEY_PATH=$BASE_PATH
ARG LOG_PATH=$BASE_PATH/logs

# Set the ARGs as ENV
ENV FLOWISE_PATH=$FLOWISE_PATH
ENV BASE_PATH=$BASE_PATH
ENV DATABASE_PATH=$DATABASE_PATH
ENV APIKEY_PATH=$APIKEY_PATH
ENV SECRETKEY_PATH=$SECRETKEY_PATH
ENV LOG_PATH=$LOG_PATH

# Install dependencies
RUN apk add --no-cache git python3 py3-pip make g++ build-base cairo-dev pango-dev chromium

ENV PUPPETEER_SKIP_DOWNLOAD=true
ENV PUPPETEER_EXECUTABLE_PATH=/usr/bin/chromium-browser

# Install Flowise globally
RUN npm install -g flowise

# Configure Flowise directories using the ARG
RUN mkdir -p $FLOWISE_PATH/uploads /data && chmod -R 777 $FLOWISE_PATH /data

WORKDIR /data

CMD ["sh", "-c", "mkdir -p $LOG_PATH && chmod -R 777 $LOG_PATH && npx flowise start"]

You just need to make sure securing access (if you want to deploy it for prod by using lower user permission, e.g add these lines at the end before running flowise):

...
# Switch to the non-root user
USER 1000

# Expose the Flowise port
EXPOSE 3000
....

[SleepingSmel]

Hello, thanks for the response! But the link of the space you provided doesn't work (error 404)

Edit: Sorry, It was on my one of the private repo.

Hello @bitsnaps , I tried with runtime error, I modified the docker like this, did I do it right? Cause maybe I'm missing something. Thanks for your help again.

[bitsnaps]

I can see your Space is working fine, you may had to wait until the process finish the work:

P.S. The Space may go to sleeping due to inactivity mode, if you don't use it for hours/days...

[SleepingSmel]

Hi, that's another space, so I couldn't make it work with persistent storage. I created a new space, added variables like user and password, and pasted the code from the screenshot in the last message. Am I doing something wrong? Thanks again for your help 🙏

[bitsnaps]

@SleepingSmel

Huggingface Spaces supports enable data persistence, to do so, you'll need to configure a persistent storage volume using Spaces capabilities and ensure Flowise is set up to use this volume for its database and other necessary files, ot use that service you'll need to enable Persistent Storage

Keep in mind, Persistent Storage is a paid service which comes with its cost:

Here is how to do so:

1. Go to your Space's Settings in Huggingface.
2. Enable Persistent Storage.
3. Configure the persistent volume path to /app/data (as specified in the Dockerfile).

You'll also need to configure environment variables DATABASE_PATH (e.g., /app/data/flowise.db).

You can use the following docker-compose.yml example:

version: '3.1'

services:
  flowise:
    build: .
    ports:
      - "7860:7860"
    environment:
      - FLOWISE_USERNAME=admin
      - FLOWISE_PASSWORD=password
      - DATABASE_PATH=/app/data/flowise.db
    volumes:
      - ./data:/app/data

If you want more control or persist data on restart, you must have a VPS with volume storage.

[SleepingSmel]

Hello @bitsnaps, huge thanks for all your help, it helped me understand how the server and databases work. I think I'm close to solving the persistent storage issue with the solution of @amansoni7477030 as he is using HF for production purposes and therefore aligns with my purpose. Happy new year and wish you the best!

[amansoni7477030]

@SleepingSmel , as @bitsnaps mentioned, Hugging Face provides persistent storage, but you have to pay for it. If you don't want to pay and still want to use persistent storage, you can consider Aiven Cloud. They offer free PostgreSQL or MySQL databases. Simply create a free account on Aiven Cloud and select the SQL option. Afterward, enter the credentials (like host, port, and other details) in your configuration, and it should work fine.

Please note that free services like Aiven Cloud are completely free, but if there is no activity for weeks or months, they may close your service. To address this, the best approach is to deploy a cron job that makes a single request to Hosted Flowise every 1-2 days. By doing this, your Hugging Face Space and persistent storage will run continuously without any issues.

In fact, my Flowise setup has been working perfectly fine in production for the past six months without any problems. If you want, you can directly duplicate my space and use it for your own purposes by simply entering the necessary credentials.

Here is my space: FlowiseAI on Hugging Face

[SleepingSmel]

Hi @amansoni7477030, your solution is exactly what I was looking for, so big thanks! I actually duplicated your space and created a database in Aiven Cloud. I have a few questions to ensure I understood correctly, I'd appreciate it if you could answer them:

• If I understood correctly, with this solution, it is not necessary to purchase persistent storage in HF, so storage will not be erased (flows, credentials...) when updating the server to use new Flowise versions, right?
• The cron job works by sending a request to one of the CURL URLs of the chatflows, not the server URL itself, right?
  Thanks again, I'm really grateful if this solution works for the production level we are seeking for 🙏

[amansoni7477030]

@SleepingSmel
Yes, both points are correct:

1. Persistent Storage: With this solution, you don't need to purchase persistent storage in HF. Using external storage like a database in Aiven Cloud ensures that your data (flows, credentials, etc.) remains intact even when updating the Flowise server to newer versions.
2. Cron Job: The cron job operates by sending requests specifically to the CURL URLs of individual chatflows, not to the server URL itself.

Let me know if you have further questions. 😊

[bitsnaps]

@amansoni7477030 Thank you for all these details, @SleepingSmel you should also consider deactivate the Sleep mode when creating the paid Space service: