Database images

[HacheEle]

Hello:
Is it possible to download the database as data dump? (as it could be done on http://ergast.com/mrd/db/)
Thank you in advance

[jolpica]

Currently we don't provide any database dumps.
While in alpha we actually retrieve our data from those ergast Database images, so for now we'd suggest downloading from there instead.

We will revisit this next season when we are using our own data, but it will likely be in a different format to Ergast.

[Vucko95]

Once new season arrives, will all the data be restructured to the new format, once Ergast is EoL ?

[jolpica]

Once the new season arrives, the existing API endpoints will continue to function as expected.

Behind the scenes, we are already storing the data in a new format, which will hopefully increase the quality of data in the long run. This is why database dumps will very likely be in a different format, if we provide them.

[Vucko95]

I don't wanna open a new topic as probably whoever looks for db dumps will go with this way.

So I have just run the steps in Data Import:

• python manage.py migrate
• make load-ergast-data
• make import-from-ergast

So this parts generates database in new format I guess from my understanding.
My question is.
How can someone now from that generate and DB Dump from new format we just generated.

I have tried the script located in scripts, but it seems broken.
I will try to figure it on my own but maybe someone has already done it.

EDIT: Made some progress with running docker exec -it devcontainer-db-1 pg_dump -U postgres -d jolpica > jolpica_dump.sql however sql file seems broken as I can't run it on solo container to populate postgres db.

[awanderingspirit]

This has to be the single most important issue with this API.

Notice that...

1. no database dumps are provided (i.e. plain files one can repeatedly access offline),
2. rate limits for requests are put in place,
3. and offset (of rows) is set to a max of 100.

Ergast empowered users with the kind provision of plain old csv files that are programming language-agnostic. Doesn't this measure address the challenge of facing insurmountable amounts of HTTP requests by leaving a large portion of users with promptly accessible offline images of the database?

One has to write an elaborate script in order to convert the JSON endpoints of this database to reusable offline files. The task is challenging given that you would encounter endless fields leading to more irreducible records (denoted in the database diagram as foreign keys, see: picture), not to mention points 2 and 3 above.

[Oldertarl]

I honestly do not see where the issue is when not running the postgres DB locally as well defined in the github definitions.

I have written a simple php script to validate if the json data (file) is present on my webserver to download the data, if not - call the API and save the data for future use.
When there is a offset, I iterate through the pages with a small delay to not hit the throttle limit.
This is faster than downloading the API calls every time they are needed.
Most of the data is static and if an update is provided on data, I simply delete the json file locally to have it downloaded again.

[jolpica]

Thanks for the feedback.

All work done on this API is done by a small team in our free time, unfortunately this means we don't always have time to do everything we'd like to.

We do plan to release database images in the future, but at the moment our focus is on ensuring we can import data in a reliable manner.

To set expectations, when we do eventually release database dumps, they will not be in an ergast compatible format, they will likely follow our own DB schema which we have a diagram of in the read me.

[gtibrett]

@jolpica Where are you importing the data from? i.e. for 2025 and on, what is the original source of data being imported into the DB? Ergast? some other source?

I would be happy to take on the task of turning that imported data into csv/sql

[harningle]

@gtibrett it's going to be PDFs in fia website, and we are working on the data import now: https://github.com/harningle/fia-doc

[ErwinBramer]

First of all, I want to thank you for your wonderful work with this API.
Answering the original question, I am working on a solution in R to update the Ergast .csv files with new data from the Jolpica F1 API. I think I'm on 90% now. I started this for my own use, but I think there are a lot of people looking for updated .csv files. I also noticed that the much referenced contribution on Kaggle by Vopani is being discontinued.
As said, I started this project updating the Ergast database for my own use, but now I am thinking about publishing it on Kaggle. Before I will do this, I want to be sure this is OK for you @jolpica.

[jolpica]

I'd ask to please refrain from polling the API to update database dumps. However I understand I haven't been helpful in providing an alternative. I'll post a sample below of a database dump we'll hope to provide in the future.
Ideally please make use of this, or use it as a basis to convert into the old ergast format yourself. I'd highly recommend using the new format though, as we aim to start providing more data than the original ergast had available.

[ErwinBramer]

Thanks for your quick reply. It would be wonderful if you could provide something like the alpha dump below. In that case I wouldn't need to poll your API. Since I am relatively new in data science, I know little to nothing of databases. My knowledge is is limited to .csv files and data files R can read. That's at least the way I used the old Ergast data.

[jolpica]

Please understand that the following is an alpha and is subject to change, whether that be file names, column names or anything else.
This dump is also currently out of date with production data (2025 data), but it will hopefully serve as a basis to help prepare for when we release these in the future.

ALPHA SUBJECT TO CHANGE DUMP.