Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ability to use complex data types and casts #118

Merged
merged 1 commit into from
Feb 4, 2024

Conversation

eta-orionis
Copy link
Contributor

Currently Sushi inserts rows straight into the database, which does not allow any "complex" data to be saved in rows, not even an array. Think of the use case

[ 
  "county" => "SomeCounty", 
  "postCodes"=>[111,222,333]
]

This PR adds the ability to use the model's $casts, which allows Eloquent to cast arrays and other data as necessary into/from a string for saving to Sushi's internal SQLite database.

Currently Sushi inserts rows straight into the database, which does not allow any "complex" data to be saved in rows, not even an array. Think of the use case 
[ "county" => "SomeCounty", "postCodes"=>[111,222,333]]

This change adds the ability to use $casts, which allows Eloquent to cast arrays and other data as necessary into/from a string for saving to the database.
@eta-orionis eta-orionis closed this Feb 2, 2024
@eta-orionis eta-orionis reopened this Feb 3, 2024
@eta-orionis
Copy link
Contributor Author

The checks seem to fail due to a database connection problem. The code works on my application.

@calebporzio
Copy link
Owner

This is pretty cool

  1. Can you add tests?
  2. Would using create() cause issues if they don't have fillable or guarded added? And is that a problem if we make all fields fillable. Maybe eloquent has a method to call create, but bypass fillable.

@calebporzio calebporzio merged commit b27ef2d into calebporzio:main Feb 4, 2024
10 of 25 checks passed
@calebporzio
Copy link
Owner

Thanks, I added tests and used forceCreate

@craigpotter
Copy link

craigpotter commented Feb 5, 2024

@calebporzio This is a cool feature but I think there needs to be a way to have casts and use the bulk insert.

We have a sushi model that has ~ 500 rows. We use casts so data is nice when we retrieve it from the model and the initial data is set up correctly for when it is inserted anyway.

Before this PR, it would be inserted as 5 queries based on the default chunk size of 100.
Now it is looping through and running 500+ inserts

@eta-orionis
Copy link
Contributor Author

@craigpotter What do you mean with:

We use casts so data is nice when we retrieve it from the model
and the initial data is set up correctly for when it is inserted anyway.

Do you mean that you had a way to use casts before this PR? Because I could not see one, certainly not one that uses the simple casts included in Laravel.

As the author of the PR, I tried to preserve the bulk-insert functionality, but could not see a way to do it.

The original bulk insert method used the underlying DB directly; this does not call any casts or events whatsoever. The only way I could see to use casts was to manipulate data through Eloquent, but then each row must be inserted separately.

Am I missing something here?

@calebporzio This is a cool feature but I think there needs to be a way to have casts and use the bulk insert.

We have a sushi model that has ~ 500 rows. We use casts so data is nice when we retrieve it from the model and the initial data is set up correctly for when it is inserted anyway.

Before this PR, it would be inserted as 5 queries based on the default chunk size of 100. Now it is looping through and running 500+ inserts

@kiwina
Copy link

kiwina commented Aug 31, 2024

Just as a note:
You can use cast just need to json encode the fields inside getRows()
Here an example:

    public function getRows(): ?array
    {
        $registry = Storage::disk('resources')->json('registry.json');
        return Arr::map($registry, function (string|array $value, string $key) {
            $slice = Arr::only($value, ['id', 'name', 'coinId', 'symbol', 'decimals', 'blockchain', 'derivation', 'curve', 'publicKeyType', 'chainId', 'explorer', 'info']);
            $slice["chainId"] = $slice["chainId"] ?? null;
            $slice['derivation'] = Json::encode($slice['derivation']);
            $slice['explorer'] = Json::encode($slice['explorer']);
            $slice['info'] = Json::encode($slice['info']);
            return $slice;
        });
    }

and set your casts

    protected function casts(): array
    {
        return [
            'derivation' => 'array',
            'explorer' => 'array',
            'info' => 'array',
        ];
    }

it isn't perfect but it works for simple stuff anything more complicated should not use arrays i would suggest

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants