Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Advanced Data Protection Support #202

Open
gsong opened this issue Mar 3, 2023 · 64 comments
Open

Advanced Data Protection Support #202

gsong opened this issue Mar 3, 2023 · 64 comments
Assignees
Labels
class(known issue) A known issue that might not be easy to fix or needs additional input. status(in progress) This item is currently in scope for the next release status(previewed) This issue or pull request should be fixed in a released beta version

Comments

@gsong
Copy link

gsong commented Mar 3, 2023

Describe the bug

$ docker compose exec photos-sync icloud-photos-sync token
-----------------------------------------------------------------------------------------------
Welcome to icloud-photos-sync, v.1.0.1!
Made with <3 by steilerDev
-----------------------------------------------------------------------------------------------
Authenticating user...
Device trusted
Sign in successful!
-----------------------------------------------------------------------------------------------
Experienced fatal error at 3/3/2023, 12:58:54 PM: TokenError (FATAL): Unable to get trust token caused by iCloudError (FATAL): Authentication failed caused by iCloudError (FATAL): Unable to get iCloud Photos service ready caused by iCloudError (FATAL): Unexpected error while setting up iCloud Photos caused by Request failed with status code 403 (Error Code: 734410aa-a8db-48ce-971f-0f6041f8751a)
-----------------------------------------------------------------------------------------------

Note that I do have Advanced Data Protection turned on.

Logs
Please paste the log file (preferably with LOG_LEVEL=debug), located in .icloud-photos-sync.log, stored in the DATA_DIR.

[2023-03-03T20:58:50.110Z] INFO i-Cloud: Initiating iCloud connection
[2023-03-03T20:58:50.652Z] INFO i-Cloud: Authenticating user
[2023-03-03T20:58:51.323Z] INFO i-Cloud: Authentication successful
[2023-03-03T20:58:51.328Z] INFO i-Cloud: Setting up iCloud connection
[2023-03-03T20:58:53.679Z] INFO i-Cloud: Getting iCloud Photos Service ready
[2023-03-03T20:58:54.627Z] ERROR Error-Handler: TokenError (FATAL): Unable to get trust token caused by iCloudError (FATAL): Authentication failed caused by iCloudError (FATAL): Unable to get iCloud Photos service ready caused by iCloudError (FATAL): Unexpected error while setting up iCloud Photos caused by Request failed with status code 403 (Error Code: 734410aa-a8db-48ce-971f-0f6041f8751a)

Operating environmnent

  • OS: macOS
  • Version: 13.2.1
  • Execution environment: docker
@gsong gsong added the status(open) A new issue - not yet classified label Mar 3, 2023
@steilerDev
Copy link
Owner

@gsong are you using iCloud Shared Photo Library?

@gsong
Copy link
Author

gsong commented Mar 5, 2023

@gsong are you using iCloud Shared Photo Library?

No, I'm not.

@krubenok
Copy link

krubenok commented Mar 6, 2023

I'm in a similar position with advanced data protection and a yubikey on my Apple ID. With this combination of security settings, the sign in prompts I receive on other devices don't have MFA codes, but rather just "ok" or "that wasn't me".

Edit; this feels like something that might have a soft dependency on #120 to do yubikey auth via a webUI.

@steilerDev
Copy link
Owner

steilerDev commented Mar 7, 2023

@gsong Any of those things enabled with you as well?

Otherwise your use case @krubenok would be a separate issue - however: I currently don't have an account setup like this (neither do I plan to do so). Most importantly, in a scenario, where Advanced Data Protection is enabled, the possibility to access your data from the WebUI needs to be enabled (this can be done optionally) - as this tool is using the APIs used by the WebUI.

In case you want me to investigate your use case further, I'd need an HAR file from your authentication against the iCloud WebUI - based on that I might be able to understand what needs to change in order to support this - #120 is not related. (Full disclosure: Keep in mind that this HAR file might contain sensitive data - unless you know how to purge it, you need to trust me that I won't abuse this - however since the MFA trust token is location/IP specific I probably won't be able to use the data from those requests anyway)
See #207

@steilerDev
Copy link
Owner

steilerDev commented Mar 8, 2023

I just realise I was skipping over this part @gsong

Note that I do have Advanced Data Protection turned on.

You will need to make sure that access through the iCloud WebUI is enabled. See Apple's support document on this. The tool is re-using those APIs (and I hope the APIs are the same when this is enabled, since I cannot test this).

Please report back - in case it does not work, I would need to ask the same of you as above.

@steilerDev steilerDev added class(known issue) A known issue that might not be easy to fix or needs additional input. status(investigating) Currently looking for the root cause of the issue and removed status(open) A new issue - not yet classified labels Mar 8, 2023
@steilerDev steilerDev changed the title Can't obtain token Advanced Data Protection Support Mar 8, 2023
@steilerDev steilerDev added the status(help needed) Help from other people is necessary to resolve this issue label Mar 9, 2023
@steilerDev
Copy link
Owner

@krubenok I've created #207 for the addition of YubiKey support - but I'll need help on that.

@steilerDev steilerDev removed the status(investigating) Currently looking for the root cause of the issue label Mar 10, 2023
@noah-guillory
Copy link

Hiya, I'm trying to set this tool up and I have Advanced Data Protection enabled on my iCloud account. I'm currently getting the following error when I try running the token command:

APP_TOKEN (FATAL): Unable to acquire trust token caused by AUTH_FAILED (FATAL): iCloud Authentication failed caused by ICLOUD_PHOTOS_SETUP_FAILED (FATAL): Unable to get iCloud Photos service ready caused by ICLOUD_PHOTOS_SETUP_ERROR (FATAL): Unexpected error while setting up iCloud Photos caused by Request failed with status code 403 (Error Code: cadf49b6-85c4-47b4-84a4-ced655765a67)

I've made sure that I have the "Access iCloud Data on the Web` option enabled in my account settings.

Let me know if I could provide you with more information to help troubleshoot this as I do not want to disable ADP.

@steilerDev
Copy link
Owner

@noah-guillory which 2FA method are you using.

Does the WebUI access work (have you ever accessed the UI from a non Safari browser, where you provided password instead of Touch ID)

@noah-guillory
Copy link

@noah-guillory which 2FA method are you using.

Does the WebUI access work (have you ever accessed the UI from a non Safari browser, where you provided password instead of Touch ID)

I am using the normal 2FA method, not using any hardware security keys or anything.

I was able to get through the process of providing my 2FA code by curling it to the MFA endpoint.

And I am able to access Photos from the WebUI using Edge as well. Though whenever I do, I do get a push notification on my Mac saying that my device is providing access to the iCloud web interface.

@steilerDev
Copy link
Owner

Do you need to confirm this notification before being able to continue?

@steilerDev
Copy link
Owner

steilerDev commented Apr 10, 2023

I need to understand how the API behaves differently from the current process, when ADP is enabled.

Best way for me to debug is by being able to see the iCloud API's behaviour here. For that I'd need a HAR file of your login on the browser. For that do the following:

  1. Open a new private window in Chrome
  2. Navigate to icloud.com
  3. Open the Network Tab of the Developer Tools (e.g. right click on the page and select 'Inspect')
  4. On the Dev Tools Network select 'Preserve log' and 'Disable cache'
  5. Clear the log and make sure logs are recorded
  6. Perform login and open iCloud Photos
  7. Once done, 'Download har' file
  8. Sent it to me so I can take a look at what's happening - feel free to sent it to my email [email protected]

Example of how to do this:

create-har-h265.mp4

@noah-guillory
Copy link

Makes sense! Whenever I get a chance I'll get you that file. Thanks for being responsive 😄

@Tomfox91
Copy link

Hi @steilerDev, is there any hope of ADP support landing soon? Did you get the input you needed?

@steilerDev
Copy link
Owner

@Tomfox91 unfortunately I have not received any feedback on my previous request - so I have not had the chance to implement this.

@steilerDev
Copy link
Owner

Thanks @Tomfox91 for sending over an HAR file - I just had a quick look - some things look different, but the good news is that the API is very close to what I am expecting :)

Unfortunately I'm not sure when I'll get around working on this as private and professional life are currently taking a lot of time :/ Anyone who wants to support on this, I'm happy to point you into directions :)

@skaeight
Copy link

Sadly I think the resolution to this issue is to buy a used / refurb m1 Mac mini.

@steilerDev
Copy link
Owner

@skaeight looking at the previously shared HAR files by @Tomfox91 I don't think this will be necessary (as long as you allow iCloud Access through the WebUI)

@steilerDev steilerDev added class(feature) Indicates work related to a new feature and removed class(feature) Indicates work related to a new feature labels Jul 5, 2023
@frprm
Copy link

frprm commented Sep 6, 2023

I tried a sync using 1.2.0-beta.4, it fails with this:

Error: APP_SYNC: Sync failed caused by AUTH_FAILED: iCloud Authentication failed caused by ICLOUD_PHOTOS_SETUP_FAILED: Unable to get iCloud Photos service ready caused by ICLOUD_PHOTOS_SETUP_ERROR: Unexpected error while setting up iCloud Photos caused by Request failed with status code 403 (error code: 67717c18-aaf8-4d72-9b1c-8b1a66068302)

I can also confirm that "Access iCloud Data on the Web" option is enabled in my account settings.

@steilerDev
Copy link
Owner

See foxt/icloud.js#4 for some research done on this topic

@steilerDev steilerDev removed the status(help needed) Help from other people is necessary to resolve this issue label Sep 25, 2023
@steilerDev
Copy link
Owner

1.4.0-nightly.1 should contain ADP support!

Copy link

This issue should be resolved with version v1.4.0-beta.1, please confirm.

@github-actions github-actions bot added the status(previewed) This issue or pull request should be fixed in a released beta version label Nov 27, 2023
@ido2
Copy link

ido2 commented Nov 27, 2023

I tested nightly (v.1.4.0-nightly.4!), was able to authenticate, got half a dozen "Your Apple ID was used to sign in to iCloud via a web browser" emails from apple, but sync command fails:

[2023-11-27T17:50:01.010Z] DEBUG iCloud: Acquired signin secrets
[2023-11-27T17:50:01.010Z] DEBUG iCloud: Response status is 200, authentication successful - device trusted
[2023-11-27T17:50:01.010Z] DEBUG ResourceManager: Reading resource file from /opt/icloud-photos-library/.icloud-photos-sync
[2023-11-27T17:50:01.011Z] INFO iCloud: Setting up iCloud connection
[2023-11-27T17:50:01.901Z] DEBUG HeaderJar: Extracted cookie from response header: X-APPLE-WEBAUTH-HSA-LOGIN (domain icloud.com) with length 0
[2023-11-27T17:50:01.901Z] DEBUG HeaderJar: Extracted cookie from response header: X-APPLE-UNIQUE-CLIENT-ID (domain icloud.com) with length 6
[2023-11-27T17:50:01.901Z] DEBUG HeaderJar: Extracted cookie from response header: X-APPLE-WEBAUTH-LOGIN (domain icloud.com) with length 220
[2023-11-27T17:50:01.901Z] DEBUG HeaderJar: Extracted cookie from response header: X-APPLE-WEBAUTH-VALIDATE (domain icloud.com) with length 220
[2023-11-27T17:50:01.902Z] DEBUG HeaderJar: Extracted cookie from response header: X-APPLE-WEBAUTH-TOKEN (domain icloud.com) with length 220
[2023-11-27T17:50:01.902Z] DEBUG HeaderJar: Extracted cookie from response header: X-APPLE-WEBAUTH-USER (domain icloud.com) with length 23
[2023-11-27T17:50:01.902Z] DEBUG HeaderJar: Extracted cookie from response header: X_APPLE_WEB_KB-A9PSKFGM1GEJWLXQCBLTPZQTYLI (domain icloud.com) with length 220
[2023-11-27T17:50:01.902Z] DEBUG HeaderJar: Extracted cookie from response header: X-APPLE-DS-WEB-SESSION-TOKEN (domain icloud.com) with length 810
[2023-11-27T17:50:01.903Z] DEBUG NetworkManager: Setting photosUrl to https://p106-ckdatabasews.icloud.com:443
[2023-11-27T17:50:01.904Z] DEBUG iCloud: PCS required, acquiring...
[2023-11-27T17:50:01.904Z] INFO iCloud: Acquiring PCS cookies
[2023-11-27T17:50:02.633Z] DEBUG NetworkManager: Settling rate limiter queue...
[2023-11-27T17:50:02.633Z] DEBUG NetworkManager: Queue has settled!
[2023-11-27T17:50:02.633Z] DEBUG NetworkManager: Settling CCY limiter queue...
[2023-11-27T17:50:02.633Z] DEBUG NetworkManager: Queue has settled!
[2023-11-27T17:50:02.633Z] DEBUG EventManager: Removed 3 listeners for source iCloudPhotos
[2023-11-27T17:50:02.633Z] DEBUG EventManager: No more listeners for source iCloudPhotos registered
[2023-11-27T17:50:02.633Z] DEBUG EventManager: Removed 12 listeners for source iCloud
[2023-11-27T17:50:02.633Z] DEBUG EventManager: No more listeners for source iCloud registered
[2023-11-27T17:50:02.636Z] INFO ErrorHandler: Handling error APP_SYNC caused by AUTH_PCS_REQUEST_FAILED
[2023-11-27T17:50:02.636Z] ERROR RuntimeError: APP_SYNC: Sync failed caused by AUTH_FAILED: iCloud Authentication failed caused by AUTH_PCS_REQUEST_FAILED: Unable to acquire PCS cookies caused by Request failed with status code 500 (error code: Enable crash reporting for error code)

@ido2
Copy link

ido2 commented Nov 27, 2023

Some additional information.

I waited some time and tried again. Now works fine.

Maybe since it tried a couple of times so fast and I hadn't had enough time to authorize on the phone, then it was banned for a while.

So maybe you ought to not try in a busy loop..

Welcome to icloud-photos-sync, v.1.4.0-nightly.4!
Made with <3 by steilerDev
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Authenticating user...
Device trusted
Advanced Data Protection requires additional cookies, acquiring...
Advanced Data Protection request not confirmed yet, retrying...
Advanced Data Protection requires additional cookies, acquiring...
Advanced Data Protection request not confirmed yet, retrying...
Advanced Data Protection requires additional cookies, acquiring...
Advanced Data Protection request not confirmed yet, retrying...
Advanced Data Protection requires additional cookies, acquiring...
Advanced Data Protection request not confirmed yet, retrying...
Advanced Data Protection requires additional cookies, acquiring...
Sign in successful!
iCloud Photos setup completed, checking indexing status...
iCloud Photos ready!
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Starting sync at 11/27/2023, 8:57:51 PM
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Loading local & fetching remote iCloud Library state...

@steilerDev
Copy link
Owner

Yeah - it seems Apple is quite quickly at limiting authentication requests when ADP is enabled - I've ran into the 500 issue as well when testing.

I might adjust the retry timeout - that's good feedback

@rggjan
Copy link

rggjan commented Apr 26, 2024

Hmm... I just tried both the nightly and the beta, but I always get:

----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Welcome to icloud-photos-sync, v.1.4.0-nightly.8!
Made with <3 by steilerDev
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Authenticating user...
MFA code required
Listening for input on port 80

and then it stops. On my iPhone, I get a notification that my icloud account is being logged in, but without asking me to allow it. When accessing iCloud on the web, I get a similar notification, but with an "allow" button, and then it works fine.

@steilerDev
Copy link
Owner

Before ADP kicks in, you will need to provide your MFA code - this is a 6-digit code either pushed to your iDevice or phone number.

See in the docs on how to submit the code to the app: https://icps.steiler.dev/get-started/#submit-mfa-code

@rggjan
Copy link

rggjan commented Apr 26, 2024

Thanks for the quick answer. I never got a code like that on my iPhone or Mac. Maybe because i have two yubikeys as 2FA, which I usually need for login? (Although I have two trusted phone numbers defined as well...)

@steilerDev
Copy link
Owner

Ahh - Yubikey authentication is currently out of scope (since I don't have an account setup with this and I don't own one) - if you've got trusted phone numbers you can have the MFA code resent to them: https://icps.steiler.dev/get-started/#re-send-mfa-code - just provide method 'sms' and a phone number id (those start at 0 and go up - it depends on how many you've got, but they should provide an error if the id is invalid)

@rggjan
Copy link

rggjan commented Apr 26, 2024

Oh, thanks!

So I tried running "docker exec photos-sync resend_mfa sms 0", but I get:

[2024-04-26T11:50:33.596Z] INFO iCloud: Resending MFA code with 'SMS' (Number ID: 0)

[2024-04-26T11:50:33.596Z] DEBUG iCloud: Requesting MFA code via URL https://idmsa.apple.com/appleauth/auth/verify/phone with data {"phoneNumber":{"id":0},"mode":"sms"}

Resending MFA code via 'SMS' (Number ID: 0)...
[2024-04-26T11:50:34.233Z] DEBUG HeaderJar: Extracted scnt from response header with length 446

[2024-04-26T11:50:34.235Z] DEBUG HeaderJar: Extracted cookie from response header: dslang (domain apple.com) with length 5

[2024-04-26T11:50:34.235Z] DEBUG HeaderJar: Extracted cookie from response header: site (domain apple.com) with length 3

[2024-04-26T11:50:34.235Z] DEBUG HeaderJar: Extracted cookie from response header: dslang (domain apple.com) with length 5

[2024-04-26T11:50:34.235Z] DEBUG HeaderJar: Extracted cookie from response header: site (domain apple.com) with length 3

[2024-04-26T11:50:34.236Z] WARN RuntimeWarning: Error within MFA flow: MFA_RESEND_FAILED: Unable to resend MFA code caused by VALIDATOR_RESEND_MFA_PHONE_RESPONSE: Unable to parse and validate resend MFA phone response (must have required property 'trustedPhoneNumber' (/data))

Warning: MFA_RESEND_FAILED: Unable to resend MFA code caused by VALIDATOR_RESEND_MFA_PHONE_RESPONSE: Unable to parse and validate resend MFA phone response (must have required property 'trustedPhoneNumber' (/data))

Do you think it's possible that SMS 2FA is disabled by Apple when security keys are used?

@steilerDev
Copy link
Owner

Can you try with id 1/2/3...?

I know that deleting and re-adding a phone number will increase this id (my demo account always needs id '3' for some reason :D )

@rggjan
Copy link

rggjan commented Apr 26, 2024

I tried up to 6, no luck.

Looking at https://discussions.apple.com/thread/254617891?sortBy=best it seems weaker options (like SMS) are disabled when security keys are added.

Too bad... seems I have to make a choice between backups and strong security 😅

@steilerDev
Copy link
Owner

Could be - to double check, you could use the PostMan collection (https://github.com/steilerDev/icloud-photos-sync/tree/main/docs/postman).

You could add your cred. and then run 01-Enter Password followed by 01--- Get list of devices (this will give you all options available for MFA)

@rggjan
Copy link

rggjan commented Apr 26, 2024

Just tried this, and I get:

{
    "cancelled": false,
    "accountName": "[email protected]",
    "keyNames": [
        "YubiKey USB-C",
        "YubiKey USB-A"
    ],
    "passkeyAutofill": false,
    "fsaChallenge": {
        "challenge": "...",
        "keyHandles": [
            "...",
            "..."
        ],
        "rpId": "apple.com",
        "allowedCredentials": "..."
    }
}

@rggjan
Copy link

rggjan commented Apr 26, 2024

I guess that means no SMS?

@steilerDev
Copy link
Owner

I'm afraid so :/

I'm getting something like this:

{
    "trustedPhoneNumbers": [
        {
            "numberWithDialCode": "+49 ••••• •••••xx",
            "pushMode": "sms",
            "obfuscatedNumber": "••••• •••••xx",
            "lastTwoDigits": "xx",
            "id": 2
        }
    ],
    "securityCode": {
        "length": 6,
        "tooManyCodesSent": false,
        "tooManyCodesValidated": false,
        "securityCodeLocked": false,
        "securityCodeCooldown": false
    },
    "authenticationType": "hsa2",
    "recoveryUrl": "https://iforgot.apple.com/phone/add?prs_account_nm=steilerdev%40web.de&autoSubmitAccount=true&appId=142",
    "cantUsePhoneNumberUrl": "https://iforgot.apple.com/iforgot/phone/add?context=cantuse&prs_account_nm=steilerdev%40web.de&autoSubmitAccount=true&appId=142",
    "recoveryWebUrl": "https://iforgot.apple.com/password/verify/appleid?prs_account_nm=steilerdev%40web.de&autoSubmitAccount=true&appId=142",
    "repairPhoneNumberUrl": "https://gsa.apple.com/appleid/account/manage/repair/verify/phone",
    "repairPhoneNumberWebUrl": "https://appleid.apple.com/widget/account/repair?#!repair",
    "aboutTwoFactorAuthenticationUrl": "https://support.apple.com/kb/HT204921",
    "autoVerified": false,
    "showAutoVerificationUI": false,
    "supportsCustodianRecovery": false,
    "hideSendSMSCodeOption": false,
    "supervisedChangePasswordFlow": false,
    "supportsRecovery": true,
    "trustedPhoneNumber": {
        "numberWithDialCode": "+49 ••••• •••••xx",
        "pushMode": "sms",
        "obfuscatedNumber": "••••• •••••xx",
        "lastTwoDigits": "xx",
        "id": 2
    },
    "hsa2Account": true,
    "restrictedAccount": false,
    "managedAccount": false
}

Unfortunately I have no idea how much work it would be to implemented - but based on a previous provided capture of a yubi key flow, it did not seem straight forward :/

Nevertheless, I'd welcome you taking a shot

@rggjan
Copy link

rggjan commented Apr 26, 2024

OK, in any case, thanks a lot for your help!

@maartenhendrix
Copy link

Any news on this?
I'm getting:

Welcome to icloud-photos-sync, v.1.4.0-nightly.8!
Made with <3 by steilerDev
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Authenticating user...
Device trusted
Error: APP_SYNC: Sync failed caused by AUTH_FAILED: iCloud Authentication failed caused by AUTH_ACCOUNT_SETUP: Unable to setup iCloud Account caused by VALIDATOR_SETUP_RESPONSE: Unable to parse and validate setup response (must be equal to constant (/data/dsInfo/isWebAccessAllowed)) (error code: c5235acd-9098-47cc-a2cb-943fd01678f6)

When I try docker exec -t photos-sync icloud-photos-sync sync
Requesting the token did finish without an error.

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Welcome to icloud-photos-sync, v.1.4.0-nightly.8!
Made with <3 by steilerDev
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Authenticating user...
MFA code required
Listening for input on port 80
MFA code received from 'Device' (...)
User authenticated
Device trusted
iCloud Photos ready!
Validated token:
<...>

@steilerDev
Copy link
Owner

@maartenhendrix have you enabled "Access iCloud data on the web"? This is necessary for the tool to read the Photo Library data.

@maartenhendrix
Copy link

@steilerDev sorry about that. It was not enabled. Works now. Thank you very much for the great tool!

@gsong
Copy link
Author

gsong commented Jul 6, 2024

A bit late to the game in providing feedback on this, but using v.1.4.0-nightly.8 I was able to successfully sync with ADP turned on. Thank you for your work on this.

@cbruegg
Copy link
Sponsor

cbruegg commented Aug 24, 2024

As this issue is still open, I'd like to report that syncing photos worked flawlessly for me using 1.4.0-beta.1 with ADP enabled. Thanks a lot!

@pudge
Copy link
Sponsor

pudge commented Aug 26, 2024

It works for me … for one hour. Then fails.

Welcome to icloud-photos-sync, v.1.4.0-beta.1!
Made with <3 by steilerDev
----------------------------------------------------------------------------------------------------------------------------------------------------------------
Authenticating user...
Device trusted
Advanced Data Protection requires additional cookies, acquiring...
Advanced Data Protection request not confirmed yet, retrying...
Advanced Data Protection requires additional cookies, acquiring...
Advanced Data Protection request not confirmed yet, retrying...
Advanced Data Protection requires additional cookies, acquiring...
Advanced Data Protection request not confirmed yet, retrying...
Advanced Data Protection requires additional cookies, acquiring...
Sign in successful!
iCloud Photos setup completed, checking indexing status...
iCloud Photos ready!
----------------------------------------------------------------------------------------------------------------------------------------------------------------
Starting sync at 8/26/2024, 2:49:08 AM
----------------------------------------------------------------------------------------------------------------------------------------------------------------
Loading local & fetching remote iCloud Library state...
Loaded local state: 0 assets & 0 albums
Fetched remote state: 35083 assets & 25 albums
Warning: Detected 5 albums, where asset counts don't match, please check the logs for more details (and see https://icps.steiler.dev/warnings/ for context)
Diffing remote with local state...
Diffing completed!
Writing diff to disk...
Syncing assets, by keeping 0 and removing 0 local assets, as well as adding 35083 remote assets...
 ■■■■■                                    13% | Elapsed: 59m16s | 4599/35083 assets downloaded
Detected error during sync: SYNC_NETWORK: Network error during sync caused by Request failed with status code 410
Refreshing iCloud connection & retrying (attempt #2)...

I retried twice more, and both times failed at 59mXXs.

@steilerDev
Copy link
Owner

steilerDev commented Aug 27, 2024

@pudge this is unfortunately a limitation of the API (staging docs):

Currently metadata needs to be updated every 60 minutes, which will lead to a re-authentication and therefore the requirement to manually re-authorize of the data access.

This re-authorization is handled when ADP is off and uses a cached trust token - ADP requires approval of the data access.

This should however only matter with the initial sync (which will probably have to load a ton of data) - the sync is incremental and existing data will not be deleted

@pudge
Copy link
Sponsor

pudge commented Aug 27, 2024

@pudge this is unfortunately a limitation of the API (staging docs):

Currently metadata needs to be updated every 60 minutes, which will lead to a re-authentication and therefore the requirement to manually re-authorize of the data access.

This re-authorization is handled when ADP is off and uses a cached trust token - ADP requires approval of the data access.

This should however only matter with the initial sync (which will probably have to load a ton of data) - the sync is incremental and existing data will not be deleted

a. I did not get a request to re-authorize (I was watching it at the time)
b. It does not look like like it makes more progress each time: each retry and run restarts with only a small handful of assets kept.

----------------------------------------------------------------------------------------------------------------------------------------------------------------
Starting sync at 8/26/2024, 2:49:08 AM
----------------------------------------------------------------------------------------------------------------------------------------------------------------
Loading local & fetching remote iCloud Library state...
Loaded local state: 0 assets & 0 albums
Fetched remote state: 35083 assets & 25 albums
Warning: Detected 5 albums, where asset counts don't match, please check the logs for more details (and see https://icps.steiler.dev/warnings/ for context)
Diffing remote with local state...
Diffing completed!
Writing diff to disk...
Syncing assets, by keeping 0 and removing 0 local assets, as well as adding 35083 remote assets...
 ■■■■■                                    13% | Elapsed: 59m16s | 4599/35083 assets downloaded
Detected error during sync: SYNC_NETWORK: Network error during sync caused by Request failed with status code 410
Refreshing iCloud connection & retrying (attempt #2)...
----------------------------------------------------------------------------------------------------------------------------------------------------------------
Sign in successful!
iCloud Photos setup completed, checking indexing status...
iCloud Photos ready!
----------------------------------------------------------------------------------------------------------------------------------------------------------------
Loading local & fetching remote iCloud Library state...
Loaded local state: 4590 assets & 0 albums
Fetched remote state: 35083 assets & 25 albums
Warning: Detected 5 albums, where asset counts don't match, please check the logs for more details (and see https://icps.steiler.dev/warnings/ for context)
Diffing remote with local state...
Diffing completed!
Writing diff to disk...
Syncing assets, by keeping 22 and removing 4568 local assets, as well as adding 35061 remote assets...
 ■                                        1% | Elapsed: 13m5s | 674/35061 assets downloaded
Detected error during sync: SYNC_UNKNOWN: Unknown error during sync caused by ENOENT: no such file or directory, open '/opt/icloud-photos-library/_All-Photos/AbxXUfjCkKroCU5tVAE0wK5HUj75.heic'
Refreshing iCloud connection & retrying (attempt #3)...
----------------------------------------------------------------------------------------------------------------------------------------------------------------
Sign in successful!
iCloud Photos setup completed, checking indexing status...
iCloud Photos ready!
----------------------------------------------------------------------------------------------------------------------------------------------------------------
Loading local & fetching remote iCloud Library state...
Loaded local state: 709 assets & 0 albums
Fetched remote state: 35083 assets & 25 albums
Fetched remote state: 35083 assets & 25 albums
Warning: Detected 5 albums, where asset counts don't match, please check the logs for more details (and see https://icps.steiler.dev/warnings/ for context)
Diffing remote with local state...
Diffing completed!
Writing diff to disk...
Syncing assets, by keeping 26 and removing 683 local assets, as well as adding 35057 remote assets...
 ■                                        1% | Elapsed: 1m57s | 539/35057 assets downloaded
Detected error during sync: SYNC_UNKNOWN: Unknown error during sync caused by ENOENT: no such file or directory, open '/opt/icloud-photos-library/_All-Photos/AWk58dykop55uk8d_1sexVwMtsAh.jpeg'

So each retry, I would be left with only a small handful of kept assets. After running through this four times (x10 “retries” each time), it kept only 109 local assets.

@steilerDev
Copy link
Owner

Do you have Advanced Data Protection enabled? Looking at the logs, it seems this is unrelated to this issue (as the re-sync errors quickly with a different error) - I think it's best to track your issue in a separate ticket.

I will lock this issue for now - it will be closed once I've released the stable release of ADP support with the next public release (which I should probably do soon).

Repository owner locked as resolved and limited conversation to collaborators Aug 27, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
class(known issue) A known issue that might not be easy to fix or needs additional input. status(in progress) This item is currently in scope for the next release status(previewed) This issue or pull request should be fixed in a released beta version
Projects
None yet
Development

No branches or pull requests