It looks like this system either does not exist or has never had any recorded downtime.
+
This could be a problem on our part. Perhaps we moved a certain resource and now it is gone. It is also possible that the resource you are trying to view is empty. But do you also mind to double check the link?
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
diff --git a/CNAME b/CNAME
new file mode 100644
index 00000000..3ee852df
--- /dev/null
+++ b/CNAME
@@ -0,0 +1 @@
+status.packit.dev
diff --git a/admin/config.yml b/admin/config.yml
new file mode 100644
index 00000000..551eb969
--- /dev/null
+++ b/admin/config.yml
@@ -0,0 +1,372 @@
+# This is the main configuration file for Netlify CMS.
+# Netlify CMS is used as the admin panel for cState.
+#
+# While some settings here can be kept as is, you may
+# want to change some for a more customized solution.
+#
+# You can look at what each option does and how to
+# toggle between them on the Netlify CMS repo docs:
+#
+# https://github.com/netlify/netlify-cms/blob/master/docs/quick-start.md
+
+backend:
+ name: git-gateway
+media_folder: "static/img"
+public_folder: "/img"
+site_url: ../
+logo_url: https://raw.githubusercontent.com/cstate/cstate/master/images/cstate-logo-bg.svg
+
+# Do not change this unless you know what you are doing!
+collections:
+ - name: "issues"
+ label: "Incidents"
+ label_singular: "Incident"
+ description: "๐ Welcome to the cState admin panel! Here you can create new incidents or manage existing ones. Go ahead and give it a shot."
+ folder: "content/issues"
+ create: true
+ slug: "{{year}}-{{month}}-{{day}}-{{slug}}"
+ view_filters:
+ - label: Active (unresolved) issues
+ field: resolved
+ pattern: false
+ - label: 'Resolved issues'
+ field: resolved
+ pattern: true
+ - label: 'Informational posts'
+ field: informational
+ pattern: true
+ - label: Drafts
+ field: draft
+ pattern: true
+ - label: Published (not a draft)
+ field: draft
+ pattern: false
+ - label: "High severity (down)"
+ field: severity
+ pattern: 'down'
+ - label: "Medium severity (disrupted)"
+ field: severity
+ pattern: 'disrupted'
+ - label: "Low severity (announcement)"
+ field: severity
+ pattern: 'announcement'
+ fields:
+ - {label: "Mark as incident", name: "section", widget: "hidden", default: "issue"}
+ - {label: "Title", name: "title", widget: "string"}
+ - {label: "Start date & time (your time) โ", name: "date", widget: "datetime"}
+ - {label: "Mark as resolved โ", name: "resolved", widget: "boolean", required: false, default: false}
+ - {label: "Hide this from the site (make it a draft) ๐", name: "draft", widget: "boolean", required: false, default: false}
+ - {label: "Mark as informational โน", name: "informational", widget: "boolean", required: false, default: false}
+ - {label: "Pin to top of page", name: "pin", widget: "boolean", required: false, default: false}
+ - {label: "End date & time (your time) โ", name: "resolvedWhen", widget: "datetime", required: false}
+ - label: "Affected systems (use exact name, separated by commas) ๐ง"
+ name: "affected"
+ widget: "list"
+ required: false
+ - label: "Severity โ "
+ name: "severity"
+ widget: "select"
+ options:
+ - { label: "High (Down) โผ ", value: "down" }
+ - { label: "Medium (Disrupted) โฒ", value: "disrupted" }
+ - { label: "Low (Announcement) โ", value: "notice" }
+ - label: "Markdown body (sequence of events, description, post-mortem)"
+ name: "body"
+ widget: "markdown"
+ default: "*Investigating* - We are investigating a potential issue that might affect the uptime of one our of services. We are sorry for any inconvenience this may cause you. This incident post will be updated once we have more information."
+ - name: "pages"
+ label: "Pages"
+ label_singular: "Page"
+ description: "๐ Create and manage pages (not connected or related to issues). Good for things like about pages, SLAs, ways of contact, and so forth."
+ folder: "content/pages"
+ create: true
+ slug: "{{slug}}"
+ view_filters:
+ - label: Drafts
+ field: draft
+ pattern: true
+ - label: Published (not a draft)
+ field: draft
+ pattern: false
+ fields:
+ - {label: "Title", name: "title", widget: "string"}
+ - {label: "Hide this from the site (make it a draft) ๐", name: "draft", widget: "boolean", required: false, default: false}
+ - {label: "Description for SEO and social media", name: "description", widget: "string", required: false}
+ - label: "Markdown (or HTML) body"
+ name: "body"
+ widget: "markdown"
+ - name: "settings"
+ label: "Settings"
+ description: "โ Please be warned that by editing settings from Netlify CMS you are _OVERWRITING_ what you have in the original Git repository, meaning any CURRENT SETTINGS OR COMMENTS left will be deleted."
+ delete: false # Prevent users from deleting documents in this collection
+ editor:
+ preview: false
+ files:
+ - name: "general"
+ label: "โ Configuration File"
+ file: "config.yml"
+ fields:
+ # MAIN
+ - label: 'Site title'
+ hint: 'What is your status page called? Shows up in the browser bar and meta.'
+ name: 'title'
+ widget: 'string'
+ - label: 'Build future posts'
+ name: 'buildFuture'
+ widget: 'boolean'
+ hint: 'Should posts, which have a publish date from the future, be built? Useful for sharing upcoming maintenance, etc.'
+ required: false
+ default: true
+ - label: 'Site display language'
+ hint: 'Docs: https://github.com/cstate/cstate/wiki/Translations#available-translations'
+ name: 'defaultContentLanguage'
+ widget: 'select'
+ default: 'en'
+ options:
+ - { label: "๐บ๐ธ English (default)", value: "en" }
+ - { label: "๐ฑ๐น Lietuviลกkai (official)", value: "lt" }
+ - { label: "๐น๐ท Turkish", value: "tr" }
+ - { label: "๐ฉ๐ช Deutsch", value: "de" }
+ - { label: "๐ณ๐ฑ Dutch", value: "nl" }
+ - { label: "๐ซ๐ท French", value: "fr" }
+ - { label: "๐ฎ๐น Italiano", value: "it" }
+ - { label: "๐ง๐ท Portuguese", value: "pt"}
+ - { label: "๐ฒ๐ฐ ะะฐะบะตะดะพะฝัะบะธ", value: "mk" }
+ - { label: "๐ณ๏ธ Tagalog", value: "tl" }
+ - { label: "๐ฏ๐ต Japanese", value: "ja" }
+ - label: 'Site language in code for html[lang]'
+ hint: 'Use the ISO 639-1 defined abbreviations. Examples: en, lt, de. Fully explained here: https://github.com/cstate/cstate/wiki/Customization#changing-site-language'
+ name: 'languageCode'
+ widget: 'string'
+ default: 'en'
+ - label: 'Base URL / Hostname'
+ name: 'baseURL'
+ widget: 'string'
+ hint: 'Where is the site hosted? Hostname (and path) to the root. Prior to version 3, a slash was used which now works in local testing, but breaks certain features of cState like RSS feeds, so a correct example for production is: https://cstate.mnts.lt'
+ default: '/'
+
+ # REQUIRED BUT HIDDEN
+ - label: 'theme'
+ name: 'theme'
+ widget: 'hidden'
+ default: 'cstate'
+ - label: 'preserveTaxonomyNames'
+ name: 'preserveTaxonomyNames'
+ widget: 'hidden'
+ default: 'true'
+ - label: "taxonomies"
+ name: taxonomies
+ widget: "hidden"
+ fields:
+ - {label: "affected", name: "affected", widget: "hidden", default: "affected"}
+ - label: "Outputs (Do not touch!)"
+ name: outputs
+ widget: "hidden"
+ fields:
+ - label: "page"
+ name: page
+ widget: "list"
+ allow_add: false
+ default: ['html', 'json']
+ - label: "section"
+ name: section
+ widget: "list"
+ allow_add: false
+ default: ['html', 'json', 'rss']
+ - label: "home"
+ name: home
+ widget: "list"
+ allow_add: false
+ default: ['html', 'json', 'rss', 'svg']
+ - label: "term"
+ name: term
+ widget: "list"
+ allow_add: false
+ default: ['html', 'json', 'rss']
+ - label: "Output formats (Do not touch!)"
+ name: outputFormats
+ widget: "hidden"
+ fields:
+ - label: "svg"
+ name: svg
+ fields:
+ - label: "isPlainText"
+ name: isPlainText
+ widget: "boolean"
+ default: true
+ - label: "mediaType"
+ name: mediaType
+ default: "image/svg+xml"
+ - label: 'Enable Git info'
+ name: 'enableGitInfo'
+ widget: 'hidden'
+ required: false
+ default: true
+
+ # PARAMS
+ - label: "Params"
+ name: "params"
+ widget: "object"
+ fields:
+ # Components
+ - label: "System components"
+ name: "systems"
+ widget: "list"
+ allow_add: true
+ fields:
+ - {label: "Name", name: "name", widget: string}
+ - {label: "Category", hint: "All systems need a category. For more, read the migration docs: https://github.com/cstate/cstate/wiki/Usage#v3-to-v4", name: "category", widget: string}
+ - {label: "Description", name: "description", widget: string, required: false}
+ - {label: "Partial", hint: "Custom HTML feature: https://github.com/cstate/cstate/wiki/Customization#custom-html", name: "partial", widget: string, required: false}
+ # Categories
+ - label: "Categories"
+ name: "categories"
+ widget: "list"
+ allow_add: true
+ fields:
+ - {label: "Name", name: "name", widget: string}
+ - {label: "Description", name: "description", widget: string, required: false}
+ - {label: "Should this category be open by default", name: "closed", widget: "boolean", required: false}
+ - {label: "Should the name of this category be shown? If you want to create the appearance of an Uncategorized category, enable this.", name: "untitled", widget: "boolean", required: false}
+ # Tabs
+ - label: "Tabs"
+ hint: 'You can add extra tabs below the main homepage summary, for example, to external sites, monitoring services, as shown in the example below. Try Uptime Robot!'
+ name: "customTabs"
+ widget: "list"
+ allow_add: true
+ fields:
+ - {label: "Name", name: "name", widget: string}
+ - {label: "Link", name: "link", widget: string}
+ # Dates
+ - label: 'Show last modified date at bottom of issues'
+ name: 'enableLastMod'
+ widget: 'boolean'
+ hint: 'Experimental feature shows when changes occured based on Git info. Read the wiki for more: https://github.com/cstate/cstate/wiki/Customization#time'
+ required: false
+ default: false
+ - label: 'Use relative time (issue began x min ago)'
+ name: 'useRelativeTime'
+ widget: 'boolean'
+ hint: 'IMPORTANT: In the frontmatter, the dates MUST be in the UTC time zone. If you use Netlify CMS, the CMS picks UTC time by default, so no need to worry. However, there may be very inaccurate times, if you manually define non-UTC time in your issues. Read the wiki for more: https://github.com/cstate/cstate/wiki/Customization#time'
+ required: false
+ default: true
+ - label: 'Skip seconds in relative time calculations?'
+ name: 'skipSeconds'
+ widget: 'boolean'
+ hint: 'For users of relative time. With option ON (true): "Last checked <1 min ago". With option OFF (false; default): "Last checked 20s ago". Read the wiki for more: https://github.com/cstate/cstate/wiki/Customization#skip-seconds-v50'
+ required: false
+ default: false
+ - label: 'Long date format'
+ name: 'dateFormat'
+ widget: 'string'
+ hint: 'Default: January 2, 2006 at 3:04 PM UTC. Docs: https://github.com/cstate/cstate/wiki/Customization#changing-date-formats-v30'
+ default: 'January 2, 2006 at 3:04 PM UTC'
+ - label: 'Short date format'
+ name: 'shortDateFormat'
+ widget: 'string'
+ hint: 'Default: 15:04 UTC โ Jan 2. Docs: https://github.com/cstate/cstate/wiki/Customization#changing-date-formats-v30'
+ default: '15:04 UTC โ Jan 2'
+ # Design site title color toggle in v4.0.1
+ - label: 'Site title text color'
+ hint: 'This is irrelevant, if you use a logo in your header.'
+ name: 'headerTextColor'
+ widget: 'select'
+ default: 'white'
+ options:
+ - { label: "Black", value: "black" }
+ - { label: "White", value: "white" }
+ # Design toggles
+ - label: 'Use large header design'
+ name: 'useLargeHeaderDesign'
+ widget: 'boolean'
+ hint: 'What header design should we use? https://github.com/cstate/cstate/wiki/Customization#changing-header'
+ required: false
+ default: false
+ - label: 'Show logo'
+ name: 'useLogo'
+ widget: 'boolean'
+ required: false
+ default: true
+ - label: 'Disable dark mode'
+ hint: 'If your OS and browser supports prefers-color-scheme and the user opted into a dark mode, the status page will be darker. Some color choices may be overriden.'
+ name: 'disableDarkMode'
+ widget: 'boolean'
+ required: false
+ default: false
+ - label: 'Logo image'
+ hint: 'Upload your logo here. If the logo option is switched off, this will not show.'
+ name: 'logo'
+ widget: 'image'
+ # Description
+ - label: 'Site description'
+ hint: 'This is the description that is shown on the footer and meta tags.'
+ name: 'description'
+ widget: 'text'
+ default: 'We continuously monitor the status of our services and if there are any interruptions, a note will be posted here.'
+ # Incidents per page
+ - label: 'Incident posts shown in one page'
+ name: 'incidentPostsPerPage'
+ hint: ''
+ widget: 'number'
+ valueType: 'int'
+ min: 1
+ max: 100
+ default: 10
+ # Incidents history
+ - label: 'Should the incident history be hidden?'
+ name: 'disableIncidentHistory'
+ hint: 'By disabling the incident history, you also disable the RSS feed. To ensure no incidents are shown, you should delete them after they are resolved. This option overrides any other options that tailor your incident historyโs look.'
+ widget: 'boolean'
+ default: false
+ # Incidents view format
+ - label: 'Should the incident history be sorted by year or month?'
+ name: 'incidentHistoryFormat'
+ hint: 'Enabling sorting disables pagination. More: https://github.com/cstate/cstate/wiki/Customization#enable-sorting--archives-by-year-or-month-v40'
+ widget: 'select'
+ options:
+ - { label: "Show by year", value: "yearly" }
+ - { label: "Show by month", value: "monthly" }
+ - { label: "No sorting; enable pagination", value: "none" }
+ # Brand color
+ - label: 'Header: always keep brand color the same'
+ name: 'alwaysKeepBrandColor'
+ hint: 'You may choose whether the brand color should stay unchanged once there are disruptions or outages on the header.'
+ widget: 'boolean'
+ default: true
+ # Colors
+ - label: 'Site HEX color: Brand'
+ hint: 'Use any color type, even rgb() works. We recommend HEX: #123456. Default: #0a0c0f'
+ name: 'brand'
+ widget: 'string'
+ default: '#0a0c0f'
+ - label: 'Site HEX color: Operational / OK'
+ hint: 'Use any color type, even rgb() works. We recommend HEX: #123456. Default: #008000'
+ name: 'ok'
+ widget: 'string'
+ default: '#008000'
+ - label: 'Site HEX color: Disrupted'
+ hint: 'Use any color type, even rgb() works. We recommend HEX: #123456. Default: #cc4400'
+ name: 'disrupted'
+ widget: 'string'
+ default: '#cc4400'
+ - label: 'Site HEX color: Down / Offline / Major'
+ hint: 'Use any color type, even rgb() works. We recommend HEX: #123456. Default: #e60000'
+ name: 'down'
+ widget: 'string'
+ default: '#e60000'
+ - label: 'Site HEX color: Notice'
+ hint: 'Use any color type, even rgb() works. We recommend HEX: #123456. Default: #24478f'
+ name: 'notice'
+ widget: 'string'
+ default: '#24478f'
+ # Google Analytics
+ - label: 'Google Analytics tracking code'
+ hint: 'By default, cState does not use Google Analytics. If you choose to use it, you may change the placeholder code below to your own and thereby enable the tracking service. If you set the value back to UA-00000000-1, the code will once again be removed.'
+ name: 'googleAnalytics'
+ widget: 'string'
+ default: 'UA-00000000-1'
+ # Enable Custom HTML
+ - label: 'Enable custom HTML'
+ name: 'enableCustomHTML'
+ widget: 'boolean'
+ default: 'false'
diff --git a/admin/index.html b/admin/index.html
new file mode 100644
index 00000000..00bc0519
--- /dev/null
+++ b/admin/index.html
@@ -0,0 +1,90 @@
+
+
+
+
+
+ cState Admin
+
+
+
+
+
+
+
+
+
+
diff --git a/affected/all/index.html b/affected/all/index.html
new file mode 100644
index 00000000..e988675a
--- /dev/null
+++ b/affected/all/index.html
@@ -0,0 +1,769 @@
+
+
+
+
+
+
+
+
+
+ all | Packit Service Status
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ Recently, based on average data, it looks like this system has gone down for about
+
+ 170
+ minutes at a time.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
diff --git a/affected/all/index.json b/affected/all/index.json
new file mode 100644
index 00000000..8ff55cef
--- /dev/null
+++ b/affected/all/index.json
@@ -0,0 +1,55 @@
+{
+ "is": "system",
+ "title": "all",
+ "permalink": "https://status.packit.dev/affected/all/",
+ "status": "ok",
+ "pages": [
+ {
+ "is": "issue",
+ "title": "Outage ",
+ "createdAt": "2024-05-07 13:00:00 +0200 +0200",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2024-05-07-outage/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2024-05-07T16:00:00+02:00",
+ "affected": ["all"],
+ "filename": "2024-05-07-outage.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Outage ",
+ "createdAt": "2024-04-25 14:30:00 +0200 +0200",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2024-04-25-outage/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2024-04-25T19:30:00+02:00",
+ "affected": ["all"],
+ "filename": "2024-04-25-outage.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Packit outage now resolved",
+ "createdAt": "2023-08-21 14:45:00 +0200 +0200",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-08-21-packit-outage/",
+ "severity": "notice",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-08-21T15:15:00+02:00",
+ "affected": ["all"],
+ "filename": "2023-08-21-packit-outage.md"
+ }
+
+ ]
+}
+
+
+
+
+
diff --git a/affected/all/index.xml b/affected/all/index.xml
new file mode 100644
index 00000000..374f685a
--- /dev/null
+++ b/affected/all/index.xml
@@ -0,0 +1,60 @@
+
+
+
+
+ all on Packit Service Status
+ https://status.packit.dev/affected/all/
+ Incident history
+ github.com/cstate
+ en
+
+ 2024-05-07T13:00:00+02:00
+ 2024-05-07T13:00:00+02:00
+
+
+
+
+
+
+
+ [Resolved] Outage
+ https://status.packit.dev/issues/2024-05-07-outage/
+ Tue, 07 May 2024 13:00:00 +0200
+ https://status.packit.dev/issues/2024-05-07-outage/
+ 2024-05-07T16:00:00+02:00
+ There has been a Packit outage. The jobs in the queue are now being picked up again and should be processed gradually (you might have experienced a delay in job execution). We are very sorry for the inconvenience!
+ <p>There has been a Packit outage.
+The jobs in the queue are now being picked up again and should be processed gradually
+(you might have experienced a delay in job execution).
+We are very sorry for the inconvenience!</p>
+
+
+
+
+ [Resolved] Outage
+ https://status.packit.dev/issues/2024-04-25-outage/
+ Thu, 25 Apr 2024 14:30:00 +0200
+ https://status.packit.dev/issues/2024-04-25-outage/
+ 2024-04-25T19:30:00+02:00
+ There has been a Packit outage. We are investigating the reason of it. The jobs in the queue are now being picked up again and should be processed gradually (you might have experienced a delay in job execution). We are very sorry for the inconvenience!
+ <p>There has been a Packit outage. We are investigating the reason of it.
+The jobs in the queue are now being picked up again and should be processed gradually
+(you might have experienced a delay in job execution).
+We are very sorry for the inconvenience!</p>
+
+
+
+
+ [Resolved] Packit outage now resolved
+ https://status.packit.dev/issues/2023-08-21-packit-outage/
+ Mon, 21 Aug 2023 14:45:00 +0200
+ https://status.packit.dev/issues/2023-08-21-packit-outage/
+ 2023-08-21T15:15:00+02:00
+ There has been a Packit outage now resolved. If your jobs are stuck, please retrigger them. We are sorry for the inconvenience.
+ <p>There has been a Packit outage now resolved. If your jobs are stuck, please retrigger them.
+We are sorry for the inconvenience.</p>
+
+
+
+
+
diff --git a/affected/all/page/1/index.html b/affected/all/page/1/index.html
new file mode 100644
index 00000000..defc2120
--- /dev/null
+++ b/affected/all/page/1/index.html
@@ -0,0 +1,10 @@
+
+
+
+ https://status.packit.dev/affected/all/
+
+
+
+
+
+
diff --git a/affected/api/index.html b/affected/api/index.html
new file mode 100644
index 00000000..ed76eaa1
--- /dev/null
+++ b/affected/api/index.html
@@ -0,0 +1,1100 @@
+
+
+
+
+
+
+
+
+
+ API | Packit Service Status
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
diff --git a/affected/api/index.json b/affected/api/index.json
new file mode 100644
index 00000000..66515960
--- /dev/null
+++ b/affected/api/index.json
@@ -0,0 +1,167 @@
+{
+ "is": "system",
+ "title": "API",
+ "permalink": "https://status.packit.dev/affected/api/",
+ "status": "ok",
+ "pages": [
+ {
+ "is": "issue",
+ "title": "GitLab outage",
+ "createdAt": "2023-12-19 05:30:00 +0100 +0100",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-12-19-gitlab-outage/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-12-20T12:00:00+01:00",
+ "affected": ["API"],
+ "filename": "2023-12-19-gitlab-outage.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Cluster upgrade",
+ "createdAt": "2023-10-02 08:00:00 +0000 UTC",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-10-02-cluster-upgrade/",
+ "severity": "notice",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-10-02T10:00:00+00:00",
+ "affected": ["API", "Workers", "Dashboard"],
+ "filename": "2023-10-02-cluster-upgrade.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Outage",
+ "createdAt": "2023-08-15 04:00:00 +0200 +0200",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-08-15-outage/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-08-15T11:03:00+02:00",
+ "affected": ["API", "Workers"],
+ "filename": "2023-08-15-outage.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Cluster Update",
+ "createdAt": "2023-05-03 10:00:00 +0200 +0200",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-05-03-cluster-update/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-05-03T11:15:00+02:00",
+ "affected": ["API", "Workers", "Dashboard"],
+ "filename": "2023-05-03-cluster-update.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Scheduled DB Cleanup",
+ "createdAt": "2023-04-26 04:40:00 +0000 UTC",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-04-26-scheduled-db-cleanup/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-04-26T05:17:00+00:00",
+ "affected": ["API", "Dashboard"],
+ "filename": "2023-04-26-scheduled-db-cleanup.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "SLO1 Error Budget issues",
+ "createdAt": "2023-03-09 10:00:00 +0100 +0100",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-03-09-slo1-error-budget/",
+ "severity": "notice",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-06-08T10:00:00+01:00",
+ "affected": ["API", "Workers", "Dashboard"],
+ "filename": "2023-03-09-slo1-error-budget.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Running grouping migrations",
+ "createdAt": "2023-02-14 02:00:00 +0100 +0100",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-02-14-running-grouping-migrations/",
+ "severity": "down",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-02-14T04:00:00+01:00",
+ "affected": ["API", "Workers", "Dashboard", "Copr", "Testing Farm", "Koji"],
+ "filename": "2023-02-14-running-grouping-migrations.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Outage",
+ "createdAt": "2022-11-24 23:24:00 +0100 +0100",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2022-11-25-outage/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2022-11-25T08:35:00+01:00",
+ "affected": ["API", "Workers"],
+ "filename": "2022-11-25-outage.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "GitHub Webhooks not working",
+ "createdAt": "2022-10-05 16:00:00 +0200 +0200",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2022-10-05-github-webhooks-degraded-status/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2022-10-05T18:00:00+02:00",
+ "affected": ["API"],
+ "filename": "2022-10-05-github-webhooks-degraded-status.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Moving production Packit Service to a new cluster",
+ "createdAt": "2021-11-18 01:00:00 +0000 UTC",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2021-11-16-auto-prod/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2021-11-18 02:33:00",
+ "affected": ["API", "Workers", "Dashboard"],
+ "filename": "2021-11-16-auto-prod.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Dashboard - unable to connect",
+ "createdAt": "2021-06-02 11:16:00 +0000 UTC",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2021-06-02-dashboard-unable-to-connect/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2021-06-02 11:40:00",
+ "affected": ["API", "Dashboard"],
+ "filename": "2021-06-02-dashboard-unable-to-connect.md"
+ }
+
+ ]
+}
+
+
+
+
+
diff --git a/affected/api/index.xml b/affected/api/index.xml
new file mode 100644
index 00000000..4eadfdc2
--- /dev/null
+++ b/affected/api/index.xml
@@ -0,0 +1,192 @@
+
+
+
+
+ API on Packit Service Status
+ https://status.packit.dev/affected/api/
+ Incident history
+ github.com/cstate
+ en
+
+ 2023-12-19T05:30:00+01:00
+ 2023-12-19T05:30:00+01:00
+
+
+
+
+
+
+
+ [Resolved] GitLab outage
+ https://status.packit.dev/issues/2023-12-19-gitlab-outage/
+ Tue, 19 Dec 2023 05:30:00 +0100
+ https://status.packit.dev/issues/2023-12-19-gitlab-outage/
+ 2023-12-20T12:00:00+01:00
+ Packit production instance has lost access to GitLab API. This most likely affects also self-hosted GitLab forges.
+The issue has been resolved and everything should work as expected.
+ <p>Packit production instance has lost access to GitLab API.
+This most likely affects also self-hosted GitLab forges.</p>
+<p>The issue has been resolved and everything should work as expected.</p>
+
+
+
+
+ [Resolved] Cluster upgrade
+ https://status.packit.dev/issues/2023-10-02-cluster-upgrade/
+ Mon, 02 Oct 2023 08:00:00 +0000
+ https://status.packit.dev/issues/2023-10-02-cluster-upgrade/
+ 2023-10-02T10:00:00+00:00
+ A cluster upgrade is scheduled to start on Mon October 2nd at 08:00 UTC and last approximately 90 minutes.
+We do not expect any significant disruption.
+The started jobs might be executed with a delay and if you think Packit does not react to your events, try to retrigger the action via comment command.
+ <p>A cluster upgrade is scheduled to start on Mon October 2nd at 08:00 UTC and last approximately 90 minutes.</p>
+<p>We do not expect any significant disruption.</p>
+<p>The started jobs might be executed with a delay
+and if you think Packit does not react to your events,
+try to retrigger the action via
+<a href="https://packit.dev/docs/guide/#how-to-re-trigger-packit-actions-in-your-pull-request">comment command</a>.</p>
+
+
+
+
+ [Resolved] Outage
+ https://status.packit.dev/issues/2023-08-15-outage/
+ Tue, 15 Aug 2023 04:00:00 +0200
+ https://status.packit.dev/issues/2023-08-15-outage/
+ 2023-08-15T11:03:00+02:00
+ We are having an outage that started on Tuesday EU early morning. Jobs during this time could get stuck in pending state. This is caused by a failed database schema migration because of a lack of memory during the automatic deployment of the new production images. We are working on the fix and apologize for the inconvenience.
+Update: Migration is completed. You might need to retrigger the jobs (i.e. via comment).
+ <p>We are having an outage that started on Tuesday EU early morning. Jobs during this time could get stuck
+in pending state.
+This is caused by a failed database schema migration because of a lack of memory during the automatic deployment
+of the new production images. We are working on the fix and apologize for the inconvenience.</p>
+<p>Update: Migration is completed. You might need to retrigger the jobs (i.e. via comment).</p>
+
+
+
+
+ [Resolved] Cluster Update
+ https://status.packit.dev/issues/2023-05-03-cluster-update/
+ Wed, 03 May 2023 10:00:00 +0200
+ https://status.packit.dev/issues/2023-05-03-cluster-update/
+ 2023-05-03T11:15:00+02:00
+ The OpenShift cluster is currently updating from 4.11.28 to 4.12.13
+ <p>The OpenShift cluster is currently updating from 4.11.28 to 4.12.13</p>
+
+
+
+
+ [Resolved] Scheduled DB Cleanup
+ https://status.packit.dev/issues/2023-04-26-scheduled-db-cleanup/
+ Wed, 26 Apr 2023 04:40:00 +0000
+ https://status.packit.dev/issues/2023-04-26-scheduled-db-cleanup/
+ 2023-04-26T05:17:00+00:00
+ We are running a DB maintenance in order to clean up old pipelines. We expect API performance to improve as a result of this.
+The maintenance window is planned for an hour, the actual clean up activities will last approximately 20 minutes. Users might experience degraded performance of the service and dashboard during this time.
+ <p>We are running a DB maintenance in order to clean up old pipelines. We expect
+API performance to improve as a result of this.</p>
+<p>The maintenance window is planned for an hour, the actual clean up activities
+will last approximately 20 minutes. Users might experience degraded
+performance of the service and dashboard during this time.</p>
+
+
+
+
+ [Resolved] SLO1 Error Budget issues
+ https://status.packit.dev/issues/2023-03-09-slo1-error-budget/
+ Thu, 09 Mar 2023 10:00:00 +0100
+ https://status.packit.dev/issues/2023-03-09-slo1-error-budget/
+ 2023-06-08T10:00:00+01:00
+ We are having SLO1 error budget issues.
+These are mostly caused by performance issues, we are working on them and prioritizing more improvements (see https://github.com/packit/packit-service/issues/1954 ).
+We are sorry for any slowness you can experience.
+ <p>We are having <a href="https://packit.dev/docs/service-level-objectives/">SLO1</a> error budget issues.</p>
+<p>These are mostly caused by performance issues, we are working on them and prioritizing more improvements (see <a href="https://github.com/packit/packit-service/issues/1954">https://github.com/packit/packit-service/issues/1954</a> ).</p>
+<p>We are sorry for any slowness you can experience.</p>
+
+
+
+
+ [Resolved] Running grouping migrations
+ https://status.packit.dev/issues/2023-02-14-running-grouping-migrations/
+ Tue, 14 Feb 2023 02:00:00 +0100
+ https://status.packit.dev/issues/2023-02-14-running-grouping-migrations/
+ 2023-02-14T04:00:00+01:00
+ We are updating our database schema to facilitate further development. As a part of this process, multiple database migrations will have to be run. The first step is grouping together related builds and test runs. The migrations will happen during the next deployment to production, on Tuesday February 14 at 2:00AM CET. This set of migrations is expected to take roughly 40 minutes, after which the service should come back up.
+ <p>We are updating our database schema to facilitate further development.
+As a part of this process, multiple database migrations will have to
+be run. The first step is grouping together related builds and test
+runs. The migrations will happen during the next deployment to production,
+on Tuesday February 14 at 2:00AM CET. This set of migrations is expected to
+take roughly 40 minutes, after which the service should come back up.</p>
+<p>Since significant refactoring was paired with these changes, there may
+be some bugs/regressions that we did not spot on staging. Please,
+contact us via the usual means if you spot anything suspicious.</p>
+
+
+
+
+ [Resolved] Outage
+ https://status.packit.dev/issues/2022-11-25-outage/
+ Thu, 24 Nov 2022 23:24:00 +0100
+ https://status.packit.dev/issues/2022-11-25-outage/
+ 2022-11-25T08:35:00+01:00
+ We are having resource issues in our cluster and particular parts of the deployment can be down.
+Updates:
+ 2022-11-25T08:35:00+01:00: Stage instance has been scaled down so our production instance has more resources. Production instance should be fine now. We are investigating more.
+ <p>We are having resource issues in our cluster and particular parts of the deployment can be down.</p>
+<p>Updates:</p>
+<ul>
+<li><code>2022-11-25T08:35:00+01:00</code>: Stage instance has been scaled down so our production instance has more resources.
+Production instance should be fine now. We are investigating more.</li>
+</ul>
+
+
+
+
+ [Resolved] GitHub Webhooks not working
+ https://status.packit.dev/issues/2022-10-05-github-webhooks-degraded-status/
+ Wed, 05 Oct 2022 16:00:00 +0200
+ https://status.packit.dev/issues/2022-10-05-github-webhooks-degraded-status/
+ 2022-10-05T18:00:00+02:00
+ Packit is not working on GitHub because the GitHub Webhooks status is ‘degraded’. See yourself: https://www.githubstatus.com/incidents/smn1qtqvbsb6
+ <p>Packit is not working on GitHub because the GitHub Webhooks status is ‘degraded’.
+See yourself: <a href="https://www.githubstatus.com/incidents/smn1qtqvbsb6">https://www.githubstatus.com/incidents/smn1qtqvbsb6</a></p>
+
+
+
+
+ [Resolved] Moving production Packit Service to a new cluster
+ https://status.packit.dev/issues/2021-11-16-auto-prod/
+ Thu, 18 Nov 2021 01:00:00 +0000
+ https://status.packit.dev/issues/2021-11-16-auto-prod/
+ 2021-11-18 02:33:00
+ We are going to move deployment of our production Packit Service to a new cluster. It should take about 1/2 an hour, during which the workers will be down and queries to API & dashboard won’t work properly because the database will be being migrated. Any requests submitted to the API meanwhile should, however, be fulfilled once the workers are up again.
+ <p>We are going to move deployment of our production Packit Service to a new cluster.
+It should take about 1/2 an hour, during which the workers will be down and queries
+to API & dashboard won’t work properly because the database will be being migrated.
+Any requests submitted to the API meanwhile should, however, be fulfilled once the workers are
+up again.</p>
+
+
+
+
+ [Resolved] Dashboard - unable to connect
+ https://status.packit.dev/issues/2021-06-02-dashboard-unable-to-connect/
+ Wed, 02 Jun 2021 11:16:00 +0000
+ https://status.packit.dev/issues/2021-06-02-dashboard-unable-to-connect/
+ 2021-06-02 11:40:00
+ API was down and therefore Dashboard was not showing any Jobs and after some time a “unable to connect …” error was shown.
+Cause: Unknown
+Fix: After restarting the API, the Dashboard started to work OK.
+ <p><strong>API</strong> was down and therefore <strong>Dashboard</strong> was not showing any Jobs
+and after some time a “unable to connect …” error was shown.</p>
+<p>Cause:
+Unknown</p>
+<p>Fix:
+After restarting the API, the Dashboard started to work OK.</p>
+
+
+
+
+
diff --git a/affected/api/page/1/index.html b/affected/api/page/1/index.html
new file mode 100644
index 00000000..84bf6a53
--- /dev/null
+++ b/affected/api/page/1/index.html
@@ -0,0 +1,10 @@
+
+
+
+ https://status.packit.dev/affected/api/
+
+
+
+
+
+
diff --git a/affected/api/page/2/index.html b/affected/api/page/2/index.html
new file mode 100644
index 00000000..d44890b2
--- /dev/null
+++ b/affected/api/page/2/index.html
@@ -0,0 +1,731 @@
+
+
+
+
+
+
+
+
+
+ API | Packit Service Status
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
diff --git a/affected/bodhi-updates/index.json b/affected/bodhi-updates/index.json
new file mode 100644
index 00000000..10885b0c
--- /dev/null
+++ b/affected/bodhi-updates/index.json
@@ -0,0 +1,27 @@
+{
+ "is": "system",
+ "title": "Bodhi updates",
+ "permalink": "https://status.packit.dev/affected/bodhi-updates/",
+ "status": "ok",
+ "pages": [
+ {
+ "is": "issue",
+ "title": "Fedora integration outage",
+ "createdAt": "2024-05-27 14:20:00 +0200 +0200",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2024-05-27-fedora-outage/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2024-05-27T18:35:00+02:00",
+ "affected": ["Pull from upstream", "Downstream Koji builds", "Bodhi updates"],
+ "filename": "2024-05-27-fedora-outage.md"
+ }
+
+ ]
+}
+
+
+
+
+
diff --git a/affected/bodhi-updates/index.xml b/affected/bodhi-updates/index.xml
new file mode 100644
index 00000000..c6a043c7
--- /dev/null
+++ b/affected/bodhi-updates/index.xml
@@ -0,0 +1,46 @@
+
+
+
+
+ Bodhi updates on Packit Service Status
+ https://status.packit.dev/affected/bodhi-updates/
+ Incident history
+ github.com/cstate
+ en
+
+ 2024-05-27T14:20:00+02:00
+ 2024-05-27T14:20:00+02:00
+
+
+
+
+
+
+
+ [Resolved] Fedora integration outage
+ https://status.packit.dev/issues/2024-05-27-fedora-outage/
+ Mon, 27 May 2024 14:20:00 +0200
+ https://status.packit.dev/issues/2024-05-27-fedora-outage/
+ 2024-05-27T18:35:00+02:00
+ One of dependencies for our Fedora Messaging consumer has not been installed during the image build which resulted in an outage of the Fedora downstream integration.
+Fedora integration is disrupted:
+ since 2pm CEST for our stage deployment since 6pm CEST for our production deployment Please trigger the respective jobs manually, if you are affected, once the outage has been resolved.
+We are very sorry for the inconvenience!
+Edit #1: Issue has been resolved for both production and staging deployments by rolling back to the previous image builds.
+ <p>One of dependencies for our Fedora Messaging consumer has not been installed
+during the image build which resulted in an outage of the Fedora downstream
+integration.</p>
+<p>Fedora integration is disrupted:</p>
+<ul>
+<li>since 2pm CEST for our stage deployment</li>
+<li>since 6pm CEST for our production deployment</li>
+</ul>
+<p>Please trigger the respective jobs manually, if you are affected, once the outage has been resolved.</p>
+<p>We are very sorry for the inconvenience!</p>
+<p><em>Edit #1</em>: Issue has been resolved for both production and staging deployments
+by rolling back to the previous image builds.</p>
+
+
+
+
+
diff --git a/affected/bodhi-updates/page/1/index.html b/affected/bodhi-updates/page/1/index.html
new file mode 100644
index 00000000..4428a253
--- /dev/null
+++ b/affected/bodhi-updates/page/1/index.html
@@ -0,0 +1,10 @@
+
+
+
+ https://status.packit.dev/affected/bodhi-updates/
+
+
+
+
+
+
diff --git a/affected/bodhi/index.html b/affected/bodhi/index.html
new file mode 100644
index 00000000..d582a4af
--- /dev/null
+++ b/affected/bodhi/index.html
@@ -0,0 +1,773 @@
+
+
+
+
+
+
+
+
+
+ Bodhi | Packit Service Status
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ Recently, based on average data, it looks like this system has gone down for about
+
+ 3439
+ minutes at a time.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
diff --git a/affected/bodhi/index.json b/affected/bodhi/index.json
new file mode 100644
index 00000000..5aea8f81
--- /dev/null
+++ b/affected/bodhi/index.json
@@ -0,0 +1,55 @@
+{
+ "is": "system",
+ "title": "Bodhi",
+ "permalink": "https://status.packit.dev/affected/bodhi/",
+ "status": "ok",
+ "pages": [
+ {
+ "is": "issue",
+ "title": "Fedora infrastructure authentication problem",
+ "createdAt": "2024-02-26 13:00:00 +0000 UTC",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2024-02-26-fedora-auth-problem/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2024-02-26T13:28:00+00:00",
+ "affected": ["Pull from upstream", "Propose downstream", "Koji", "Bodhi"],
+ "filename": "2024-02-26-fedora-auth-problem.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Fedora infrastructure authentication problem",
+ "createdAt": "2024-01-22 22:00:00 +0100 +0100",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2024-01-23-fedora-auth-problem/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2024-01-23T21:00:00+01:00",
+ "affected": ["Workers", "Koji", "Bodhi"],
+ "filename": "2024-01-23-fedora-auth-problem.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Bodhi Updates not created",
+ "createdAt": "2023-05-04 11:30:00 +0200 +0200",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-05-05-bodhi-updates/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-05-10T16:00:00+02:00",
+ "affected": ["Bodhi"],
+ "filename": "2023-05-05-Bodhi-updates.md"
+ }
+
+ ]
+}
+
+
+
+
+
diff --git a/affected/bodhi/index.xml b/affected/bodhi/index.xml
new file mode 100644
index 00000000..0a352efe
--- /dev/null
+++ b/affected/bodhi/index.xml
@@ -0,0 +1,78 @@
+
+
+
+
+ Bodhi on Packit Service Status
+ https://status.packit.dev/affected/bodhi/
+ Incident history
+ github.com/cstate
+ en
+
+ 2024-02-26T13:00:00+00:00
+ 2024-02-26T13:00:00+00:00
+
+
+
+
+
+
+
+ [Resolved] Fedora infrastructure authentication problem
+ https://status.packit.dev/issues/2024-02-26-fedora-auth-problem/
+ Mon, 26 Feb 2024 13:00:00 +0000
+ https://status.packit.dev/issues/2024-02-26-fedora-auth-problem/
+ 2024-02-26T13:28:00+00:00
+ There is a problem with authentication in Fedora Infrastructure, leading to failures of jobs that interact with it (mostly propose_downstream, pull_from_upstream, koji_build, bodhi_update) as Packit is unable to authenticate. To stay updated on the root cause of this problem, please track this issue.
+The problem should be now resolved. You can retrigger the failed jobs by:
+ /packit propose-downstream comment in any upstream issue /packit pull-from-upstream / /packit koji-build / /packit create-update comment in any dist-git pull-request
+ <p>There is a problem with authentication in Fedora Infrastructure, leading to failures of jobs that interact with it
+(mostly <code>propose_downstream</code>, <code>pull_from_upstream</code>, <code>koji_build</code>, <code>bodhi_update</code>)
+as Packit is unable to authenticate. To stay updated on the root cause of this problem, please
+track <a href="https://pagure.io/fedora-infrastructure/issue/11793">this issue</a>.</p>
+<p>The problem should be now resolved. You can retrigger the failed jobs by:</p>
+<ul>
+<li><code>/packit propose-downstream</code> comment in any upstream issue</li>
+<li><code>/packit pull-from-upstream</code> / <code>/packit koji-build</code> / <code>/packit create-update</code> comment in any dist-git pull-request</li>
+</ul>
+
+
+
+
+ [Resolved] Fedora infrastructure authentication problem
+ https://status.packit.dev/issues/2024-01-23-fedora-auth-problem/
+ Mon, 22 Jan 2024 22:00:00 +0100
+ https://status.packit.dev/issues/2024-01-23-fedora-auth-problem/
+ 2024-01-23T21:00:00+01:00
+ The Fedora account system is currently down, leading to failures of jobs that interact with it (mostly propose_downstream, pull_from_upstream, koji_build, bodhi_update) as Packit is unable to authenticate. To stay updated on the root cause of this problem, please track this issue.
+The problem should be now resolved. You can retrigger the failed jobs by:
+ /packit propose-downstream comment in any upstream issue /packit pull-from-upstream / /packit koji-build / /packit create-update comment in any dist-git pull-request
+ <p>The Fedora account system is currently down, leading to failures of jobs that interact with it
+(mostly <code>propose_downstream</code>, <code>pull_from_upstream</code>, <code>koji_build</code>, <code>bodhi_update</code>)
+as Packit is unable to authenticate. To stay updated on the root cause of this problem, please
+track <a href="https://pagure.io/fedora-infrastructure/issue/11733">this issue</a>.</p>
+<p>The problem should be now resolved. You can retrigger the failed jobs by:</p>
+<ul>
+<li><code>/packit propose-downstream</code> comment in any upstream issue</li>
+<li><code>/packit pull-from-upstream</code> / <code>/packit koji-build</code> / <code>/packit create-update</code> comment in any dist-git pull-request</li>
+</ul>
+
+
+
+
+ [Resolved] Bodhi Updates not created
+ https://status.packit.dev/issues/2023-05-05-bodhi-updates/
+ Thu, 04 May 2023 11:30:00 +0200
+ https://status.packit.dev/issues/2023-05-05-bodhi-updates/
+ 2023-05-10T16:00:00+02:00
+ We’re experiencing issues with creating Bodhi updates due to failing authentication.
+You can try to retrigger an update by a comment in a dist-git pull request:
+/packit create-update
+ <p>We’re experiencing issues with creating Bodhi updates due to failing authentication.</p>
+<p>You can try to retrigger an update by a comment in a dist-git pull request:</p>
+<pre><code>/packit create-update
+</code></pre>
+
+
+
+
+
diff --git a/affected/bodhi/page/1/index.html b/affected/bodhi/page/1/index.html
new file mode 100644
index 00000000..26056d11
--- /dev/null
+++ b/affected/bodhi/page/1/index.html
@@ -0,0 +1,10 @@
+
+
+
+ https://status.packit.dev/affected/bodhi/
+
+
+
+
+
+
diff --git a/affected/copr/index.html b/affected/copr/index.html
new file mode 100644
index 00000000..bcaba81b
--- /dev/null
+++ b/affected/copr/index.html
@@ -0,0 +1,1093 @@
+
+
+
+
+
+
+
+
+
+ Copr | Packit Service Status
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
diff --git a/affected/copr/index.json b/affected/copr/index.json
new file mode 100644
index 00000000..72b63952
--- /dev/null
+++ b/affected/copr/index.json
@@ -0,0 +1,209 @@
+{
+ "is": "system",
+ "title": "Copr",
+ "permalink": "https://status.packit.dev/affected/copr/",
+ "status": "ok",
+ "pages": [
+ {
+ "is": "issue",
+ "title": "Many Copr builds pending",
+ "createdAt": "2024-04-23 14:00:00 +0100 +0100",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2024-04-23-copr-pending-queue/",
+ "severity": "notice",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2024-04-23T21:00:00+01:00",
+ "affected": ["Copr"],
+ "filename": "2024-04-23-copr-pending-queue.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Copr builds stuck in pending or running state",
+ "createdAt": "2023-12-06 12:00:00 +0100 +0100",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-12-06-copr-outage/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-12-06T16:00:00+01:00",
+ "affected": ["Copr"],
+ "filename": "2023-12-06-copr-outage.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Copr Upgrade",
+ "createdAt": "2023-11-28 13:30:00 +0000 UTC",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-11-28-copr-upgrade/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-11-28T16:45:00+00:00",
+ "affected": ["Copr"],
+ "filename": "2023-11-28-copr-upgrade.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "GitLab.com outage",
+ "createdAt": "2023-10-10 04:00:00 +0200 +0200",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-10-10-gitlab-outage/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-10-11T14:00:00+02:00",
+ "affected": ["Copr", "Testing Farm"],
+ "filename": "2023-10-10-gitlab-outage.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "COPR build failures",
+ "createdAt": "2023-09-09 12:00:00 +0000 UTC",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-09-09-copr-outage/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-09-11T15:00:00+00:00",
+ "affected": ["Copr"],
+ "filename": "2023-09-09-copr-outage.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Copr frontend instability",
+ "createdAt": "2023-09-05 15:15:00 +0200 +0200",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-09-05-copr-outage/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-04-05T15:00:00+00:00",
+ "affected": ["Copr"],
+ "filename": "2023-09-05-copr-outage.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Copr Upgrade",
+ "createdAt": "2023-08-16 14:30:00 +0200 +0200",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-08-16-copr-upgrade/",
+ "severity": "down",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-08-16T16:30:00+02:00",
+ "affected": ["Copr"],
+ "filename": "2023-08-16-copr-upgrade.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Upgrade of Copr servers",
+ "createdAt": "2023-05-25 10:00:00 +0000 UTC",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-05-25-copr-upgrade/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-05-25T13:00:00+00:00",
+ "affected": ["Copr"],
+ "filename": "2023-05-25-copr-upgrade.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Upgrade of Copr servers",
+ "createdAt": "2023-04-05 12:00:00 +0000 UTC",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-04-05-copr-outage/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-04-05T15:00:00+00:00",
+ "affected": ["Copr"],
+ "filename": "2023-04-05-copr-outage.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Running grouping migrations",
+ "createdAt": "2023-02-14 02:00:00 +0100 +0100",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-02-14-running-grouping-migrations/",
+ "severity": "down",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-02-14T04:00:00+01:00",
+ "affected": ["API", "Workers", "Dashboard", "Copr", "Testing Farm", "Koji"],
+ "filename": "2023-02-14-running-grouping-migrations.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Upgrade of Copr Servers",
+ "createdAt": "2022-11-28 16:00:00 +0100 +0100",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2022-11-28-copr-outage/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2022-11-28T21:00:00+01:00",
+ "affected": ["Copr"],
+ "filename": "2022-11-28-copr-outage.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Copr storage move",
+ "createdAt": "2022-10-28 21:00:00 +0200 +0200",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2022-10-28-30-copr-storage-move/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2022-10-30T08:00:00+02:00",
+ "affected": ["Copr"],
+ "filename": "2022-10-28-30-copr-storage-move.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Issues with the task execution",
+ "createdAt": "2022-10-11 13:57:00 +0200 +0200",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2022-10-11-outage/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2022-10-11T15:12:00+02:00",
+ "affected": ["Workers", "Copr"],
+ "filename": "2022-10-11-outage.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Upgrade of Copr Servers",
+ "createdAt": "2022-06-22 14:00:00 +0200 +0200",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2022-06-22-upgrade-of-copr-servers/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2022-06-22T16:10:00+02:00",
+ "affected": ["Copr"],
+ "filename": "2022-06-22-upgrade-of-copr-servers.md"
+ }
+
+ ]
+}
+
+
+
+
+
diff --git a/affected/copr/index.xml b/affected/copr/index.xml
new file mode 100644
index 00000000..fb20c278
--- /dev/null
+++ b/affected/copr/index.xml
@@ -0,0 +1,238 @@
+
+
+
+
+ Copr on Packit Service Status
+ https://status.packit.dev/affected/copr/
+ Incident history
+ github.com/cstate
+ en
+
+ 2024-04-23T14:00:00+01:00
+ 2024-04-23T14:00:00+01:00
+
+
+
+
+
+
+
+ [Resolved] Many Copr builds pending
+ https://status.packit.dev/issues/2024-04-23-copr-pending-queue/
+ Tue, 23 Apr 2024 14:00:00 +0100
+ https://status.packit.dev/issues/2024-04-23-copr-pending-queue/
+ 2024-04-23T21:00:00+01:00
+ The Copr backend is under a huge load and many builds are still in a pending state. The Copr builds will take more time than usual to finish. You can monitor the pending queue here: https://copr.fedorainfracloud.org/status/stats/.
+ <p>The Copr backend is under a huge load and many builds are still in a pending state.
+The Copr builds will take more time than usual to finish.
+You can monitor the pending queue here: <a href="https://copr.fedorainfracloud.org/status/stats/">https://copr.fedorainfracloud.org/status/stats/</a>.</p>
+
+
+
+
+ [Resolved] Copr builds stuck in pending or running state
+ https://status.packit.dev/issues/2023-12-06-copr-outage/
+ Wed, 06 Dec 2023 12:00:00 +0100
+ https://status.packit.dev/issues/2023-12-06-copr-outage/
+ 2023-12-06T16:00:00+01:00
+ Copr builds could have been stuck in a pending or running state. The issue has been resolved, it may take a while for the affected Copr builds to finish, but there is no need to retrigger them.
+ <p>Copr builds could have been stuck in a pending or running state.
+The issue has been resolved, it may take a while for the affected Copr builds to finish,
+but there is no need to retrigger them.</p>
+
+
+
+
+ [Resolved] Copr Upgrade
+ https://status.packit.dev/issues/2023-11-28-copr-upgrade/
+ Tue, 28 Nov 2023 13:30:00 +0000
+ https://status.packit.dev/issues/2023-11-28-copr-upgrade/
+ 2023-11-28T16:45:00+00:00
+ Due to an upgrade of the Fedora Copr Servers, Copr builds might have been delayed or failing.
+ <p>Due to an <a href="https://pagure.io/fedora-infrastructure/issue/11648">upgrade</a> of the Fedora
+Copr Servers, Copr builds might have been delayed or failing.</p>
+
+
+
+
+ [Resolved] GitLab.com outage
+ https://status.packit.dev/issues/2023-10-10-gitlab-outage/
+ Tue, 10 Oct 2023 04:00:00 +0200
+ https://status.packit.dev/issues/2023-10-10-gitlab-outage/
+ 2023-10-11T14:00:00+02:00
+ GitLab.com workflows are disrupted due to the issue with detecting the upstream/downstream repository in the current working directory.
+Self-hosted GitLab forges are not affected.
+ <p>GitLab.com workflows are disrupted due to the <a href="https://github.com/packit/packit/issues/2118">issue</a> with detecting the upstream/downstream repository in the current working directory.</p>
+<p>Self-hosted GitLab forges are not affected.</p>
+
+
+
+
+ [Resolved] COPR build failures
+ https://status.packit.dev/issues/2023-09-09-copr-outage/
+ Sat, 09 Sep 2023 12:00:00 +0000
+ https://status.packit.dev/issues/2023-09-09-copr-outage/
+ 2023-09-11T15:00:00+00:00
+ Copr is experiencing issues with builds. We are sorry about that. You can follow the progress in the Copr issue.
+The failure in the COPR logs looks like the following: Machine 5b8689e424334255ab9ace8a58343076 still running. Killing...
+In case you are building in your own COPR project you can go to your project settings and change Mock bootstrap from default to Enable for a quick workaround.
+ <p>Copr is experiencing issues with builds. We are sorry about that. You can follow the progress in the <a href="https://github.com/fedora-copr/copr/issues/2906">Copr issue</a>.</p>
+<p>The failure in the COPR logs looks like the following: <code>Machine 5b8689e424334255ab9ace8a58343076 still running. Killing...</code></p>
+<p>In case you are building in your own COPR project you can go to your project settings and change <code>Mock bootstrap</code> from default to <code>Enable</code> for a quick workaround.</p>
+
+
+
+
+ [Resolved] Copr frontend instability
+ https://status.packit.dev/issues/2023-09-05-copr-outage/
+ Tue, 05 Sep 2023 15:15:00 +0200
+ https://status.packit.dev/issues/2023-09-05-copr-outage/
+ 2023-04-05T15:00:00+00:00
+ Copr is experiencing issues with its frontend which is currently mitigated while Copr team investigates.
+In case some of your builds were not submitted, please retrigger.
+ <p>Copr is experiencing issues with its <a href="https://github.com/fedora-copr/copr/issues/2897">frontend</a>
+which is currently mitigated while Copr team investigates.</p>
+<p>In case some of your builds were not submitted, please retrigger.</p>
+
+
+
+
+ [Resolved] Copr Upgrade
+ https://status.packit.dev/issues/2023-08-16-copr-upgrade/
+ Wed, 16 Aug 2023 14:30:00 +0200
+ https://status.packit.dev/issues/2023-08-16-copr-upgrade/
+ 2023-08-16T16:30:00+02:00
+ There will be a Copr outage starting at
+date --date "2023-08-16 12:30 UTC" which will last approximately 2 hours. The build queue processing will be stopped. During the outage time, we expect a very short downtime for the DNF package storage (less than 10 minutes).
+Reason for outage: We will update the infrastructure machines to the latest packages that are currently being developed.
+Affected Services: https://copr.fedorainfracloud.org/ https://download.copr.fedorainfracloud.org/results/
+Upstream ticket: https://github.com/fedora-copr/copr/issues/2824
+ <p>There will be a Copr outage starting at</p>
+<pre tabindex="0"><code>date --date "2023-08-16 12:30 UTC"
+</code></pre><p>which will last approximately 2 hours. The build queue processing will be stopped. During the outage time, we expect a very short downtime for the DNF package storage (less than 10 minutes).</p>
+<h2 id="reason-for-outage">Reason for outage:</h2>
+<p>We will update the infrastructure machines to the latest packages that are currently being developed.</p>
+<h2 id="affected-services">Affected Services:</h2>
+<p><a href="https://copr.fedorainfracloud.org/">https://copr.fedorainfracloud.org/</a>
+<a href="https://download.copr.fedorainfracloud.org/results/">https://download.copr.fedorainfracloud.org/results/</a></p>
+<h2 id="upstream-ticket">Upstream ticket:</h2>
+<p><a href="https://github.com/fedora-copr/copr/issues/2824">https://github.com/fedora-copr/copr/issues/2824</a></p>
+<p>Please join #fedora-buildsys, #fedora-admin or #fedora-noc on irc.libera.chat
+or add comments to this ticket.</p>
+
+
+
+
+ [Resolved] Upgrade of Copr servers
+ https://status.packit.dev/issues/2023-05-25-copr-upgrade/
+ Thu, 25 May 2023 10:00:00 +0000
+ https://status.packit.dev/issues/2023-05-25-copr-upgrade/
+ 2023-05-25T13:00:00+00:00
+ Due to an upgrade of the Fedora Copr Servers, Copr builds might be delayed or failing. The upgrade should last approximately 3 hours.
+ <p>Due to an <a href="https://pagure.io/fedora-infrastructure/issue/11332">upgrade</a> of the Fedora
+Copr Servers, Copr builds might be delayed or failing. The upgrade should last approximately 3 hours.</p>
+
+
+
+
+ [Resolved] Upgrade of Copr servers
+ https://status.packit.dev/issues/2023-04-05-copr-outage/
+ Wed, 05 Apr 2023 12:00:00 +0000
+ https://status.packit.dev/issues/2023-04-05-copr-outage/
+ 2023-04-05T15:00:00+00:00
+ Due to an upgrade of the Fedora Copr Servers, Copr builds might be delayed or failing. The upgrade should last approximately 3 hours.
+ <p>Due to an <a href="https://pagure.io/fedora-infrastructure/issue/11201">upgrade</a> of the Fedora
+Copr Servers, Copr builds might be delayed or failing. The upgrade should last approximately 3 hours.</p>
+
+
+
+
+ [Resolved] Running grouping migrations
+ https://status.packit.dev/issues/2023-02-14-running-grouping-migrations/
+ Tue, 14 Feb 2023 02:00:00 +0100
+ https://status.packit.dev/issues/2023-02-14-running-grouping-migrations/
+ 2023-02-14T04:00:00+01:00
+ We are updating our database schema to facilitate further development. As a part of this process, multiple database migrations will have to be run. The first step is grouping together related builds and test runs. The migrations will happen during the next deployment to production, on Tuesday February 14 at 2:00AM CET. This set of migrations is expected to take roughly 40 minutes, after which the service should come back up.
+ <p>We are updating our database schema to facilitate further development.
+As a part of this process, multiple database migrations will have to
+be run. The first step is grouping together related builds and test
+runs. The migrations will happen during the next deployment to production,
+on Tuesday February 14 at 2:00AM CET. This set of migrations is expected to
+take roughly 40 minutes, after which the service should come back up.</p>
+<p>Since significant refactoring was paired with these changes, there may
+be some bugs/regressions that we did not spot on staging. Please,
+contact us via the usual means if you spot anything suspicious.</p>
+
+
+
+
+ [Resolved] Upgrade of Copr Servers
+ https://status.packit.dev/issues/2022-11-28-copr-outage/
+ Mon, 28 Nov 2022 16:00:00 +0100
+ https://status.packit.dev/issues/2022-11-28-copr-outage/
+ 2022-11-28T21:00:00+01:00
+ Due to an upgrade of the Fedora Copr Servers, Copr builds might have been failing.
+ <p>Due to an <a href="https://pagure.io/fedora-infrastructure/issue/11011">upgrade</a> of the Fedora
+Copr Servers, Copr builds might have been failing.</p>
+
+
+
+
+ [Resolved] Copr storage move
+ https://status.packit.dev/issues/2022-10-28-30-copr-storage-move/
+ Fri, 28 Oct 2022 21:00:00 +0200
+ https://status.packit.dev/issues/2022-10-28-30-copr-storage-move/
+ 2022-10-30T08:00:00+02:00
+ Copr team is performing a backend storage move over the weekend of Oct 28-30th. Building in Copr will be disrupted or unavailable during this period.
+For more info, please follow the Copr outage announcement.
+Update: the upgrade was successful and Copr is fully operational.
+ <p>Copr team is performing a backend storage move over the weekend of Oct 28-30th.
+Building in Copr will be disrupted or unavailable during this period.</p>
+<p>For more info, please follow the <a href="https://lists.fedoraproject.org/archives/list/copr-devel@lists.fedorahosted.org/thread/TRNW5JVCFIQJDUCND4KZWZVM7XVCYLVL/">Copr outage announcement</a>.</p>
+<p>Update: the upgrade was successful and Copr is fully operational.</p>
+
+
+
+
+ [Resolved] Issues with the task execution
+ https://status.packit.dev/issues/2022-10-11-outage/
+ Tue, 11 Oct 2022 13:57:00 +0200
+ https://status.packit.dev/issues/2022-10-11-outage/
+ 2022-10-11T15:12:00+02:00
+ We are experiencing issues with the execution of the tasks. (The statuses are waiting in the The task was accepted. state.) We are investigating the cause. Sorry about the possible troubles. We will try to fix this as soon as possible.
+Updates:
+ Tue Oct 11 12:24:02 UTC: We are slowly going through the queue. Please, be patient. Tue Oct 11 13:12:36 UTC: We have gone through the queue and you should experience regular response times.
+ <p>We are experiencing issues with the execution of the tasks.
+(The statuses are waiting in the <code>The task was accepted.</code> state.)
+We are investigating the cause.
+Sorry about the possible troubles.
+We will try to fix this as soon as possible.</p>
+<p>Updates:</p>
+<ul>
+<li>Tue Oct 11 12:24:02 UTC: We are slowly going through the queue. Please, be patient.</li>
+<li>Tue Oct 11 13:12:36 UTC: We have gone through the queue and you should experience regular response times.
+The original cause is still not known but the problem probably occured during an SRPM upload to Copr.
+(It can be a fault on our or Copr side.)</li>
+<li>Wed Oct 12 05:27:46 UTC: Service operated normally since the last update.
+To reduce the chance of this, we <a href="https://packit.dev/posts/copr-srpms/">plan to move the building of SRPMs to Copr</a>.
+(Currently default for new installations and opt in for other projects.)</li>
+</ul>
+
+
+
+
+ [Resolved] Upgrade of Copr Servers
+ https://status.packit.dev/issues/2022-06-22-upgrade-of-copr-servers/
+ Wed, 22 Jun 2022 14:00:00 +0200
+ https://status.packit.dev/issues/2022-06-22-upgrade-of-copr-servers/
+ 2022-06-22T16:10:00+02:00
+ Due to an ongoing upgrade of the Fedora Copr Servers, building might be delayed or failing. The upgrade should last ~3 hours.
+ <p>Due to an <a href="https://pagure.io/fedora-infrastructure/issue/10780">ongoing
+upgrade</a> of the Fedora
+Copr Servers, building might be delayed or failing. The upgrade should last ~3
+hours.</p>
+
+
+
+
+
diff --git a/affected/copr/page/1/index.html b/affected/copr/page/1/index.html
new file mode 100644
index 00000000..5428d39d
--- /dev/null
+++ b/affected/copr/page/1/index.html
@@ -0,0 +1,10 @@
+
+
+
+ https://status.packit.dev/affected/copr/
+
+
+
+
+
+
diff --git a/affected/copr/page/2/index.html b/affected/copr/page/2/index.html
new file mode 100644
index 00000000..c4abe8ff
--- /dev/null
+++ b/affected/copr/page/2/index.html
@@ -0,0 +1,854 @@
+
+
+
+
+
+
+
+
+
+ Copr | Packit Service Status
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
diff --git a/affected/dashboard/index.json b/affected/dashboard/index.json
new file mode 100644
index 00000000..1a7701bf
--- /dev/null
+++ b/affected/dashboard/index.json
@@ -0,0 +1,153 @@
+{
+ "is": "system",
+ "title": "Dashboard",
+ "permalink": "https://status.packit.dev/affected/dashboard/",
+ "status": "ok",
+ "pages": [
+ {
+ "is": "issue",
+ "title": "Cluster upgrade",
+ "createdAt": "2023-10-02 08:00:00 +0000 UTC",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-10-02-cluster-upgrade/",
+ "severity": "notice",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-10-02T10:00:00+00:00",
+ "affected": ["API", "Workers", "Dashboard"],
+ "filename": "2023-10-02-cluster-upgrade.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Cluster Update",
+ "createdAt": "2023-05-03 10:00:00 +0200 +0200",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-05-03-cluster-update/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-05-03T11:15:00+02:00",
+ "affected": ["API", "Workers", "Dashboard"],
+ "filename": "2023-05-03-cluster-update.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Scheduled DB Cleanup",
+ "createdAt": "2023-04-26 04:40:00 +0000 UTC",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-04-26-scheduled-db-cleanup/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-04-26T05:17:00+00:00",
+ "affected": ["API", "Dashboard"],
+ "filename": "2023-04-26-scheduled-db-cleanup.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "SLO1 Error Budget issues",
+ "createdAt": "2023-03-09 10:00:00 +0100 +0100",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-03-09-slo1-error-budget/",
+ "severity": "notice",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-06-08T10:00:00+01:00",
+ "affected": ["API", "Workers", "Dashboard"],
+ "filename": "2023-03-09-slo1-error-budget.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Running grouping migrations",
+ "createdAt": "2023-02-14 02:00:00 +0100 +0100",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-02-14-running-grouping-migrations/",
+ "severity": "down",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-02-14T04:00:00+01:00",
+ "affected": ["API", "Workers", "Dashboard", "Copr", "Testing Farm", "Koji"],
+ "filename": "2023-02-14-running-grouping-migrations.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "SLO2 Error Budget Depleted in a Day",
+ "createdAt": "2023-01-31 10:00:00 +0100 +0100",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-02-17-slo2-error-budget-depleted-in-a-day/",
+ "severity": "notice",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-02-01T10:00:00+01:00",
+ "affected": ["Dashboard"],
+ "filename": "2023-02-17-slo2-error-budget-depleted-in-a-day.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Problems with production database",
+ "createdAt": "2021-11-18 02:33:00 +0000 UTC",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2021-11-18-production-db-problems/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2021-11-19 00:10:00",
+ "affected": ["Dashboard"],
+ "filename": "2021-11-18-production-db-problems.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Moving production Packit Service to a new cluster",
+ "createdAt": "2021-11-18 01:00:00 +0000 UTC",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2021-11-16-auto-prod/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2021-11-18 02:33:00",
+ "affected": ["API", "Workers", "Dashboard"],
+ "filename": "2021-11-16-auto-prod.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "PostgreSQL db upgrade",
+ "createdAt": "2021-06-04 07:00:00 +0000 UTC",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2021-06-04-postgresql-12/",
+ "severity": "down",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2021-06-04 09:10:40+00:00",
+ "affected": ["Workers", "Dashboard"],
+ "filename": "2021-06-04-postgresql-12.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Dashboard - unable to connect",
+ "createdAt": "2021-06-02 11:16:00 +0000 UTC",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2021-06-02-dashboard-unable-to-connect/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2021-06-02 11:40:00",
+ "affected": ["API", "Dashboard"],
+ "filename": "2021-06-02-dashboard-unable-to-connect.md"
+ }
+
+ ]
+}
+
+
+
+
+
diff --git a/affected/dashboard/index.xml b/affected/dashboard/index.xml
new file mode 100644
index 00000000..e2f03ff8
--- /dev/null
+++ b/affected/dashboard/index.xml
@@ -0,0 +1,174 @@
+
+
+
+
+ Dashboard on Packit Service Status
+ https://status.packit.dev/affected/dashboard/
+ Incident history
+ github.com/cstate
+ en
+
+ 2023-10-02T08:00:00+00:00
+ 2023-10-02T08:00:00+00:00
+
+
+
+
+
+
+
+ [Resolved] Cluster upgrade
+ https://status.packit.dev/issues/2023-10-02-cluster-upgrade/
+ Mon, 02 Oct 2023 08:00:00 +0000
+ https://status.packit.dev/issues/2023-10-02-cluster-upgrade/
+ 2023-10-02T10:00:00+00:00
+ A cluster upgrade is scheduled to start on Mon October 2nd at 08:00 UTC and last approximately 90 minutes.
+We do not expect any significant disruption.
+The started jobs might be executed with a delay and if you think Packit does not react to your events, try to retrigger the action via comment command.
+ <p>A cluster upgrade is scheduled to start on Mon October 2nd at 08:00 UTC and last approximately 90 minutes.</p>
+<p>We do not expect any significant disruption.</p>
+<p>The started jobs might be executed with a delay
+and if you think Packit does not react to your events,
+try to retrigger the action via
+<a href="https://packit.dev/docs/guide/#how-to-re-trigger-packit-actions-in-your-pull-request">comment command</a>.</p>
+
+
+
+
+ [Resolved] Cluster Update
+ https://status.packit.dev/issues/2023-05-03-cluster-update/
+ Wed, 03 May 2023 10:00:00 +0200
+ https://status.packit.dev/issues/2023-05-03-cluster-update/
+ 2023-05-03T11:15:00+02:00
+ The OpenShift cluster is currently updating from 4.11.28 to 4.12.13
+ <p>The OpenShift cluster is currently updating from 4.11.28 to 4.12.13</p>
+
+
+
+
+ [Resolved] Scheduled DB Cleanup
+ https://status.packit.dev/issues/2023-04-26-scheduled-db-cleanup/
+ Wed, 26 Apr 2023 04:40:00 +0000
+ https://status.packit.dev/issues/2023-04-26-scheduled-db-cleanup/
+ 2023-04-26T05:17:00+00:00
+ We are running a DB maintenance in order to clean up old pipelines. We expect API performance to improve as a result of this.
+The maintenance window is planned for an hour, the actual clean up activities will last approximately 20 minutes. Users might experience degraded performance of the service and dashboard during this time.
+ <p>We are running a DB maintenance in order to clean up old pipelines. We expect
+API performance to improve as a result of this.</p>
+<p>The maintenance window is planned for an hour, the actual clean up activities
+will last approximately 20 minutes. Users might experience degraded
+performance of the service and dashboard during this time.</p>
+
+
+
+
+ [Resolved] SLO1 Error Budget issues
+ https://status.packit.dev/issues/2023-03-09-slo1-error-budget/
+ Thu, 09 Mar 2023 10:00:00 +0100
+ https://status.packit.dev/issues/2023-03-09-slo1-error-budget/
+ 2023-06-08T10:00:00+01:00
+ We are having SLO1 error budget issues.
+These are mostly caused by performance issues, we are working on them and prioritizing more improvements (see https://github.com/packit/packit-service/issues/1954 ).
+We are sorry for any slowness you can experience.
+ <p>We are having <a href="https://packit.dev/docs/service-level-objectives/">SLO1</a> error budget issues.</p>
+<p>These are mostly caused by performance issues, we are working on them and prioritizing more improvements (see <a href="https://github.com/packit/packit-service/issues/1954">https://github.com/packit/packit-service/issues/1954</a> ).</p>
+<p>We are sorry for any slowness you can experience.</p>
+
+
+
+
+ [Resolved] Running grouping migrations
+ https://status.packit.dev/issues/2023-02-14-running-grouping-migrations/
+ Tue, 14 Feb 2023 02:00:00 +0100
+ https://status.packit.dev/issues/2023-02-14-running-grouping-migrations/
+ 2023-02-14T04:00:00+01:00
+ We are updating our database schema to facilitate further development. As a part of this process, multiple database migrations will have to be run. The first step is grouping together related builds and test runs. The migrations will happen during the next deployment to production, on Tuesday February 14 at 2:00AM CET. This set of migrations is expected to take roughly 40 minutes, after which the service should come back up.
+ <p>We are updating our database schema to facilitate further development.
+As a part of this process, multiple database migrations will have to
+be run. The first step is grouping together related builds and test
+runs. The migrations will happen during the next deployment to production,
+on Tuesday February 14 at 2:00AM CET. This set of migrations is expected to
+take roughly 40 minutes, after which the service should come back up.</p>
+<p>Since significant refactoring was paired with these changes, there may
+be some bugs/regressions that we did not spot on staging. Please,
+contact us via the usual means if you spot anything suspicious.</p>
+
+
+
+
+ [Resolved] SLO2 Error Budget Depleted in a Day
+ https://status.packit.dev/issues/2023-02-17-slo2-error-budget-depleted-in-a-day/
+ Tue, 31 Jan 2023 10:00:00 +0100
+ https://status.packit.dev/issues/2023-02-17-slo2-error-budget-depleted-in-a-day/
+ 2023-02-01T10:00:00+01:00
+ The SLO2 error budget dropped from 85% all the way down to zero in one day. The likely cause was that the fedora-messaging consumer was stuck, so we were not getting messages from Copr about finished SRPM builds. Luckily, the babysit tasks proved to be very useful and caught those later.
+We’ve added a liveness probe to the messaging consumer to prevent this from happening again.
+ <p>The SLO2 error budget dropped from 85% all the way down to zero in one day.
+<img src="https://status.packit.dev/images/SLO2-error-budget-depleted.png" alt="SLO2"></p>
+<p>The likely cause was that the fedora-messaging consumer was stuck, so we
+were not getting messages from Copr about finished SRPM builds.
+Luckily, the babysit tasks proved to be very useful and caught those later.</p>
+<p>We’ve <a href="https://github.com/packit/packit-service-fedmsg/pull/76">added a liveness probe to the messaging consumer</a>
+to prevent this from happening again.</p>
+
+
+
+
+ [Resolved] Problems with production database
+ https://status.packit.dev/issues/2021-11-18-production-db-problems/
+ Thu, 18 Nov 2021 02:33:00 +0000
+ https://status.packit.dev/issues/2021-11-18-production-db-problems/
+ 2021-11-19 00:10:00
+ After moving our production service to a new cluster, we’re seeing intermittent issues with connecting to the database. They are mostly manifesting in the dashboard. We need to decide next steps.
+ <p>After moving our production service to a new cluster, we’re seeing intermittent issues
+with connecting to the database. They are mostly manifesting in the dashboard.
+We need to decide next steps.</p>
+
+
+
+
+ [Resolved] Moving production Packit Service to a new cluster
+ https://status.packit.dev/issues/2021-11-16-auto-prod/
+ Thu, 18 Nov 2021 01:00:00 +0000
+ https://status.packit.dev/issues/2021-11-16-auto-prod/
+ 2021-11-18 02:33:00
+ We are going to move deployment of our production Packit Service to a new cluster. It should take about 1/2 an hour, during which the workers will be down and queries to API & dashboard won’t work properly because the database will be being migrated. Any requests submitted to the API meanwhile should, however, be fulfilled once the workers are up again.
+ <p>We are going to move deployment of our production Packit Service to a new cluster.
+It should take about 1/2 an hour, during which the workers will be down and queries
+to API & dashboard won’t work properly because the database will be being migrated.
+Any requests submitted to the API meanwhile should, however, be fulfilled once the workers are
+up again.</p>
+
+
+
+
+ [Resolved] PostgreSQL db upgrade
+ https://status.packit.dev/issues/2021-06-04-postgresql-12/
+ Fri, 04 Jun 2021 07:00:00 +0000
+ https://status.packit.dev/issues/2021-06-04-postgresql-12/
+ 2021-06-04 09:10:40+00:00
+ Upgrading our PostgreSQL database from version 10 to version 12.
+ <p>Upgrading our PostgreSQL database from version 10 to version 12.</p>
+
+
+
+
+ [Resolved] Dashboard - unable to connect
+ https://status.packit.dev/issues/2021-06-02-dashboard-unable-to-connect/
+ Wed, 02 Jun 2021 11:16:00 +0000
+ https://status.packit.dev/issues/2021-06-02-dashboard-unable-to-connect/
+ 2021-06-02 11:40:00
+ API was down and therefore Dashboard was not showing any Jobs and after some time a “unable to connect …” error was shown.
+Cause: Unknown
+Fix: After restarting the API, the Dashboard started to work OK.
+ <p><strong>API</strong> was down and therefore <strong>Dashboard</strong> was not showing any Jobs
+and after some time a “unable to connect …” error was shown.</p>
+<p>Cause:
+Unknown</p>
+<p>Fix:
+After restarting the API, the Dashboard started to work OK.</p>
+
+
+
+
+
diff --git a/affected/dashboard/page/1/index.html b/affected/dashboard/page/1/index.html
new file mode 100644
index 00000000..e0510fb8
--- /dev/null
+++ b/affected/dashboard/page/1/index.html
@@ -0,0 +1,10 @@
+
+
+
+ https://status.packit.dev/affected/dashboard/
+
+
+
+
+
+
diff --git a/affected/downstream-koji-builds/index.html b/affected/downstream-koji-builds/index.html
new file mode 100644
index 00000000..6cc8c3c9
--- /dev/null
+++ b/affected/downstream-koji-builds/index.html
@@ -0,0 +1,721 @@
+
+
+
+
+
+
+
+
+
+ Downstream Koji builds | Packit Service Status
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ Recently, based on average data, it looks like this system has gone down for about
+
+ 3182
+ minutes at a time.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
diff --git a/affected/downstream-koji-builds/index.json b/affected/downstream-koji-builds/index.json
new file mode 100644
index 00000000..80196855
--- /dev/null
+++ b/affected/downstream-koji-builds/index.json
@@ -0,0 +1,41 @@
+{
+ "is": "system",
+ "title": "Downstream Koji builds",
+ "permalink": "https://status.packit.dev/affected/downstream-koji-builds/",
+ "status": "ok",
+ "pages": [
+ {
+ "is": "issue",
+ "title": "Fedora integration outage",
+ "createdAt": "2024-05-27 14:20:00 +0200 +0200",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2024-05-27-fedora-outage/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2024-05-27T18:35:00+02:00",
+ "affected": ["Pull from upstream", "Downstream Koji builds", "Bodhi updates"],
+ "filename": "2024-05-27-fedora-outage.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Outage of automatic triggering of downstream Koji builds",
+ "createdAt": "2024-01-11 09:30:00 +0100 +0100",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2024-01-11-downstream-koji-builds-outage/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2024-01-15T15:20:00+01:00",
+ "affected": ["Downstream Koji builds"],
+ "filename": "2024-01-11-downstream-koji-builds-outage.md"
+ }
+
+ ]
+}
+
+
+
+
+
diff --git a/affected/downstream-koji-builds/index.xml b/affected/downstream-koji-builds/index.xml
new file mode 100644
index 00000000..63a9fb82
--- /dev/null
+++ b/affected/downstream-koji-builds/index.xml
@@ -0,0 +1,59 @@
+
+
+
+
+ Downstream Koji builds on Packit Service Status
+ https://status.packit.dev/affected/downstream-koji-builds/
+ Incident history
+ github.com/cstate
+ en
+
+ 2024-05-27T14:20:00+02:00
+ 2024-05-27T14:20:00+02:00
+
+
+
+
+
+
+
+ [Resolved] Fedora integration outage
+ https://status.packit.dev/issues/2024-05-27-fedora-outage/
+ Mon, 27 May 2024 14:20:00 +0200
+ https://status.packit.dev/issues/2024-05-27-fedora-outage/
+ 2024-05-27T18:35:00+02:00
+ One of dependencies for our Fedora Messaging consumer has not been installed during the image build which resulted in an outage of the Fedora downstream integration.
+Fedora integration is disrupted:
+ since 2pm CEST for our stage deployment since 6pm CEST for our production deployment Please trigger the respective jobs manually, if you are affected, once the outage has been resolved.
+We are very sorry for the inconvenience!
+Edit #1: Issue has been resolved for both production and staging deployments by rolling back to the previous image builds.
+ <p>One of dependencies for our Fedora Messaging consumer has not been installed
+during the image build which resulted in an outage of the Fedora downstream
+integration.</p>
+<p>Fedora integration is disrupted:</p>
+<ul>
+<li>since 2pm CEST for our stage deployment</li>
+<li>since 6pm CEST for our production deployment</li>
+</ul>
+<p>Please trigger the respective jobs manually, if you are affected, once the outage has been resolved.</p>
+<p>We are very sorry for the inconvenience!</p>
+<p><em>Edit #1</em>: Issue has been resolved for both production and staging deployments
+by rolling back to the previous image builds.</p>
+
+
+
+
+ [Resolved] Outage of automatic triggering of downstream Koji builds
+ https://status.packit.dev/issues/2024-01-11-downstream-koji-builds-outage/
+ Thu, 11 Jan 2024 09:30:00 +0100
+ https://status.packit.dev/issues/2024-01-11-downstream-koji-builds-outage/
+ 2024-01-15T15:20:00+01:00
+ Since 2024-01-11 until 2024-01-15 downstream Koji builds weren’t automatically triggered due to Fedora Infrastructure changes. Please trigger your builds manually if you are affected.
+ <p>Since 2024-01-11 until 2024-01-15 downstream Koji builds weren’t automatically triggered
+due to Fedora Infrastructure changes. Please <a href="https://packit.dev/docs/fedora-releases-guide#retriggering-2">trigger</a>
+your builds manually if you are affected.</p>
+
+
+
+
+
diff --git a/affected/downstream-koji-builds/page/1/index.html b/affected/downstream-koji-builds/page/1/index.html
new file mode 100644
index 00000000..22d11de7
--- /dev/null
+++ b/affected/downstream-koji-builds/page/1/index.html
@@ -0,0 +1,10 @@
+
+
+
+ https://status.packit.dev/affected/downstream-koji-builds/
+
+
+
+
+
+
diff --git a/affected/github/index.html b/affected/github/index.html
new file mode 100644
index 00000000..2c90bc85
--- /dev/null
+++ b/affected/github/index.html
@@ -0,0 +1,865 @@
+
+
+
+
+
+
+
+
+
+ GitHub | Packit Service Status
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
diff --git a/affected/koji-builds/index.json b/affected/koji-builds/index.json
new file mode 100644
index 00000000..24ee39af
--- /dev/null
+++ b/affected/koji-builds/index.json
@@ -0,0 +1,27 @@
+{
+ "is": "system",
+ "title": "Koji builds",
+ "permalink": "https://status.packit.dev/affected/koji-builds/",
+ "status": "ok",
+ "pages": [
+ {
+ "is": "issue",
+ "title": "Koji Outage",
+ "createdAt": "2024-02-21 21:00:00 +0000 UTC",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2024-02-21-koji-outage/",
+ "severity": "notice",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2024-02-22T00:00:00+00:00",
+ "affected": ["Koji builds"],
+ "filename": "2024-02-21-koji-outage.md"
+ }
+
+ ]
+}
+
+
+
+
+
diff --git a/affected/koji-builds/index.xml b/affected/koji-builds/index.xml
new file mode 100644
index 00000000..d94e7b81
--- /dev/null
+++ b/affected/koji-builds/index.xml
@@ -0,0 +1,46 @@
+
+
+
+
+ Koji builds on Packit Service Status
+ https://status.packit.dev/affected/koji-builds/
+ Incident history
+ github.com/cstate
+ en
+
+ 2024-02-21T21:00:00+00:00
+ 2024-02-21T21:00:00+00:00
+
+
+
+
+
+
+
+ [Resolved] Koji Outage
+ https://status.packit.dev/issues/2024-02-21-koji-outage/
+ Wed, 21 Feb 2024 21:00:00 +0000
+ https://status.packit.dev/issues/2024-02-21-koji-outage/
+ 2024-02-22T00:00:00+00:00
+ Planned Outage:
+Koji upgrade at 2024-02-21 21:00 UTC
+There will be an outage starting at 2024-02-21 21:00 UTC, which will last approximately 3 hours.
+Reason for outage:
+Koji will be upgraded to 1.34.0, which requires a schema update that touches many rows. Koji team estimate this will take about 45minutes to complete and during that time, Koji service will be completely offline.
+You should avoid to trigger Koji builds in Packit (e.
+ <p>Planned Outage:</p>
+<p>Koji upgrade at 2024-02-21 21:00 UTC</p>
+<p>There will be an outage starting at 2024-02-21 21:00 UTC,
+which will last approximately 3 hours.</p>
+<p>Reason for outage:</p>
+<p>Koji will be upgraded to 1.34.0, which requires a schema update that touches many rows.
+Koji team estimate this will take about 45minutes to complete and during that time,
+Koji service will be completely offline.</p>
+<p>You should avoid to trigger Koji builds in Packit (e.g. merging distgit Packit pull requests).
+You can retrigger the build, if it is missing, commenting a pull request with:</p>
+<p>/packit koji-build</p>
+
+
+
+
+
diff --git a/affected/koji-builds/page/1/index.html b/affected/koji-builds/page/1/index.html
new file mode 100644
index 00000000..57933ddf
--- /dev/null
+++ b/affected/koji-builds/page/1/index.html
@@ -0,0 +1,10 @@
+
+
+
+ https://status.packit.dev/affected/koji-builds/
+
+
+
+
+
+
diff --git a/affected/koji/index.html b/affected/koji/index.html
new file mode 100644
index 00000000..eafa70f1
--- /dev/null
+++ b/affected/koji/index.html
@@ -0,0 +1,773 @@
+
+
+
+
+
+
+
+
+
+ Koji | Packit Service Status
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ Recently, based on average data, it looks like this system has gone down for about
+
+ 509
+ minutes at a time.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
diff --git a/affected/koji/index.json b/affected/koji/index.json
new file mode 100644
index 00000000..80d291e8
--- /dev/null
+++ b/affected/koji/index.json
@@ -0,0 +1,55 @@
+{
+ "is": "system",
+ "title": "Koji",
+ "permalink": "https://status.packit.dev/affected/koji/",
+ "status": "ok",
+ "pages": [
+ {
+ "is": "issue",
+ "title": "Fedora infrastructure authentication problem",
+ "createdAt": "2024-02-26 13:00:00 +0000 UTC",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2024-02-26-fedora-auth-problem/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2024-02-26T13:28:00+00:00",
+ "affected": ["Pull from upstream", "Propose downstream", "Koji", "Bodhi"],
+ "filename": "2024-02-26-fedora-auth-problem.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Fedora infrastructure authentication problem",
+ "createdAt": "2024-01-22 22:00:00 +0100 +0100",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2024-01-23-fedora-auth-problem/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2024-01-23T21:00:00+01:00",
+ "affected": ["Workers", "Koji", "Bodhi"],
+ "filename": "2024-01-23-fedora-auth-problem.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Running grouping migrations",
+ "createdAt": "2023-02-14 02:00:00 +0100 +0100",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-02-14-running-grouping-migrations/",
+ "severity": "down",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-02-14T04:00:00+01:00",
+ "affected": ["API", "Workers", "Dashboard", "Copr", "Testing Farm", "Koji"],
+ "filename": "2023-02-14-running-grouping-migrations.md"
+ }
+
+ ]
+}
+
+
+
+
+
diff --git a/affected/koji/index.xml b/affected/koji/index.xml
new file mode 100644
index 00000000..151f7018
--- /dev/null
+++ b/affected/koji/index.xml
@@ -0,0 +1,81 @@
+
+
+
+
+ Koji on Packit Service Status
+ https://status.packit.dev/affected/koji/
+ Incident history
+ github.com/cstate
+ en
+
+ 2024-02-26T13:00:00+00:00
+ 2024-02-26T13:00:00+00:00
+
+
+
+
+
+
+
+ [Resolved] Fedora infrastructure authentication problem
+ https://status.packit.dev/issues/2024-02-26-fedora-auth-problem/
+ Mon, 26 Feb 2024 13:00:00 +0000
+ https://status.packit.dev/issues/2024-02-26-fedora-auth-problem/
+ 2024-02-26T13:28:00+00:00
+ There is a problem with authentication in Fedora Infrastructure, leading to failures of jobs that interact with it (mostly propose_downstream, pull_from_upstream, koji_build, bodhi_update) as Packit is unable to authenticate. To stay updated on the root cause of this problem, please track this issue.
+The problem should be now resolved. You can retrigger the failed jobs by:
+ /packit propose-downstream comment in any upstream issue /packit pull-from-upstream / /packit koji-build / /packit create-update comment in any dist-git pull-request
+ <p>There is a problem with authentication in Fedora Infrastructure, leading to failures of jobs that interact with it
+(mostly <code>propose_downstream</code>, <code>pull_from_upstream</code>, <code>koji_build</code>, <code>bodhi_update</code>)
+as Packit is unable to authenticate. To stay updated on the root cause of this problem, please
+track <a href="https://pagure.io/fedora-infrastructure/issue/11793">this issue</a>.</p>
+<p>The problem should be now resolved. You can retrigger the failed jobs by:</p>
+<ul>
+<li><code>/packit propose-downstream</code> comment in any upstream issue</li>
+<li><code>/packit pull-from-upstream</code> / <code>/packit koji-build</code> / <code>/packit create-update</code> comment in any dist-git pull-request</li>
+</ul>
+
+
+
+
+ [Resolved] Fedora infrastructure authentication problem
+ https://status.packit.dev/issues/2024-01-23-fedora-auth-problem/
+ Mon, 22 Jan 2024 22:00:00 +0100
+ https://status.packit.dev/issues/2024-01-23-fedora-auth-problem/
+ 2024-01-23T21:00:00+01:00
+ The Fedora account system is currently down, leading to failures of jobs that interact with it (mostly propose_downstream, pull_from_upstream, koji_build, bodhi_update) as Packit is unable to authenticate. To stay updated on the root cause of this problem, please track this issue.
+The problem should be now resolved. You can retrigger the failed jobs by:
+ /packit propose-downstream comment in any upstream issue /packit pull-from-upstream / /packit koji-build / /packit create-update comment in any dist-git pull-request
+ <p>The Fedora account system is currently down, leading to failures of jobs that interact with it
+(mostly <code>propose_downstream</code>, <code>pull_from_upstream</code>, <code>koji_build</code>, <code>bodhi_update</code>)
+as Packit is unable to authenticate. To stay updated on the root cause of this problem, please
+track <a href="https://pagure.io/fedora-infrastructure/issue/11733">this issue</a>.</p>
+<p>The problem should be now resolved. You can retrigger the failed jobs by:</p>
+<ul>
+<li><code>/packit propose-downstream</code> comment in any upstream issue</li>
+<li><code>/packit pull-from-upstream</code> / <code>/packit koji-build</code> / <code>/packit create-update</code> comment in any dist-git pull-request</li>
+</ul>
+
+
+
+
+ [Resolved] Running grouping migrations
+ https://status.packit.dev/issues/2023-02-14-running-grouping-migrations/
+ Tue, 14 Feb 2023 02:00:00 +0100
+ https://status.packit.dev/issues/2023-02-14-running-grouping-migrations/
+ 2023-02-14T04:00:00+01:00
+ We are updating our database schema to facilitate further development. As a part of this process, multiple database migrations will have to be run. The first step is grouping together related builds and test runs. The migrations will happen during the next deployment to production, on Tuesday February 14 at 2:00AM CET. This set of migrations is expected to take roughly 40 minutes, after which the service should come back up.
+ <p>We are updating our database schema to facilitate further development.
+As a part of this process, multiple database migrations will have to
+be run. The first step is grouping together related builds and test
+runs. The migrations will happen during the next deployment to production,
+on Tuesday February 14 at 2:00AM CET. This set of migrations is expected to
+take roughly 40 minutes, after which the service should come back up.</p>
+<p>Since significant refactoring was paired with these changes, there may
+be some bugs/regressions that we did not spot on staging. Please,
+contact us via the usual means if you spot anything suspicious.</p>
+
+
+
+
+
diff --git a/affected/koji/page/1/index.html b/affected/koji/page/1/index.html
new file mode 100644
index 00000000..a22ddcc7
--- /dev/null
+++ b/affected/koji/page/1/index.html
@@ -0,0 +1,10 @@
+
+
+
+ https://status.packit.dev/affected/koji/
+
+
+
+
+
+
diff --git a/affected/page/1/index.html b/affected/page/1/index.html
new file mode 100644
index 00000000..4586b088
--- /dev/null
+++ b/affected/page/1/index.html
@@ -0,0 +1,10 @@
+
+
+
+ https://status.packit.dev/affected/
+
+
+
+
+
+
diff --git a/affected/page/2/index.html b/affected/page/2/index.html
new file mode 100644
index 00000000..88b69cd8
--- /dev/null
+++ b/affected/page/2/index.html
@@ -0,0 +1,650 @@
+
+
+
+
+
+
+
+
+
+ Affected | Packit Service Status
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
diff --git a/affected/propose-downstream/index.json b/affected/propose-downstream/index.json
new file mode 100644
index 00000000..87c562d0
--- /dev/null
+++ b/affected/propose-downstream/index.json
@@ -0,0 +1,27 @@
+{
+ "is": "system",
+ "title": "Propose downstream",
+ "permalink": "https://status.packit.dev/affected/propose-downstream/",
+ "status": "ok",
+ "pages": [
+ {
+ "is": "issue",
+ "title": "Fedora infrastructure authentication problem",
+ "createdAt": "2024-02-26 13:00:00 +0000 UTC",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2024-02-26-fedora-auth-problem/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2024-02-26T13:28:00+00:00",
+ "affected": ["Pull from upstream", "Propose downstream", "Koji", "Bodhi"],
+ "filename": "2024-02-26-fedora-auth-problem.md"
+ }
+
+ ]
+}
+
+
+
+
+
diff --git a/affected/propose-downstream/index.xml b/affected/propose-downstream/index.xml
new file mode 100644
index 00000000..02e5701b
--- /dev/null
+++ b/affected/propose-downstream/index.xml
@@ -0,0 +1,41 @@
+
+
+
+
+ Propose downstream on Packit Service Status
+ https://status.packit.dev/affected/propose-downstream/
+ Incident history
+ github.com/cstate
+ en
+
+ 2024-02-26T13:00:00+00:00
+ 2024-02-26T13:00:00+00:00
+
+
+
+
+
+
+
+ [Resolved] Fedora infrastructure authentication problem
+ https://status.packit.dev/issues/2024-02-26-fedora-auth-problem/
+ Mon, 26 Feb 2024 13:00:00 +0000
+ https://status.packit.dev/issues/2024-02-26-fedora-auth-problem/
+ 2024-02-26T13:28:00+00:00
+ There is a problem with authentication in Fedora Infrastructure, leading to failures of jobs that interact with it (mostly propose_downstream, pull_from_upstream, koji_build, bodhi_update) as Packit is unable to authenticate. To stay updated on the root cause of this problem, please track this issue.
+The problem should be now resolved. You can retrigger the failed jobs by:
+ /packit propose-downstream comment in any upstream issue /packit pull-from-upstream / /packit koji-build / /packit create-update comment in any dist-git pull-request
+ <p>There is a problem with authentication in Fedora Infrastructure, leading to failures of jobs that interact with it
+(mostly <code>propose_downstream</code>, <code>pull_from_upstream</code>, <code>koji_build</code>, <code>bodhi_update</code>)
+as Packit is unable to authenticate. To stay updated on the root cause of this problem, please
+track <a href="https://pagure.io/fedora-infrastructure/issue/11793">this issue</a>.</p>
+<p>The problem should be now resolved. You can retrigger the failed jobs by:</p>
+<ul>
+<li><code>/packit propose-downstream</code> comment in any upstream issue</li>
+<li><code>/packit pull-from-upstream</code> / <code>/packit koji-build</code> / <code>/packit create-update</code> comment in any dist-git pull-request</li>
+</ul>
+
+
+
+
+
diff --git a/affected/propose-downstream/page/1/index.html b/affected/propose-downstream/page/1/index.html
new file mode 100644
index 00000000..b9f169fd
--- /dev/null
+++ b/affected/propose-downstream/page/1/index.html
@@ -0,0 +1,10 @@
+
+
+
+ https://status.packit.dev/affected/propose-downstream/
+
+
+
+
+
+
diff --git a/affected/pull-from-upstream/index.html b/affected/pull-from-upstream/index.html
new file mode 100644
index 00000000..be3bb849
--- /dev/null
+++ b/affected/pull-from-upstream/index.html
@@ -0,0 +1,865 @@
+
+
+
+
+
+
+
+
+
+ Pull from upstream | Packit Service Status
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
diff --git a/affected/pull-from-upstream/index.json b/affected/pull-from-upstream/index.json
new file mode 100644
index 00000000..5260d1c1
--- /dev/null
+++ b/affected/pull-from-upstream/index.json
@@ -0,0 +1,83 @@
+{
+ "is": "system",
+ "title": "Pull from upstream",
+ "permalink": "https://status.packit.dev/affected/pull-from-upstream/",
+ "status": "ok",
+ "pages": [
+ {
+ "is": "issue",
+ "title": "release-monitoring.org outage",
+ "createdAt": "2024-06-26 08:00:00 +0100 +0100",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2024-07-02-release-monitoring.org-outage/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2024-07-03T16:00:00+02:00",
+ "affected": ["Pull from upstream"],
+ "filename": "2024-07-02-release-monitoring.org-outage.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Fedora integration outage",
+ "createdAt": "2024-05-27 14:20:00 +0200 +0200",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2024-05-27-fedora-outage/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2024-05-27T18:35:00+02:00",
+ "affected": ["Pull from upstream", "Downstream Koji builds", "Bodhi updates"],
+ "filename": "2024-05-27-fedora-outage.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "New Hotness outage ",
+ "createdAt": "2024-05-09 03:00:00 +0000 UTC",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2024-05-16-new-hotness-outage/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2024-05-16T14:30:00+02:00",
+ "affected": ["Pull from upstream"],
+ "filename": "2024-05-16-new-hotness-outage.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Fedora infrastructure authentication problem",
+ "createdAt": "2024-02-26 13:00:00 +0000 UTC",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2024-02-26-fedora-auth-problem/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2024-02-26T13:28:00+00:00",
+ "affected": ["Pull from upstream", "Propose downstream", "Koji", "Bodhi"],
+ "filename": "2024-02-26-fedora-auth-problem.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Pull From Upstream Outage",
+ "createdAt": "2024-02-21 00:00:00 +0000 UTC",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2024-02-22-pull-from-upstream-outage/",
+ "severity": "notice",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2024-02-22T14:40:00+00:00",
+ "affected": ["Pull from upstream"],
+ "filename": "2024-02-22-pull-from-upstream-outage.md"
+ }
+
+ ]
+}
+
+
+
+
+
diff --git a/affected/pull-from-upstream/index.xml b/affected/pull-from-upstream/index.xml
new file mode 100644
index 00000000..8db010c3
--- /dev/null
+++ b/affected/pull-from-upstream/index.xml
@@ -0,0 +1,120 @@
+
+
+
+
+ Pull from upstream on Packit Service Status
+ https://status.packit.dev/affected/pull-from-upstream/
+ Incident history
+ github.com/cstate
+ en
+
+ 2024-06-26T08:00:00+01:00
+ 2024-06-26T08:00:00+01:00
+
+
+
+
+
+
+
+ [Resolved] release-monitoring.org outage
+ https://status.packit.dev/issues/2024-07-02-release-monitoring.org-outage/
+ Wed, 26 Jun 2024 08:00:00 +0100
+ https://status.packit.dev/issues/2024-07-02-release-monitoring.org-outage/
+ 2024-07-03T16:00:00+02:00
+ There is a New Hotness outage resulting in no Bugzillas about new releases being filed. As a result, no Packit update dist-git pull requests will be created.
+There is a workaround: edit your release-monitoring.org project and tick the Check latest release on submit checkbox.
+Update: It seems that the issue has been resolved and new release bugzillas are being filed automatically again.
+ <p>There is a New Hotness outage resulting in no Bugzillas about new releases being filed.
+As a result, no Packit update dist-git pull requests will be created.</p>
+<p>There is a workaround: edit your release-monitoring.org project and tick the <code>Check latest release on submit</code> checkbox.</p>
+<p><em>Update</em>: It seems that the issue has been resolved and new release bugzillas are being filed automatically again.</p>
+
+
+
+
+ [Resolved] Fedora integration outage
+ https://status.packit.dev/issues/2024-05-27-fedora-outage/
+ Mon, 27 May 2024 14:20:00 +0200
+ https://status.packit.dev/issues/2024-05-27-fedora-outage/
+ 2024-05-27T18:35:00+02:00
+ One of dependencies for our Fedora Messaging consumer has not been installed during the image build which resulted in an outage of the Fedora downstream integration.
+Fedora integration is disrupted:
+ since 2pm CEST for our stage deployment since 6pm CEST for our production deployment Please trigger the respective jobs manually, if you are affected, once the outage has been resolved.
+We are very sorry for the inconvenience!
+Edit #1: Issue has been resolved for both production and staging deployments by rolling back to the previous image builds.
+ <p>One of dependencies for our Fedora Messaging consumer has not been installed
+during the image build which resulted in an outage of the Fedora downstream
+integration.</p>
+<p>Fedora integration is disrupted:</p>
+<ul>
+<li>since 2pm CEST for our stage deployment</li>
+<li>since 6pm CEST for our production deployment</li>
+</ul>
+<p>Please trigger the respective jobs manually, if you are affected, once the outage has been resolved.</p>
+<p>We are very sorry for the inconvenience!</p>
+<p><em>Edit #1</em>: Issue has been resolved for both production and staging deployments
+by rolling back to the previous image builds.</p>
+
+
+
+
+ [Resolved] New Hotness outage
+ https://status.packit.dev/issues/2024-05-16-new-hotness-outage/
+ Thu, 09 May 2024 03:00:00 +0000
+ https://status.packit.dev/issues/2024-05-16-new-hotness-outage/
+ 2024-05-16T14:30:00+02:00
+ There was a New Hotness outage resulting in no Bugzillas about new release being filed. As a result, no Packit update dist-git pull requests were created.
+Please trigger the release syncs manually if you are affected by commenting in any dist-git pull request:
+/packit pull-from-upstream We are very sorry for the inconvenience!
+ <p>There was a New Hotness outage resulting in no Bugzillas about new release being filed.
+As a result, no Packit update dist-git pull requests were created.</p>
+<p>Please <a href="https://packit.dev/docs/fedora-releases-guide/dist-git-onboarding#pull_from_upstream">trigger</a>
+the release syncs manually if you are affected by commenting in any dist-git pull request:</p>
+<pre><code>/packit pull-from-upstream
+</code></pre>
+<p>We are very sorry for the inconvenience!</p>
+
+
+
+
+ [Resolved] Fedora infrastructure authentication problem
+ https://status.packit.dev/issues/2024-02-26-fedora-auth-problem/
+ Mon, 26 Feb 2024 13:00:00 +0000
+ https://status.packit.dev/issues/2024-02-26-fedora-auth-problem/
+ 2024-02-26T13:28:00+00:00
+ There is a problem with authentication in Fedora Infrastructure, leading to failures of jobs that interact with it (mostly propose_downstream, pull_from_upstream, koji_build, bodhi_update) as Packit is unable to authenticate. To stay updated on the root cause of this problem, please track this issue.
+The problem should be now resolved. You can retrigger the failed jobs by:
+ /packit propose-downstream comment in any upstream issue /packit pull-from-upstream / /packit koji-build / /packit create-update comment in any dist-git pull-request
+ <p>There is a problem with authentication in Fedora Infrastructure, leading to failures of jobs that interact with it
+(mostly <code>propose_downstream</code>, <code>pull_from_upstream</code>, <code>koji_build</code>, <code>bodhi_update</code>)
+as Packit is unable to authenticate. To stay updated on the root cause of this problem, please
+track <a href="https://pagure.io/fedora-infrastructure/issue/11793">this issue</a>.</p>
+<p>The problem should be now resolved. You can retrigger the failed jobs by:</p>
+<ul>
+<li><code>/packit propose-downstream</code> comment in any upstream issue</li>
+<li><code>/packit pull-from-upstream</code> / <code>/packit koji-build</code> / <code>/packit create-update</code> comment in any dist-git pull-request</li>
+</ul>
+
+
+
+
+ [Resolved] Pull From Upstream Outage
+ https://status.packit.dev/issues/2024-02-22-pull-from-upstream-outage/
+ Wed, 21 Feb 2024 00:00:00 +0000
+ https://status.packit.dev/issues/2024-02-22-pull-from-upstream-outage/
+ 2024-02-22T14:40:00+00:00
+ pull_from_upstream jobs were not executed properly at the given time frame because of an expired token.
+Please trigger the release syncs manually if you are affected by commenting in any dist-git pull request:
+/packit pull-from-upstream We are very sorry for the inconvenience!
+ <p>pull_from_upstream jobs were not executed properly at the given time frame because of an expired token.</p>
+<p>Please <a href="https://packit.dev/docs/fedora-releases-guide#retriggering-1">trigger</a>
+the release syncs manually if you are affected by commenting in any dist-git pull request:</p>
+<pre><code>/packit pull-from-upstream
+</code></pre>
+<p>We are very sorry for the inconvenience!</p>
+
+
+
+
+
diff --git a/affected/pull-from-upstream/page/1/index.html b/affected/pull-from-upstream/page/1/index.html
new file mode 100644
index 00000000..23bebbdd
--- /dev/null
+++ b/affected/pull-from-upstream/page/1/index.html
@@ -0,0 +1,10 @@
+
+
+
+ https://status.packit.dev/affected/pull-from-upstream/
+
+
+
+
+
+
diff --git a/affected/testing-farm/index.html b/affected/testing-farm/index.html
new file mode 100644
index 00000000..5a133750
--- /dev/null
+++ b/affected/testing-farm/index.html
@@ -0,0 +1,1099 @@
+
+
+
+
+
+
+
+
+
+ Testing Farm | Packit Service Status
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
diff --git a/affected/testing-farm/index.json b/affected/testing-farm/index.json
new file mode 100644
index 00000000..27c8c66c
--- /dev/null
+++ b/affected/testing-farm/index.json
@@ -0,0 +1,181 @@
+{
+ "is": "system",
+ "title": "Testing Farm",
+ "permalink": "https://status.packit.dev/affected/testing-farm/",
+ "status": "disrupted",
+ "pages": [
+ {
+ "is": "issue",
+ "title": "Testing Farm outage",
+ "createdAt": "2024-07-22 00:16:00 +0000 UTC",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2024-07-22-tf-outage/",
+ "severity": "disrupted",
+ "resolved": false,
+ "informational": false,
+ "resolvedAt": "",
+ "affected": ["Testing Farm"],
+ "filename": "2024-07-22-tf-outage.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Testing Farm outage",
+ "createdAt": "2023-12-10 20:40:00 +0100 +0100",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-12-10-tf-outage/",
+ "severity": "down",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-12-11T12:00:00+01:00",
+ "affected": ["Testing Farm"],
+ "filename": "2023-12-10-tf-outage.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "GitLab.com outage",
+ "createdAt": "2023-10-10 04:00:00 +0200 +0200",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-10-10-gitlab-outage/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-10-11T14:00:00+02:00",
+ "affected": ["Copr", "Testing Farm"],
+ "filename": "2023-10-10-gitlab-outage.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Testing Farm outage",
+ "createdAt": "2023-08-24 15:40:00 +0200 +0200",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-08-24-tf-outage/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-08-24T16:00:00+02:00",
+ "affected": ["Testing Farm"],
+ "filename": "2023-08-24-tf-outage.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Testing Farm testing blocked",
+ "createdAt": "2023-07-19 08:00:00 +0000 UTC",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-07-19-testing-farm-failing-on-rawhide/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-07-19T11:00:00+00:00",
+ "affected": ["Testing Farm"],
+ "filename": "2023-07-19-testing-farm-failing-on-rawhide.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Guest couldn't be provisioned: Artemis resource ended in 'error' state",
+ "createdAt": "2023-07-03 08:00:00 +0200 +0200",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-07-03-testing-provisioning-failed/",
+ "severity": "notice",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-07-03T10:00:00+00:00",
+ "affected": ["Testing Farm"],
+ "filename": "2023-07-03-testing-provisioning-failed.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Fedora Rawhide Testing Blocked on dnf5",
+ "createdAt": "2023-06-22 09:30:00 +0200 +0200",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-06-22-fedora-rawhide-testing-blocked-on-dnf5/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-07-18T20:30:00+02:00",
+ "affected": ["Testing Farm"],
+ "filename": "2023-06-22-fedora-rawhide-testing-blocked-on-dnf5.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Testing Farm errors",
+ "createdAt": "2023-05-10 14:00:00 +0200 +0200",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-05-11-tf-merge-issue/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-05-11T15:00:00+02:00",
+ "affected": ["Testing Farm"],
+ "filename": "2023-05-11-tf-merge-issue.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Running grouping migrations",
+ "createdAt": "2023-02-14 02:00:00 +0100 +0100",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-02-14-running-grouping-migrations/",
+ "severity": "down",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-02-14T04:00:00+01:00",
+ "affected": ["API", "Workers", "Dashboard", "Copr", "Testing Farm", "Koji"],
+ "filename": "2023-02-14-running-grouping-migrations.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Testing Farm outage",
+ "createdAt": "2023-01-24 15:00:00 +0100 +0100",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-01-24-tf-outage/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-01-25T10:00:00+01:00",
+ "affected": ["Testing Farm"],
+ "filename": "2023-01-24-tf-outage.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Internal Testing Farm infrastructure is having an outage",
+ "createdAt": "2022-10-10 15:14:00 +0200 +0200",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2022-10-10-outage/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2022-10-11T21:34:00+02:00",
+ "affected": ["Testing Farm"],
+ "filename": "2022-10-10-outage.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "September 22th outage",
+ "createdAt": "2022-09-22 08:00:00 +0200 +0200",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2022-09-22-outage/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2022-09-23T08:00:00+02:00",
+ "affected": ["Testing Farm"],
+ "filename": "2022-09-22-outage.md"
+ }
+
+ ]
+}
+
+
+
+
+
diff --git a/affected/testing-farm/index.xml b/affected/testing-farm/index.xml
new file mode 100644
index 00000000..adffe9fe
--- /dev/null
+++ b/affected/testing-farm/index.xml
@@ -0,0 +1,195 @@
+
+
+
+
+ Testing Farm on Packit Service Status
+ https://status.packit.dev/affected/testing-farm/
+ Incident history
+ github.com/cstate
+ en
+
+ 2024-07-22T00:16:00+00:00
+ 2024-07-22T00:16:00+00:00
+
+
+
+
+
+
+
+ Testing Farm outage
+ https://status.packit.dev/issues/2024-07-22-tf-outage/
+ Mon, 22 Jul 2024 00:16:00 +0000
+ https://status.packit.dev/issues/2024-07-22-tf-outage/
+
+ Testing Farm infrastructure is having an outage resulting in tests stuck in a queued state. You can watch the status of the outage here: https://status.testing-farm.io/issues/2024-07-22-public-ranch-outage/
+The tests should be executed once the outage is resolved.
+ <p>Testing Farm infrastructure is having an outage resulting in tests stuck in a queued state.
+You can watch the status of the outage here: <a href="https://status.testing-farm.io/issues/2024-07-22-public-ranch-outage/">https://status.testing-farm.io/issues/2024-07-22-public-ranch-outage/</a></p>
+<p>The tests should be executed once the outage is resolved.</p>
+
+
+
+
+ [Resolved] Testing Farm outage
+ https://status.packit.dev/issues/2023-12-10-tf-outage/
+ Sun, 10 Dec 2023 20:40:00 +0100
+ https://status.packit.dev/issues/2023-12-10-tf-outage/
+ 2023-12-11T12:00:00+01:00
+ Testing Farm provisioning is currently failing (due to failing DNS resolution). All runs created on the Testing Farm end with:
+โ Guest couldn't be provisioned: Artemis resource ended in 'error' state In case it affects you, please retrigger the tests via /packit test comment command in the pull requests, once the outage is resolved.
+ <p>Testing Farm provisioning is currently failing (due to failing DNS resolution). All runs created on the Testing Farm end with:</p>
+<pre tabindex="0"><code>โ Guest couldn't be provisioned: Artemis resource ended in 'error' state
+</code></pre><p>In case it affects you, please retrigger the tests via <code>/packit test</code> comment command in the pull requests, <em>once the outage is resolved</em>.</p>
+
+
+
+
+ [Resolved] GitLab.com outage
+ https://status.packit.dev/issues/2023-10-10-gitlab-outage/
+ Tue, 10 Oct 2023 04:00:00 +0200
+ https://status.packit.dev/issues/2023-10-10-gitlab-outage/
+ 2023-10-11T14:00:00+02:00
+ GitLab.com workflows are disrupted due to the issue with detecting the upstream/downstream repository in the current working directory.
+Self-hosted GitLab forges are not affected.
+ <p>GitLab.com workflows are disrupted due to the <a href="https://github.com/packit/packit/issues/2118">issue</a> with detecting the upstream/downstream repository in the current working directory.</p>
+<p>Self-hosted GitLab forges are not affected.</p>
+
+
+
+
+ [Resolved] Testing Farm outage
+ https://status.packit.dev/issues/2023-08-24-tf-outage/
+ Thu, 24 Aug 2023 15:40:00 +0200
+ https://status.packit.dev/issues/2023-08-24-tf-outage/
+ 2023-08-24T16:00:00+02:00
+ Testing Farm infrastructure had a small outage that caused few failing test runs. In case it affected you, please retrigger the tests via /packit test comment command in the pull requests.
+ <p>Testing Farm infrastructure had a small outage that caused few failing test runs.
+In case it affected you, please retrigger the tests via <code>/packit test</code> comment command in the pull requests.</p>
+
+
+
+
+ [Resolved] Testing Farm testing blocked
+ https://status.packit.dev/issues/2023-07-19-testing-farm-failing-on-rawhide/
+ Wed, 19 Jul 2023 08:00:00 +0000
+ https://status.packit.dev/issues/2023-07-19-testing-farm-failing-on-rawhide/
+ 2023-07-19T11:00:00+00:00
+ Testing via Testing Farm is blocked: https://status.testing-farm.io/issues/2023-07-19-public-ranch-outage/.
+Testing Farm team is working on the fix (ETA 2 hours).
+ <p>Testing via Testing Farm is blocked: <a href="https://status.testing-farm.io/issues/2023-07-19-public-ranch-outage/">https://status.testing-farm.io/issues/2023-07-19-public-ranch-outage/</a>.</p>
+<p>Testing Farm team is working on the fix (ETA 2 hours).</p>
+
+
+
+
+ [Resolved] Guest couldn't be provisioned: Artemis resource ended in 'error' state
+ https://status.packit.dev/issues/2023-07-03-testing-provisioning-failed/
+ Mon, 03 Jul 2023 08:00:00 +0200
+ https://status.packit.dev/issues/2023-07-03-testing-provisioning-failed/
+ 2023-07-03T10:00:00+00:00
+ Some jobs submitted by Packit to the Testing Farm are pending with this error: "Guest couldn't be provisioned: Artemis resource ended in 'error' state".
+Testing Farm has already resolved the issue; please re-submit the job with a comment like /packit test (more info on retriggering here) or through the GitHub checks button.
+ <p>Some jobs submitted by Packit to the Testing Farm are pending with this error:
+<code>"Guest couldn't be provisioned: Artemis resource ended in 'error' state"</code>.</p>
+<p>Testing Farm has already resolved the issue; please re-submit the job with a
+comment like <code>/packit test</code> (more info on retriggering
+<a href="https://packit.dev/docs/configuration/upstream/tests/#restart-testing">here</a>)
+or through the GitHub checks button.</p>
+
+
+
+
+ [Resolved] Fedora Rawhide Testing Blocked on dnf5
+ https://status.packit.dev/issues/2023-06-22-fedora-rawhide-testing-blocked-on-dnf5/
+ Thu, 22 Jun 2023 09:30:00 +0200
+ https://status.packit.dev/issues/2023-06-22-fedora-rawhide-testing-blocked-on-dnf5/
+ 2023-07-18T20:30:00+02:00
+ dnf has been upgraded to dnf5 in Rawhide, resulting in various issues when running tests in Testing Farm.
+The Testing Farm team together with dnf5 team is working on fixing the issues and trying to resolve the situation ASAP.
+If you want to help, here are upstream issues related to this problem:
+ https://github.com/rpm-software-management/dnf5/issues/549 https://github.com/rpm-software-management/dnf5/issues/168
+ <p>dnf has been upgraded to dnf5 in Rawhide, resulting in various issues
+when running tests in Testing Farm.</p>
+<p>The Testing Farm team together with dnf5 team is working on fixing the issues
+and trying to resolve the situation ASAP.</p>
+<p>If you want to help, here are upstream issues related to this problem:</p>
+<ul>
+<li><a href="https://github.com/rpm-software-management/dnf5/issues/549">https://github.com/rpm-software-management/dnf5/issues/549</a></li>
+<li><a href="https://github.com/rpm-software-management/dnf5/issues/168">https://github.com/rpm-software-management/dnf5/issues/168</a></li>
+</ul>
+
+
+
+
+ [Resolved] Testing Farm errors
+ https://status.packit.dev/issues/2023-05-11-tf-merge-issue/
+ Wed, 10 May 2023 14:00:00 +0200
+ https://status.packit.dev/issues/2023-05-11-tf-merge-issue/
+ 2023-05-11T15:00:00+02:00
+ After the deployment of the new version of Testing Farm, you may have encountered Testing Farm runs that fail with the status Failed to checkout ref {sha}. Together with the Testing Farm team, we are working on a fix.
+ <p>After the deployment of the new version of Testing Farm, you may have encountered Testing Farm runs that
+fail with the status <code>Failed to checkout ref {sha}</code>. Together with the Testing Farm team, we are working on a fix.</p>
+
+
+
+
+ [Resolved] Running grouping migrations
+ https://status.packit.dev/issues/2023-02-14-running-grouping-migrations/
+ Tue, 14 Feb 2023 02:00:00 +0100
+ https://status.packit.dev/issues/2023-02-14-running-grouping-migrations/
+ 2023-02-14T04:00:00+01:00
+ We are updating our database schema to facilitate further development. As a part of this process, multiple database migrations will have to be run. The first step is grouping together related builds and test runs. The migrations will happen during the next deployment to production, on Tuesday February 14 at 2:00AM CET. This set of migrations is expected to take roughly 40 minutes, after which the service should come back up.
+ <p>We are updating our database schema to facilitate further development.
+As a part of this process, multiple database migrations will have to
+be run. The first step is grouping together related builds and test
+runs. The migrations will happen during the next deployment to production,
+on Tuesday February 14 at 2:00AM CET. This set of migrations is expected to
+take roughly 40 minutes, after which the service should come back up.</p>
+<p>Since significant refactoring was paired with these changes, there may
+be some bugs/regressions that we did not spot on staging. Please,
+contact us via the usual means if you spot anything suspicious.</p>
+
+
+
+
+ [Resolved] Testing Farm outage
+ https://status.packit.dev/issues/2023-01-24-tf-outage/
+ Tue, 24 Jan 2023 15:00:00 +0100
+ https://status.packit.dev/issues/2023-01-24-tf-outage/
+ 2023-01-25T10:00:00+01:00
+ Testing Farm infrastructure had an outage which caused the status of the test runs to be stuck in the queued state. This is now resolved and tests can be retriggered by /packit test comment command in the pull requests.
+ <p>Testing Farm infrastructure had an <a href="https://status.testing-farm.io/issues/2023-01-24-public-ranch-outage/">outage</a>
+which caused the status of the test runs to be stuck in the <code>queued</code> state.
+This is now resolved and tests can be retriggered by <code>/packit test</code> comment command in the pull requests.</p>
+
+
+
+
+ [Resolved] Internal Testing Farm infrastructure is having an outage
+ https://status.packit.dev/issues/2022-10-10-outage/
+ Mon, 10 Oct 2022 15:14:00 +0200
+ https://status.packit.dev/issues/2022-10-10-outage/
+ 2022-10-11T21:34:00+02:00
+ Due to a wide outage in Red Hat internal infrastructure the Red Hat Ranch (=internal ranch) is disabled in the Testing Farm (=our test runner). Currently there is no ETA for resolving the problem. Follow the progress on the Testing Farm status page: https://status.testing-farm.io/issues/2022-10-10-redhat-ranch/
+ <p>Due to a wide outage in Red Hat internal infrastructure the Red Hat Ranch (=internal ranch) is disabled in the Testing Farm (=our test runner).
+Currently there is no ETA for resolving the problem.
+Follow the progress on the Testing Farm status page: <a href="https://status.testing-farm.io/issues/2022-10-10-redhat-ranch/">https://status.testing-farm.io/issues/2022-10-10-redhat-ranch/</a></p>
+
+
+
+
+ [Resolved] September 22th outage
+ https://status.packit.dev/issues/2022-09-22-outage/
+ Thu, 22 Sep 2022 08:00:00 +0200
+ https://status.packit.dev/issues/2022-09-22-outage/
+ 2022-09-23T08:00:00+02:00
+ Testing Farm is failing to run tests submitted by Packit which use Copr builds. Tests with the skip_build flag are not be affected by this issue. We are working together with the Testing Farm team to fix this as soon as possible.
+ <p>Testing Farm is failing to run tests submitted by Packit which use <code>Copr</code> builds.
+Tests with the <code>skip_build</code> flag are not be affected by this issue.
+We are working together with the Testing Farm team to fix this as soon as possible.</p>
+
+
+
+
+
diff --git a/affected/testing-farm/page/1/index.html b/affected/testing-farm/page/1/index.html
new file mode 100644
index 00000000..4fc5ec54
--- /dev/null
+++ b/affected/testing-farm/page/1/index.html
@@ -0,0 +1,10 @@
+
+
+
+ https://status.packit.dev/affected/testing-farm/
+
+
+
+
+
+
diff --git a/affected/testing-farm/page/2/index.html b/affected/testing-farm/page/2/index.html
new file mode 100644
index 00000000..80b571d6
--- /dev/null
+++ b/affected/testing-farm/page/2/index.html
@@ -0,0 +1,772 @@
+
+
+
+
+
+
+
+
+
+ Testing Farm | Packit Service Status
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
diff --git a/affected/workers/index.json b/affected/workers/index.json
new file mode 100644
index 00000000..e6cc3443
--- /dev/null
+++ b/affected/workers/index.json
@@ -0,0 +1,237 @@
+{
+ "is": "system",
+ "title": "Workers",
+ "permalink": "https://status.packit.dev/affected/workers/",
+ "status": "ok",
+ "pages": [
+ {
+ "is": "issue",
+ "title": "Fedora infrastructure authentication problem",
+ "createdAt": "2024-01-22 22:00:00 +0100 +0100",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2024-01-23-fedora-auth-problem/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2024-01-23T21:00:00+01:00",
+ "affected": ["Workers", "Koji", "Bodhi"],
+ "filename": "2024-01-23-fedora-auth-problem.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Cluster upgrade",
+ "createdAt": "2023-10-02 08:00:00 +0000 UTC",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-10-02-cluster-upgrade/",
+ "severity": "notice",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-10-02T10:00:00+00:00",
+ "affected": ["API", "Workers", "Dashboard"],
+ "filename": "2023-10-02-cluster-upgrade.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Outage",
+ "createdAt": "2023-08-15 04:00:00 +0200 +0200",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-08-15-outage/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-08-15T11:03:00+02:00",
+ "affected": ["API", "Workers"],
+ "filename": "2023-08-15-outage.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Cluster Update",
+ "createdAt": "2023-05-03 10:00:00 +0200 +0200",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-05-03-cluster-update/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-05-03T11:15:00+02:00",
+ "affected": ["API", "Workers", "Dashboard"],
+ "filename": "2023-05-03-cluster-update.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Issue with pushing to dist-git",
+ "createdAt": "2023-04-11 18:28:00 +0200 +0200",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-04-12-push-to-distgit-issue/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-04-12T16:30:00+02:00",
+ "affected": ["Workers"],
+ "filename": "2023-04-12-push-to-distgit-issue.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "SLO1 Error Budget issues",
+ "createdAt": "2023-03-09 10:00:00 +0100 +0100",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-03-09-slo1-error-budget/",
+ "severity": "notice",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-06-08T10:00:00+01:00",
+ "affected": ["API", "Workers", "Dashboard"],
+ "filename": "2023-03-09-slo1-error-budget.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Running grouping migrations",
+ "createdAt": "2023-02-14 02:00:00 +0100 +0100",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-02-14-running-grouping-migrations/",
+ "severity": "down",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-02-14T04:00:00+01:00",
+ "affected": ["API", "Workers", "Dashboard", "Copr", "Testing Farm", "Koji"],
+ "filename": "2023-02-14-running-grouping-migrations.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "SRPM builds stuck in Submitted state",
+ "createdAt": "2023-01-20 21:19:06 +0100 +0100",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-01-21-srpm-builds-stuck-in-submitted-state/",
+ "severity": "down",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-01-21T9:51:30+01:00",
+ "affected": ["Workers"],
+ "filename": "2023-01-21-srpm-builds-stuck-in-submitted-state.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Outage",
+ "createdAt": "2022-11-24 23:24:00 +0100 +0100",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2022-11-25-outage/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2022-11-25T08:35:00+01:00",
+ "affected": ["API", "Workers"],
+ "filename": "2022-11-25-outage.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Copr builds and tests on commits and releases ignored",
+ "createdAt": "2022-11-01 03:00:00 +0100 +0100",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2022-11-01-commit-release-trigger/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2022-11-02T10:30:00+01:00",
+ "affected": ["Workers"],
+ "filename": "2022-11-01-commit-release-trigger.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Issues with the task execution",
+ "createdAt": "2022-10-11 13:57:00 +0200 +0200",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2022-10-11-outage/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2022-10-11T15:12:00+02:00",
+ "affected": ["Workers", "Copr"],
+ "filename": "2022-10-11-outage.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Failing SRPM builds done in Copr",
+ "createdAt": "2022-08-23 04:00:00 +0200 +0200",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2022-08-23-srpm-builds-failing/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2022-08-23T09:00:00+02:00",
+ "affected": ["Workers"],
+ "filename": "2022-08-23-srpm-builds-failing.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "SRPM builds done in Copr are failing",
+ "createdAt": "2022-06-21 10:18:22 +0200 +0200",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2022-06-21-srpm-builds-failing/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2022-06-21T11:10:22+02:00",
+ "affected": ["Workers"],
+ "filename": "2022-06-21-srpm-builds-failing.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Longer response times",
+ "createdAt": "2021-12-07 09:06:14 +0100 +0100",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2021-12-07-response-delays/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2021-12-07T11:49:14+01:00",
+ "affected": ["workers"],
+ "filename": "2021-12-07-response-delays.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Moving production Packit Service to a new cluster",
+ "createdAt": "2021-11-18 01:00:00 +0000 UTC",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2021-11-16-auto-prod/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2021-11-18 02:33:00",
+ "affected": ["API", "Workers", "Dashboard"],
+ "filename": "2021-11-16-auto-prod.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "PostgreSQL db upgrade",
+ "createdAt": "2021-06-04 07:00:00 +0000 UTC",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2021-06-04-postgresql-12/",
+ "severity": "down",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2021-06-04 09:10:40+00:00",
+ "affected": ["Workers", "Dashboard"],
+ "filename": "2021-06-04-postgresql-12.md"
+ }
+
+ ]
+}
+
+
+
+
+
diff --git a/affected/workers/index.xml b/affected/workers/index.xml
new file mode 100644
index 00000000..b46aa983
--- /dev/null
+++ b/affected/workers/index.xml
@@ -0,0 +1,275 @@
+
+
+
+
+ Workers on Packit Service Status
+ https://status.packit.dev/affected/workers/
+ Incident history
+ github.com/cstate
+ en
+
+ 2024-01-22T22:00:00+01:00
+ 2024-01-22T22:00:00+01:00
+
+
+
+
+
+
+
+ [Resolved] Fedora infrastructure authentication problem
+ https://status.packit.dev/issues/2024-01-23-fedora-auth-problem/
+ Mon, 22 Jan 2024 22:00:00 +0100
+ https://status.packit.dev/issues/2024-01-23-fedora-auth-problem/
+ 2024-01-23T21:00:00+01:00
+ The Fedora account system is currently down, leading to failures of jobs that interact with it (mostly propose_downstream, pull_from_upstream, koji_build, bodhi_update) as Packit is unable to authenticate. To stay updated on the root cause of this problem, please track this issue.
+The problem should be now resolved. You can retrigger the failed jobs by:
+ /packit propose-downstream comment in any upstream issue /packit pull-from-upstream / /packit koji-build / /packit create-update comment in any dist-git pull-request
+ <p>The Fedora account system is currently down, leading to failures of jobs that interact with it
+(mostly <code>propose_downstream</code>, <code>pull_from_upstream</code>, <code>koji_build</code>, <code>bodhi_update</code>)
+as Packit is unable to authenticate. To stay updated on the root cause of this problem, please
+track <a href="https://pagure.io/fedora-infrastructure/issue/11733">this issue</a>.</p>
+<p>The problem should be now resolved. You can retrigger the failed jobs by:</p>
+<ul>
+<li><code>/packit propose-downstream</code> comment in any upstream issue</li>
+<li><code>/packit pull-from-upstream</code> / <code>/packit koji-build</code> / <code>/packit create-update</code> comment in any dist-git pull-request</li>
+</ul>
+
+
+
+
+ [Resolved] Cluster upgrade
+ https://status.packit.dev/issues/2023-10-02-cluster-upgrade/
+ Mon, 02 Oct 2023 08:00:00 +0000
+ https://status.packit.dev/issues/2023-10-02-cluster-upgrade/
+ 2023-10-02T10:00:00+00:00
+ A cluster upgrade is scheduled to start on Mon October 2nd at 08:00 UTC and last approximately 90 minutes.
+We do not expect any significant disruption.
+The started jobs might be executed with a delay and if you think Packit does not react to your events, try to retrigger the action via comment command.
+ <p>A cluster upgrade is scheduled to start on Mon October 2nd at 08:00 UTC and last approximately 90 minutes.</p>
+<p>We do not expect any significant disruption.</p>
+<p>The started jobs might be executed with a delay
+and if you think Packit does not react to your events,
+try to retrigger the action via
+<a href="https://packit.dev/docs/guide/#how-to-re-trigger-packit-actions-in-your-pull-request">comment command</a>.</p>
+
+
+
+
+ [Resolved] Outage
+ https://status.packit.dev/issues/2023-08-15-outage/
+ Tue, 15 Aug 2023 04:00:00 +0200
+ https://status.packit.dev/issues/2023-08-15-outage/
+ 2023-08-15T11:03:00+02:00
+ We are having an outage that started on Tuesday EU early morning. Jobs during this time could get stuck in pending state. This is caused by a failed database schema migration because of a lack of memory during the automatic deployment of the new production images. We are working on the fix and apologize for the inconvenience.
+Update: Migration is completed. You might need to retrigger the jobs (i.e. via comment).
+ <p>We are having an outage that started on Tuesday EU early morning. Jobs during this time could get stuck
+in pending state.
+This is caused by a failed database schema migration because of a lack of memory during the automatic deployment
+of the new production images. We are working on the fix and apologize for the inconvenience.</p>
+<p>Update: Migration is completed. You might need to retrigger the jobs (i.e. via comment).</p>
+
+
+
+
+ [Resolved] Cluster Update
+ https://status.packit.dev/issues/2023-05-03-cluster-update/
+ Wed, 03 May 2023 10:00:00 +0200
+ https://status.packit.dev/issues/2023-05-03-cluster-update/
+ 2023-05-03T11:15:00+02:00
+ The OpenShift cluster is currently updating from 4.11.28 to 4.12.13
+ <p>The OpenShift cluster is currently updating from 4.11.28 to 4.12.13</p>
+
+
+
+
+ [Resolved] Issue with pushing to dist-git
+ https://status.packit.dev/issues/2023-04-12-push-to-distgit-issue/
+ Tue, 11 Apr 2023 18:28:00 +0200
+ https://status.packit.dev/issues/2023-04-12-push-to-distgit-issue/
+ 2023-04-12T16:30:00+02:00
+ Due to a new pre-push hook being applied in dist-git, we are unable to create pull-requests from fork as part of propose-downstream and pull-from-upstream jobs.
+We are sorry about that and are working on a way how to resolve this.
+ <p>Due to <a href="https://docs.pagure.org/rpkg/releases/1.66.html#checking-a-repo-configuration-before-git-push-with-a-git-hook-script">a new pre-push hook</a>
+being applied in dist-git, we are unable to create pull-requests
+from fork as part of <code>propose-downstream</code> and <code>pull-from-upstream</code> jobs.</p>
+<p>We are sorry about that and are working on a way how to resolve this.</p>
+
+
+
+
+ [Resolved] SLO1 Error Budget issues
+ https://status.packit.dev/issues/2023-03-09-slo1-error-budget/
+ Thu, 09 Mar 2023 10:00:00 +0100
+ https://status.packit.dev/issues/2023-03-09-slo1-error-budget/
+ 2023-06-08T10:00:00+01:00
+ We are having SLO1 error budget issues.
+These are mostly caused by performance issues, we are working on them and prioritizing more improvements (see https://github.com/packit/packit-service/issues/1954 ).
+We are sorry for any slowness you can experience.
+ <p>We are having <a href="https://packit.dev/docs/service-level-objectives/">SLO1</a> error budget issues.</p>
+<p>These are mostly caused by performance issues, we are working on them and prioritizing more improvements (see <a href="https://github.com/packit/packit-service/issues/1954">https://github.com/packit/packit-service/issues/1954</a> ).</p>
+<p>We are sorry for any slowness you can experience.</p>
+
+
+
+
+ [Resolved] Running grouping migrations
+ https://status.packit.dev/issues/2023-02-14-running-grouping-migrations/
+ Tue, 14 Feb 2023 02:00:00 +0100
+ https://status.packit.dev/issues/2023-02-14-running-grouping-migrations/
+ 2023-02-14T04:00:00+01:00
+ We are updating our database schema to facilitate further development. As a part of this process, multiple database migrations will have to be run. The first step is grouping together related builds and test runs. The migrations will happen during the next deployment to production, on Tuesday February 14 at 2:00AM CET. This set of migrations is expected to take roughly 40 minutes, after which the service should come back up.
+ <p>We are updating our database schema to facilitate further development.
+As a part of this process, multiple database migrations will have to
+be run. The first step is grouping together related builds and test
+runs. The migrations will happen during the next deployment to production,
+on Tuesday February 14 at 2:00AM CET. This set of migrations is expected to
+take roughly 40 minutes, after which the service should come back up.</p>
+<p>Since significant refactoring was paired with these changes, there may
+be some bugs/regressions that we did not spot on staging. Please,
+contact us via the usual means if you spot anything suspicious.</p>
+
+
+
+
+ [Resolved] SRPM builds stuck in Submitted state
+ https://status.packit.dev/issues/2023-01-21-srpm-builds-stuck-in-submitted-state/
+ Fri, 20 Jan 2023 21:19:06 +0100
+ https://status.packit.dev/issues/2023-01-21-srpm-builds-stuck-in-submitted-state/
+ 2023-01-21T9:51:30+01:00
+ Workers for processing long-running tasks got stuck on Friday evening with to most visible consequence being the SRPM builds stuck in ‘Submitted’ state.
+The workers were restarted on Sat morning and all SRPMs were sent to Copr to build and the jobs pipelines were processed.
+The cause is not known at the time of resolving the issue.
+ <p>Workers for processing long-running tasks got stuck on Friday evening
+with to most visible consequence being the SRPM builds stuck in ‘Submitted’ state.</p>
+<p>The workers were restarted on Sat morning and all SRPMs were sent to Copr
+to build and the jobs pipelines were processed.</p>
+<p><a href="https://github.com/packit/packit-service/issues/1858">The cause is not known at the time of resolving the issue.</a></p>
+
+
+
+
+ [Resolved] Outage
+ https://status.packit.dev/issues/2022-11-25-outage/
+ Thu, 24 Nov 2022 23:24:00 +0100
+ https://status.packit.dev/issues/2022-11-25-outage/
+ 2022-11-25T08:35:00+01:00
+ We are having resource issues in our cluster and particular parts of the deployment can be down.
+Updates:
+ 2022-11-25T08:35:00+01:00: Stage instance has been scaled down so our production instance has more resources. Production instance should be fine now. We are investigating more.
+ <p>We are having resource issues in our cluster and particular parts of the deployment can be down.</p>
+<p>Updates:</p>
+<ul>
+<li><code>2022-11-25T08:35:00+01:00</code>: Stage instance has been scaled down so our production instance has more resources.
+Production instance should be fine now. We are investigating more.</li>
+</ul>
+
+
+
+
+ [Resolved] Copr builds and tests on commits and releases ignored
+ https://status.packit.dev/issues/2022-11-01-commit-release-trigger/
+ Tue, 01 Nov 2022 03:00:00 +0100
+ https://status.packit.dev/issues/2022-11-01-commit-release-trigger/
+ 2022-11-02T10:30:00+01:00
+ There was a bug introduced with Tuesday’s production update causing Copr builds and tests for commit and release triggers were not executed.
+ <p>There was a bug introduced with Tuesday’s production update causing Copr builds and tests
+for <code>commit</code> and <code>release</code> triggers were not executed.</p>
+
+
+
+
+ [Resolved] Issues with the task execution
+ https://status.packit.dev/issues/2022-10-11-outage/
+ Tue, 11 Oct 2022 13:57:00 +0200
+ https://status.packit.dev/issues/2022-10-11-outage/
+ 2022-10-11T15:12:00+02:00
+ We are experiencing issues with the execution of the tasks. (The statuses are waiting in the The task was accepted. state.) We are investigating the cause. Sorry about the possible troubles. We will try to fix this as soon as possible.
+Updates:
+ Tue Oct 11 12:24:02 UTC: We are slowly going through the queue. Please, be patient. Tue Oct 11 13:12:36 UTC: We have gone through the queue and you should experience regular response times.
+ <p>We are experiencing issues with the execution of the tasks.
+(The statuses are waiting in the <code>The task was accepted.</code> state.)
+We are investigating the cause.
+Sorry about the possible troubles.
+We will try to fix this as soon as possible.</p>
+<p>Updates:</p>
+<ul>
+<li>Tue Oct 11 12:24:02 UTC: We are slowly going through the queue. Please, be patient.</li>
+<li>Tue Oct 11 13:12:36 UTC: We have gone through the queue and you should experience regular response times.
+The original cause is still not known but the problem probably occured during an SRPM upload to Copr.
+(It can be a fault on our or Copr side.)</li>
+<li>Wed Oct 12 05:27:46 UTC: Service operated normally since the last update.
+To reduce the chance of this, we <a href="https://packit.dev/posts/copr-srpms/">plan to move the building of SRPMs to Copr</a>.
+(Currently default for new installations and opt in for other projects.)</li>
+</ul>
+
+
+
+
+ [Resolved] Failing SRPM builds done in Copr
+ https://status.packit.dev/issues/2022-08-23-srpm-builds-failing/
+ Tue, 23 Aug 2022 04:00:00 +0200
+ https://status.packit.dev/issues/2022-08-23-srpm-builds-failing/
+ 2022-08-23T09:00:00+02:00
+ SRPM builds happening in Copr (for packages using the srpm_build_deps config key) were failing due to a dependency issue.
+ <p>SRPM builds happening in Copr (for packages using the <code>srpm_build_deps</code> config
+key) were failing due to a dependency issue.</p>
+
+
+
+
+ [Resolved] SRPM builds done in Copr are failing
+ https://status.packit.dev/issues/2022-06-21-srpm-builds-failing/
+ Tue, 21 Jun 2022 10:18:22 +0200
+ https://status.packit.dev/issues/2022-06-21-srpm-builds-failing/
+ 2022-06-21T11:10:22+02:00
+ SRPM builds happening in Copr (for packages using the srpm_build_deps config key) are failing due to a dependency issue. We are working on a fix.
+ <p>SRPM builds happening in Copr (for packages using the <code>srpm_build_deps</code> config
+key) are failing due to a dependency issue. We are working on a fix.</p>
+
+
+
+
+ [Resolved] Longer response times
+ https://status.packit.dev/issues/2021-12-07-response-delays/
+ Tue, 07 Dec 2021 09:06:14 +0100
+ https://status.packit.dev/issues/2021-12-07-response-delays/
+ 2021-12-07T11:49:14+01:00
+ Our service got overwhelmed by the events coming from a Fedora dist-git (caused by the epel-9 branching) causing the longer responses.
+We have scaled up the workers and the queue is getting shorter. We are planning the next steps to be resilient next time. Sorry about the disturbance.
+update: The queue is now empty and you can expect regular operational times.
+ <p>Our service got overwhelmed by the events coming from a Fedora dist-git (caused by the <code>epel-9</code> branching) causing the longer responses.</p>
+<p>We have scaled up the workers and the queue is getting shorter.
+We are planning the next steps to be resilient next time. Sorry about the disturbance.</p>
+<p>update: The queue is now empty and you can expect regular operational times.</p>
+
+
+
+
+ [Resolved] Moving production Packit Service to a new cluster
+ https://status.packit.dev/issues/2021-11-16-auto-prod/
+ Thu, 18 Nov 2021 01:00:00 +0000
+ https://status.packit.dev/issues/2021-11-16-auto-prod/
+ 2021-11-18 02:33:00
+ We are going to move deployment of our production Packit Service to a new cluster. It should take about 1/2 an hour, during which the workers will be down and queries to API & dashboard won’t work properly because the database will be being migrated. Any requests submitted to the API meanwhile should, however, be fulfilled once the workers are up again.
+ <p>We are going to move deployment of our production Packit Service to a new cluster.
+It should take about 1/2 an hour, during which the workers will be down and queries
+to API & dashboard won’t work properly because the database will be being migrated.
+Any requests submitted to the API meanwhile should, however, be fulfilled once the workers are
+up again.</p>
+
+
+
+
+ [Resolved] PostgreSQL db upgrade
+ https://status.packit.dev/issues/2021-06-04-postgresql-12/
+ Fri, 04 Jun 2021 07:00:00 +0000
+ https://status.packit.dev/issues/2021-06-04-postgresql-12/
+ 2021-06-04 09:10:40+00:00
+ Upgrading our PostgreSQL database from version 10 to version 12.
+ <p>Upgrading our PostgreSQL database from version 10 to version 12.</p>
+
+
+
+
+
diff --git a/affected/workers/page/1/index.html b/affected/workers/page/1/index.html
new file mode 100644
index 00000000..1998ddb7
--- /dev/null
+++ b/affected/workers/page/1/index.html
@@ -0,0 +1,10 @@
+
+
+
+ https://status.packit.dev/affected/workers/
+
+
+
+
+
+
diff --git a/affected/workers/page/2/index.html b/affected/workers/page/2/index.html
new file mode 100644
index 00000000..1870528d
--- /dev/null
+++ b/affected/workers/page/2/index.html
@@ -0,0 +1,936 @@
+
+
+
+
+
+
+
+
+
+ Workers | Packit Service Status
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
diff --git a/index.json b/index.json
new file mode 100644
index 00000000..60bb1808
--- /dev/null
+++ b/index.json
@@ -0,0 +1,149 @@
+{
+ "is": "index",
+ "cStateVersion": "5.5.1",
+ "apiVersion": "2.0",
+ "title": "Packit Service Status",
+ "languageCodeHTML": "en",
+ "languageCode": "en",
+ "baseURL": "https://status.packit.dev",
+ "description": "We continuously monitor the status of our services and if there are any interruptions, a note will be posted here.",
+ "summaryStatus": "disrupted",
+ "categories": [
+ {
+ "name": "Core",
+ "description": "Systems we have complete control over",
+ "hideTitle": false,
+ "closedByDefault": false
+ }
+ ,
+ {
+ "name": "Others",
+ "description": "Systems we use, but have no direct control over",
+ "hideTitle": false,
+ "closedByDefault": false
+ }
+
+ ],
+ "systems": [
+
+ {
+ "name": "API",
+ "description": "prod.packit.dev/api - If API is down, we're not able to receive your requests.",
+ "category": "Core",
+
+ "status": "ok",
+
+ "unresolvedIssues": []
+
+ },
+ {
+ "name": "Workers",
+ "description": "If workers are down, received requests are waiting in a queue to be proceeded.",
+ "category": "Core",
+
+ "status": "ok",
+
+ "unresolvedIssues": []
+
+ },
+ {
+ "name": "Dashboard",
+ "description": "dashboard.packit.dev",
+ "category": "Core",
+
+ "status": "ok",
+
+ "unresolvedIssues": []
+
+ },
+ {
+ "name": "Copr",
+ "description": "copr.fedorainfracloud.org - If it's down, we can't build your packages.",
+ "category": "Others",
+
+ "status": "ok",
+
+ "unresolvedIssues": []
+
+ },
+ {
+ "name": "Testing Farm",
+ "description": "api.dev.testing-farm.io - If it's down, we can't run tests.",
+ "category": "Others",
+
+ "status": "disrupted",
+
+ "unresolvedIssues": [
+ {
+ "is": "issue",
+ "title": "Testing Farm outage",
+ "createdAt": "2024-07-22 00:16:00 +0000 UTC",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2024-07-22-tf-outage/",
+ "severity": "disrupted",
+ "resolved": false,
+ "informational": false,
+ "resolvedAt": "",
+ "affected": ["Testing Farm"],
+ "filename": "2024-07-22-tf-outage.md"
+ }
+
+ ]
+ },
+ {
+ "name": "Koji",
+ "description": "koji.fedoraproject.org",
+ "category": "Others",
+
+ "status": "ok",
+
+ "unresolvedIssues": []
+
+ },
+ {
+ "name": "Bodhi",
+ "description": "bodhi.fedoraproject.org",
+ "category": "Others",
+
+ "status": "ok",
+
+ "unresolvedIssues": []
+
+ }
+ ],
+
+ "tabs": [
+ {
+ "name": "UptimeRobot",
+ "link": ""
+ }
+ ,
+ {
+ "name": "Testing Farm Status",
+ "link": ""
+ }
+ ,
+ {
+ "name": "Packit docs",
+ "link": ""
+ }
+ ,
+ {
+ "name": "Packit dashboard",
+ "link": ""
+ }
+
+ ],
+
+ "buildDate": "2024-07-22",
+ "buildTime": "07:42",
+ "buildTimezone": "UTC",
+ "colorBrand": "#0a0c0f",
+ "colorOk": "#008000",
+ "colorDisrupted": "#cc4400",
+ "colorDown": "#e60000",
+ "colorNotice": "#24478f",
+ "alwaysKeepBrandColor": "true",
+ "logo": "https://status.packit.dev/logo.png",
+ "googleAnalytics": "UA-00000000-1"
+}
diff --git a/index.svg b/index.svg
new file mode 100644
index 00000000..68ba9572
--- /dev/null
+++ b/index.svg
@@ -0,0 +1,19 @@
+
diff --git a/index.xml b/index.xml
new file mode 100644
index 00000000..589e5cfb
--- /dev/null
+++ b/index.xml
@@ -0,0 +1,1133 @@
+
+
+
+
+ Packit Service Status
+ https://status.packit.dev/
+ Incident history
+ github.com/cstate
+ en
+
+ Mon, 22 Jul 2024 00:16:00 +0000
+
+
+
+
+
+
+
+
+ Testing Farm outage
+ https://status.packit.dev/issues/2024-07-22-tf-outage/
+ Mon, 22 Jul 2024 00:16:00 +0000
+ https://status.packit.dev/issues/2024-07-22-tf-outage/
+
+ <p>Testing Farm infrastructure is having an outage resulting in tests stuck in a queued state.
+You can watch the status of the outage here: <a href="https://status.testing-farm.io/issues/2024-07-22-public-ranch-outage/">https://status.testing-farm.io/issues/2024-07-22-public-ranch-outage/</a></p>
+<p>The tests should be executed once the outage is resolved.</p>
+
+
+
+
+ [Resolved] release-monitoring.org outage
+ https://status.packit.dev/issues/2024-07-02-release-monitoring.org-outage/
+ Wed, 26 Jun 2024 08:00:00 +0100
+ https://status.packit.dev/issues/2024-07-02-release-monitoring.org-outage/
+ 2024-07-03T16:00:00+02:00
+ <p>There is a New Hotness outage resulting in no Bugzillas about new releases being filed.
+As a result, no Packit update dist-git pull requests will be created.</p>
+<p>There is a workaround: edit your release-monitoring.org project and tick the <code>Check latest release on submit</code> checkbox.</p>
+<p><em>Update</em>: It seems that the issue has been resolved and new release bugzillas are being filed automatically again.</p>
+
+
+
+
+ [Resolved] Fedora integration outage
+ https://status.packit.dev/issues/2024-05-27-fedora-outage/
+ Mon, 27 May 2024 14:20:00 +0200
+ https://status.packit.dev/issues/2024-05-27-fedora-outage/
+ 2024-05-27T18:35:00+02:00
+ <p>One of dependencies for our Fedora Messaging consumer has not been installed
+during the image build which resulted in an outage of the Fedora downstream
+integration.</p>
+<p>Fedora integration is disrupted:</p>
+<ul>
+<li>since 2pm CEST for our stage deployment</li>
+<li>since 6pm CEST for our production deployment</li>
+</ul>
+<p>Please trigger the respective jobs manually, if you are affected, once the outage has been resolved.</p>
+<p>We are very sorry for the inconvenience!</p>
+<p><em>Edit #1</em>: Issue has been resolved for both production and staging deployments
+by rolling back to the previous image builds.</p>
+
+
+
+
+ [Resolved] New Hotness outage
+ https://status.packit.dev/issues/2024-05-16-new-hotness-outage/
+ Thu, 09 May 2024 03:00:00 +0000
+ https://status.packit.dev/issues/2024-05-16-new-hotness-outage/
+ 2024-05-16T14:30:00+02:00
+ <p>There was a New Hotness outage resulting in no Bugzillas about new release being filed.
+As a result, no Packit update dist-git pull requests were created.</p>
+<p>Please <a href="https://packit.dev/docs/fedora-releases-guide/dist-git-onboarding#pull_from_upstream">trigger</a>
+the release syncs manually if you are affected by commenting in any dist-git pull request:</p>
+<pre><code>/packit pull-from-upstream
+</code></pre>
+<p>We are very sorry for the inconvenience!</p>
+
+
+
+
+ [Resolved] Outage
+ https://status.packit.dev/issues/2024-05-07-outage/
+ Tue, 07 May 2024 13:00:00 +0200
+ https://status.packit.dev/issues/2024-05-07-outage/
+ 2024-05-07T16:00:00+02:00
+ <p>There has been a Packit outage.
+The jobs in the queue are now being picked up again and should be processed gradually
+(you might have experienced a delay in job execution).
+We are very sorry for the inconvenience!</p>
+
+
+
+
+ [Resolved] Outage
+ https://status.packit.dev/issues/2024-04-25-outage/
+ Thu, 25 Apr 2024 14:30:00 +0200
+ https://status.packit.dev/issues/2024-04-25-outage/
+ 2024-04-25T19:30:00+02:00
+ <p>There has been a Packit outage. We are investigating the reason of it.
+The jobs in the queue are now being picked up again and should be processed gradually
+(you might have experienced a delay in job execution).
+We are very sorry for the inconvenience!</p>
+
+
+
+
+ [Resolved] Many Copr builds pending
+ https://status.packit.dev/issues/2024-04-23-copr-pending-queue/
+ Tue, 23 Apr 2024 14:00:00 +0100
+ https://status.packit.dev/issues/2024-04-23-copr-pending-queue/
+ 2024-04-23T21:00:00+01:00
+ <p>The Copr backend is under a huge load and many builds are still in a pending state.
+The Copr builds will take more time than usual to finish.
+You can monitor the pending queue here: <a href="https://copr.fedorainfracloud.org/status/stats/">https://copr.fedorainfracloud.org/status/stats/</a>.</p>
+
+
+
+
+ [Resolved] GitHub outage
+ https://status.packit.dev/issues/2024-04-05-github-outage/
+ Fri, 05 Apr 2024 10:15:00 +0100
+ https://status.packit.dev/issues/2024-04-05-github-outage/
+ 2024-04-05T11:15:00+01:00
+ <p>There is an ongoing GitHub outage.
+You can monitor the status and subscribe for updates here: <a href="https://www.githubstatus.com">https://www.githubstatus.com</a></p>
+
+
+
+
+ [Resolved] Fedora infrastructure authentication problem
+ https://status.packit.dev/issues/2024-02-26-fedora-auth-problem/
+ Mon, 26 Feb 2024 13:00:00 +0000
+ https://status.packit.dev/issues/2024-02-26-fedora-auth-problem/
+ 2024-02-26T13:28:00+00:00
+ <p>There is a problem with authentication in Fedora Infrastructure, leading to failures of jobs that interact with it
+(mostly <code>propose_downstream</code>, <code>pull_from_upstream</code>, <code>koji_build</code>, <code>bodhi_update</code>)
+as Packit is unable to authenticate. To stay updated on the root cause of this problem, please
+track <a href="https://pagure.io/fedora-infrastructure/issue/11793">this issue</a>.</p>
+<p>The problem should be now resolved. You can retrigger the failed jobs by:</p>
+<ul>
+<li><code>/packit propose-downstream</code> comment in any upstream issue</li>
+<li><code>/packit pull-from-upstream</code> / <code>/packit koji-build</code> / <code>/packit create-update</code> comment in any dist-git pull-request</li>
+</ul>
+
+
+
+
+ [Resolved] Koji Outage
+ https://status.packit.dev/issues/2024-02-21-koji-outage/
+ Wed, 21 Feb 2024 21:00:00 +0000
+ https://status.packit.dev/issues/2024-02-21-koji-outage/
+ 2024-02-22T00:00:00+00:00
+ <p>Planned Outage:</p>
+<p>Koji upgrade at 2024-02-21 21:00 UTC</p>
+<p>There will be an outage starting at 2024-02-21 21:00 UTC,
+which will last approximately 3 hours.</p>
+<p>Reason for outage:</p>
+<p>Koji will be upgraded to 1.34.0, which requires a schema update that touches many rows.
+Koji team estimate this will take about 45minutes to complete and during that time,
+Koji service will be completely offline.</p>
+<p>You should avoid to trigger Koji builds in Packit (e.g. merging distgit Packit pull requests).
+You can retrigger the build, if it is missing, commenting a pull request with:</p>
+<p>/packit koji-build</p>
+
+
+
+
+ [Resolved] Pull From Upstream Outage
+ https://status.packit.dev/issues/2024-02-22-pull-from-upstream-outage/
+ Wed, 21 Feb 2024 00:00:00 +0000
+ https://status.packit.dev/issues/2024-02-22-pull-from-upstream-outage/
+ 2024-02-22T14:40:00+00:00
+ <p>pull_from_upstream jobs were not executed properly at the given time frame because of an expired token.</p>
+<p>Please <a href="https://packit.dev/docs/fedora-releases-guide#retriggering-1">trigger</a>
+the release syncs manually if you are affected by commenting in any dist-git pull request:</p>
+<pre><code>/packit pull-from-upstream
+</code></pre>
+<p>We are very sorry for the inconvenience!</p>
+
+
+
+
+ [Resolved] Fedora infrastructure authentication problem
+ https://status.packit.dev/issues/2024-01-23-fedora-auth-problem/
+ Mon, 22 Jan 2024 22:00:00 +0100
+ https://status.packit.dev/issues/2024-01-23-fedora-auth-problem/
+ 2024-01-23T21:00:00+01:00
+ <p>The Fedora account system is currently down, leading to failures of jobs that interact with it
+(mostly <code>propose_downstream</code>, <code>pull_from_upstream</code>, <code>koji_build</code>, <code>bodhi_update</code>)
+as Packit is unable to authenticate. To stay updated on the root cause of this problem, please
+track <a href="https://pagure.io/fedora-infrastructure/issue/11733">this issue</a>.</p>
+<p>The problem should be now resolved. You can retrigger the failed jobs by:</p>
+<ul>
+<li><code>/packit propose-downstream</code> comment in any upstream issue</li>
+<li><code>/packit pull-from-upstream</code> / <code>/packit koji-build</code> / <code>/packit create-update</code> comment in any dist-git pull-request</li>
+</ul>
+
+
+
+
+ [Resolved] Outage of automatic triggering of downstream Koji builds
+ https://status.packit.dev/issues/2024-01-11-downstream-koji-builds-outage/
+ Thu, 11 Jan 2024 09:30:00 +0100
+ https://status.packit.dev/issues/2024-01-11-downstream-koji-builds-outage/
+ 2024-01-15T15:20:00+01:00
+ <p>Since 2024-01-11 until 2024-01-15 downstream Koji builds weren’t automatically triggered
+due to Fedora Infrastructure changes. Please <a href="https://packit.dev/docs/fedora-releases-guide#retriggering-2">trigger</a>
+your builds manually if you are affected.</p>
+
+
+
+
+ [Resolved] GitHub outage
+ https://status.packit.dev/issues/2024-01-09-github-outage/
+ Tue, 09 Jan 2024 14:45:00 +0100
+ https://status.packit.dev/issues/2024-01-09-github-outage/
+ 2024-11-09T15:40:00+01:00
+ <p>There is an ongoing GitHub outage.
+You can monitor the status and subscribe for updates here: <a href="https://www.githubstatus.com">https://www.githubstatus.com</a></p>
+
+
+
+
+ [Resolved] Christmas is here ๐
+ https://status.packit.dev/issues/2023-12-23-christmas-2023/
+ Fri, 22 Dec 2023 19:00:00 +0100
+ https://status.packit.dev/issues/2023-12-23-christmas-2023/
+ 2024-01-02T09:00:00+01:00
+ <p>All the team is on vacation.
+Give us some time to get refreshed and be better prepared to work hard on Packit’s new features and bugs.</p>
+<p>–</p>
+<p>Have a wonderful year 2024 everyone! We’re back.</p>
+
+
+
+
+ [Resolved] GitLab outage
+ https://status.packit.dev/issues/2023-12-19-gitlab-outage/
+ Tue, 19 Dec 2023 05:30:00 +0100
+ https://status.packit.dev/issues/2023-12-19-gitlab-outage/
+ 2023-12-20T12:00:00+01:00
+ <p>Packit production instance has lost access to GitLab API.
+This most likely affects also self-hosted GitLab forges.</p>
+<p>The issue has been resolved and everything should work as expected.</p>
+
+
+
+
+ [Resolved] Production instance firewall
+ https://status.packit.dev/issues/2023-12-10-prod-fw/
+ Sun, 10 Dec 2023 20:45:00 +0100
+ https://status.packit.dev/issues/2023-12-10-prod-fw/
+ 2024-02-15T08:00:00+00:00
+ <p>We have moved our production instance to another OpenShift cluster which may result in some jobs failing.
+This issue should mainly affect jobs that involve syncing to the dist-git, namely:</p>
+<ul>
+<li><code>propose-downstream</code></li>
+<li><code>pull-from-upstream</code></li>
+</ul>
+<p>In case you hit any issues when running <code>propose-downstream</code> or <code>pull-from-upstream</code>, please let us know.</p>
+<h2 id="known-issues">Known Issues</h2>
+<ul>
+<li>none as of now (2023-12-11 17:00 UTC)</li>
+</ul>
+
+
+
+
+ [Resolved] Testing Farm outage
+ https://status.packit.dev/issues/2023-12-10-tf-outage/
+ Sun, 10 Dec 2023 20:40:00 +0100
+ https://status.packit.dev/issues/2023-12-10-tf-outage/
+ 2023-12-11T12:00:00+01:00
+ <p>Testing Farm provisioning is currently failing (due to failing DNS resolution). All runs created on the Testing Farm end with:</p>
+<pre tabindex="0"><code>โ Guest couldn't be provisioned: Artemis resource ended in 'error' state
+</code></pre><p>In case it affects you, please retrigger the tests via <code>/packit test</code> comment command in the pull requests, <em>once the outage is resolved</em>.</p>
+
+
+
+
+ [Resolved] Production move
+ https://status.packit.dev/issues/2023-12-10-mpp-move/
+ Sun, 10 Dec 2023 20:15:00 +0100
+ https://status.packit.dev/issues/2023-12-10-mpp-move/
+ 2023-12-11T18:00:00+01:00
+ <p><em>UPDATE #2</em>: No issues have been noticed so far.</p>
+<p><em>UPDATE</em>: Production is deployed again. We are monitoring the deployment for any potential issues.</p>
+<p>Our production deployment is being moved to a different cluster, you may experience outage for at most two hours.</p>
+<p>In case you experience any issues, please retrigger the jobs manually via comment.</p>
+
+
+
+
+ [Resolved] Copr builds stuck in pending or running state
+ https://status.packit.dev/issues/2023-12-06-copr-outage/
+ Wed, 06 Dec 2023 12:00:00 +0100
+ https://status.packit.dev/issues/2023-12-06-copr-outage/
+ 2023-12-06T16:00:00+01:00
+ <p>Copr builds could have been stuck in a pending or running state.
+The issue has been resolved, it may take a while for the affected Copr builds to finish,
+but there is no need to retrigger them.</p>
+
+
+
+
+ [Resolved] Copr Upgrade
+ https://status.packit.dev/issues/2023-11-28-copr-upgrade/
+ Tue, 28 Nov 2023 13:30:00 +0000
+ https://status.packit.dev/issues/2023-11-28-copr-upgrade/
+ 2023-11-28T16:45:00+00:00
+ <p>Due to an <a href="https://pagure.io/fedora-infrastructure/issue/11648">upgrade</a> of the Fedora
+Copr Servers, Copr builds might have been delayed or failing.</p>
+
+
+
+
+ [Resolved] COPR builds failing for epel8
+ https://status.packit.dev/issues/2023-11-14-copr-outage-for-epel8/
+ Tue, 14 Nov 2023 10:00:00 +0100
+ https://status.packit.dev/issues/2023-11-14-copr-outage-for-epel8/
+ 2023-11-14T17:15:00+01:00
+ <p>Building on COPR for <em>epel8</em> chroot will be broken today for several hours.
+Today is release day of RHEL 8.9 and everyone is seeing 404 not found error when trying to access
+RHEL 8 repositories.</p>
+
+
+
+
+ [Resolved] Production move
+ https://status.packit.dev/issues/2023-11-12-mpp-move/
+ Sun, 12 Nov 2023 15:00:00 +0100
+ https://status.packit.dev/issues/2023-11-12-mpp-move/
+ 2023-11-12T17:15:00+01:00
+ <p>Our production deployment is being moved to different cluster, you may experience outage for at most an hour.</p>
+<p>In case you experience any issues, please retrigger the jobs manually via comment.</p>
+<p><em>UPDATE</em>: Migration has been reverted due to issues with Fedora infra (authentication & message bus).</p>
+
+
+
+
+ [Resolved] GitHub outage
+ https://status.packit.dev/issues/2023-11-07-github-outage/
+ Tue, 07 Nov 2023 14:45:00 +0100
+ https://status.packit.dev/issues/2023-11-07-github-outage/
+ 2023-11-07T15:30:00+01:00
+ <p>There is an ongoing GitHub outage.
+You can monitor the status and subscribe for updates here: <a href="https://www.githubstatus.com">https://www.githubstatus.com</a></p>
+
+
+
+
+ [Resolved] GitLab.com outage
+ https://status.packit.dev/issues/2023-10-10-gitlab-outage/
+ Tue, 10 Oct 2023 04:00:00 +0200
+ https://status.packit.dev/issues/2023-10-10-gitlab-outage/
+ 2023-10-11T14:00:00+02:00
+ <p>GitLab.com workflows are disrupted due to the <a href="https://github.com/packit/packit/issues/2118">issue</a> with detecting the upstream/downstream repository in the current working directory.</p>
+<p>Self-hosted GitLab forges are not affected.</p>
+
+
+
+
+ [Resolved] Cluster upgrade
+ https://status.packit.dev/issues/2023-10-02-cluster-upgrade/
+ Mon, 02 Oct 2023 08:00:00 +0000
+ https://status.packit.dev/issues/2023-10-02-cluster-upgrade/
+ 2023-10-02T10:00:00+00:00
+ <p>A cluster upgrade is scheduled to start on Mon October 2nd at 08:00 UTC and last approximately 90 minutes.</p>
+<p>We do not expect any significant disruption.</p>
+<p>The started jobs might be executed with a delay
+and if you think Packit does not react to your events,
+try to retrigger the action via
+<a href="https://packit.dev/docs/guide/#how-to-re-trigger-packit-actions-in-your-pull-request">comment command</a>.</p>
+
+
+
+
+ [Resolved] COPR build failures
+ https://status.packit.dev/issues/2023-09-09-copr-outage/
+ Sat, 09 Sep 2023 12:00:00 +0000
+ https://status.packit.dev/issues/2023-09-09-copr-outage/
+ 2023-09-11T15:00:00+00:00
+ <p>Copr is experiencing issues with builds. We are sorry about that. You can follow the progress in the <a href="https://github.com/fedora-copr/copr/issues/2906">Copr issue</a>.</p>
+<p>The failure in the COPR logs looks like the following: <code>Machine 5b8689e424334255ab9ace8a58343076 still running. Killing...</code></p>
+<p>In case you are building in your own COPR project you can go to your project settings and change <code>Mock bootstrap</code> from default to <code>Enable</code> for a quick workaround.</p>
+
+
+
+
+ [Resolved] Copr frontend instability
+ https://status.packit.dev/issues/2023-09-05-copr-outage/
+ Tue, 05 Sep 2023 15:15:00 +0200
+ https://status.packit.dev/issues/2023-09-05-copr-outage/
+ 2023-04-05T15:00:00+00:00
+ <p>Copr is experiencing issues with its <a href="https://github.com/fedora-copr/copr/issues/2897">frontend</a>
+which is currently mitigated while Copr team investigates.</p>
+<p>In case some of your builds were not submitted, please retrigger.</p>
+
+
+
+
+ [Resolved] System upgrade scheduled
+ https://status.packit.dev/issues/2023-08-28-cluster-upgrade/
+ Mon, 28 Aug 2023 10:00:00 +0200
+ https://status.packit.dev/issues/2023-08-28-cluster-upgrade/
+ 2023-08-28T11:45:00+02:00
+ <p>A system upgrade was scheduled and lasted approximately 90 minutes.</p>
+<p>The started jobs might’ve been executed with a delay or dropped,
+if you think Packit does not react to your events, try to retrigger
+the action via
+<a href="https://packit.dev/docs/guide/#how-to-re-trigger-packit-actions-in-your-pull-request">comment command</a>.</p>
+
+
+
+
+ [Resolved] Testing Farm outage
+ https://status.packit.dev/issues/2023-08-24-tf-outage/
+ Thu, 24 Aug 2023 15:40:00 +0200
+ https://status.packit.dev/issues/2023-08-24-tf-outage/
+ 2023-08-24T16:00:00+02:00
+ <p>Testing Farm infrastructure had a small outage that caused few failing test runs.
+In case it affected you, please retrigger the tests via <code>/packit test</code> comment command in the pull requests.</p>
+
+
+
+
+ [Resolved] Packit outage now resolved
+ https://status.packit.dev/issues/2023-08-21-packit-outage/
+ Mon, 21 Aug 2023 14:45:00 +0200
+ https://status.packit.dev/issues/2023-08-21-packit-outage/
+ 2023-08-21T15:15:00+02:00
+ <p>There has been a Packit outage now resolved. If your jobs are stuck, please retrigger them.
+We are sorry for the inconvenience.</p>
+
+
+
+
+ [Resolved] Copr Upgrade
+ https://status.packit.dev/issues/2023-08-16-copr-upgrade/
+ Wed, 16 Aug 2023 14:30:00 +0200
+ https://status.packit.dev/issues/2023-08-16-copr-upgrade/
+ 2023-08-16T16:30:00+02:00
+ <p>There will be a Copr outage starting at</p>
+<pre tabindex="0"><code>date --date "2023-08-16 12:30 UTC"
+</code></pre><p>which will last approximately 2 hours. The build queue processing will be stopped. During the outage time, we expect a very short downtime for the DNF package storage (less than 10 minutes).</p>
+<h2 id="reason-for-outage">Reason for outage:</h2>
+<p>We will update the infrastructure machines to the latest packages that are currently being developed.</p>
+<h2 id="affected-services">Affected Services:</h2>
+<p><a href="https://copr.fedorainfracloud.org/">https://copr.fedorainfracloud.org/</a>
+<a href="https://download.copr.fedorainfracloud.org/results/">https://download.copr.fedorainfracloud.org/results/</a></p>
+<h2 id="upstream-ticket">Upstream ticket:</h2>
+<p><a href="https://github.com/fedora-copr/copr/issues/2824">https://github.com/fedora-copr/copr/issues/2824</a></p>
+<p>Please join #fedora-buildsys, #fedora-admin or #fedora-noc on irc.libera.chat
+or add comments to this ticket.</p>
+
+
+
+
+ [Resolved] Outage
+ https://status.packit.dev/issues/2023-08-15-outage/
+ Tue, 15 Aug 2023 04:00:00 +0200
+ https://status.packit.dev/issues/2023-08-15-outage/
+ 2023-08-15T11:03:00+02:00
+ <p>We are having an outage that started on Tuesday EU early morning. Jobs during this time could get stuck
+in pending state.
+This is caused by a failed database schema migration because of a lack of memory during the automatic deployment
+of the new production images. We are working on the fix and apologize for the inconvenience.</p>
+<p>Update: Migration is completed. You might need to retrigger the jobs (i.e. via comment).</p>
+
+
+
+
+ [Resolved] Staging instance firewall
+ https://status.packit.dev/issues/2023-07-24-stage-fw/
+ Mon, 24 Jul 2023 14:30:00 +0200
+ https://status.packit.dev/issues/2023-07-24-stage-fw/
+ 2023-12-10T20:00:00+01:00
+ <p>We have moved our staging instance to another OpenShift cluster which may result in some jobs failing.
+This issue should mainly affect jobs that involve syncing to the dist-git, namely:</p>
+<ul>
+<li><code>propose-downstream</code></li>
+<li><code>pull-from-upstream</code></li>
+</ul>
+<p>In case you are using our staging instance and you hit any issues when running <code>propose-downstream</code> or <code>pull-from-upstream</code>, please let us know.</p>
+<h2 id="known-issues">Known Issues</h2>
+<ul>
+<li><del>Testing Farm is currently not available on the staging instance.</del>
+<em>Edit</em>: This issue has been fixed as of July 26th 10am.</li>
+</ul>
+
+
+
+
+ [Resolved] Testing Farm testing blocked
+ https://status.packit.dev/issues/2023-07-19-testing-farm-failing-on-rawhide/
+ Wed, 19 Jul 2023 08:00:00 +0000
+ https://status.packit.dev/issues/2023-07-19-testing-farm-failing-on-rawhide/
+ 2023-07-19T11:00:00+00:00
+ <p>Testing via Testing Farm is blocked: <a href="https://status.testing-farm.io/issues/2023-07-19-public-ranch-outage/">https://status.testing-farm.io/issues/2023-07-19-public-ranch-outage/</a>.</p>
+<p>Testing Farm team is working on the fix (ETA 2 hours).</p>
+
+
+
+
+ [Resolved] Guest couldn't be provisioned: Artemis resource ended in 'error' state
+ https://status.packit.dev/issues/2023-07-03-testing-provisioning-failed/
+ Mon, 03 Jul 2023 08:00:00 +0200
+ https://status.packit.dev/issues/2023-07-03-testing-provisioning-failed/
+ 2023-07-03T10:00:00+00:00
+ <p>Some jobs submitted by Packit to the Testing Farm are pending with this error:
+<code>"Guest couldn't be provisioned: Artemis resource ended in 'error' state"</code>.</p>
+<p>Testing Farm has already resolved the issue; please re-submit the job with a
+comment like <code>/packit test</code> (more info on retriggering
+<a href="https://packit.dev/docs/configuration/upstream/tests/#restart-testing">here</a>)
+or through the GitHub checks button.</p>
+
+
+
+
+ [Resolved] Fedora Rawhide Testing Blocked on dnf5
+ https://status.packit.dev/issues/2023-06-22-fedora-rawhide-testing-blocked-on-dnf5/
+ Thu, 22 Jun 2023 09:30:00 +0200
+ https://status.packit.dev/issues/2023-06-22-fedora-rawhide-testing-blocked-on-dnf5/
+ 2023-07-18T20:30:00+02:00
+ <p>dnf has been upgraded to dnf5 in Rawhide, resulting in various issues
+when running tests in Testing Farm.</p>
+<p>The Testing Farm team together with dnf5 team is working on fixing the issues
+and trying to resolve the situation ASAP.</p>
+<p>If you want to help, here are upstream issues related to this problem:</p>
+<ul>
+<li><a href="https://github.com/rpm-software-management/dnf5/issues/549">https://github.com/rpm-software-management/dnf5/issues/549</a></li>
+<li><a href="https://github.com/rpm-software-management/dnf5/issues/168">https://github.com/rpm-software-management/dnf5/issues/168</a></li>
+</ul>
+
+
+
+
+ [Resolved] Upgrade of Copr servers
+ https://status.packit.dev/issues/2023-05-25-copr-upgrade/
+ Thu, 25 May 2023 10:00:00 +0000
+ https://status.packit.dev/issues/2023-05-25-copr-upgrade/
+ 2023-05-25T13:00:00+00:00
+ <p>Due to an <a href="https://pagure.io/fedora-infrastructure/issue/11332">upgrade</a> of the Fedora
+Copr Servers, Copr builds might be delayed or failing. The upgrade should last approximately 3 hours.</p>
+
+
+
+
+ [Resolved] GitHub outage
+ https://status.packit.dev/issues/2023-05-11-github-outage/
+ Thu, 11 May 2023 15:30:00 +0200
+ https://status.packit.dev/issues/2023-05-11-github-outage/
+ 2023-05-11T21:00:00+02:00
+ <p>There is an ongoing GitHub outage.
+You can monitor the status and subscribe for updates here: <a href="https://www.githubstatus.com">https://www.githubstatus.com</a></p>
+<p>EDIT: <a href="https://github.blog/2023-05-16-addressing-githubs-recent-availability-issues">GitHub’s post mortem</a></p>
+
+
+
+
+ [Resolved] Testing Farm errors
+ https://status.packit.dev/issues/2023-05-11-tf-merge-issue/
+ Wed, 10 May 2023 14:00:00 +0200
+ https://status.packit.dev/issues/2023-05-11-tf-merge-issue/
+ 2023-05-11T15:00:00+02:00
+ <p>After the deployment of the new version of Testing Farm, you may have encountered Testing Farm runs that
+fail with the status <code>Failed to checkout ref {sha}</code>. Together with the Testing Farm team, we are working on a fix.</p>
+
+
+
+
+ [Resolved] GitHub outage
+ https://status.packit.dev/issues/2023-05-09-github-outage/
+ Tue, 09 May 2023 13:50:00 +0200
+ https://status.packit.dev/issues/2023-05-09-github-outage/
+ 2023-05-09T23:15:00+02:00
+ <p>There is an ongoing GitHub outage.
+You can monitor the status and subscribe for updates here: <a href="https://www.githubstatus.com">https://www.githubstatus.com</a></p>
+
+
+
+
+ [Resolved] Bodhi Updates not created
+ https://status.packit.dev/issues/2023-05-05-bodhi-updates/
+ Thu, 04 May 2023 11:30:00 +0200
+ https://status.packit.dev/issues/2023-05-05-bodhi-updates/
+ 2023-05-10T16:00:00+02:00
+ <p>We’re experiencing issues with creating Bodhi updates due to failing authentication.</p>
+<p>You can try to retrigger an update by a comment in a dist-git pull request:</p>
+<pre><code>/packit create-update
+</code></pre>
+
+
+
+
+ [Resolved] Cluster Update
+ https://status.packit.dev/issues/2023-05-03-cluster-update/
+ Wed, 03 May 2023 10:00:00 +0200
+ https://status.packit.dev/issues/2023-05-03-cluster-update/
+ 2023-05-03T11:15:00+02:00
+ <p>The OpenShift cluster is currently updating from 4.11.28 to 4.12.13</p>
+
+
+
+
+ [Resolved] Scheduled DB Cleanup
+ https://status.packit.dev/issues/2023-04-26-scheduled-db-cleanup/
+ Wed, 26 Apr 2023 04:40:00 +0000
+ https://status.packit.dev/issues/2023-04-26-scheduled-db-cleanup/
+ 2023-04-26T05:17:00+00:00
+ <p>We are running a DB maintenance in order to clean up old pipelines. We expect
+API performance to improve as a result of this.</p>
+<p>The maintenance window is planned for an hour, the actual clean up activities
+will last approximately 20 minutes. Users might experience degraded
+performance of the service and dashboard during this time.</p>
+
+
+
+
+ [Resolved] Issue with pushing to dist-git
+ https://status.packit.dev/issues/2023-04-12-push-to-distgit-issue/
+ Tue, 11 Apr 2023 18:28:00 +0200
+ https://status.packit.dev/issues/2023-04-12-push-to-distgit-issue/
+ 2023-04-12T16:30:00+02:00
+ <p>Due to <a href="https://docs.pagure.org/rpkg/releases/1.66.html#checking-a-repo-configuration-before-git-push-with-a-git-hook-script">a new pre-push hook</a>
+being applied in dist-git, we are unable to create pull-requests
+from fork as part of <code>propose-downstream</code> and <code>pull-from-upstream</code> jobs.</p>
+<p>We are sorry about that and are working on a way how to resolve this.</p>
+
+
+
+
+ [Resolved] Upgrade of Copr servers
+ https://status.packit.dev/issues/2023-04-05-copr-outage/
+ Wed, 05 Apr 2023 12:00:00 +0000
+ https://status.packit.dev/issues/2023-04-05-copr-outage/
+ 2023-04-05T15:00:00+00:00
+ <p>Due to an <a href="https://pagure.io/fedora-infrastructure/issue/11201">upgrade</a> of the Fedora
+Copr Servers, Copr builds might be delayed or failing. The upgrade should last approximately 3 hours.</p>
+
+
+
+
+ [Resolved] SLO1 Error Budget issues
+ https://status.packit.dev/issues/2023-03-09-slo1-error-budget/
+ Thu, 09 Mar 2023 10:00:00 +0100
+ https://status.packit.dev/issues/2023-03-09-slo1-error-budget/
+ 2023-06-08T10:00:00+01:00
+ <p>We are having <a href="https://packit.dev/docs/service-level-objectives/">SLO1</a> error budget issues.</p>
+<p>These are mostly caused by performance issues, we are working on them and prioritizing more improvements (see <a href="https://github.com/packit/packit-service/issues/1954">https://github.com/packit/packit-service/issues/1954</a> ).</p>
+<p>We are sorry for any slowness you can experience.</p>
+
+
+
+
+ [Resolved] Running grouping migrations
+ https://status.packit.dev/issues/2023-02-14-running-grouping-migrations/
+ Tue, 14 Feb 2023 02:00:00 +0100
+ https://status.packit.dev/issues/2023-02-14-running-grouping-migrations/
+ 2023-02-14T04:00:00+01:00
+ <p>We are updating our database schema to facilitate further development.
+As a part of this process, multiple database migrations will have to
+be run. The first step is grouping together related builds and test
+runs. The migrations will happen during the next deployment to production,
+on Tuesday February 14 at 2:00AM CET. This set of migrations is expected to
+take roughly 40 minutes, after which the service should come back up.</p>
+<p>Since significant refactoring was paired with these changes, there may
+be some bugs/regressions that we did not spot on staging. Please,
+contact us via the usual means if you spot anything suspicious.</p>
+
+
+
+
+ [Resolved] SLO2 Error Budget Depleted in a Day
+ https://status.packit.dev/issues/2023-02-17-slo2-error-budget-depleted-in-a-day/
+ Tue, 31 Jan 2023 10:00:00 +0100
+ https://status.packit.dev/issues/2023-02-17-slo2-error-budget-depleted-in-a-day/
+ 2023-02-01T10:00:00+01:00
+ <p>The SLO2 error budget dropped from 85% all the way down to zero in one day.
+<img src="https://status.packit.dev/images/SLO2-error-budget-depleted.png" alt="SLO2"></p>
+<p>The likely cause was that the fedora-messaging consumer was stuck, so we
+were not getting messages from Copr about finished SRPM builds.
+Luckily, the babysit tasks proved to be very useful and caught those later.</p>
+<p>We’ve <a href="https://github.com/packit/packit-service-fedmsg/pull/76">added a liveness probe to the messaging consumer</a>
+to prevent this from happening again.</p>
+
+
+
+
+ [Resolved] Testing Farm outage
+ https://status.packit.dev/issues/2023-01-24-tf-outage/
+ Tue, 24 Jan 2023 15:00:00 +0100
+ https://status.packit.dev/issues/2023-01-24-tf-outage/
+ 2023-01-25T10:00:00+01:00
+ <p>Testing Farm infrastructure had an <a href="https://status.testing-farm.io/issues/2023-01-24-public-ranch-outage/">outage</a>
+which caused the status of the test runs to be stuck in the <code>queued</code> state.
+This is now resolved and tests can be retriggered by <code>/packit test</code> comment command in the pull requests.</p>
+
+
+
+
+ [Resolved] SRPM builds stuck in Submitted state
+ https://status.packit.dev/issues/2023-01-21-srpm-builds-stuck-in-submitted-state/
+ Fri, 20 Jan 2023 21:19:06 +0100
+ https://status.packit.dev/issues/2023-01-21-srpm-builds-stuck-in-submitted-state/
+ 2023-01-21T9:51:30+01:00
+ <p>Workers for processing long-running tasks got stuck on Friday evening
+with to most visible consequence being the SRPM builds stuck in ‘Submitted’ state.</p>
+<p>The workers were restarted on Sat morning and all SRPMs were sent to Copr
+to build and the jobs pipelines were processed.</p>
+<p><a href="https://github.com/packit/packit-service/issues/1858">The cause is not known at the time of resolving the issue.</a></p>
+
+
+
+
+ [Resolved] System upgrade scheduled
+ https://status.packit.dev/issues/2023-01-06-cluster-upgrade/
+ Fri, 06 Jan 2023 10:00:00 +0000
+ https://status.packit.dev/issues/2023-01-06-cluster-upgrade/
+ 2023-01-06T22:00:00+00:00
+ <p>A system upgrade is scheduled to start on Fri Jan 06 20:00 UTC and last approximately 90 minutes.</p>
+<p>We do not expect any significant disruption.</p>
+<p>The started jobs might be executed with a delay
+and if you think Packit does not react to your events,
+try to retrigger the action via
+<a href="https://packit.dev/docs/guide/#how-to-re-trigger-packit-actions-in-your-pull-request">comment command</a>.</p>
+
+
+
+
+ [Resolved] Christmas is here ๐
+ https://status.packit.dev/issues/2022-12-23-christmas-2022/
+ Fri, 23 Dec 2022 18:00:00 +0000
+ https://status.packit.dev/issues/2022-12-23-christmas-2022/
+ 2023-01-02T08:00:00+00:00
+ <p>All the team is on vacation.
+Give us some time to get refreshed and be better prepared to work hard on Packit’s new features and bugs.</p>
+<p>The Packit staging instance (<a href="https://dashboard.stg.packit.dev/">https://dashboard.stg.packit.dev/</a>) will not run during this week.
+It will be back on the 2nd of January.</p>
+<p>–</p>
+<p>Have a wonderful year 2023 everyone! We’re back, staging instance is up and serving.</p>
+
+
+
+
+ [Resolved] Upgrade of Copr Servers
+ https://status.packit.dev/issues/2022-11-28-copr-outage/
+ Mon, 28 Nov 2022 16:00:00 +0100
+ https://status.packit.dev/issues/2022-11-28-copr-outage/
+ 2022-11-28T21:00:00+01:00
+ <p>Due to an <a href="https://pagure.io/fedora-infrastructure/issue/11011">upgrade</a> of the Fedora
+Copr Servers, Copr builds might have been failing.</p>
+
+
+
+
+ [Resolved] Staging is unstable
+ https://status.packit.dev/issues/2022-11-28-staging-unstable/
+ Fri, 25 Nov 2022 08:35:00 +0100
+ https://status.packit.dev/issues/2022-11-28-staging-unstable/
+ 2022-11-30T14:06:00+01:00
+ <p>Since the last cluster update, we’ve been running our staging instance
+on a separate node pool which uses <a href="https://aws.amazon.com/ec2/spot">AWS Spot instances</a>.
+It causes the staging instance to be quite unreliable.
+We’re testing it in order to find balance between price and usefulness.</p>
+
+
+
+
+ [Resolved] Outage
+ https://status.packit.dev/issues/2022-11-25-outage/
+ Thu, 24 Nov 2022 23:24:00 +0100
+ https://status.packit.dev/issues/2022-11-25-outage/
+ 2022-11-25T08:35:00+01:00
+ <p>We are having resource issues in our cluster and particular parts of the deployment can be down.</p>
+<p>Updates:</p>
+<ul>
+<li><code>2022-11-25T08:35:00+01:00</code>: Stage instance has been scaled down so our production instance has more resources.
+Production instance should be fine now. We are investigating more.</li>
+</ul>
+
+
+
+
+ [Resolved] System upgrade scheduled
+ https://status.packit.dev/issues/2022-11-24-cluster-upgrade/
+ Thu, 24 Nov 2022 19:00:00 +0000
+ https://status.packit.dev/issues/2022-11-24-cluster-upgrade/
+ 2022-11-24T23:00:00+00:00
+ <p>A system upgrade is scheduled to start on Thu Nov 24 21:00 UTC and last approximately 90 minutes.</p>
+<p>We do not expect any significant disruption.</p>
+<p>The started jobs might be executed with a delay
+and if you think Packit does not react to your events,
+try to retrigger the action via
+<a href="https://packit.dev/docs/guide/#how-to-re-trigger-packit-actions-in-your-pull-request">comment command</a>.</p>
+
+
+
+
+ [Resolved] Copr builds and tests on commits and releases ignored
+ https://status.packit.dev/issues/2022-11-01-commit-release-trigger/
+ Tue, 01 Nov 2022 03:00:00 +0100
+ https://status.packit.dev/issues/2022-11-01-commit-release-trigger/
+ 2022-11-02T10:30:00+01:00
+ <p>There was a bug introduced with Tuesday’s production update causing Copr builds and tests
+for <code>commit</code> and <code>release</code> triggers were not executed.</p>
+
+
+
+
+ [Resolved] Copr storage move
+ https://status.packit.dev/issues/2022-10-28-30-copr-storage-move/
+ Fri, 28 Oct 2022 21:00:00 +0200
+ https://status.packit.dev/issues/2022-10-28-30-copr-storage-move/
+ 2022-10-30T08:00:00+02:00
+ <p>Copr team is performing a backend storage move over the weekend of Oct 28-30th.
+Building in Copr will be disrupted or unavailable during this period.</p>
+<p>For more info, please follow the <a href="https://lists.fedoraproject.org/archives/list/copr-devel@lists.fedorahosted.org/thread/TRNW5JVCFIQJDUCND4KZWZVM7XVCYLVL/">Copr outage announcement</a>.</p>
+<p>Update: the upgrade was successful and Copr is fully operational.</p>
+
+
+
+
+ [Resolved] System upgrade scheduled
+ https://status.packit.dev/issues/2022-10-13-cluster-upgrade/
+ Thu, 13 Oct 2022 19:00:00 +0000
+ https://status.packit.dev/issues/2022-10-13-cluster-upgrade/
+ 2022-10-13T20:37:00+00:00
+ <p>A system upgrade is scheduled to start on Wed Oct 13 19:00 UTC and last approximately 90 minutes.</p>
+<p>We do not expect any significant disruption.</p>
+<p>The started jobs might be executed with a delay
+and if you think Packit does not react to your events,
+try to retrigger the action via
+<a href="https://packit.dev/docs/guide/#how-to-re-trigger-packit-actions-in-your-pull-request">comment command</a>.</p>
+
+
+
+
+ [Resolved] Issues with the task execution
+ https://status.packit.dev/issues/2022-10-11-outage/
+ Tue, 11 Oct 2022 13:57:00 +0200
+ https://status.packit.dev/issues/2022-10-11-outage/
+ 2022-10-11T15:12:00+02:00
+ <p>We are experiencing issues with the execution of the tasks.
+(The statuses are waiting in the <code>The task was accepted.</code> state.)
+We are investigating the cause.
+Sorry about the possible troubles.
+We will try to fix this as soon as possible.</p>
+<p>Updates:</p>
+<ul>
+<li>Tue Oct 11 12:24:02 UTC: We are slowly going through the queue. Please, be patient.</li>
+<li>Tue Oct 11 13:12:36 UTC: We have gone through the queue and you should experience regular response times.
+The original cause is still not known but the problem probably occured during an SRPM upload to Copr.
+(It can be a fault on our or Copr side.)</li>
+<li>Wed Oct 12 05:27:46 UTC: Service operated normally since the last update.
+To reduce the chance of this, we <a href="https://packit.dev/posts/copr-srpms/">plan to move the building of SRPMs to Copr</a>.
+(Currently default for new installations and opt in for other projects.)</li>
+</ul>
+
+
+
+
+ [Resolved] Internal Testing Farm infrastructure is having an outage
+ https://status.packit.dev/issues/2022-10-10-outage/
+ Mon, 10 Oct 2022 15:14:00 +0200
+ https://status.packit.dev/issues/2022-10-10-outage/
+ 2022-10-11T21:34:00+02:00
+ <p>Due to a wide outage in Red Hat internal infrastructure the Red Hat Ranch (=internal ranch) is disabled in the Testing Farm (=our test runner).
+Currently there is no ETA for resolving the problem.
+Follow the progress on the Testing Farm status page: <a href="https://status.testing-farm.io/issues/2022-10-10-redhat-ranch/">https://status.testing-farm.io/issues/2022-10-10-redhat-ranch/</a></p>
+
+
+
+
+ [Resolved] GitHub Webhooks not working
+ https://status.packit.dev/issues/2022-10-05-github-webhooks-degraded-status/
+ Wed, 05 Oct 2022 16:00:00 +0200
+ https://status.packit.dev/issues/2022-10-05-github-webhooks-degraded-status/
+ 2022-10-05T18:00:00+02:00
+ <p>Packit is not working on GitHub because the GitHub Webhooks status is ‘degraded’.
+See yourself: <a href="https://www.githubstatus.com/incidents/smn1qtqvbsb6">https://www.githubstatus.com/incidents/smn1qtqvbsb6</a></p>
+
+
+
+
+ [Resolved] September 22th outage
+ https://status.packit.dev/issues/2022-09-22-outage/
+ Thu, 22 Sep 2022 08:00:00 +0200
+ https://status.packit.dev/issues/2022-09-22-outage/
+ 2022-09-23T08:00:00+02:00
+ <p>Testing Farm is failing to run tests submitted by Packit which use <code>Copr</code> builds.
+Tests with the <code>skip_build</code> flag are not be affected by this issue.
+We are working together with the Testing Farm team to fix this as soon as possible.</p>
+
+
+
+
+ [Resolved] 3/4 of SLO1 Error Budget consumed in 6 weeks
+ https://status.packit.dev/issues/2022-10-27-slo1-error-budget-drop-postmortem/
+ Thu, 01 Sep 2022 01:00:00 +0200
+ https://status.packit.dev/issues/2022-10-27-slo1-error-budget-drop-postmortem/
+ 2022-10-12T01:00:00+02:00
+ <p>(Mostly for our internal use since the source data are not publicly available.)</p>
+<p>During the 6 weeks since the beginning of September the <a href="https://packit.dev/docs/service-level-objectives/#slo1-changes-to-github-prs-receive-a-status-update-within-15-seconds-in-99-of-cases">SLO1</a>
+error budget dropped to 25%. After that (in the middle of October)
+the trend turned and now, at the end of October, it’s at 50%.</p>
+<p>When looking at metrics in our (not public) Grafana
+(<code>Packit boards</code> -> <code>(Prod/Stg) Accepted status time)</code>
+we can see that the average value (of the “accepted status time”) indeed
+increased during the beginning of September by approx 1 second and
+cases of >15s started to appear so the error budget started to be consumed.</p>
+<p>The cause is yet unknown, but the changes we did the last week of August
+and which could thus contribute to the problem were:</p>
+<ul>
+<li>lots of SQLAlchemy-related changes - contribution unknown</li>
+<li>switched 4 single-threaded workers to 1 multithreaded - there might be some
+correlation, but it’s not clear because for example the first 2 weeks of October
+the change was reverted (i.e. there were again the 4 single-threaded workers)
+and even the average value seemed to drop a bit, the error budget continued
+to be consumed (meaning there were still some cases of >15s).</li>
+</ul>
+<p>We need to continue watching the metrics, experiment with the workers
+(numbers and types) and give the changes more time (2 weeks at least)
+to be able to tell whether they change the trend.</p>
+<p>Related:</p>
+<ul>
+<li><a href="https://github.com/packit/packit-service/issues/1364">https://github.com/packit/packit-service/issues/1364</a></li>
+<li><a href="https://github.com/packit/packit-service/issues/1677">https://github.com/packit/packit-service/issues/1677</a></li>
+<li><a href="https://github.com/packit/packit-service/issues/1728">https://github.com/packit/packit-service/issues/1728</a></li>
+</ul>
+
+
+
+
+ [Resolved] Failing SRPM builds done in Copr
+ https://status.packit.dev/issues/2022-08-23-srpm-builds-failing/
+ Tue, 23 Aug 2022 04:00:00 +0200
+ https://status.packit.dev/issues/2022-08-23-srpm-builds-failing/
+ 2022-08-23T09:00:00+02:00
+ <p>SRPM builds happening in Copr (for packages using the <code>srpm_build_deps</code> config
+key) were failing due to a dependency issue.</p>
+
+
+
+
+ [Resolved] Summer is here โ๏ธ
+ https://status.packit.dev/issues/2022-07-01-team-on-vacation/
+ Mon, 04 Jul 2022 08:00:00 +0000
+ https://status.packit.dev/issues/2022-07-01-team-on-vacation/
+ 2022-07-08T18:00:00+00:00
+ <p>Most of the team is on vacation. We try our best to solve issues with limited resources.
+Give us some time to get refreshed and be better prepared to work hard on Packit’s new features and bugs.</p>
+
+
+
+
+ [Resolved] Upgrade of Copr Servers
+ https://status.packit.dev/issues/2022-06-22-upgrade-of-copr-servers/
+ Wed, 22 Jun 2022 14:00:00 +0200
+ https://status.packit.dev/issues/2022-06-22-upgrade-of-copr-servers/
+ 2022-06-22T16:10:00+02:00
+ <p>Due to an <a href="https://pagure.io/fedora-infrastructure/issue/10780">ongoing
+upgrade</a> of the Fedora
+Copr Servers, building might be delayed or failing. The upgrade should last ~3
+hours.</p>
+
+
+
+
+ [Resolved] SRPM builds done in Copr are failing
+ https://status.packit.dev/issues/2022-06-21-srpm-builds-failing/
+ Tue, 21 Jun 2022 10:18:22 +0200
+ https://status.packit.dev/issues/2022-06-21-srpm-builds-failing/
+ 2022-06-21T11:10:22+02:00
+ <p>SRPM builds happening in Copr (for packages using the <code>srpm_build_deps</code> config
+key) are failing due to a dependency issue. We are working on a fix.</p>
+
+
+
+
+ [Resolved] Networking Issue
+ https://status.packit.dev/issues/2022-02-03-networking-issue/
+ Thu, 03 Feb 2022 08:37:21 +0100
+ https://status.packit.dev/issues/2022-02-03-networking-issue/
+ 2022-02-03T11:17:21+01:00
+ <p>Our production instance is getting various network issues.
+In logs, you can see errors with <code>rsync</code>, <code>kerberos</code> and other network related commands.
+We are sorry about the current state and are investigating the cause.
+We will let you informed once we know more.</p>
+<p>edit: The problem was with one of our task runners.
+Most probably because of a memory issue we are now trying to find and resolve.
+The runner was recreated and is working fine currently.
+We are getting an eye on it; please let us know if you find anything suspicious.</p>
+
+
+
+
+ [Resolved] February 1st outage
+ https://status.packit.dev/issues/2022-02-01-outage/
+ Tue, 01 Feb 2022 02:00:00 +0000
+ https://status.packit.dev/issues/2022-02-01-outage/
+ 2022-02-01T04:15:00+00:00
+ <p>We had an outage on Tuesday EU night/early morning. Jobs during this time could get stuck
+in pending state.
+This was caused by a failed database schema migration because of a lack of memory during the automatic deployment of the new production images.
+We apologize for the inconvenience.</p>
+
+
+
+
+ [Resolved] New production deployment
+ https://status.packit.dev/issues/2021-12-08-new-production-deployment/
+ Wed, 08 Dec 2021 02:00:00 +0000
+ https://status.packit.dev/issues/2021-12-08-new-production-deployment/
+ 2022-01-01T00:00:00+00:00
+ <p>We’ve deployed production instance of Packit Service to a new Openshift cluster.
+If you spot anything unusual, <a href="https://packit.dev/docs/faq/#how-can-i-contact-you">let us know</a>.</p>
+
+
+
+
+ [Resolved] Longer response times
+ https://status.packit.dev/issues/2021-12-07-response-delays/
+ Tue, 07 Dec 2021 09:06:14 +0100
+ https://status.packit.dev/issues/2021-12-07-response-delays/
+ 2021-12-07T11:49:14+01:00
+ <p>Our service got overwhelmed by the events coming from a Fedora dist-git (caused by the <code>epel-9</code> branching) causing the longer responses.</p>
+<p>We have scaled up the workers and the queue is getting shorter.
+We are planning the next steps to be resilient next time. Sorry about the disturbance.</p>
+<p>update: The queue is now empty and you can expect regular operational times.</p>
+
+
+
+
+ [Resolved] Problems with production database
+ https://status.packit.dev/issues/2021-11-18-production-db-problems/
+ Thu, 18 Nov 2021 02:33:00 +0000
+ https://status.packit.dev/issues/2021-11-18-production-db-problems/
+ 2021-11-19 00:10:00
+ <p>After moving our production service to a new cluster, we’re seeing intermittent issues
+with connecting to the database. They are mostly manifesting in the dashboard.
+We need to decide next steps.</p>
+
+
+
+
+ [Resolved] Moving production Packit Service to a new cluster
+ https://status.packit.dev/issues/2021-11-16-auto-prod/
+ Thu, 18 Nov 2021 01:00:00 +0000
+ https://status.packit.dev/issues/2021-11-16-auto-prod/
+ 2021-11-18 02:33:00
+ <p>We are going to move deployment of our production Packit Service to a new cluster.
+It should take about 1/2 an hour, during which the workers will be down and queries
+to API & dashboard won’t work properly because the database will be being migrated.
+Any requests submitted to the API meanwhile should, however, be fulfilled once the workers are
+up again.</p>
+
+
+
+
+ [Resolved] Slow SRPM builds
+ https://status.packit.dev/issues/2021-09-08-outage/
+ Tue, 07 Sep 2021 20:30:00 +0200
+ https://status.packit.dev/issues/2021-09-08-outage/
+ 2021-09-16T13:54:56+00:00
+ <p>Building of SRPMs for specific projects needs more memory and blocks the worker for a longer time,
+this causes other jobs to be slow and delayed. We are working to resolve this in the week of September 13th.</p>
+
+
+
+
+ [Resolved] August 27th outage
+ https://status.packit.dev/issues/2021-08-27-outage/
+ Fri, 27 Aug 2021 15:00:00 +0200
+ https://status.packit.dev/issues/2021-08-27-outage/
+ 2021-08-28 08:00:00+00:00
+ <p>We had an outage which started in the Friday EU afternoon and was fully
+resolved on Saturday EU morning. Most of the jobs were failing during this
+time. This was caused by a misconfiguration on our side. We apologize for the
+inconvenience. We’re going to take steps to prevent the misconfiguration from
+happening in future.</p>
+
+
+
+
+ [Resolved] Summer is here โ๏ธ #2
+ https://status.packit.dev/issues/2021-07-56-vacation/
+ Thu, 01 Jul 2021 10:55:00 +0200
+ https://status.packit.dev/issues/2021-07-56-vacation/
+ 2021-07-06 19:10:40+00:00
+ <p>There are 2 days of public holidays on July 5th and 6th in Czech Republic and since most of the team is located in CZ, we’ll be unavailable during these two days.</p>
+
+
+
+
+ [Resolved] Summer is here โ๏ธ
+ https://status.packit.dev/issues/2021-06-28-vacation/
+ Fri, 25 Jun 2021 10:55:00 +0200
+ https://status.packit.dev/issues/2021-06-28-vacation/
+ 2021-06-28 19:10:40+00:00
+ <p>Most of the team will be unavailable on Monday, June 28th. We’ll try our best to resolve issues as soon as we’re back.</p>
+
+
+
+
+ [Resolved] PostgreSQL db upgrade
+ https://status.packit.dev/issues/2021-06-04-postgresql-12/
+ Fri, 04 Jun 2021 07:00:00 +0000
+ https://status.packit.dev/issues/2021-06-04-postgresql-12/
+ 2021-06-04 09:10:40+00:00
+ <p>Upgrading our PostgreSQL database from version 10 to version 12.</p>
+
+
+
+
+ [Resolved] Dashboard - unable to connect
+ https://status.packit.dev/issues/2021-06-02-dashboard-unable-to-connect/
+ Wed, 02 Jun 2021 11:16:00 +0000
+ https://status.packit.dev/issues/2021-06-02-dashboard-unable-to-connect/
+ 2021-06-02 11:40:00
+ <p><strong>API</strong> was down and therefore <strong>Dashboard</strong> was not showing any Jobs
+and after some time a “unable to connect …” error was shown.</p>
+<p>Cause:
+Unknown</p>
+<p>Fix:
+After restarting the API, the Dashboard started to work OK.</p>
+
+
+
+
+
+
diff --git a/issues/2021-06-02-dashboard-unable-to-connect/index.html b/issues/2021-06-02-dashboard-unable-to-connect/index.html
new file mode 100644
index 00000000..de8e140e
--- /dev/null
+++ b/issues/2021-06-02-dashboard-unable-to-connect/index.html
@@ -0,0 +1,652 @@
+
+
+
+
+
+
+
+
+
+ Dashboard - unable to connect | Packit Service Status
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ Resolved after
+
+
+
+ 130h
+ 15m of downtime.
+
+
+
+
+ July 6, 2021 at 7:10 PM UTC
+
+
+
+
+
+
+
+
+
There are 2 days of public holidays on July 5th and 6th in Czech Republic and since most of the team is located in CZ, we’ll be unavailable during these two days.
+
+
+
+
+
+
+
+
+
Last updated:
+
+
+ July 22, 2024 at 9:41 AM UTC
+
+
+
+ Resolved after
+
+
+
+ 19h
+ 0m of downtime.
+
+
+
+
+ August 28, 2021 at 8:00 AM UTC
+
+
+
+
+
+
+
+
+
We had an outage which started in the Friday EU afternoon and was fully
+resolved on Saturday EU morning. Most of the jobs were failing during this
+time. This was caused by a misconfiguration on our side. We apologize for the
+inconvenience. We’re going to take steps to prevent the misconfiguration from
+happening in future.
+
+
+
+
+
+
+
+
+
Last updated:
+
+
+ July 22, 2024 at 9:41 AM UTC
+
+
+
+ Resolved after
+
+
+
+ 211h
+ 24m of downtime.
+
+
+
+
+ September 16, 2021 at 1:54 PM UTC
+
+
+
+
+
+
+
+
+
Building of SRPMs for specific projects needs more memory and blocks the worker for a longer time,
+this causes other jobs to be slow and delayed. We are working to resolve this in the week of September 13th.
+
+
+
+
+
+
+
+
+
Last updated:
+
+
+ July 22, 2024 at 9:41 AM UTC
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
diff --git a/issues/2021-09-08-outage/index.json b/issues/2021-09-08-outage/index.json
new file mode 100644
index 00000000..9894970e
--- /dev/null
+++ b/issues/2021-09-08-outage/index.json
@@ -0,0 +1,13 @@
+{
+ "is": "issue",
+ "title": "Slow SRPM builds",
+ "createdAt": "2021-09-07 20:30:00 +0200 +0200",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2021-09-08-outage/",
+ "severity": "notice",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2021-09-16T13:54:56+00:00",
+ "affected": [],
+ "filename": "2021-09-08-outage.md"
+}
\ No newline at end of file
diff --git a/issues/2021-11-16-auto-prod/index.html b/issues/2021-11-16-auto-prod/index.html
new file mode 100644
index 00000000..e12a96e8
--- /dev/null
+++ b/issues/2021-11-16-auto-prod/index.html
@@ -0,0 +1,651 @@
+
+
+
+
+
+
+
+
+
+ Moving production Packit Service to a new cluster | Packit Service Status
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ Resolved after
+
+
+
+ 1h
+ 33m of downtime.
+
+
+
+
+ November 18, 2021 at 2:33 AM UTC
+
+
+
+
+
+
+
+
+
We are going to move deployment of our production Packit Service to a new cluster.
+It should take about 1/2 an hour, during which the workers will be down and queries
+to API & dashboard won’t work properly because the database will be being migrated.
+Any requests submitted to the API meanwhile should, however, be fulfilled once the workers are
+up again.
+
+
+
+
+
+
+
+
+
Last updated:
+
+
+ July 22, 2024 at 9:41 AM UTC
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
diff --git a/issues/2021-11-16-auto-prod/index.json b/issues/2021-11-16-auto-prod/index.json
new file mode 100644
index 00000000..0c0cc8c3
--- /dev/null
+++ b/issues/2021-11-16-auto-prod/index.json
@@ -0,0 +1,13 @@
+{
+ "is": "issue",
+ "title": "Moving production Packit Service to a new cluster",
+ "createdAt": "2021-11-18 01:00:00 +0000 UTC",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2021-11-16-auto-prod/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2021-11-18 02:33:00",
+ "affected": ["API", "Workers", "Dashboard"],
+ "filename": "2021-11-16-auto-prod.md"
+}
\ No newline at end of file
diff --git a/issues/2021-11-18-production-db-problems/index.html b/issues/2021-11-18-production-db-problems/index.html
new file mode 100644
index 00000000..6e5cfc35
--- /dev/null
+++ b/issues/2021-11-18-production-db-problems/index.html
@@ -0,0 +1,645 @@
+
+
+
+
+
+
+
+
+
+ Problems with production database | Packit Service Status
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ Resolved after
+
+
+
+ 21h
+ 37m of downtime.
+
+
+
+
+ November 19, 2021 at 12:10 AM UTC
+
+
+
+
+
+
+
+
+
After moving our production service to a new cluster, we’re seeing intermittent issues
+with connecting to the database. They are mostly manifesting in the dashboard.
+We need to decide next steps.
+
+
+
+
+
+
+
+
+
Last updated:
+
+
+ July 22, 2024 at 9:41 AM UTC
+
+
+
+ Resolved after
+
+
+
+ 2h
+ 43m of downtime.
+
+
+
+
+ December 7, 2021 at 11:49 AM UTC
+
+
+
+
+
+
+
+
+
Our service got overwhelmed by the events coming from a Fedora dist-git (caused by the epel-9 branching) causing the longer responses.
+
We have scaled up the workers and the queue is getting shorter.
+We are planning the next steps to be resilient next time. Sorry about the disturbance.
+
update: The queue is now empty and you can expect regular operational times.
+
+
+
+
+
+
+
+
+
Last updated:
+
+
+ July 22, 2024 at 9:41 AM UTC
+
+
+
+ Resolved after
+
+
+
+ 2h
+ 15m of downtime.
+
+
+
+
+ February 1, 2022 at 4:15 AM UTC
+
+
+
+
+
+
+
+
+
We had an outage on Tuesday EU night/early morning. Jobs during this time could get stuck
+in pending state.
+This was caused by a failed database schema migration because of a lack of memory during the automatic deployment of the new production images.
+We apologize for the inconvenience.
+
+
+
+
+
+
+
+
+
Last updated:
+
+
+ July 22, 2024 at 9:41 AM UTC
+
+
+
+ Resolved after
+
+
+
+ 2h
+ 40m of downtime.
+
+
+
+
+ February 3, 2022 at 11:17 AM UTC
+
+
+
+
+
+
+
+
+
Our production instance is getting various network issues.
+In logs, you can see errors with rsync, kerberos and other network related commands.
+We are sorry about the current state and are investigating the cause.
+We will let you informed once we know more.
+
edit: The problem was with one of our task runners.
+Most probably because of a memory issue we are now trying to find and resolve.
+The runner was recreated and is working fine currently.
+We are getting an eye on it; please let us know if you find anything suspicious.
+
+
+
+
+
+
+
+
+
Last updated:
+
+
+ July 22, 2024 at 9:41 AM UTC
+
+
+
+ Resolved after
+
+
+
+ 106h
+ 0m of downtime.
+
+
+
+
+ July 8, 2022 at 6:00 PM UTC
+
+
+
+
+
+
+
+
+
Most of the team is on vacation. We try our best to solve issues with limited resources.
+Give us some time to get refreshed and be better prepared to work hard on Packit’s new features and bugs.
+
+
+
+
+
+
+
+
+
Last updated:
+
+
+ July 22, 2024 at 9:41 AM UTC
+
+
+
+ Resolved after
+
+
+
+ 24h
+ 0m of downtime.
+
+
+
+
+ September 23, 2022 at 8:00 AM UTC
+
+
+
+
+
+
+
+
+
Testing Farm is failing to run tests submitted by Packit which use Copr builds.
+Tests with the skip_build flag are not be affected by this issue.
+We are working together with the Testing Farm team to fix this as soon as possible.
+
+
+
+
+
+
+
+
+
Last updated:
+
+
+ July 22, 2024 at 9:41 AM UTC
+
+
+
+ Resolved after
+
+
+
+ 30h
+ 20m of downtime.
+
+
+
+
+ October 11, 2022 at 9:34 PM UTC
+
+
+
+
+
+
+
+
+
Due to a wide outage in Red Hat internal infrastructure the Red Hat Ranch (=internal ranch) is disabled in the Testing Farm (=our test runner).
+Currently there is no ETA for resolving the problem.
+Follow the progress on the Testing Farm status page: https://status.testing-farm.io/issues/2022-10-10-redhat-ranch/
+
+
+
+
+
+
+
+
+
Last updated:
+
+
+ July 22, 2024 at 9:41 AM UTC
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
diff --git a/issues/2022-10-10-outage/index.json b/issues/2022-10-10-outage/index.json
new file mode 100644
index 00000000..8574b395
--- /dev/null
+++ b/issues/2022-10-10-outage/index.json
@@ -0,0 +1,13 @@
+{
+ "is": "issue",
+ "title": "Internal Testing Farm infrastructure is having an outage",
+ "createdAt": "2022-10-10 15:14:00 +0200 +0200",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2022-10-10-outage/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2022-10-11T21:34:00+02:00",
+ "affected": ["Testing Farm"],
+ "filename": "2022-10-10-outage.md"
+}
\ No newline at end of file
diff --git a/issues/2022-10-11-outage/index.html b/issues/2022-10-11-outage/index.html
new file mode 100644
index 00000000..1bb6b8fa
--- /dev/null
+++ b/issues/2022-10-11-outage/index.html
@@ -0,0 +1,661 @@
+
+
+
+
+
+
+
+
+
+ Issues with the task execution | Packit Service Status
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ Resolved after
+
+
+
+ 1h
+ 15m of downtime.
+
+
+
+
+ October 11, 2022 at 3:12 PM UTC
+
+
+
+
+
+
+
+
+
We are experiencing issues with the execution of the tasks.
+(The statuses are waiting in the The task was accepted. state.)
+We are investigating the cause.
+Sorry about the possible troubles.
+We will try to fix this as soon as possible.
+
Updates:
+
+
Tue Oct 11 12:24:02 UTC: We are slowly going through the queue. Please, be patient.
+
Tue Oct 11 13:12:36 UTC: We have gone through the queue and you should experience regular response times.
+The original cause is still not known but the problem probably occured during an SRPM upload to Copr.
+(It can be a fault on our or Copr side.)
+
Wed Oct 12 05:27:46 UTC: Service operated normally since the last update.
+To reduce the chance of this, we plan to move the building of SRPMs to Copr.
+(Currently default for new installations and opt in for other projects.)
+
+
+
+
+
+
+
+
+
+
Last updated:
+
+
+ July 22, 2024 at 9:41 AM UTC
+
+
+
+ Resolved after
+
+
+
+ 1h
+ 37m of downtime.
+
+
+
+
+ October 13, 2022 at 8:37 PM UTC
+
+
+
+
+
+
+
+
+
A system upgrade is scheduled to start on Wed Oct 13 19:00 UTC and last approximately 90 minutes.
+
We do not expect any significant disruption.
+
The started jobs might be executed with a delay
+and if you think Packit does not react to your events,
+try to retrigger the action via
+comment command.
+
+
+
+
+
+
+
+
+
Last updated:
+
+
+ July 22, 2024 at 9:41 AM UTC
+
+
+
+ Resolved after
+
+
+
+ 984h
+ 0m of downtime.
+
+
+
+
+ October 12, 2022 at 1:00 AM UTC
+
+
+
+
+
+
+
+
+
(Mostly for our internal use since the source data are not publicly available.)
+
During the 6 weeks since the beginning of September the SLO1
+error budget dropped to 25%. After that (in the middle of October)
+the trend turned and now, at the end of October, it’s at 50%.
+
When looking at metrics in our (not public) Grafana
+(Packit boards -> (Prod/Stg) Accepted status time)
+we can see that the average value (of the “accepted status time”) indeed
+increased during the beginning of September by approx 1 second and
+cases of >15s started to appear so the error budget started to be consumed.
+
The cause is yet unknown, but the changes we did the last week of August
+and which could thus contribute to the problem were:
+
+
lots of SQLAlchemy-related changes - contribution unknown
+
switched 4 single-threaded workers to 1 multithreaded - there might be some
+correlation, but it’s not clear because for example the first 2 weeks of October
+the change was reverted (i.e. there were again the 4 single-threaded workers)
+and even the average value seemed to drop a bit, the error budget continued
+to be consumed (meaning there were still some cases of >15s).
+
+
We need to continue watching the metrics, experiment with the workers
+(numbers and types) and give the changes more time (2 weeks at least)
+to be able to tell whether they change the trend.
+
+ Resolved after
+
+
+
+ 4h
+ 0m of downtime.
+
+
+
+
+ November 24, 2022 at 11:00 PM UTC
+
+
+
+
+
+
+
+
+
A system upgrade is scheduled to start on Thu Nov 24 21:00 UTC and last approximately 90 minutes.
+
We do not expect any significant disruption.
+
The started jobs might be executed with a delay
+and if you think Packit does not react to your events,
+try to retrigger the action via
+comment command.
+
+
+
+
+
+
+
+
+
Last updated:
+
+
+ July 22, 2024 at 9:41 AM UTC
+
+
+
+ Resolved after
+
+
+
+ 9h
+ 11m of downtime.
+
+
+
+
+ November 25, 2022 at 8:35 AM UTC
+
+
+
+
+
+
+
+
+
We are having resource issues in our cluster and particular parts of the deployment can be down.
+
Updates:
+
+
2022-11-25T08:35:00+01:00: Stage instance has been scaled down so our production instance has more resources.
+Production instance should be fine now. We are investigating more.
+
+
+
+
+
+
+
+
+
+
Last updated:
+
+
+ July 22, 2024 at 9:41 AM UTC
+
+
+
+ Resolved after
+
+
+
+ 125h
+ 31m of downtime.
+
+
+
+
+ November 30, 2022 at 2:06 PM UTC
+
+
+
+
+
+
+
+
+
Since the last cluster update, we’ve been running our staging instance
+on a separate node pool which uses AWS Spot instances.
+It causes the staging instance to be quite unreliable.
+We’re testing it in order to find balance between price and usefulness.
+
+
+
+
+
+
+
+
+
Last updated:
+
+
+ July 22, 2024 at 9:41 AM UTC
+
+
+
+ Resolved after
+
+
+
+ 12h
+ 0m of downtime.
+
+
+
+
+ January 6, 2023 at 10:00 PM UTC
+
+
+
+
+
+
+
+
+
A system upgrade is scheduled to start on Fri Jan 06 20:00 UTC and last approximately 90 minutes.
+
We do not expect any significant disruption.
+
The started jobs might be executed with a delay
+and if you think Packit does not react to your events,
+try to retrigger the action via
+comment command.
+
+
+
+
+
+
+
+
+
Last updated:
+
+
+ July 22, 2024 at 9:41 AM UTC
+
+
+
+ Resolved after
+
+
+
+ 12h
+ 32m of downtime.
+
+
+
+
+ January 21, 2023 at 9:51 AM UTC
+
+
+
+
+
+
+
+
+
Workers for processing long-running tasks got stuck on Friday evening
+with to most visible consequence being the SRPM builds stuck in ‘Submitted’ state.
+
The workers were restarted on Sat morning and all SRPMs were sent to Copr
+to build and the jobs pipelines were processed.
+
+ Resolved after
+
+
+
+ 19h
+ 0m of downtime.
+
+
+
+
+ January 25, 2023 at 10:00 AM UTC
+
+
+
+
+
+
+
+
+
Testing Farm infrastructure had an outage
+which caused the status of the test runs to be stuck in the queued state.
+This is now resolved and tests can be retriggered by /packit test comment command in the pull requests.
+
+
+
+
+
+
+
+
+
Last updated:
+
+
+ July 22, 2024 at 9:41 AM UTC
+
+
+
+ Resolved after
+
+
+
+ 2h
+ 0m of downtime.
+
+
+
+
+ February 14, 2023 at 4:00 AM UTC
+
+
+
+
+
+
+
+
+
We are updating our database schema to facilitate further development.
+As a part of this process, multiple database migrations will have to
+be run. The first step is grouping together related builds and test
+runs. The migrations will happen during the next deployment to production,
+on Tuesday February 14 at 2:00AM CET. This set of migrations is expected to
+take roughly 40 minutes, after which the service should come back up.
+
Since significant refactoring was paired with these changes, there may
+be some bugs/regressions that we did not spot on staging. Please,
+contact us via the usual means if you spot anything suspicious.
+
+
+
+
+
+
+
+
+
Last updated:
+
+
+ July 22, 2024 at 9:41 AM UTC
+
+
+
+ Resolved after
+
+
+
+ 24h
+ 0m of downtime.
+
+
+
+
+ February 1, 2023 at 10:00 AM UTC
+
+
+
+
+
+
+
+
+
The SLO2 error budget dropped from 85% all the way down to zero in one day.
+
+
The likely cause was that the fedora-messaging consumer was stuck, so we
+were not getting messages from Copr about finished SRPM builds.
+Luckily, the babysit tasks proved to be very useful and caught those later.
+
+ Resolved after
+
+
+
+ 22h
+ 2m of downtime.
+
+
+
+
+ April 12, 2023 at 4:30 PM UTC
+
+
+
+
+
+
+
+
+
Due to a new pre-push hook
+being applied in dist-git, we are unable to create pull-requests
+from fork as part of propose-downstream and pull-from-upstream jobs.
+
We are sorry about that and are working on a way how to resolve this.
+
+
+
+
+
+
+
+
+
Last updated:
+
+
+ July 22, 2024 at 9:41 AM UTC
+
+
+
+ Resolved after
+
+
+
+ 37m
+ of downtime.
+
+
+
+
+ April 26, 2023 at 5:17 AM UTC
+
+
+
+
+
+
+
+
+
We are running a DB maintenance in order to clean up old pipelines. We expect
+API performance to improve as a result of this.
+
The maintenance window is planned for an hour, the actual clean up activities
+will last approximately 20 minutes. Users might experience degraded
+performance of the service and dashboard during this time.
+
+
+
+
+
+
+
+
+
Last updated:
+
+
+ July 22, 2024 at 9:41 AM UTC
+
+
+
+ Resolved after
+
+
+
+ 25h
+ 0m of downtime.
+
+
+
+
+ May 11, 2023 at 3:00 PM UTC
+
+
+
+
+
+
+
+
+
After the deployment of the new version of Testing Farm, you may have encountered Testing Farm runs that
+fail with the status Failed to checkout ref {sha}. Together with the Testing Farm team, we are working on a fix.
+
+
+
+
+
+
+
+
+
Last updated:
+
+
+ July 22, 2024 at 9:41 AM UTC
+
+
+
+ Resolved after
+
+
+
+ 4h
+ 0m of downtime.
+
+
+
+
+ July 3, 2023 at 10:00 AM UTC
+
+
+
+
+
+
+
+
+
Some jobs submitted by Packit to the Testing Farm are pending with this error:
+"Guest couldn't be provisioned: Artemis resource ended in 'error' state".
+
Testing Farm has already resolved the issue; please re-submit the job with a
+comment like /packit test (more info on retriggering
+here)
+or through the GitHub checks button.
+
+
+
+
+
+
+
+
+
Last updated:
+
+
+ July 22, 2024 at 9:41 AM UTC
+
+
+
+ Resolved after
+
+
+
+ 3342h
+ 30m of downtime.
+
+
+
+
+ December 10, 2023 at 8:00 PM UTC
+
+
+
+
+
+
+
+
+
We have moved our staging instance to another OpenShift cluster which may result in some jobs failing.
+This issue should mainly affect jobs that involve syncing to the dist-git, namely:
+
+
propose-downstream
+
pull-from-upstream
+
+
In case you are using our staging instance and you hit any issues when running propose-downstream or pull-from-upstream, please let us know.
+
Known Issues
+
+
Testing Farm is currently not available on the staging instance.
+Edit: This issue has been fixed as of July 26th 10am.
+
+
+
+
+
+
+
+
+
+
Last updated:
+
+
+ July 22, 2024 at 9:41 AM UTC
+
+
+
+ Resolved after
+
+
+
+ 7h
+ 3m of downtime.
+
+
+
+
+ August 15, 2023 at 11:03 AM UTC
+
+
+
+
+
+
+
+
+
We are having an outage that started on Tuesday EU early morning. Jobs during this time could get stuck
+in pending state.
+This is caused by a failed database schema migration because of a lack of memory during the automatic deployment
+of the new production images. We are working on the fix and apologize for the inconvenience.
+
Update: Migration is completed. You might need to retrigger the jobs (i.e. via comment).
+
+
+
+
+
+
+
+
+
Last updated:
+
+
+ July 22, 2024 at 9:41 AM UTC
+
+
+
+ Resolved after
+
+
+
+ 2h
+ 0m of downtime.
+
+
+
+
+ August 16, 2023 at 4:30 PM UTC
+
+
+
+
+
+
+
+
+
There will be a Copr outage starting at
+
date --date "2023-08-16 12:30 UTC"
+
which will last approximately 2 hours. The build queue processing will be stopped. During the outage time, we expect a very short downtime for the DNF package storage (less than 10 minutes).
+
Reason for outage:
+
We will update the infrastructure machines to the latest packages that are currently being developed.
+
+ Resolved after
+
+
+
+ 20m
+ of downtime.
+
+
+
+
+ August 24, 2023 at 4:00 PM UTC
+
+
+
+
+
+
+
+
+
Testing Farm infrastructure had a small outage that caused few failing test runs.
+In case it affected you, please retrigger the tests via /packit test comment command in the pull requests.
+
+
+
+
+
+
+
+
+
Last updated:
+
+
+ July 22, 2024 at 9:41 AM UTC
+
+
+
+ Resolved after
+
+
+
+ 1h
+ 45m of downtime.
+
+
+
+
+ August 28, 2023 at 11:45 AM UTC
+
+
+
+
+
+
+
+
+
A system upgrade was scheduled and lasted approximately 90 minutes.
+
The started jobs might’ve been executed with a delay or dropped,
+if you think Packit does not react to your events, try to retrigger
+the action via
+comment command.
+
+
+
+
+
+
+
+
+
Last updated:
+
+
+ July 22, 2024 at 9:41 AM UTC
+
+
+
+ Resolved after
+
+
+
+ 51h
+ 0m of downtime.
+
+
+
+
+ September 11, 2023 at 3:00 PM UTC
+
+
+
+
+
+
+
+
+
Copr is experiencing issues with builds. We are sorry about that. You can follow the progress in the Copr issue.
+
The failure in the COPR logs looks like the following: Machine 5b8689e424334255ab9ace8a58343076 still running. Killing...
+
In case you are building in your own COPR project you can go to your project settings and change Mock bootstrap from default to Enable for a quick workaround.
+
+
+
+
+
+
+
+
+
Last updated:
+
+
+ July 22, 2024 at 9:41 AM UTC
+
+
+
+ Resolved after
+
+
+
+ 2h
+ 0m of downtime.
+
+
+
+
+ October 2, 2023 at 10:00 AM UTC
+
+
+
+
+
+
+
+
+
A cluster upgrade is scheduled to start on Mon October 2nd at 08:00 UTC and last approximately 90 minutes.
+
We do not expect any significant disruption.
+
The started jobs might be executed with a delay
+and if you think Packit does not react to your events,
+try to retrigger the action via
+comment command.
+
+
+
+
+
+
+
+
+
Last updated:
+
+
+ July 22, 2024 at 9:41 AM UTC
+
+
+
+ Resolved after
+
+
+
+ 7h
+ 15m of downtime.
+
+
+
+
+ November 14, 2023 at 5:15 PM UTC
+
+
+
+
+
+
+
+
+
Building on COPR for epel8 chroot will be broken today for several hours.
+Today is release day of RHEL 8.9 and everyone is seeing 404 not found error when trying to access
+RHEL 8 repositories.
+
+
+
+
+
+
+
+
+
Last updated:
+
+
+ July 22, 2024 at 9:41 AM UTC
+
+
+
+ Resolved after
+
+
+
+ 4h
+ 0m of downtime.
+
+
+
+
+ December 6, 2023 at 4:00 PM UTC
+
+
+
+
+
+
+
+
+
Copr builds could have been stuck in a pending or running state.
+The issue has been resolved, it may take a while for the affected Copr builds to finish,
+but there is no need to retrigger them.
+
+
+
+
+
+
+
+
+
Last updated:
+
+
+ July 22, 2024 at 9:41 AM UTC
+
+
+
+ Resolved after
+
+
+
+ 1596h
+ 15m of downtime.
+
+
+
+
+ February 15, 2024 at 8:00 AM UTC
+
+
+
+
+
+
+
+
+
We have moved our production instance to another OpenShift cluster which may result in some jobs failing.
+This issue should mainly affect jobs that involve syncing to the dist-git, namely:
+
+
propose-downstream
+
pull-from-upstream
+
+
In case you hit any issues when running propose-downstream or pull-from-upstream, please let us know.
+
Known Issues
+
+
none as of now (2023-12-11 17:00 UTC)
+
+
+
+
+
+
+
+
+
+
Last updated:
+
+
+ July 22, 2024 at 9:41 AM UTC
+
+
+
+ Resolved after
+
+
+
+ 101h
+ 50m of downtime.
+
+
+
+
+ January 15, 2024 at 3:20 PM UTC
+
+
+
+
+
+
+
+
+
Since 2024-01-11 until 2024-01-15 downstream Koji builds weren’t automatically triggered
+due to Fedora Infrastructure changes. Please trigger
+your builds manually if you are affected.
+
+
+
+
+
+
+
+
+
Last updated:
+
+
+ July 22, 2024 at 9:41 AM UTC
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
diff --git a/issues/2024-01-11-downstream-koji-builds-outage/index.json b/issues/2024-01-11-downstream-koji-builds-outage/index.json
new file mode 100644
index 00000000..851e4316
--- /dev/null
+++ b/issues/2024-01-11-downstream-koji-builds-outage/index.json
@@ -0,0 +1,13 @@
+{
+ "is": "issue",
+ "title": "Outage of automatic triggering of downstream Koji builds",
+ "createdAt": "2024-01-11 09:30:00 +0100 +0100",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2024-01-11-downstream-koji-builds-outage/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2024-01-15T15:20:00+01:00",
+ "affected": ["Downstream Koji builds"],
+ "filename": "2024-01-11-downstream-koji-builds-outage.md"
+}
\ No newline at end of file
diff --git a/issues/2024-01-23-fedora-auth-problem/index.html b/issues/2024-01-23-fedora-auth-problem/index.html
new file mode 100644
index 00000000..5e8d6846
--- /dev/null
+++ b/issues/2024-01-23-fedora-auth-problem/index.html
@@ -0,0 +1,657 @@
+
+
+
+
+
+
+
+
+
+ Fedora infrastructure authentication problem | Packit Service Status
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ Resolved after
+
+
+
+ 23h
+ 0m of downtime.
+
+
+
+
+ January 23, 2024 at 9:00 PM UTC
+
+
+
+
+
+
+
+
+
The Fedora account system is currently down, leading to failures of jobs that interact with it
+(mostly propose_downstream, pull_from_upstream, koji_build, bodhi_update)
+as Packit is unable to authenticate. To stay updated on the root cause of this problem, please
+track this issue.
+
The problem should be now resolved. You can retrigger the failed jobs by:
+
+
/packit propose-downstream comment in any upstream issue
+
/packit pull-from-upstream / /packit koji-build / /packit create-update comment in any dist-git pull-request
+
+
+
+
+
+
+
+
+
+
Last updated:
+
+
+ July 22, 2024 at 9:41 AM UTC
+
+
+
+ Resolved after
+
+
+
+ 3h
+ 0m of downtime.
+
+
+
+
+ February 22, 2024 at 12:00 AM UTC
+
+
+
+
+
+
+
+
+
Planned Outage:
+
Koji upgrade at 2024-02-21 21:00 UTC
+
There will be an outage starting at 2024-02-21 21:00 UTC,
+which will last approximately 3 hours.
+
Reason for outage:
+
Koji will be upgraded to 1.34.0, which requires a schema update that touches many rows.
+Koji team estimate this will take about 45minutes to complete and during that time,
+Koji service will be completely offline.
+
You should avoid to trigger Koji builds in Packit (e.g. merging distgit Packit pull requests).
+You can retrigger the build, if it is missing, commenting a pull request with:
+
/packit koji-build
+
+
+
+
+
+
+
+
+
Last updated:
+
+
+ July 22, 2024 at 9:41 AM UTC
+
+
+
+ Resolved after
+
+
+
+ 28m
+ of downtime.
+
+
+
+
+ February 26, 2024 at 1:28 PM UTC
+
+
+
+
+
+
+
+
+
There is a problem with authentication in Fedora Infrastructure, leading to failures of jobs that interact with it
+(mostly propose_downstream, pull_from_upstream, koji_build, bodhi_update)
+as Packit is unable to authenticate. To stay updated on the root cause of this problem, please
+track this issue.
+
The problem should be now resolved. You can retrigger the failed jobs by:
+
+
/packit propose-downstream comment in any upstream issue
+
/packit pull-from-upstream / /packit koji-build / /packit create-update comment in any dist-git pull-request
+
+
+
+
+
+
+
+
+
+
Last updated:
+
+
+ July 22, 2024 at 9:41 AM UTC
+
+
+
+ Resolved after
+
+
+
+ 7h
+ 0m of downtime.
+
+
+
+
+ April 23, 2024 at 9:00 PM UTC
+
+
+
+
+
+
+
+
+
The Copr backend is under a huge load and many builds are still in a pending state.
+The Copr builds will take more time than usual to finish.
+You can monitor the pending queue here: https://copr.fedorainfracloud.org/status/stats/.
+
+
+
+
+
+
+
+
+
Last updated:
+
+
+ July 22, 2024 at 9:41 AM UTC
+
+
+
+ Resolved after
+
+
+
+ 5h
+ 0m of downtime.
+
+
+
+
+ April 25, 2024 at 7:30 PM UTC
+
+
+
+
+
+
+
+
+
There has been a Packit outage. We are investigating the reason of it.
+The jobs in the queue are now being picked up again and should be processed gradually
+(you might have experienced a delay in job execution).
+We are very sorry for the inconvenience!
+
+
+
+
+
+
+
+
+
Last updated:
+
+
+ July 22, 2024 at 9:41 AM UTC
+
+
+
+ Resolved after
+
+
+
+ 3h
+ 0m of downtime.
+
+
+
+
+ May 7, 2024 at 4:00 PM UTC
+
+
+
+
+
+
+
+
+
There has been a Packit outage.
+The jobs in the queue are now being picked up again and should be processed gradually
+(you might have experienced a delay in job execution).
+We are very sorry for the inconvenience!
+
+
+
+
+
+
+
+
+
Last updated:
+
+
+ July 22, 2024 at 9:41 AM UTC
+
+
+
+ Resolved after
+
+
+
+ 177h
+ 30m of downtime.
+
+
+
+
+ May 16, 2024 at 2:30 PM UTC
+
+
+
+
+
+
+
+
+
There was a New Hotness outage resulting in no Bugzillas about new release being filed.
+As a result, no Packit update dist-git pull requests were created.
+
Please trigger
+the release syncs manually if you are affected by commenting in any dist-git pull request:
+
/packit pull-from-upstream
+
+
We are very sorry for the inconvenience!
+
+
+
+
+
+
+
+
+
Last updated:
+
+
+ July 22, 2024 at 9:41 AM UTC
+
+
+
+ Resolved after
+
+
+
+ 4h
+ 15m of downtime.
+
+
+
+
+ May 27, 2024 at 6:35 PM UTC
+
+
+
+
+
+
+
+
+
One of dependencies for our Fedora Messaging consumer has not been installed
+during the image build which resulted in an outage of the Fedora downstream
+integration.
+
Fedora integration is disrupted:
+
+
since 2pm CEST for our stage deployment
+
since 6pm CEST for our production deployment
+
+
Please trigger the respective jobs manually, if you are affected, once the outage has been resolved.
+
We are very sorry for the inconvenience!
+
Edit #1: Issue has been resolved for both production and staging deployments
+by rolling back to the previous image builds.
+
+
+
+
+
+
+
+
+
Last updated:
+
+
+ July 22, 2024 at 9:41 AM UTC
+
+
+
+ Resolved after
+
+
+
+ 175h
+ 0m of downtime.
+
+
+
+
+ July 3, 2024 at 4:00 PM UTC
+
+
+
+
+
+
+
+
+
There is a New Hotness outage resulting in no Bugzillas about new releases being filed.
+As a result, no Packit update dist-git pull requests will be created.
+
There is a workaround: edit your release-monitoring.org project and tick the Check latest release on submit checkbox.
+
Update: It seems that the issue has been resolved and new release bugzillas are being filed automatically again.
+
+
+
+
+
+
+
+
+
Last updated:
+
+
+ July 22, 2024 at 9:41 AM UTC
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
diff --git a/issues/index.json b/issues/index.json
new file mode 100644
index 00000000..54d508d2
--- /dev/null
+++ b/issues/index.json
@@ -0,0 +1,1156 @@
+{
+ "is": "issues",
+ "title": "Issues",
+ "baseURL": "https://status.packit.dev",
+ "description": "",
+ "pages": [
+ {
+ "is": "issue",
+ "title": "Testing Farm outage",
+ "createdAt": "2024-07-22 00:16:00 +0000 UTC",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2024-07-22-tf-outage/",
+ "severity": "disrupted",
+ "resolved": false,
+ "informational": false,
+ "resolvedAt": "",
+ "affected": ["Testing Farm"],
+ "filename": "2024-07-22-tf-outage.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "release-monitoring.org outage",
+ "createdAt": "2024-06-26 08:00:00 +0100 +0100",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2024-07-02-release-monitoring.org-outage/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2024-07-03T16:00:00+02:00",
+ "affected": ["Pull from upstream"],
+ "filename": "2024-07-02-release-monitoring.org-outage.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Fedora integration outage",
+ "createdAt": "2024-05-27 14:20:00 +0200 +0200",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2024-05-27-fedora-outage/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2024-05-27T18:35:00+02:00",
+ "affected": ["Pull from upstream", "Downstream Koji builds", "Bodhi updates"],
+ "filename": "2024-05-27-fedora-outage.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "New Hotness outage ",
+ "createdAt": "2024-05-09 03:00:00 +0000 UTC",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2024-05-16-new-hotness-outage/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2024-05-16T14:30:00+02:00",
+ "affected": ["Pull from upstream"],
+ "filename": "2024-05-16-new-hotness-outage.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Outage ",
+ "createdAt": "2024-05-07 13:00:00 +0200 +0200",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2024-05-07-outage/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2024-05-07T16:00:00+02:00",
+ "affected": ["all"],
+ "filename": "2024-05-07-outage.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Outage ",
+ "createdAt": "2024-04-25 14:30:00 +0200 +0200",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2024-04-25-outage/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2024-04-25T19:30:00+02:00",
+ "affected": ["all"],
+ "filename": "2024-04-25-outage.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Many Copr builds pending",
+ "createdAt": "2024-04-23 14:00:00 +0100 +0100",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2024-04-23-copr-pending-queue/",
+ "severity": "notice",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2024-04-23T21:00:00+01:00",
+ "affected": ["Copr"],
+ "filename": "2024-04-23-copr-pending-queue.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "GitHub outage",
+ "createdAt": "2024-04-05 10:15:00 +0100 +0100",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2024-04-05-github-outage/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2024-04-05T11:15:00+01:00",
+ "affected": ["GitHub"],
+ "filename": "2024-04-05-github-outage.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Fedora infrastructure authentication problem",
+ "createdAt": "2024-02-26 13:00:00 +0000 UTC",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2024-02-26-fedora-auth-problem/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2024-02-26T13:28:00+00:00",
+ "affected": ["Pull from upstream", "Propose downstream", "Koji", "Bodhi"],
+ "filename": "2024-02-26-fedora-auth-problem.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Koji Outage",
+ "createdAt": "2024-02-21 21:00:00 +0000 UTC",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2024-02-21-koji-outage/",
+ "severity": "notice",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2024-02-22T00:00:00+00:00",
+ "affected": ["Koji builds"],
+ "filename": "2024-02-21-koji-outage.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Pull From Upstream Outage",
+ "createdAt": "2024-02-21 00:00:00 +0000 UTC",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2024-02-22-pull-from-upstream-outage/",
+ "severity": "notice",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2024-02-22T14:40:00+00:00",
+ "affected": ["Pull from upstream"],
+ "filename": "2024-02-22-pull-from-upstream-outage.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Fedora infrastructure authentication problem",
+ "createdAt": "2024-01-22 22:00:00 +0100 +0100",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2024-01-23-fedora-auth-problem/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2024-01-23T21:00:00+01:00",
+ "affected": ["Workers", "Koji", "Bodhi"],
+ "filename": "2024-01-23-fedora-auth-problem.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Outage of automatic triggering of downstream Koji builds",
+ "createdAt": "2024-01-11 09:30:00 +0100 +0100",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2024-01-11-downstream-koji-builds-outage/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2024-01-15T15:20:00+01:00",
+ "affected": ["Downstream Koji builds"],
+ "filename": "2024-01-11-downstream-koji-builds-outage.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "GitHub outage",
+ "createdAt": "2024-01-09 14:45:00 +0100 +0100",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2024-01-09-github-outage/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2024-11-09T15:40:00+01:00",
+ "affected": ["GitHub"],
+ "filename": "2024-01-09-GitHub-outage.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Christmas is here ๐",
+ "createdAt": "2023-12-22 19:00:00 +0100 +0100",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-12-23-christmas-2023/",
+ "severity": "notice",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2024-01-02T09:00:00+01:00",
+ "affected": [],
+ "filename": "2023-12-23-christmas-2023.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "GitLab outage",
+ "createdAt": "2023-12-19 05:30:00 +0100 +0100",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-12-19-gitlab-outage/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-12-20T12:00:00+01:00",
+ "affected": ["API"],
+ "filename": "2023-12-19-gitlab-outage.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Production instance firewall",
+ "createdAt": "2023-12-10 20:45:00 +0100 +0100",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-12-10-prod-fw/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2024-02-15T08:00:00+00:00",
+ "affected": [],
+ "filename": "2023-12-10-prod-fw.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Testing Farm outage",
+ "createdAt": "2023-12-10 20:40:00 +0100 +0100",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-12-10-tf-outage/",
+ "severity": "down",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-12-11T12:00:00+01:00",
+ "affected": ["Testing Farm"],
+ "filename": "2023-12-10-tf-outage.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Production move",
+ "createdAt": "2023-12-10 20:15:00 +0100 +0100",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-12-10-mpp-move/",
+ "severity": "notice",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-12-11T18:00:00+01:00",
+ "affected": [],
+ "filename": "2023-12-10-mpp-move.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Copr builds stuck in pending or running state",
+ "createdAt": "2023-12-06 12:00:00 +0100 +0100",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-12-06-copr-outage/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-12-06T16:00:00+01:00",
+ "affected": ["Copr"],
+ "filename": "2023-12-06-copr-outage.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Copr Upgrade",
+ "createdAt": "2023-11-28 13:30:00 +0000 UTC",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-11-28-copr-upgrade/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-11-28T16:45:00+00:00",
+ "affected": ["Copr"],
+ "filename": "2023-11-28-copr-upgrade.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "COPR builds failing for epel8",
+ "createdAt": "2023-11-14 10:00:00 +0100 +0100",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-11-14-copr-outage-for-epel8/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-11-14T17:15:00+01:00",
+ "affected": [],
+ "filename": "2023-11-14-copr-outage-for-epel8.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Production move",
+ "createdAt": "2023-11-12 15:00:00 +0100 +0100",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-11-12-mpp-move/",
+ "severity": "notice",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-11-12T17:15:00+01:00",
+ "affected": [],
+ "filename": "2023-11-12-mpp-move.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "GitHub outage",
+ "createdAt": "2023-11-07 14:45:00 +0100 +0100",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-11-07-github-outage/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-11-07T15:30:00+01:00",
+ "affected": ["GitHub"],
+ "filename": "2023-11-07-GitHub-outage.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "GitLab.com outage",
+ "createdAt": "2023-10-10 04:00:00 +0200 +0200",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-10-10-gitlab-outage/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-10-11T14:00:00+02:00",
+ "affected": ["Copr", "Testing Farm"],
+ "filename": "2023-10-10-gitlab-outage.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Cluster upgrade",
+ "createdAt": "2023-10-02 08:00:00 +0000 UTC",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-10-02-cluster-upgrade/",
+ "severity": "notice",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-10-02T10:00:00+00:00",
+ "affected": ["API", "Workers", "Dashboard"],
+ "filename": "2023-10-02-cluster-upgrade.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "COPR build failures",
+ "createdAt": "2023-09-09 12:00:00 +0000 UTC",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-09-09-copr-outage/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-09-11T15:00:00+00:00",
+ "affected": ["Copr"],
+ "filename": "2023-09-09-copr-outage.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Copr frontend instability",
+ "createdAt": "2023-09-05 15:15:00 +0200 +0200",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-09-05-copr-outage/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-04-05T15:00:00+00:00",
+ "affected": ["Copr"],
+ "filename": "2023-09-05-copr-outage.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "System upgrade scheduled",
+ "createdAt": "2023-08-28 10:00:00 +0200 +0200",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-08-28-cluster-upgrade/",
+ "severity": "notice",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-08-28T11:45:00+02:00",
+ "affected": [],
+ "filename": "2023-08-28-cluster-upgrade.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Testing Farm outage",
+ "createdAt": "2023-08-24 15:40:00 +0200 +0200",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-08-24-tf-outage/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-08-24T16:00:00+02:00",
+ "affected": ["Testing Farm"],
+ "filename": "2023-08-24-tf-outage.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Packit outage now resolved",
+ "createdAt": "2023-08-21 14:45:00 +0200 +0200",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-08-21-packit-outage/",
+ "severity": "notice",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-08-21T15:15:00+02:00",
+ "affected": ["all"],
+ "filename": "2023-08-21-packit-outage.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Copr Upgrade",
+ "createdAt": "2023-08-16 14:30:00 +0200 +0200",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-08-16-copr-upgrade/",
+ "severity": "down",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-08-16T16:30:00+02:00",
+ "affected": ["Copr"],
+ "filename": "2023-08-16-copr-upgrade.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Outage",
+ "createdAt": "2023-08-15 04:00:00 +0200 +0200",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-08-15-outage/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-08-15T11:03:00+02:00",
+ "affected": ["API", "Workers"],
+ "filename": "2023-08-15-outage.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Staging instance firewall",
+ "createdAt": "2023-07-24 14:30:00 +0200 +0200",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-07-24-stage-fw/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-12-10T20:00:00+01:00",
+ "affected": [],
+ "filename": "2023-07-24-stage-fw.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Testing Farm testing blocked",
+ "createdAt": "2023-07-19 08:00:00 +0000 UTC",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-07-19-testing-farm-failing-on-rawhide/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-07-19T11:00:00+00:00",
+ "affected": ["Testing Farm"],
+ "filename": "2023-07-19-testing-farm-failing-on-rawhide.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Guest couldn't be provisioned: Artemis resource ended in 'error' state",
+ "createdAt": "2023-07-03 08:00:00 +0200 +0200",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-07-03-testing-provisioning-failed/",
+ "severity": "notice",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-07-03T10:00:00+00:00",
+ "affected": ["Testing Farm"],
+ "filename": "2023-07-03-testing-provisioning-failed.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Fedora Rawhide Testing Blocked on dnf5",
+ "createdAt": "2023-06-22 09:30:00 +0200 +0200",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-06-22-fedora-rawhide-testing-blocked-on-dnf5/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-07-18T20:30:00+02:00",
+ "affected": ["Testing Farm"],
+ "filename": "2023-06-22-fedora-rawhide-testing-blocked-on-dnf5.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Upgrade of Copr servers",
+ "createdAt": "2023-05-25 10:00:00 +0000 UTC",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-05-25-copr-upgrade/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-05-25T13:00:00+00:00",
+ "affected": ["Copr"],
+ "filename": "2023-05-25-copr-upgrade.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "GitHub outage",
+ "createdAt": "2023-05-11 15:30:00 +0200 +0200",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-05-11-github-outage/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-05-11T21:00:00+02:00",
+ "affected": ["GitHub"],
+ "filename": "2023-05-11-GitHub-outage.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Testing Farm errors",
+ "createdAt": "2023-05-10 14:00:00 +0200 +0200",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-05-11-tf-merge-issue/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-05-11T15:00:00+02:00",
+ "affected": ["Testing Farm"],
+ "filename": "2023-05-11-tf-merge-issue.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "GitHub outage",
+ "createdAt": "2023-05-09 13:50:00 +0200 +0200",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-05-09-github-outage/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-05-09T23:15:00+02:00",
+ "affected": ["GitHub"],
+ "filename": "2023-05-09-GitHub-outage.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Bodhi Updates not created",
+ "createdAt": "2023-05-04 11:30:00 +0200 +0200",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-05-05-bodhi-updates/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-05-10T16:00:00+02:00",
+ "affected": ["Bodhi"],
+ "filename": "2023-05-05-Bodhi-updates.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Cluster Update",
+ "createdAt": "2023-05-03 10:00:00 +0200 +0200",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-05-03-cluster-update/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-05-03T11:15:00+02:00",
+ "affected": ["API", "Workers", "Dashboard"],
+ "filename": "2023-05-03-cluster-update.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Scheduled DB Cleanup",
+ "createdAt": "2023-04-26 04:40:00 +0000 UTC",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-04-26-scheduled-db-cleanup/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-04-26T05:17:00+00:00",
+ "affected": ["API", "Dashboard"],
+ "filename": "2023-04-26-scheduled-db-cleanup.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Issue with pushing to dist-git",
+ "createdAt": "2023-04-11 18:28:00 +0200 +0200",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-04-12-push-to-distgit-issue/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-04-12T16:30:00+02:00",
+ "affected": ["Workers"],
+ "filename": "2023-04-12-push-to-distgit-issue.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Upgrade of Copr servers",
+ "createdAt": "2023-04-05 12:00:00 +0000 UTC",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-04-05-copr-outage/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-04-05T15:00:00+00:00",
+ "affected": ["Copr"],
+ "filename": "2023-04-05-copr-outage.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "SLO1 Error Budget issues",
+ "createdAt": "2023-03-09 10:00:00 +0100 +0100",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-03-09-slo1-error-budget/",
+ "severity": "notice",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-06-08T10:00:00+01:00",
+ "affected": ["API", "Workers", "Dashboard"],
+ "filename": "2023-03-09-slo1-error-budget.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Running grouping migrations",
+ "createdAt": "2023-02-14 02:00:00 +0100 +0100",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-02-14-running-grouping-migrations/",
+ "severity": "down",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-02-14T04:00:00+01:00",
+ "affected": ["API", "Workers", "Dashboard", "Copr", "Testing Farm", "Koji"],
+ "filename": "2023-02-14-running-grouping-migrations.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "SLO2 Error Budget Depleted in a Day",
+ "createdAt": "2023-01-31 10:00:00 +0100 +0100",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-02-17-slo2-error-budget-depleted-in-a-day/",
+ "severity": "notice",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-02-01T10:00:00+01:00",
+ "affected": ["Dashboard"],
+ "filename": "2023-02-17-slo2-error-budget-depleted-in-a-day.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Testing Farm outage",
+ "createdAt": "2023-01-24 15:00:00 +0100 +0100",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-01-24-tf-outage/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-01-25T10:00:00+01:00",
+ "affected": ["Testing Farm"],
+ "filename": "2023-01-24-tf-outage.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "SRPM builds stuck in Submitted state",
+ "createdAt": "2023-01-20 21:19:06 +0100 +0100",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-01-21-srpm-builds-stuck-in-submitted-state/",
+ "severity": "down",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-01-21T9:51:30+01:00",
+ "affected": ["Workers"],
+ "filename": "2023-01-21-srpm-builds-stuck-in-submitted-state.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "System upgrade scheduled",
+ "createdAt": "2023-01-06 10:00:00 +0000 UTC",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-01-06-cluster-upgrade/",
+ "severity": "notice",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-01-06T22:00:00+00:00",
+ "affected": [],
+ "filename": "2023-01-06-cluster-upgrade.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Christmas is here ๐",
+ "createdAt": "2022-12-23 18:00:00 +0000 UTC",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2022-12-23-christmas-2022/",
+ "severity": "notice",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2023-01-02T08:00:00+00:00",
+ "affected": [],
+ "filename": "2022-12-23-christmas-2022.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Upgrade of Copr Servers",
+ "createdAt": "2022-11-28 16:00:00 +0100 +0100",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2022-11-28-copr-outage/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2022-11-28T21:00:00+01:00",
+ "affected": ["Copr"],
+ "filename": "2022-11-28-copr-outage.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Staging is unstable",
+ "createdAt": "2022-11-25 08:35:00 +0100 +0100",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2022-11-28-staging-unstable/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2022-11-30T14:06:00+01:00",
+ "affected": [],
+ "filename": "2022-11-28-staging-unstable.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Outage",
+ "createdAt": "2022-11-24 23:24:00 +0100 +0100",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2022-11-25-outage/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2022-11-25T08:35:00+01:00",
+ "affected": ["API", "Workers"],
+ "filename": "2022-11-25-outage.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "System upgrade scheduled",
+ "createdAt": "2022-11-24 19:00:00 +0000 UTC",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2022-11-24-cluster-upgrade/",
+ "severity": "notice",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2022-11-24T23:00:00+00:00",
+ "affected": [],
+ "filename": "2022-11-24-cluster-upgrade.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Copr builds and tests on commits and releases ignored",
+ "createdAt": "2022-11-01 03:00:00 +0100 +0100",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2022-11-01-commit-release-trigger/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2022-11-02T10:30:00+01:00",
+ "affected": ["Workers"],
+ "filename": "2022-11-01-commit-release-trigger.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Copr storage move",
+ "createdAt": "2022-10-28 21:00:00 +0200 +0200",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2022-10-28-30-copr-storage-move/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2022-10-30T08:00:00+02:00",
+ "affected": ["Copr"],
+ "filename": "2022-10-28-30-copr-storage-move.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "System upgrade scheduled",
+ "createdAt": "2022-10-13 19:00:00 +0000 UTC",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2022-10-13-cluster-upgrade/",
+ "severity": "notice",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2022-10-13T20:37:00+00:00",
+ "affected": [],
+ "filename": "2022-10-13-cluster-upgrade.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Issues with the task execution",
+ "createdAt": "2022-10-11 13:57:00 +0200 +0200",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2022-10-11-outage/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2022-10-11T15:12:00+02:00",
+ "affected": ["Workers", "Copr"],
+ "filename": "2022-10-11-outage.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Internal Testing Farm infrastructure is having an outage",
+ "createdAt": "2022-10-10 15:14:00 +0200 +0200",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2022-10-10-outage/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2022-10-11T21:34:00+02:00",
+ "affected": ["Testing Farm"],
+ "filename": "2022-10-10-outage.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "GitHub Webhooks not working",
+ "createdAt": "2022-10-05 16:00:00 +0200 +0200",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2022-10-05-github-webhooks-degraded-status/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2022-10-05T18:00:00+02:00",
+ "affected": ["API"],
+ "filename": "2022-10-05-github-webhooks-degraded-status.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "September 22th outage",
+ "createdAt": "2022-09-22 08:00:00 +0200 +0200",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2022-09-22-outage/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2022-09-23T08:00:00+02:00",
+ "affected": ["Testing Farm"],
+ "filename": "2022-09-22-outage.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "3/4 of SLO1 Error Budget consumed in 6 weeks",
+ "createdAt": "2022-09-01 01:00:00 +0200 +0200",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2022-10-27-slo1-error-budget-drop-postmortem/",
+ "severity": "notice",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2022-10-12T01:00:00+02:00",
+ "affected": [],
+ "filename": "2022-10-27-slo1-error-budget-drop-postmortem.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Failing SRPM builds done in Copr",
+ "createdAt": "2022-08-23 04:00:00 +0200 +0200",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2022-08-23-srpm-builds-failing/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2022-08-23T09:00:00+02:00",
+ "affected": ["Workers"],
+ "filename": "2022-08-23-srpm-builds-failing.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Summer is here โ๏ธ",
+ "createdAt": "2022-07-04 08:00:00 +0000 UTC",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2022-07-01-team-on-vacation/",
+ "severity": "notice",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2022-07-08T18:00:00+00:00",
+ "affected": [],
+ "filename": "2022-07-01-team-on-vacation.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Upgrade of Copr Servers",
+ "createdAt": "2022-06-22 14:00:00 +0200 +0200",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2022-06-22-upgrade-of-copr-servers/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2022-06-22T16:10:00+02:00",
+ "affected": ["Copr"],
+ "filename": "2022-06-22-upgrade-of-copr-servers.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "SRPM builds done in Copr are failing",
+ "createdAt": "2022-06-21 10:18:22 +0200 +0200",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2022-06-21-srpm-builds-failing/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2022-06-21T11:10:22+02:00",
+ "affected": ["Workers"],
+ "filename": "2022-06-21-srpm-builds-failing.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Networking Issue",
+ "createdAt": "2022-02-03 08:37:21 +0100 +0100",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2022-02-03-networking-issue/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2022-02-03T11:17:21+01:00",
+ "affected": [],
+ "filename": "2022-02-03-networking-issue.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "February 1st outage",
+ "createdAt": "2022-02-01 02:00:00 +0000 UTC",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2022-02-01-outage/",
+ "severity": "notice",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2022-02-01T04:15:00+00:00",
+ "affected": [],
+ "filename": "2022-02-01-outage.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "New production deployment",
+ "createdAt": "2021-12-08 02:00:00 +0000 UTC",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2021-12-08-new-production-deployment/",
+ "severity": "notice",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2022-01-01T00:00:00+00:00",
+ "affected": [],
+ "filename": "2021-12-08-new-production-deployment.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Longer response times",
+ "createdAt": "2021-12-07 09:06:14 +0100 +0100",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2021-12-07-response-delays/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2021-12-07T11:49:14+01:00",
+ "affected": ["workers"],
+ "filename": "2021-12-07-response-delays.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Problems with production database",
+ "createdAt": "2021-11-18 02:33:00 +0000 UTC",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2021-11-18-production-db-problems/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2021-11-19 00:10:00",
+ "affected": ["Dashboard"],
+ "filename": "2021-11-18-production-db-problems.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Moving production Packit Service to a new cluster",
+ "createdAt": "2021-11-18 01:00:00 +0000 UTC",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2021-11-16-auto-prod/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2021-11-18 02:33:00",
+ "affected": ["API", "Workers", "Dashboard"],
+ "filename": "2021-11-16-auto-prod.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Slow SRPM builds",
+ "createdAt": "2021-09-07 20:30:00 +0200 +0200",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2021-09-08-outage/",
+ "severity": "notice",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2021-09-16T13:54:56+00:00",
+ "affected": [],
+ "filename": "2021-09-08-outage.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "August 27th outage",
+ "createdAt": "2021-08-27 15:00:00 +0200 +0200",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2021-08-27-outage/",
+ "severity": "notice",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2021-08-28 08:00:00+00:00",
+ "affected": [],
+ "filename": "2021-08-27-outage.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Summer is here โ๏ธ #2",
+ "createdAt": "2021-07-01 10:55:00 +0200 +0200",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2021-07-56-vacation/",
+ "severity": "notice",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2021-07-06 19:10:40+00:00",
+ "affected": [],
+ "filename": "2021-07-56-vacation.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Summer is here โ๏ธ",
+ "createdAt": "2021-06-25 10:55:00 +0200 +0200",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2021-06-28-vacation/",
+ "severity": "notice",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2021-06-28 19:10:40+00:00",
+ "affected": [],
+ "filename": "2021-06-28-vacation.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "PostgreSQL db upgrade",
+ "createdAt": "2021-06-04 07:00:00 +0000 UTC",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2021-06-04-postgresql-12/",
+ "severity": "down",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2021-06-04 09:10:40+00:00",
+ "affected": ["Workers", "Dashboard"],
+ "filename": "2021-06-04-postgresql-12.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "Dashboard - unable to connect",
+ "createdAt": "2021-06-02 11:16:00 +0000 UTC",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2021-06-02-dashboard-unable-to-connect/",
+ "severity": "disrupted",
+ "resolved": true,
+ "informational": false,
+ "resolvedAt": "2021-06-02 11:40:00",
+ "affected": ["API", "Dashboard"],
+ "filename": "2021-06-02-dashboard-unable-to-connect.md"
+ }
+ ,
+ {
+ "is": "issue",
+ "title": "",
+ "createdAt": "0001-01-01 00:00:00 +0000 UTC",
+ "lastMod": "2024-07-22 09:41:48 +0200 +0200",
+ "permalink": "https://status.packit.dev/issues/2023-10-31-release-monitoring.org-outage/",
+ "severity": "",
+ "resolved": false,
+ "informational": false,
+ "resolvedAt": "",
+ "affected": [],
+ "filename": "2023-10-31-release-monitoring.org-outage.md"
+ }
+
+ ]
+}
diff --git a/issues/index.xml b/issues/index.xml
new file mode 100644
index 00000000..73f28360
--- /dev/null
+++ b/issues/index.xml
@@ -0,0 +1,1318 @@
+
+
+
+
+ Issues on Packit Service Status
+ https://status.packit.dev/issues/
+ Incident history
+ github.com/cstate
+ en
+
+ 2024-07-22T00:16:00+00:00
+ 2024-07-22T00:16:00+00:00
+
+
+
+
+
+
+
+ Testing Farm outage
+ https://status.packit.dev/issues/2024-07-22-tf-outage/
+ Mon, 22 Jul 2024 00:16:00 +0000
+ https://status.packit.dev/issues/2024-07-22-tf-outage/
+
+ Testing Farm infrastructure is having an outage resulting in tests stuck in a queued state. You can watch the status of the outage here: https://status.testing-farm.io/issues/2024-07-22-public-ranch-outage/
+The tests should be executed once the outage is resolved.
+ <p>Testing Farm infrastructure is having an outage resulting in tests stuck in a queued state.
+You can watch the status of the outage here: <a href="https://status.testing-farm.io/issues/2024-07-22-public-ranch-outage/">https://status.testing-farm.io/issues/2024-07-22-public-ranch-outage/</a></p>
+<p>The tests should be executed once the outage is resolved.</p>
+
+
+
+
+ [Resolved] release-monitoring.org outage
+ https://status.packit.dev/issues/2024-07-02-release-monitoring.org-outage/
+ Wed, 26 Jun 2024 08:00:00 +0100
+ https://status.packit.dev/issues/2024-07-02-release-monitoring.org-outage/
+ 2024-07-03T16:00:00+02:00
+ There is a New Hotness outage resulting in no Bugzillas about new releases being filed. As a result, no Packit update dist-git pull requests will be created.
+There is a workaround: edit your release-monitoring.org project and tick the Check latest release on submit checkbox.
+Update: It seems that the issue has been resolved and new release bugzillas are being filed automatically again.
+ <p>There is a New Hotness outage resulting in no Bugzillas about new releases being filed.
+As a result, no Packit update dist-git pull requests will be created.</p>
+<p>There is a workaround: edit your release-monitoring.org project and tick the <code>Check latest release on submit</code> checkbox.</p>
+<p><em>Update</em>: It seems that the issue has been resolved and new release bugzillas are being filed automatically again.</p>
+
+
+
+
+ [Resolved] Fedora integration outage
+ https://status.packit.dev/issues/2024-05-27-fedora-outage/
+ Mon, 27 May 2024 14:20:00 +0200
+ https://status.packit.dev/issues/2024-05-27-fedora-outage/
+ 2024-05-27T18:35:00+02:00
+ One of dependencies for our Fedora Messaging consumer has not been installed during the image build which resulted in an outage of the Fedora downstream integration.
+Fedora integration is disrupted:
+ since 2pm CEST for our stage deployment since 6pm CEST for our production deployment Please trigger the respective jobs manually, if you are affected, once the outage has been resolved.
+We are very sorry for the inconvenience!
+Edit #1: Issue has been resolved for both production and staging deployments by rolling back to the previous image builds.
+ <p>One of dependencies for our Fedora Messaging consumer has not been installed
+during the image build which resulted in an outage of the Fedora downstream
+integration.</p>
+<p>Fedora integration is disrupted:</p>
+<ul>
+<li>since 2pm CEST for our stage deployment</li>
+<li>since 6pm CEST for our production deployment</li>
+</ul>
+<p>Please trigger the respective jobs manually, if you are affected, once the outage has been resolved.</p>
+<p>We are very sorry for the inconvenience!</p>
+<p><em>Edit #1</em>: Issue has been resolved for both production and staging deployments
+by rolling back to the previous image builds.</p>
+
+
+
+
+ [Resolved] New Hotness outage
+ https://status.packit.dev/issues/2024-05-16-new-hotness-outage/
+ Thu, 09 May 2024 03:00:00 +0000
+ https://status.packit.dev/issues/2024-05-16-new-hotness-outage/
+ 2024-05-16T14:30:00+02:00
+ There was a New Hotness outage resulting in no Bugzillas about new release being filed. As a result, no Packit update dist-git pull requests were created.
+Please trigger the release syncs manually if you are affected by commenting in any dist-git pull request:
+/packit pull-from-upstream We are very sorry for the inconvenience!
+ <p>There was a New Hotness outage resulting in no Bugzillas about new release being filed.
+As a result, no Packit update dist-git pull requests were created.</p>
+<p>Please <a href="https://packit.dev/docs/fedora-releases-guide/dist-git-onboarding#pull_from_upstream">trigger</a>
+the release syncs manually if you are affected by commenting in any dist-git pull request:</p>
+<pre><code>/packit pull-from-upstream
+</code></pre>
+<p>We are very sorry for the inconvenience!</p>
+
+
+
+
+ [Resolved] Outage
+ https://status.packit.dev/issues/2024-05-07-outage/
+ Tue, 07 May 2024 13:00:00 +0200
+ https://status.packit.dev/issues/2024-05-07-outage/
+ 2024-05-07T16:00:00+02:00
+ There has been a Packit outage. The jobs in the queue are now being picked up again and should be processed gradually (you might have experienced a delay in job execution). We are very sorry for the inconvenience!
+ <p>There has been a Packit outage.
+The jobs in the queue are now being picked up again and should be processed gradually
+(you might have experienced a delay in job execution).
+We are very sorry for the inconvenience!</p>
+
+
+
+
+ [Resolved] Outage
+ https://status.packit.dev/issues/2024-04-25-outage/
+ Thu, 25 Apr 2024 14:30:00 +0200
+ https://status.packit.dev/issues/2024-04-25-outage/
+ 2024-04-25T19:30:00+02:00
+ There has been a Packit outage. We are investigating the reason of it. The jobs in the queue are now being picked up again and should be processed gradually (you might have experienced a delay in job execution). We are very sorry for the inconvenience!
+ <p>There has been a Packit outage. We are investigating the reason of it.
+The jobs in the queue are now being picked up again and should be processed gradually
+(you might have experienced a delay in job execution).
+We are very sorry for the inconvenience!</p>
+
+
+
+
+ [Resolved] Many Copr builds pending
+ https://status.packit.dev/issues/2024-04-23-copr-pending-queue/
+ Tue, 23 Apr 2024 14:00:00 +0100
+ https://status.packit.dev/issues/2024-04-23-copr-pending-queue/
+ 2024-04-23T21:00:00+01:00
+ The Copr backend is under a huge load and many builds are still in a pending state. The Copr builds will take more time than usual to finish. You can monitor the pending queue here: https://copr.fedorainfracloud.org/status/stats/.
+ <p>The Copr backend is under a huge load and many builds are still in a pending state.
+The Copr builds will take more time than usual to finish.
+You can monitor the pending queue here: <a href="https://copr.fedorainfracloud.org/status/stats/">https://copr.fedorainfracloud.org/status/stats/</a>.</p>
+
+
+
+
+ [Resolved] GitHub outage
+ https://status.packit.dev/issues/2024-04-05-github-outage/
+ Fri, 05 Apr 2024 10:15:00 +0100
+ https://status.packit.dev/issues/2024-04-05-github-outage/
+ 2024-04-05T11:15:00+01:00
+ There is an ongoing GitHub outage. You can monitor the status and subscribe for updates here: https://www.githubstatus.com
+ <p>There is an ongoing GitHub outage.
+You can monitor the status and subscribe for updates here: <a href="https://www.githubstatus.com">https://www.githubstatus.com</a></p>
+
+
+
+
+ [Resolved] Fedora infrastructure authentication problem
+ https://status.packit.dev/issues/2024-02-26-fedora-auth-problem/
+ Mon, 26 Feb 2024 13:00:00 +0000
+ https://status.packit.dev/issues/2024-02-26-fedora-auth-problem/
+ 2024-02-26T13:28:00+00:00
+ There is a problem with authentication in Fedora Infrastructure, leading to failures of jobs that interact with it (mostly propose_downstream, pull_from_upstream, koji_build, bodhi_update) as Packit is unable to authenticate. To stay updated on the root cause of this problem, please track this issue.
+The problem should be now resolved. You can retrigger the failed jobs by:
+ /packit propose-downstream comment in any upstream issue /packit pull-from-upstream / /packit koji-build / /packit create-update comment in any dist-git pull-request
+ <p>There is a problem with authentication in Fedora Infrastructure, leading to failures of jobs that interact with it
+(mostly <code>propose_downstream</code>, <code>pull_from_upstream</code>, <code>koji_build</code>, <code>bodhi_update</code>)
+as Packit is unable to authenticate. To stay updated on the root cause of this problem, please
+track <a href="https://pagure.io/fedora-infrastructure/issue/11793">this issue</a>.</p>
+<p>The problem should be now resolved. You can retrigger the failed jobs by:</p>
+<ul>
+<li><code>/packit propose-downstream</code> comment in any upstream issue</li>
+<li><code>/packit pull-from-upstream</code> / <code>/packit koji-build</code> / <code>/packit create-update</code> comment in any dist-git pull-request</li>
+</ul>
+
+
+
+
+ [Resolved] Koji Outage
+ https://status.packit.dev/issues/2024-02-21-koji-outage/
+ Wed, 21 Feb 2024 21:00:00 +0000
+ https://status.packit.dev/issues/2024-02-21-koji-outage/
+ 2024-02-22T00:00:00+00:00
+ Planned Outage:
+Koji upgrade at 2024-02-21 21:00 UTC
+There will be an outage starting at 2024-02-21 21:00 UTC, which will last approximately 3 hours.
+Reason for outage:
+Koji will be upgraded to 1.34.0, which requires a schema update that touches many rows. Koji team estimate this will take about 45minutes to complete and during that time, Koji service will be completely offline.
+You should avoid to trigger Koji builds in Packit (e.
+ <p>Planned Outage:</p>
+<p>Koji upgrade at 2024-02-21 21:00 UTC</p>
+<p>There will be an outage starting at 2024-02-21 21:00 UTC,
+which will last approximately 3 hours.</p>
+<p>Reason for outage:</p>
+<p>Koji will be upgraded to 1.34.0, which requires a schema update that touches many rows.
+Koji team estimate this will take about 45minutes to complete and during that time,
+Koji service will be completely offline.</p>
+<p>You should avoid to trigger Koji builds in Packit (e.g. merging distgit Packit pull requests).
+You can retrigger the build, if it is missing, commenting a pull request with:</p>
+<p>/packit koji-build</p>
+
+
+
+
+ [Resolved] Pull From Upstream Outage
+ https://status.packit.dev/issues/2024-02-22-pull-from-upstream-outage/
+ Wed, 21 Feb 2024 00:00:00 +0000
+ https://status.packit.dev/issues/2024-02-22-pull-from-upstream-outage/
+ 2024-02-22T14:40:00+00:00
+ pull_from_upstream jobs were not executed properly at the given time frame because of an expired token.
+Please trigger the release syncs manually if you are affected by commenting in any dist-git pull request:
+/packit pull-from-upstream We are very sorry for the inconvenience!
+ <p>pull_from_upstream jobs were not executed properly at the given time frame because of an expired token.</p>
+<p>Please <a href="https://packit.dev/docs/fedora-releases-guide#retriggering-1">trigger</a>
+the release syncs manually if you are affected by commenting in any dist-git pull request:</p>
+<pre><code>/packit pull-from-upstream
+</code></pre>
+<p>We are very sorry for the inconvenience!</p>
+
+
+
+
+ [Resolved] Fedora infrastructure authentication problem
+ https://status.packit.dev/issues/2024-01-23-fedora-auth-problem/
+ Mon, 22 Jan 2024 22:00:00 +0100
+ https://status.packit.dev/issues/2024-01-23-fedora-auth-problem/
+ 2024-01-23T21:00:00+01:00
+ The Fedora account system is currently down, leading to failures of jobs that interact with it (mostly propose_downstream, pull_from_upstream, koji_build, bodhi_update) as Packit is unable to authenticate. To stay updated on the root cause of this problem, please track this issue.
+The problem should be now resolved. You can retrigger the failed jobs by:
+ /packit propose-downstream comment in any upstream issue /packit pull-from-upstream / /packit koji-build / /packit create-update comment in any dist-git pull-request
+ <p>The Fedora account system is currently down, leading to failures of jobs that interact with it
+(mostly <code>propose_downstream</code>, <code>pull_from_upstream</code>, <code>koji_build</code>, <code>bodhi_update</code>)
+as Packit is unable to authenticate. To stay updated on the root cause of this problem, please
+track <a href="https://pagure.io/fedora-infrastructure/issue/11733">this issue</a>.</p>
+<p>The problem should be now resolved. You can retrigger the failed jobs by:</p>
+<ul>
+<li><code>/packit propose-downstream</code> comment in any upstream issue</li>
+<li><code>/packit pull-from-upstream</code> / <code>/packit koji-build</code> / <code>/packit create-update</code> comment in any dist-git pull-request</li>
+</ul>
+
+
+
+
+ [Resolved] Outage of automatic triggering of downstream Koji builds
+ https://status.packit.dev/issues/2024-01-11-downstream-koji-builds-outage/
+ Thu, 11 Jan 2024 09:30:00 +0100
+ https://status.packit.dev/issues/2024-01-11-downstream-koji-builds-outage/
+ 2024-01-15T15:20:00+01:00
+ Since 2024-01-11 until 2024-01-15 downstream Koji builds weren’t automatically triggered due to Fedora Infrastructure changes. Please trigger your builds manually if you are affected.
+ <p>Since 2024-01-11 until 2024-01-15 downstream Koji builds weren’t automatically triggered
+due to Fedora Infrastructure changes. Please <a href="https://packit.dev/docs/fedora-releases-guide#retriggering-2">trigger</a>
+your builds manually if you are affected.</p>
+
+
+
+
+ [Resolved] GitHub outage
+ https://status.packit.dev/issues/2024-01-09-github-outage/
+ Tue, 09 Jan 2024 14:45:00 +0100
+ https://status.packit.dev/issues/2024-01-09-github-outage/
+ 2024-11-09T15:40:00+01:00
+ There is an ongoing GitHub outage. You can monitor the status and subscribe for updates here: https://www.githubstatus.com
+ <p>There is an ongoing GitHub outage.
+You can monitor the status and subscribe for updates here: <a href="https://www.githubstatus.com">https://www.githubstatus.com</a></p>
+
+
+
+
+ [Resolved] Christmas is here ๐
+ https://status.packit.dev/issues/2023-12-23-christmas-2023/
+ Fri, 22 Dec 2023 19:00:00 +0100
+ https://status.packit.dev/issues/2023-12-23-christmas-2023/
+ 2024-01-02T09:00:00+01:00
+ All the team is on vacation. Give us some time to get refreshed and be better prepared to work hard on Packit’s new features and bugs.
+–
+Have a wonderful year 2024 everyone! We’re back.
+ <p>All the team is on vacation.
+Give us some time to get refreshed and be better prepared to work hard on Packit’s new features and bugs.</p>
+<p>–</p>
+<p>Have a wonderful year 2024 everyone! We’re back.</p>
+
+
+
+
+ [Resolved] GitLab outage
+ https://status.packit.dev/issues/2023-12-19-gitlab-outage/
+ Tue, 19 Dec 2023 05:30:00 +0100
+ https://status.packit.dev/issues/2023-12-19-gitlab-outage/
+ 2023-12-20T12:00:00+01:00
+ Packit production instance has lost access to GitLab API. This most likely affects also self-hosted GitLab forges.
+The issue has been resolved and everything should work as expected.
+ <p>Packit production instance has lost access to GitLab API.
+This most likely affects also self-hosted GitLab forges.</p>
+<p>The issue has been resolved and everything should work as expected.</p>
+
+
+
+
+ [Resolved] Production instance firewall
+ https://status.packit.dev/issues/2023-12-10-prod-fw/
+ Sun, 10 Dec 2023 20:45:00 +0100
+ https://status.packit.dev/issues/2023-12-10-prod-fw/
+ 2024-02-15T08:00:00+00:00
+ We have moved our production instance to another OpenShift cluster which may result in some jobs failing. This issue should mainly affect jobs that involve syncing to the dist-git, namely:
+ propose-downstream pull-from-upstream In case you hit any issues when running propose-downstream or pull-from-upstream, please let us know.
+Known Issues none as of now (2023-12-11 17:00 UTC)
+ <p>We have moved our production instance to another OpenShift cluster which may result in some jobs failing.
+This issue should mainly affect jobs that involve syncing to the dist-git, namely:</p>
+<ul>
+<li><code>propose-downstream</code></li>
+<li><code>pull-from-upstream</code></li>
+</ul>
+<p>In case you hit any issues when running <code>propose-downstream</code> or <code>pull-from-upstream</code>, please let us know.</p>
+<h2 id="known-issues">Known Issues</h2>
+<ul>
+<li>none as of now (2023-12-11 17:00 UTC)</li>
+</ul>
+
+
+
+
+ [Resolved] Testing Farm outage
+ https://status.packit.dev/issues/2023-12-10-tf-outage/
+ Sun, 10 Dec 2023 20:40:00 +0100
+ https://status.packit.dev/issues/2023-12-10-tf-outage/
+ 2023-12-11T12:00:00+01:00
+ Testing Farm provisioning is currently failing (due to failing DNS resolution). All runs created on the Testing Farm end with:
+โ Guest couldn't be provisioned: Artemis resource ended in 'error' state In case it affects you, please retrigger the tests via /packit test comment command in the pull requests, once the outage is resolved.
+ <p>Testing Farm provisioning is currently failing (due to failing DNS resolution). All runs created on the Testing Farm end with:</p>
+<pre tabindex="0"><code>โ Guest couldn't be provisioned: Artemis resource ended in 'error' state
+</code></pre><p>In case it affects you, please retrigger the tests via <code>/packit test</code> comment command in the pull requests, <em>once the outage is resolved</em>.</p>
+
+
+
+
+ [Resolved] Production move
+ https://status.packit.dev/issues/2023-12-10-mpp-move/
+ Sun, 10 Dec 2023 20:15:00 +0100
+ https://status.packit.dev/issues/2023-12-10-mpp-move/
+ 2023-12-11T18:00:00+01:00
+ UPDATE #2: No issues have been noticed so far.
+UPDATE: Production is deployed again. We are monitoring the deployment for any potential issues.
+Our production deployment is being moved to a different cluster, you may experience outage for at most two hours.
+In case you experience any issues, please retrigger the jobs manually via comment.
+ <p><em>UPDATE #2</em>: No issues have been noticed so far.</p>
+<p><em>UPDATE</em>: Production is deployed again. We are monitoring the deployment for any potential issues.</p>
+<p>Our production deployment is being moved to a different cluster, you may experience outage for at most two hours.</p>
+<p>In case you experience any issues, please retrigger the jobs manually via comment.</p>
+
+
+
+
+ [Resolved] Copr builds stuck in pending or running state
+ https://status.packit.dev/issues/2023-12-06-copr-outage/
+ Wed, 06 Dec 2023 12:00:00 +0100
+ https://status.packit.dev/issues/2023-12-06-copr-outage/
+ 2023-12-06T16:00:00+01:00
+ Copr builds could have been stuck in a pending or running state. The issue has been resolved, it may take a while for the affected Copr builds to finish, but there is no need to retrigger them.
+ <p>Copr builds could have been stuck in a pending or running state.
+The issue has been resolved, it may take a while for the affected Copr builds to finish,
+but there is no need to retrigger them.</p>
+
+
+
+
+ [Resolved] Copr Upgrade
+ https://status.packit.dev/issues/2023-11-28-copr-upgrade/
+ Tue, 28 Nov 2023 13:30:00 +0000
+ https://status.packit.dev/issues/2023-11-28-copr-upgrade/
+ 2023-11-28T16:45:00+00:00
+ Due to an upgrade of the Fedora Copr Servers, Copr builds might have been delayed or failing.
+ <p>Due to an <a href="https://pagure.io/fedora-infrastructure/issue/11648">upgrade</a> of the Fedora
+Copr Servers, Copr builds might have been delayed or failing.</p>
+
+
+
+
+ [Resolved] COPR builds failing for epel8
+ https://status.packit.dev/issues/2023-11-14-copr-outage-for-epel8/
+ Tue, 14 Nov 2023 10:00:00 +0100
+ https://status.packit.dev/issues/2023-11-14-copr-outage-for-epel8/
+ 2023-11-14T17:15:00+01:00
+ Building on COPR for epel8 chroot will be broken today for several hours. Today is release day of RHEL 8.9 and everyone is seeing 404 not found error when trying to access RHEL 8 repositories.
+ <p>Building on COPR for <em>epel8</em> chroot will be broken today for several hours.
+Today is release day of RHEL 8.9 and everyone is seeing 404 not found error when trying to access
+RHEL 8 repositories.</p>
+
+
+
+
+ [Resolved] Production move
+ https://status.packit.dev/issues/2023-11-12-mpp-move/
+ Sun, 12 Nov 2023 15:00:00 +0100
+ https://status.packit.dev/issues/2023-11-12-mpp-move/
+ 2023-11-12T17:15:00+01:00
+ Our production deployment is being moved to different cluster, you may experience outage for at most an hour.
+In case you experience any issues, please retrigger the jobs manually via comment.
+UPDATE: Migration has been reverted due to issues with Fedora infra (authentication & message bus).
+ <p>Our production deployment is being moved to different cluster, you may experience outage for at most an hour.</p>
+<p>In case you experience any issues, please retrigger the jobs manually via comment.</p>
+<p><em>UPDATE</em>: Migration has been reverted due to issues with Fedora infra (authentication & message bus).</p>
+
+
+
+
+ [Resolved] GitHub outage
+ https://status.packit.dev/issues/2023-11-07-github-outage/
+ Tue, 07 Nov 2023 14:45:00 +0100
+ https://status.packit.dev/issues/2023-11-07-github-outage/
+ 2023-11-07T15:30:00+01:00
+ There is an ongoing GitHub outage. You can monitor the status and subscribe for updates here: https://www.githubstatus.com
+ <p>There is an ongoing GitHub outage.
+You can monitor the status and subscribe for updates here: <a href="https://www.githubstatus.com">https://www.githubstatus.com</a></p>
+
+
+
+
+ [Resolved] GitLab.com outage
+ https://status.packit.dev/issues/2023-10-10-gitlab-outage/
+ Tue, 10 Oct 2023 04:00:00 +0200
+ https://status.packit.dev/issues/2023-10-10-gitlab-outage/
+ 2023-10-11T14:00:00+02:00
+ GitLab.com workflows are disrupted due to the issue with detecting the upstream/downstream repository in the current working directory.
+Self-hosted GitLab forges are not affected.
+ <p>GitLab.com workflows are disrupted due to the <a href="https://github.com/packit/packit/issues/2118">issue</a> with detecting the upstream/downstream repository in the current working directory.</p>
+<p>Self-hosted GitLab forges are not affected.</p>
+
+
+
+
+ [Resolved] Cluster upgrade
+ https://status.packit.dev/issues/2023-10-02-cluster-upgrade/
+ Mon, 02 Oct 2023 08:00:00 +0000
+ https://status.packit.dev/issues/2023-10-02-cluster-upgrade/
+ 2023-10-02T10:00:00+00:00
+ A cluster upgrade is scheduled to start on Mon October 2nd at 08:00 UTC and last approximately 90 minutes.
+We do not expect any significant disruption.
+The started jobs might be executed with a delay and if you think Packit does not react to your events, try to retrigger the action via comment command.
+ <p>A cluster upgrade is scheduled to start on Mon October 2nd at 08:00 UTC and last approximately 90 minutes.</p>
+<p>We do not expect any significant disruption.</p>
+<p>The started jobs might be executed with a delay
+and if you think Packit does not react to your events,
+try to retrigger the action via
+<a href="https://packit.dev/docs/guide/#how-to-re-trigger-packit-actions-in-your-pull-request">comment command</a>.</p>
+
+
+
+
+ [Resolved] COPR build failures
+ https://status.packit.dev/issues/2023-09-09-copr-outage/
+ Sat, 09 Sep 2023 12:00:00 +0000
+ https://status.packit.dev/issues/2023-09-09-copr-outage/
+ 2023-09-11T15:00:00+00:00
+ Copr is experiencing issues with builds. We are sorry about that. You can follow the progress in the Copr issue.
+The failure in the COPR logs looks like the following: Machine 5b8689e424334255ab9ace8a58343076 still running. Killing...
+In case you are building in your own COPR project you can go to your project settings and change Mock bootstrap from default to Enable for a quick workaround.
+ <p>Copr is experiencing issues with builds. We are sorry about that. You can follow the progress in the <a href="https://github.com/fedora-copr/copr/issues/2906">Copr issue</a>.</p>
+<p>The failure in the COPR logs looks like the following: <code>Machine 5b8689e424334255ab9ace8a58343076 still running. Killing...</code></p>
+<p>In case you are building in your own COPR project you can go to your project settings and change <code>Mock bootstrap</code> from default to <code>Enable</code> for a quick workaround.</p>
+
+
+
+
+ [Resolved] Copr frontend instability
+ https://status.packit.dev/issues/2023-09-05-copr-outage/
+ Tue, 05 Sep 2023 15:15:00 +0200
+ https://status.packit.dev/issues/2023-09-05-copr-outage/
+ 2023-04-05T15:00:00+00:00
+ Copr is experiencing issues with its frontend which is currently mitigated while Copr team investigates.
+In case some of your builds were not submitted, please retrigger.
+ <p>Copr is experiencing issues with its <a href="https://github.com/fedora-copr/copr/issues/2897">frontend</a>
+which is currently mitigated while Copr team investigates.</p>
+<p>In case some of your builds were not submitted, please retrigger.</p>
+
+
+
+
+ [Resolved] System upgrade scheduled
+ https://status.packit.dev/issues/2023-08-28-cluster-upgrade/
+ Mon, 28 Aug 2023 10:00:00 +0200
+ https://status.packit.dev/issues/2023-08-28-cluster-upgrade/
+ 2023-08-28T11:45:00+02:00
+ A system upgrade was scheduled and lasted approximately 90 minutes.
+The started jobs might’ve been executed with a delay or dropped, if you think Packit does not react to your events, try to retrigger the action via comment command.
+ <p>A system upgrade was scheduled and lasted approximately 90 minutes.</p>
+<p>The started jobs might’ve been executed with a delay or dropped,
+if you think Packit does not react to your events, try to retrigger
+the action via
+<a href="https://packit.dev/docs/guide/#how-to-re-trigger-packit-actions-in-your-pull-request">comment command</a>.</p>
+
+
+
+
+ [Resolved] Testing Farm outage
+ https://status.packit.dev/issues/2023-08-24-tf-outage/
+ Thu, 24 Aug 2023 15:40:00 +0200
+ https://status.packit.dev/issues/2023-08-24-tf-outage/
+ 2023-08-24T16:00:00+02:00
+ Testing Farm infrastructure had a small outage that caused few failing test runs. In case it affected you, please retrigger the tests via /packit test comment command in the pull requests.
+ <p>Testing Farm infrastructure had a small outage that caused few failing test runs.
+In case it affected you, please retrigger the tests via <code>/packit test</code> comment command in the pull requests.</p>
+
+
+
+
+ [Resolved] Packit outage now resolved
+ https://status.packit.dev/issues/2023-08-21-packit-outage/
+ Mon, 21 Aug 2023 14:45:00 +0200
+ https://status.packit.dev/issues/2023-08-21-packit-outage/
+ 2023-08-21T15:15:00+02:00
+ There has been a Packit outage now resolved. If your jobs are stuck, please retrigger them. We are sorry for the inconvenience.
+ <p>There has been a Packit outage now resolved. If your jobs are stuck, please retrigger them.
+We are sorry for the inconvenience.</p>
+
+
+
+
+ [Resolved] Copr Upgrade
+ https://status.packit.dev/issues/2023-08-16-copr-upgrade/
+ Wed, 16 Aug 2023 14:30:00 +0200
+ https://status.packit.dev/issues/2023-08-16-copr-upgrade/
+ 2023-08-16T16:30:00+02:00
+ There will be a Copr outage starting at
+date --date "2023-08-16 12:30 UTC" which will last approximately 2 hours. The build queue processing will be stopped. During the outage time, we expect a very short downtime for the DNF package storage (less than 10 minutes).
+Reason for outage: We will update the infrastructure machines to the latest packages that are currently being developed.
+Affected Services: https://copr.fedorainfracloud.org/ https://download.copr.fedorainfracloud.org/results/
+Upstream ticket: https://github.com/fedora-copr/copr/issues/2824
+ <p>There will be a Copr outage starting at</p>
+<pre tabindex="0"><code>date --date "2023-08-16 12:30 UTC"
+</code></pre><p>which will last approximately 2 hours. The build queue processing will be stopped. During the outage time, we expect a very short downtime for the DNF package storage (less than 10 minutes).</p>
+<h2 id="reason-for-outage">Reason for outage:</h2>
+<p>We will update the infrastructure machines to the latest packages that are currently being developed.</p>
+<h2 id="affected-services">Affected Services:</h2>
+<p><a href="https://copr.fedorainfracloud.org/">https://copr.fedorainfracloud.org/</a>
+<a href="https://download.copr.fedorainfracloud.org/results/">https://download.copr.fedorainfracloud.org/results/</a></p>
+<h2 id="upstream-ticket">Upstream ticket:</h2>
+<p><a href="https://github.com/fedora-copr/copr/issues/2824">https://github.com/fedora-copr/copr/issues/2824</a></p>
+<p>Please join #fedora-buildsys, #fedora-admin or #fedora-noc on irc.libera.chat
+or add comments to this ticket.</p>
+
+
+
+
+ [Resolved] Outage
+ https://status.packit.dev/issues/2023-08-15-outage/
+ Tue, 15 Aug 2023 04:00:00 +0200
+ https://status.packit.dev/issues/2023-08-15-outage/
+ 2023-08-15T11:03:00+02:00
+ We are having an outage that started on Tuesday EU early morning. Jobs during this time could get stuck in pending state. This is caused by a failed database schema migration because of a lack of memory during the automatic deployment of the new production images. We are working on the fix and apologize for the inconvenience.
+Update: Migration is completed. You might need to retrigger the jobs (i.e. via comment).
+ <p>We are having an outage that started on Tuesday EU early morning. Jobs during this time could get stuck
+in pending state.
+This is caused by a failed database schema migration because of a lack of memory during the automatic deployment
+of the new production images. We are working on the fix and apologize for the inconvenience.</p>
+<p>Update: Migration is completed. You might need to retrigger the jobs (i.e. via comment).</p>
+
+
+
+
+ [Resolved] Staging instance firewall
+ https://status.packit.dev/issues/2023-07-24-stage-fw/
+ Mon, 24 Jul 2023 14:30:00 +0200
+ https://status.packit.dev/issues/2023-07-24-stage-fw/
+ 2023-12-10T20:00:00+01:00
+ We have moved our staging instance to another OpenShift cluster which may result in some jobs failing. This issue should mainly affect jobs that involve syncing to the dist-git, namely:
+ propose-downstream pull-from-upstream In case you are using our staging instance and you hit any issues when running propose-downstream or pull-from-upstream, please let us know.
+Known Issues Testing Farm is currently not available on the staging instance. Edit: This issue has been fixed as of July 26th 10am.
+ <p>We have moved our staging instance to another OpenShift cluster which may result in some jobs failing.
+This issue should mainly affect jobs that involve syncing to the dist-git, namely:</p>
+<ul>
+<li><code>propose-downstream</code></li>
+<li><code>pull-from-upstream</code></li>
+</ul>
+<p>In case you are using our staging instance and you hit any issues when running <code>propose-downstream</code> or <code>pull-from-upstream</code>, please let us know.</p>
+<h2 id="known-issues">Known Issues</h2>
+<ul>
+<li><del>Testing Farm is currently not available on the staging instance.</del>
+<em>Edit</em>: This issue has been fixed as of July 26th 10am.</li>
+</ul>
+
+
+
+
+ [Resolved] Testing Farm testing blocked
+ https://status.packit.dev/issues/2023-07-19-testing-farm-failing-on-rawhide/
+ Wed, 19 Jul 2023 08:00:00 +0000
+ https://status.packit.dev/issues/2023-07-19-testing-farm-failing-on-rawhide/
+ 2023-07-19T11:00:00+00:00
+ Testing via Testing Farm is blocked: https://status.testing-farm.io/issues/2023-07-19-public-ranch-outage/.
+Testing Farm team is working on the fix (ETA 2 hours).
+ <p>Testing via Testing Farm is blocked: <a href="https://status.testing-farm.io/issues/2023-07-19-public-ranch-outage/">https://status.testing-farm.io/issues/2023-07-19-public-ranch-outage/</a>.</p>
+<p>Testing Farm team is working on the fix (ETA 2 hours).</p>
+
+
+
+
+ [Resolved] Guest couldn't be provisioned: Artemis resource ended in 'error' state
+ https://status.packit.dev/issues/2023-07-03-testing-provisioning-failed/
+ Mon, 03 Jul 2023 08:00:00 +0200
+ https://status.packit.dev/issues/2023-07-03-testing-provisioning-failed/
+ 2023-07-03T10:00:00+00:00
+ Some jobs submitted by Packit to the Testing Farm are pending with this error: "Guest couldn't be provisioned: Artemis resource ended in 'error' state".
+Testing Farm has already resolved the issue; please re-submit the job with a comment like /packit test (more info on retriggering here) or through the GitHub checks button.
+ <p>Some jobs submitted by Packit to the Testing Farm are pending with this error:
+<code>"Guest couldn't be provisioned: Artemis resource ended in 'error' state"</code>.</p>
+<p>Testing Farm has already resolved the issue; please re-submit the job with a
+comment like <code>/packit test</code> (more info on retriggering
+<a href="https://packit.dev/docs/configuration/upstream/tests/#restart-testing">here</a>)
+or through the GitHub checks button.</p>
+
+
+
+
+ [Resolved] Fedora Rawhide Testing Blocked on dnf5
+ https://status.packit.dev/issues/2023-06-22-fedora-rawhide-testing-blocked-on-dnf5/
+ Thu, 22 Jun 2023 09:30:00 +0200
+ https://status.packit.dev/issues/2023-06-22-fedora-rawhide-testing-blocked-on-dnf5/
+ 2023-07-18T20:30:00+02:00
+ dnf has been upgraded to dnf5 in Rawhide, resulting in various issues when running tests in Testing Farm.
+The Testing Farm team together with dnf5 team is working on fixing the issues and trying to resolve the situation ASAP.
+If you want to help, here are upstream issues related to this problem:
+ https://github.com/rpm-software-management/dnf5/issues/549 https://github.com/rpm-software-management/dnf5/issues/168
+ <p>dnf has been upgraded to dnf5 in Rawhide, resulting in various issues
+when running tests in Testing Farm.</p>
+<p>The Testing Farm team together with dnf5 team is working on fixing the issues
+and trying to resolve the situation ASAP.</p>
+<p>If you want to help, here are upstream issues related to this problem:</p>
+<ul>
+<li><a href="https://github.com/rpm-software-management/dnf5/issues/549">https://github.com/rpm-software-management/dnf5/issues/549</a></li>
+<li><a href="https://github.com/rpm-software-management/dnf5/issues/168">https://github.com/rpm-software-management/dnf5/issues/168</a></li>
+</ul>
+
+
+
+
+ [Resolved] Upgrade of Copr servers
+ https://status.packit.dev/issues/2023-05-25-copr-upgrade/
+ Thu, 25 May 2023 10:00:00 +0000
+ https://status.packit.dev/issues/2023-05-25-copr-upgrade/
+ 2023-05-25T13:00:00+00:00
+ Due to an upgrade of the Fedora Copr Servers, Copr builds might be delayed or failing. The upgrade should last approximately 3 hours.
+ <p>Due to an <a href="https://pagure.io/fedora-infrastructure/issue/11332">upgrade</a> of the Fedora
+Copr Servers, Copr builds might be delayed or failing. The upgrade should last approximately 3 hours.</p>
+
+
+
+
+ [Resolved] GitHub outage
+ https://status.packit.dev/issues/2023-05-11-github-outage/
+ Thu, 11 May 2023 15:30:00 +0200
+ https://status.packit.dev/issues/2023-05-11-github-outage/
+ 2023-05-11T21:00:00+02:00
+ There is an ongoing GitHub outage. You can monitor the status and subscribe for updates here: https://www.githubstatus.com
+EDIT: GitHub’s post mortem
+ <p>There is an ongoing GitHub outage.
+You can monitor the status and subscribe for updates here: <a href="https://www.githubstatus.com">https://www.githubstatus.com</a></p>
+<p>EDIT: <a href="https://github.blog/2023-05-16-addressing-githubs-recent-availability-issues">GitHub’s post mortem</a></p>
+
+
+
+
+ [Resolved] Testing Farm errors
+ https://status.packit.dev/issues/2023-05-11-tf-merge-issue/
+ Wed, 10 May 2023 14:00:00 +0200
+ https://status.packit.dev/issues/2023-05-11-tf-merge-issue/
+ 2023-05-11T15:00:00+02:00
+ After the deployment of the new version of Testing Farm, you may have encountered Testing Farm runs that fail with the status Failed to checkout ref {sha}. Together with the Testing Farm team, we are working on a fix.
+ <p>After the deployment of the new version of Testing Farm, you may have encountered Testing Farm runs that
+fail with the status <code>Failed to checkout ref {sha}</code>. Together with the Testing Farm team, we are working on a fix.</p>
+
+
+
+
+ [Resolved] GitHub outage
+ https://status.packit.dev/issues/2023-05-09-github-outage/
+ Tue, 09 May 2023 13:50:00 +0200
+ https://status.packit.dev/issues/2023-05-09-github-outage/
+ 2023-05-09T23:15:00+02:00
+ There is an ongoing GitHub outage. You can monitor the status and subscribe for updates here: https://www.githubstatus.com
+ <p>There is an ongoing GitHub outage.
+You can monitor the status and subscribe for updates here: <a href="https://www.githubstatus.com">https://www.githubstatus.com</a></p>
+
+
+
+
+ [Resolved] Bodhi Updates not created
+ https://status.packit.dev/issues/2023-05-05-bodhi-updates/
+ Thu, 04 May 2023 11:30:00 +0200
+ https://status.packit.dev/issues/2023-05-05-bodhi-updates/
+ 2023-05-10T16:00:00+02:00
+ We’re experiencing issues with creating Bodhi updates due to failing authentication.
+You can try to retrigger an update by a comment in a dist-git pull request:
+/packit create-update
+ <p>We’re experiencing issues with creating Bodhi updates due to failing authentication.</p>
+<p>You can try to retrigger an update by a comment in a dist-git pull request:</p>
+<pre><code>/packit create-update
+</code></pre>
+
+
+
+
+ [Resolved] Cluster Update
+ https://status.packit.dev/issues/2023-05-03-cluster-update/
+ Wed, 03 May 2023 10:00:00 +0200
+ https://status.packit.dev/issues/2023-05-03-cluster-update/
+ 2023-05-03T11:15:00+02:00
+ The OpenShift cluster is currently updating from 4.11.28 to 4.12.13
+ <p>The OpenShift cluster is currently updating from 4.11.28 to 4.12.13</p>
+
+
+
+
+ [Resolved] Scheduled DB Cleanup
+ https://status.packit.dev/issues/2023-04-26-scheduled-db-cleanup/
+ Wed, 26 Apr 2023 04:40:00 +0000
+ https://status.packit.dev/issues/2023-04-26-scheduled-db-cleanup/
+ 2023-04-26T05:17:00+00:00
+ We are running a DB maintenance in order to clean up old pipelines. We expect API performance to improve as a result of this.
+The maintenance window is planned for an hour, the actual clean up activities will last approximately 20 minutes. Users might experience degraded performance of the service and dashboard during this time.
+ <p>We are running a DB maintenance in order to clean up old pipelines. We expect
+API performance to improve as a result of this.</p>
+<p>The maintenance window is planned for an hour, the actual clean up activities
+will last approximately 20 minutes. Users might experience degraded
+performance of the service and dashboard during this time.</p>
+
+
+
+
+ [Resolved] Issue with pushing to dist-git
+ https://status.packit.dev/issues/2023-04-12-push-to-distgit-issue/
+ Tue, 11 Apr 2023 18:28:00 +0200
+ https://status.packit.dev/issues/2023-04-12-push-to-distgit-issue/
+ 2023-04-12T16:30:00+02:00
+ Due to a new pre-push hook being applied in dist-git, we are unable to create pull-requests from fork as part of propose-downstream and pull-from-upstream jobs.
+We are sorry about that and are working on a way how to resolve this.
+ <p>Due to <a href="https://docs.pagure.org/rpkg/releases/1.66.html#checking-a-repo-configuration-before-git-push-with-a-git-hook-script">a new pre-push hook</a>
+being applied in dist-git, we are unable to create pull-requests
+from fork as part of <code>propose-downstream</code> and <code>pull-from-upstream</code> jobs.</p>
+<p>We are sorry about that and are working on a way how to resolve this.</p>
+
+
+
+
+ [Resolved] Upgrade of Copr servers
+ https://status.packit.dev/issues/2023-04-05-copr-outage/
+ Wed, 05 Apr 2023 12:00:00 +0000
+ https://status.packit.dev/issues/2023-04-05-copr-outage/
+ 2023-04-05T15:00:00+00:00
+ Due to an upgrade of the Fedora Copr Servers, Copr builds might be delayed or failing. The upgrade should last approximately 3 hours.
+ <p>Due to an <a href="https://pagure.io/fedora-infrastructure/issue/11201">upgrade</a> of the Fedora
+Copr Servers, Copr builds might be delayed or failing. The upgrade should last approximately 3 hours.</p>
+
+
+
+
+ [Resolved] SLO1 Error Budget issues
+ https://status.packit.dev/issues/2023-03-09-slo1-error-budget/
+ Thu, 09 Mar 2023 10:00:00 +0100
+ https://status.packit.dev/issues/2023-03-09-slo1-error-budget/
+ 2023-06-08T10:00:00+01:00
+ We are having SLO1 error budget issues.
+These are mostly caused by performance issues, we are working on them and prioritizing more improvements (see https://github.com/packit/packit-service/issues/1954 ).
+We are sorry for any slowness you can experience.
+ <p>We are having <a href="https://packit.dev/docs/service-level-objectives/">SLO1</a> error budget issues.</p>
+<p>These are mostly caused by performance issues, we are working on them and prioritizing more improvements (see <a href="https://github.com/packit/packit-service/issues/1954">https://github.com/packit/packit-service/issues/1954</a> ).</p>
+<p>We are sorry for any slowness you can experience.</p>
+
+
+
+
+ [Resolved] Running grouping migrations
+ https://status.packit.dev/issues/2023-02-14-running-grouping-migrations/
+ Tue, 14 Feb 2023 02:00:00 +0100
+ https://status.packit.dev/issues/2023-02-14-running-grouping-migrations/
+ 2023-02-14T04:00:00+01:00
+ We are updating our database schema to facilitate further development. As a part of this process, multiple database migrations will have to be run. The first step is grouping together related builds and test runs. The migrations will happen during the next deployment to production, on Tuesday February 14 at 2:00AM CET. This set of migrations is expected to take roughly 40 minutes, after which the service should come back up.
+ <p>We are updating our database schema to facilitate further development.
+As a part of this process, multiple database migrations will have to
+be run. The first step is grouping together related builds and test
+runs. The migrations will happen during the next deployment to production,
+on Tuesday February 14 at 2:00AM CET. This set of migrations is expected to
+take roughly 40 minutes, after which the service should come back up.</p>
+<p>Since significant refactoring was paired with these changes, there may
+be some bugs/regressions that we did not spot on staging. Please,
+contact us via the usual means if you spot anything suspicious.</p>
+
+
+
+
+ [Resolved] SLO2 Error Budget Depleted in a Day
+ https://status.packit.dev/issues/2023-02-17-slo2-error-budget-depleted-in-a-day/
+ Tue, 31 Jan 2023 10:00:00 +0100
+ https://status.packit.dev/issues/2023-02-17-slo2-error-budget-depleted-in-a-day/
+ 2023-02-01T10:00:00+01:00
+ The SLO2 error budget dropped from 85% all the way down to zero in one day. The likely cause was that the fedora-messaging consumer was stuck, so we were not getting messages from Copr about finished SRPM builds. Luckily, the babysit tasks proved to be very useful and caught those later.
+We’ve added a liveness probe to the messaging consumer to prevent this from happening again.
+ <p>The SLO2 error budget dropped from 85% all the way down to zero in one day.
+<img src="https://status.packit.dev/images/SLO2-error-budget-depleted.png" alt="SLO2"></p>
+<p>The likely cause was that the fedora-messaging consumer was stuck, so we
+were not getting messages from Copr about finished SRPM builds.
+Luckily, the babysit tasks proved to be very useful and caught those later.</p>
+<p>We’ve <a href="https://github.com/packit/packit-service-fedmsg/pull/76">added a liveness probe to the messaging consumer</a>
+to prevent this from happening again.</p>
+
+
+
+
+ [Resolved] Testing Farm outage
+ https://status.packit.dev/issues/2023-01-24-tf-outage/
+ Tue, 24 Jan 2023 15:00:00 +0100
+ https://status.packit.dev/issues/2023-01-24-tf-outage/
+ 2023-01-25T10:00:00+01:00
+ Testing Farm infrastructure had an outage which caused the status of the test runs to be stuck in the queued state. This is now resolved and tests can be retriggered by /packit test comment command in the pull requests.
+ <p>Testing Farm infrastructure had an <a href="https://status.testing-farm.io/issues/2023-01-24-public-ranch-outage/">outage</a>
+which caused the status of the test runs to be stuck in the <code>queued</code> state.
+This is now resolved and tests can be retriggered by <code>/packit test</code> comment command in the pull requests.</p>
+
+
+
+
+ [Resolved] SRPM builds stuck in Submitted state
+ https://status.packit.dev/issues/2023-01-21-srpm-builds-stuck-in-submitted-state/
+ Fri, 20 Jan 2023 21:19:06 +0100
+ https://status.packit.dev/issues/2023-01-21-srpm-builds-stuck-in-submitted-state/
+ 2023-01-21T9:51:30+01:00
+ Workers for processing long-running tasks got stuck on Friday evening with to most visible consequence being the SRPM builds stuck in ‘Submitted’ state.
+The workers were restarted on Sat morning and all SRPMs were sent to Copr to build and the jobs pipelines were processed.
+The cause is not known at the time of resolving the issue.
+ <p>Workers for processing long-running tasks got stuck on Friday evening
+with to most visible consequence being the SRPM builds stuck in ‘Submitted’ state.</p>
+<p>The workers were restarted on Sat morning and all SRPMs were sent to Copr
+to build and the jobs pipelines were processed.</p>
+<p><a href="https://github.com/packit/packit-service/issues/1858">The cause is not known at the time of resolving the issue.</a></p>
+
+
+
+
+ [Resolved] System upgrade scheduled
+ https://status.packit.dev/issues/2023-01-06-cluster-upgrade/
+ Fri, 06 Jan 2023 10:00:00 +0000
+ https://status.packit.dev/issues/2023-01-06-cluster-upgrade/
+ 2023-01-06T22:00:00+00:00
+ A system upgrade is scheduled to start on Fri Jan 06 20:00 UTC and last approximately 90 minutes.
+We do not expect any significant disruption.
+The started jobs might be executed with a delay and if you think Packit does not react to your events, try to retrigger the action via comment command.
+ <p>A system upgrade is scheduled to start on Fri Jan 06 20:00 UTC and last approximately 90 minutes.</p>
+<p>We do not expect any significant disruption.</p>
+<p>The started jobs might be executed with a delay
+and if you think Packit does not react to your events,
+try to retrigger the action via
+<a href="https://packit.dev/docs/guide/#how-to-re-trigger-packit-actions-in-your-pull-request">comment command</a>.</p>
+
+
+
+
+ [Resolved] Christmas is here ๐
+ https://status.packit.dev/issues/2022-12-23-christmas-2022/
+ Fri, 23 Dec 2022 18:00:00 +0000
+ https://status.packit.dev/issues/2022-12-23-christmas-2022/
+ 2023-01-02T08:00:00+00:00
+ All the team is on vacation. Give us some time to get refreshed and be better prepared to work hard on Packit’s new features and bugs.
+The Packit staging instance (https://dashboard.stg.packit.dev/) will not run during this week. It will be back on the 2nd of January.
+–
+Have a wonderful year 2023 everyone! We’re back, staging instance is up and serving.
+ <p>All the team is on vacation.
+Give us some time to get refreshed and be better prepared to work hard on Packit’s new features and bugs.</p>
+<p>The Packit staging instance (<a href="https://dashboard.stg.packit.dev/">https://dashboard.stg.packit.dev/</a>) will not run during this week.
+It will be back on the 2nd of January.</p>
+<p>–</p>
+<p>Have a wonderful year 2023 everyone! We’re back, staging instance is up and serving.</p>
+
+
+
+
+ [Resolved] Upgrade of Copr Servers
+ https://status.packit.dev/issues/2022-11-28-copr-outage/
+ Mon, 28 Nov 2022 16:00:00 +0100
+ https://status.packit.dev/issues/2022-11-28-copr-outage/
+ 2022-11-28T21:00:00+01:00
+ Due to an upgrade of the Fedora Copr Servers, Copr builds might have been failing.
+ <p>Due to an <a href="https://pagure.io/fedora-infrastructure/issue/11011">upgrade</a> of the Fedora
+Copr Servers, Copr builds might have been failing.</p>
+
+
+
+
+ [Resolved] Staging is unstable
+ https://status.packit.dev/issues/2022-11-28-staging-unstable/
+ Fri, 25 Nov 2022 08:35:00 +0100
+ https://status.packit.dev/issues/2022-11-28-staging-unstable/
+ 2022-11-30T14:06:00+01:00
+ Since the last cluster update, we’ve been running our staging instance on a separate node pool which uses AWS Spot instances. It causes the staging instance to be quite unreliable. We’re testing it in order to find balance between price and usefulness.
+ <p>Since the last cluster update, we’ve been running our staging instance
+on a separate node pool which uses <a href="https://aws.amazon.com/ec2/spot">AWS Spot instances</a>.
+It causes the staging instance to be quite unreliable.
+We’re testing it in order to find balance between price and usefulness.</p>
+
+
+
+
+ [Resolved] Outage
+ https://status.packit.dev/issues/2022-11-25-outage/
+ Thu, 24 Nov 2022 23:24:00 +0100
+ https://status.packit.dev/issues/2022-11-25-outage/
+ 2022-11-25T08:35:00+01:00
+ We are having resource issues in our cluster and particular parts of the deployment can be down.
+Updates:
+ 2022-11-25T08:35:00+01:00: Stage instance has been scaled down so our production instance has more resources. Production instance should be fine now. We are investigating more.
+ <p>We are having resource issues in our cluster and particular parts of the deployment can be down.</p>
+<p>Updates:</p>
+<ul>
+<li><code>2022-11-25T08:35:00+01:00</code>: Stage instance has been scaled down so our production instance has more resources.
+Production instance should be fine now. We are investigating more.</li>
+</ul>
+
+
+
+
+ [Resolved] System upgrade scheduled
+ https://status.packit.dev/issues/2022-11-24-cluster-upgrade/
+ Thu, 24 Nov 2022 19:00:00 +0000
+ https://status.packit.dev/issues/2022-11-24-cluster-upgrade/
+ 2022-11-24T23:00:00+00:00
+ A system upgrade is scheduled to start on Thu Nov 24 21:00 UTC and last approximately 90 minutes.
+We do not expect any significant disruption.
+The started jobs might be executed with a delay and if you think Packit does not react to your events, try to retrigger the action via comment command.
+ <p>A system upgrade is scheduled to start on Thu Nov 24 21:00 UTC and last approximately 90 minutes.</p>
+<p>We do not expect any significant disruption.</p>
+<p>The started jobs might be executed with a delay
+and if you think Packit does not react to your events,
+try to retrigger the action via
+<a href="https://packit.dev/docs/guide/#how-to-re-trigger-packit-actions-in-your-pull-request">comment command</a>.</p>
+
+
+
+
+ [Resolved] Copr builds and tests on commits and releases ignored
+ https://status.packit.dev/issues/2022-11-01-commit-release-trigger/
+ Tue, 01 Nov 2022 03:00:00 +0100
+ https://status.packit.dev/issues/2022-11-01-commit-release-trigger/
+ 2022-11-02T10:30:00+01:00
+ There was a bug introduced with Tuesday’s production update causing Copr builds and tests for commit and release triggers were not executed.
+ <p>There was a bug introduced with Tuesday’s production update causing Copr builds and tests
+for <code>commit</code> and <code>release</code> triggers were not executed.</p>
+
+
+
+
+ [Resolved] Copr storage move
+ https://status.packit.dev/issues/2022-10-28-30-copr-storage-move/
+ Fri, 28 Oct 2022 21:00:00 +0200
+ https://status.packit.dev/issues/2022-10-28-30-copr-storage-move/
+ 2022-10-30T08:00:00+02:00
+ Copr team is performing a backend storage move over the weekend of Oct 28-30th. Building in Copr will be disrupted or unavailable during this period.
+For more info, please follow the Copr outage announcement.
+Update: the upgrade was successful and Copr is fully operational.
+ <p>Copr team is performing a backend storage move over the weekend of Oct 28-30th.
+Building in Copr will be disrupted or unavailable during this period.</p>
+<p>For more info, please follow the <a href="https://lists.fedoraproject.org/archives/list/copr-devel@lists.fedorahosted.org/thread/TRNW5JVCFIQJDUCND4KZWZVM7XVCYLVL/">Copr outage announcement</a>.</p>
+<p>Update: the upgrade was successful and Copr is fully operational.</p>
+
+
+
+
+ [Resolved] System upgrade scheduled
+ https://status.packit.dev/issues/2022-10-13-cluster-upgrade/
+ Thu, 13 Oct 2022 19:00:00 +0000
+ https://status.packit.dev/issues/2022-10-13-cluster-upgrade/
+ 2022-10-13T20:37:00+00:00
+ A system upgrade is scheduled to start on Wed Oct 13 19:00 UTC and last approximately 90 minutes.
+We do not expect any significant disruption.
+The started jobs might be executed with a delay and if you think Packit does not react to your events, try to retrigger the action via comment command.
+ <p>A system upgrade is scheduled to start on Wed Oct 13 19:00 UTC and last approximately 90 minutes.</p>
+<p>We do not expect any significant disruption.</p>
+<p>The started jobs might be executed with a delay
+and if you think Packit does not react to your events,
+try to retrigger the action via
+<a href="https://packit.dev/docs/guide/#how-to-re-trigger-packit-actions-in-your-pull-request">comment command</a>.</p>
+
+
+
+
+ [Resolved] Issues with the task execution
+ https://status.packit.dev/issues/2022-10-11-outage/
+ Tue, 11 Oct 2022 13:57:00 +0200
+ https://status.packit.dev/issues/2022-10-11-outage/
+ 2022-10-11T15:12:00+02:00
+ We are experiencing issues with the execution of the tasks. (The statuses are waiting in the The task was accepted. state.) We are investigating the cause. Sorry about the possible troubles. We will try to fix this as soon as possible.
+Updates:
+ Tue Oct 11 12:24:02 UTC: We are slowly going through the queue. Please, be patient. Tue Oct 11 13:12:36 UTC: We have gone through the queue and you should experience regular response times.
+ <p>We are experiencing issues with the execution of the tasks.
+(The statuses are waiting in the <code>The task was accepted.</code> state.)
+We are investigating the cause.
+Sorry about the possible troubles.
+We will try to fix this as soon as possible.</p>
+<p>Updates:</p>
+<ul>
+<li>Tue Oct 11 12:24:02 UTC: We are slowly going through the queue. Please, be patient.</li>
+<li>Tue Oct 11 13:12:36 UTC: We have gone through the queue and you should experience regular response times.
+The original cause is still not known but the problem probably occured during an SRPM upload to Copr.
+(It can be a fault on our or Copr side.)</li>
+<li>Wed Oct 12 05:27:46 UTC: Service operated normally since the last update.
+To reduce the chance of this, we <a href="https://packit.dev/posts/copr-srpms/">plan to move the building of SRPMs to Copr</a>.
+(Currently default for new installations and opt in for other projects.)</li>
+</ul>
+
+
+
+
+ [Resolved] Internal Testing Farm infrastructure is having an outage
+ https://status.packit.dev/issues/2022-10-10-outage/
+ Mon, 10 Oct 2022 15:14:00 +0200
+ https://status.packit.dev/issues/2022-10-10-outage/
+ 2022-10-11T21:34:00+02:00
+ Due to a wide outage in Red Hat internal infrastructure the Red Hat Ranch (=internal ranch) is disabled in the Testing Farm (=our test runner). Currently there is no ETA for resolving the problem. Follow the progress on the Testing Farm status page: https://status.testing-farm.io/issues/2022-10-10-redhat-ranch/
+ <p>Due to a wide outage in Red Hat internal infrastructure the Red Hat Ranch (=internal ranch) is disabled in the Testing Farm (=our test runner).
+Currently there is no ETA for resolving the problem.
+Follow the progress on the Testing Farm status page: <a href="https://status.testing-farm.io/issues/2022-10-10-redhat-ranch/">https://status.testing-farm.io/issues/2022-10-10-redhat-ranch/</a></p>
+
+
+
+
+ [Resolved] GitHub Webhooks not working
+ https://status.packit.dev/issues/2022-10-05-github-webhooks-degraded-status/
+ Wed, 05 Oct 2022 16:00:00 +0200
+ https://status.packit.dev/issues/2022-10-05-github-webhooks-degraded-status/
+ 2022-10-05T18:00:00+02:00
+ Packit is not working on GitHub because the GitHub Webhooks status is ‘degraded’. See yourself: https://www.githubstatus.com/incidents/smn1qtqvbsb6
+ <p>Packit is not working on GitHub because the GitHub Webhooks status is ‘degraded’.
+See yourself: <a href="https://www.githubstatus.com/incidents/smn1qtqvbsb6">https://www.githubstatus.com/incidents/smn1qtqvbsb6</a></p>
+
+
+
+
+ [Resolved] September 22th outage
+ https://status.packit.dev/issues/2022-09-22-outage/
+ Thu, 22 Sep 2022 08:00:00 +0200
+ https://status.packit.dev/issues/2022-09-22-outage/
+ 2022-09-23T08:00:00+02:00
+ Testing Farm is failing to run tests submitted by Packit which use Copr builds. Tests with the skip_build flag are not be affected by this issue. We are working together with the Testing Farm team to fix this as soon as possible.
+ <p>Testing Farm is failing to run tests submitted by Packit which use <code>Copr</code> builds.
+Tests with the <code>skip_build</code> flag are not be affected by this issue.
+We are working together with the Testing Farm team to fix this as soon as possible.</p>
+
+
+
+
+ [Resolved] 3/4 of SLO1 Error Budget consumed in 6 weeks
+ https://status.packit.dev/issues/2022-10-27-slo1-error-budget-drop-postmortem/
+ Thu, 01 Sep 2022 01:00:00 +0200
+ https://status.packit.dev/issues/2022-10-27-slo1-error-budget-drop-postmortem/
+ 2022-10-12T01:00:00+02:00
+ (Mostly for our internal use since the source data are not publicly available.)
+During the 6 weeks since the beginning of September the SLO1 error budget dropped to 25%. After that (in the middle of October) the trend turned and now, at the end of October, it’s at 50%.
+When looking at metrics in our (not public) Grafana (Packit boards -> (Prod/Stg) Accepted status time) we can see that the average value (of the “accepted status time”) indeed increased during the beginning of September by approx 1 second and cases of >15s started to appear so the error budget started to be consumed.
+ <p>(Mostly for our internal use since the source data are not publicly available.)</p>
+<p>During the 6 weeks since the beginning of September the <a href="https://packit.dev/docs/service-level-objectives/#slo1-changes-to-github-prs-receive-a-status-update-within-15-seconds-in-99-of-cases">SLO1</a>
+error budget dropped to 25%. After that (in the middle of October)
+the trend turned and now, at the end of October, it’s at 50%.</p>
+<p>When looking at metrics in our (not public) Grafana
+(<code>Packit boards</code> -> <code>(Prod/Stg) Accepted status time)</code>
+we can see that the average value (of the “accepted status time”) indeed
+increased during the beginning of September by approx 1 second and
+cases of >15s started to appear so the error budget started to be consumed.</p>
+<p>The cause is yet unknown, but the changes we did the last week of August
+and which could thus contribute to the problem were:</p>
+<ul>
+<li>lots of SQLAlchemy-related changes - contribution unknown</li>
+<li>switched 4 single-threaded workers to 1 multithreaded - there might be some
+correlation, but it’s not clear because for example the first 2 weeks of October
+the change was reverted (i.e. there were again the 4 single-threaded workers)
+and even the average value seemed to drop a bit, the error budget continued
+to be consumed (meaning there were still some cases of >15s).</li>
+</ul>
+<p>We need to continue watching the metrics, experiment with the workers
+(numbers and types) and give the changes more time (2 weeks at least)
+to be able to tell whether they change the trend.</p>
+<p>Related:</p>
+<ul>
+<li><a href="https://github.com/packit/packit-service/issues/1364">https://github.com/packit/packit-service/issues/1364</a></li>
+<li><a href="https://github.com/packit/packit-service/issues/1677">https://github.com/packit/packit-service/issues/1677</a></li>
+<li><a href="https://github.com/packit/packit-service/issues/1728">https://github.com/packit/packit-service/issues/1728</a></li>
+</ul>
+
+
+
+
+ [Resolved] Failing SRPM builds done in Copr
+ https://status.packit.dev/issues/2022-08-23-srpm-builds-failing/
+ Tue, 23 Aug 2022 04:00:00 +0200
+ https://status.packit.dev/issues/2022-08-23-srpm-builds-failing/
+ 2022-08-23T09:00:00+02:00
+ SRPM builds happening in Copr (for packages using the srpm_build_deps config key) were failing due to a dependency issue.
+ <p>SRPM builds happening in Copr (for packages using the <code>srpm_build_deps</code> config
+key) were failing due to a dependency issue.</p>
+
+
+
+
+ [Resolved] Summer is here โ๏ธ
+ https://status.packit.dev/issues/2022-07-01-team-on-vacation/
+ Mon, 04 Jul 2022 08:00:00 +0000
+ https://status.packit.dev/issues/2022-07-01-team-on-vacation/
+ 2022-07-08T18:00:00+00:00
+ Most of the team is on vacation. We try our best to solve issues with limited resources. Give us some time to get refreshed and be better prepared to work hard on Packit’s new features and bugs.
+ <p>Most of the team is on vacation. We try our best to solve issues with limited resources.
+Give us some time to get refreshed and be better prepared to work hard on Packit’s new features and bugs.</p>
+
+
+
+
+ [Resolved] Upgrade of Copr Servers
+ https://status.packit.dev/issues/2022-06-22-upgrade-of-copr-servers/
+ Wed, 22 Jun 2022 14:00:00 +0200
+ https://status.packit.dev/issues/2022-06-22-upgrade-of-copr-servers/
+ 2022-06-22T16:10:00+02:00
+ Due to an ongoing upgrade of the Fedora Copr Servers, building might be delayed or failing. The upgrade should last ~3 hours.
+ <p>Due to an <a href="https://pagure.io/fedora-infrastructure/issue/10780">ongoing
+upgrade</a> of the Fedora
+Copr Servers, building might be delayed or failing. The upgrade should last ~3
+hours.</p>
+
+
+
+
+ [Resolved] SRPM builds done in Copr are failing
+ https://status.packit.dev/issues/2022-06-21-srpm-builds-failing/
+ Tue, 21 Jun 2022 10:18:22 +0200
+ https://status.packit.dev/issues/2022-06-21-srpm-builds-failing/
+ 2022-06-21T11:10:22+02:00
+ SRPM builds happening in Copr (for packages using the srpm_build_deps config key) are failing due to a dependency issue. We are working on a fix.
+ <p>SRPM builds happening in Copr (for packages using the <code>srpm_build_deps</code> config
+key) are failing due to a dependency issue. We are working on a fix.</p>
+
+
+
+
+ [Resolved] Networking Issue
+ https://status.packit.dev/issues/2022-02-03-networking-issue/
+ Thu, 03 Feb 2022 08:37:21 +0100
+ https://status.packit.dev/issues/2022-02-03-networking-issue/
+ 2022-02-03T11:17:21+01:00
+ Our production instance is getting various network issues. In logs, you can see errors with rsync, kerberos and other network related commands. We are sorry about the current state and are investigating the cause. We will let you informed once we know more.
+edit: The problem was with one of our task runners. Most probably because of a memory issue we are now trying to find and resolve. The runner was recreated and is working fine currently.
+ <p>Our production instance is getting various network issues.
+In logs, you can see errors with <code>rsync</code>, <code>kerberos</code> and other network related commands.
+We are sorry about the current state and are investigating the cause.
+We will let you informed once we know more.</p>
+<p>edit: The problem was with one of our task runners.
+Most probably because of a memory issue we are now trying to find and resolve.
+The runner was recreated and is working fine currently.
+We are getting an eye on it; please let us know if you find anything suspicious.</p>
+
+
+
+
+ [Resolved] February 1st outage
+ https://status.packit.dev/issues/2022-02-01-outage/
+ Tue, 01 Feb 2022 02:00:00 +0000
+ https://status.packit.dev/issues/2022-02-01-outage/
+ 2022-02-01T04:15:00+00:00
+ We had an outage on Tuesday EU night/early morning. Jobs during this time could get stuck in pending state. This was caused by a failed database schema migration because of a lack of memory during the automatic deployment of the new production images. We apologize for the inconvenience.
+ <p>We had an outage on Tuesday EU night/early morning. Jobs during this time could get stuck
+in pending state.
+This was caused by a failed database schema migration because of a lack of memory during the automatic deployment of the new production images.
+We apologize for the inconvenience.</p>
+
+
+
+
+ [Resolved] New production deployment
+ https://status.packit.dev/issues/2021-12-08-new-production-deployment/
+ Wed, 08 Dec 2021 02:00:00 +0000
+ https://status.packit.dev/issues/2021-12-08-new-production-deployment/
+ 2022-01-01T00:00:00+00:00
+ We’ve deployed production instance of Packit Service to a new Openshift cluster. If you spot anything unusual, let us know.
+ <p>We’ve deployed production instance of Packit Service to a new Openshift cluster.
+If you spot anything unusual, <a href="https://packit.dev/docs/faq/#how-can-i-contact-you">let us know</a>.</p>
+
+
+
+
+ [Resolved] Longer response times
+ https://status.packit.dev/issues/2021-12-07-response-delays/
+ Tue, 07 Dec 2021 09:06:14 +0100
+ https://status.packit.dev/issues/2021-12-07-response-delays/
+ 2021-12-07T11:49:14+01:00
+ Our service got overwhelmed by the events coming from a Fedora dist-git (caused by the epel-9 branching) causing the longer responses.
+We have scaled up the workers and the queue is getting shorter. We are planning the next steps to be resilient next time. Sorry about the disturbance.
+update: The queue is now empty and you can expect regular operational times.
+ <p>Our service got overwhelmed by the events coming from a Fedora dist-git (caused by the <code>epel-9</code> branching) causing the longer responses.</p>
+<p>We have scaled up the workers and the queue is getting shorter.
+We are planning the next steps to be resilient next time. Sorry about the disturbance.</p>
+<p>update: The queue is now empty and you can expect regular operational times.</p>
+
+
+
+
+ [Resolved] Problems with production database
+ https://status.packit.dev/issues/2021-11-18-production-db-problems/
+ Thu, 18 Nov 2021 02:33:00 +0000
+ https://status.packit.dev/issues/2021-11-18-production-db-problems/
+ 2021-11-19 00:10:00
+ After moving our production service to a new cluster, we’re seeing intermittent issues with connecting to the database. They are mostly manifesting in the dashboard. We need to decide next steps.
+ <p>After moving our production service to a new cluster, we’re seeing intermittent issues
+with connecting to the database. They are mostly manifesting in the dashboard.
+We need to decide next steps.</p>
+
+
+
+
+ [Resolved] Moving production Packit Service to a new cluster
+ https://status.packit.dev/issues/2021-11-16-auto-prod/
+ Thu, 18 Nov 2021 01:00:00 +0000
+ https://status.packit.dev/issues/2021-11-16-auto-prod/
+ 2021-11-18 02:33:00
+ We are going to move deployment of our production Packit Service to a new cluster. It should take about 1/2 an hour, during which the workers will be down and queries to API & dashboard won’t work properly because the database will be being migrated. Any requests submitted to the API meanwhile should, however, be fulfilled once the workers are up again.
+ <p>We are going to move deployment of our production Packit Service to a new cluster.
+It should take about 1/2 an hour, during which the workers will be down and queries
+to API & dashboard won’t work properly because the database will be being migrated.
+Any requests submitted to the API meanwhile should, however, be fulfilled once the workers are
+up again.</p>
+
+
+
+
+ [Resolved] Slow SRPM builds
+ https://status.packit.dev/issues/2021-09-08-outage/
+ Tue, 07 Sep 2021 20:30:00 +0200
+ https://status.packit.dev/issues/2021-09-08-outage/
+ 2021-09-16T13:54:56+00:00
+ Building of SRPMs for specific projects needs more memory and blocks the worker for a longer time, this causes other jobs to be slow and delayed. We are working to resolve this in the week of September 13th.
+ <p>Building of SRPMs for specific projects needs more memory and blocks the worker for a longer time,
+this causes other jobs to be slow and delayed. We are working to resolve this in the week of September 13th.</p>
+
+
+
+
+ [Resolved] August 27th outage
+ https://status.packit.dev/issues/2021-08-27-outage/
+ Fri, 27 Aug 2021 15:00:00 +0200
+ https://status.packit.dev/issues/2021-08-27-outage/
+ 2021-08-28 08:00:00+00:00
+ We had an outage which started in the Friday EU afternoon and was fully resolved on Saturday EU morning. Most of the jobs were failing during this time. This was caused by a misconfiguration on our side. We apologize for the inconvenience. We’re going to take steps to prevent the misconfiguration from happening in future.
+ <p>We had an outage which started in the Friday EU afternoon and was fully
+resolved on Saturday EU morning. Most of the jobs were failing during this
+time. This was caused by a misconfiguration on our side. We apologize for the
+inconvenience. We’re going to take steps to prevent the misconfiguration from
+happening in future.</p>
+
+
+
+
+ [Resolved] Summer is here โ๏ธ #2
+ https://status.packit.dev/issues/2021-07-56-vacation/
+ Thu, 01 Jul 2021 10:55:00 +0200
+ https://status.packit.dev/issues/2021-07-56-vacation/
+ 2021-07-06 19:10:40+00:00
+ There are 2 days of public holidays on July 5th and 6th in Czech Republic and since most of the team is located in CZ, we’ll be unavailable during these two days.
+ <p>There are 2 days of public holidays on July 5th and 6th in Czech Republic and since most of the team is located in CZ, we’ll be unavailable during these two days.</p>
+
+
+
+
+ [Resolved] Summer is here โ๏ธ
+ https://status.packit.dev/issues/2021-06-28-vacation/
+ Fri, 25 Jun 2021 10:55:00 +0200
+ https://status.packit.dev/issues/2021-06-28-vacation/
+ 2021-06-28 19:10:40+00:00
+ Most of the team will be unavailable on Monday, June 28th. We’ll try our best to resolve issues as soon as we’re back.
+ <p>Most of the team will be unavailable on Monday, June 28th. We’ll try our best to resolve issues as soon as we’re back.</p>
+
+
+
+
+ [Resolved] PostgreSQL db upgrade
+ https://status.packit.dev/issues/2021-06-04-postgresql-12/
+ Fri, 04 Jun 2021 07:00:00 +0000
+ https://status.packit.dev/issues/2021-06-04-postgresql-12/
+ 2021-06-04 09:10:40+00:00
+ Upgrading our PostgreSQL database from version 10 to version 12.
+ <p>Upgrading our PostgreSQL database from version 10 to version 12.</p>
+
+
+
+
+ [Resolved] Dashboard - unable to connect
+ https://status.packit.dev/issues/2021-06-02-dashboard-unable-to-connect/
+ Wed, 02 Jun 2021 11:16:00 +0000
+ https://status.packit.dev/issues/2021-06-02-dashboard-unable-to-connect/
+ 2021-06-02 11:40:00
+ API was down and therefore Dashboard was not showing any Jobs and after some time a “unable to connect …” error was shown.
+Cause: Unknown
+Fix: After restarting the API, the Dashboard started to work OK.
+ <p><strong>API</strong> was down and therefore <strong>Dashboard</strong> was not showing any Jobs
+and after some time a “unable to connect …” error was shown.</p>
+<p>Cause:
+Unknown</p>
+<p>Fix:
+After restarting the API, the Dashboard started to work OK.</p>
+
+
+
+
+
+ https://status.packit.dev/issues/2023-10-31-release-monitoring.org-outage/
+ Mon, 01 Jan 0001 00:00:00 +0000
+ https://status.packit.dev/issues/2023-10-31-release-monitoring.org-outage/
+
+ title: “release-monitoring.org outage” date: “2023-10-31T14:00:00+01:00” affected:
+ Copr resolved: true resolvedWhen: “2023-10-31T15:15:00+01:00” section: issue severity: disrupted release-monitoring.org cerfiticate has expired and that is causing Copr builds to fail.
+ <p>title: “release-monitoring.org outage”
+date: “2023-10-31T14:00:00+01:00”
+affected:</p>
+<ul>
+<li>Copr
+resolved: true
+resolvedWhen: “2023-10-31T15:15:00+01:00”
+section: issue
+severity: disrupted</li>
+</ul>
+<hr>
+<p>release-monitoring.org cerfiticate has expired and that is causing Copr builds to fail.</p>
+
+
+
+
+
diff --git a/issues/page/1/index.html b/issues/page/1/index.html
new file mode 100644
index 00000000..c6f68a10
--- /dev/null
+++ b/issues/page/1/index.html
@@ -0,0 +1,10 @@
+
+
+
+ https://status.packit.dev/issues/
+
+
+
+
+
+
diff --git a/issues/page/2/index.html b/issues/page/2/index.html
new file mode 100644
index 00000000..4be10e08
--- /dev/null
+++ b/issues/page/2/index.html
@@ -0,0 +1,1096 @@
+
+
+
+
+
+
+
+
+
+ Issues | Packit Service Status
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 