Koji is unavailable during F43 mass branching for external users. All builds that will be running at that time for the rawhide will be canceled and can be resubmitted by maintainers after the branching.
Once Fedora Linux 43 is branched we will reenable builds in Koji.
We will be updating and rebooting various servers. Services will be up or down during the outage window.
We are currently under DDoS affecting most of the fedoraproject.org services as it's affecting proxies and those are redirecting most of the traffic to other fedoraproject.org services.
This also affects authentication and obtaining kerberos tickets
Update: 17:00UTC: All services should be up and functional. Please let us know if you still see any issues.
The Datacenter move has been completed. Almost all services are back online and processing. Followups and known minor issues are being tracked in the above ticket.
If you notice anything amiss, please use our usual issue reporting path:
https://docs.fedoraproject.org/en-US/infra/day_to_day_fedora/
Thanks everyone for your support, we hope that the faster hardware in the new datacenter is helping empower you to do more great things for the fedora project.
Update: Aside a few smaller issues, everything should be up and working. Please report issues found in the normal process
We will be moving services and applications from our IAD2 datacenter to a new RDU3 one.
End user services such as: docs, mirrorlists, dns, pagure.io, torrent, fedorapeople, fedoraproject.org website, and tier0 download server will be unaffected and should continue to work normally through the outage window.
Other services may be up and down during the outage window.
Contributors are advised to wait until after the outage window to resume work and report issues with services.
Update 2025-07-01 01:00UTC:
Many services have been migrated, but there's still a number to bring up and validate. Tomorrow the buildsystem (koji) and related services will be migrated, then will we work to bring everything on line. Thanks again for everyone's patience during this move.
Please be aware that CentOS Stream infrastructure may be affected by the move as well.
Update 2025-07-02 2:30UTC:
We have migrated all our data and deployed all instances in the new datacenter, and now it's just a matter of bringing everything back online.
More services are back online, including: src.fedoraproject.org wiki authentication matrix bots openshift cluster consoles fasjson elections fmn accounts mote lists/mailman rabbitmq clusters registry downloads ...and others.
Unfortunately today we hit some network issues and were not able to bring koji back up. It should be back after some firewall changes tomorrow we hope. After that we plan to bring up the entire build/sign/compose pipeline as well as all the remaining applications.
Update 2025-07-03 01:00UTC:
Most services are up and running, but we are still working on bringing the build system fully back up. More details at: https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedoraproject.org/thread/CKIQPKWLISZNJZWWFFWVDENBUGHJW6R7/
We will be moving staging services and applications from our IAD2 datacenter to a new RDU3 one.
This only affects staging, end user services are unaffected.
Contributors who need to use staging are advised to wait until after the outage window to resume work and report issues with services.
We will be applying updates to all our servers and rebooting.
As part of this we will be doing a large upgrade to the Wiki, which will be down at least two hours.
The other services will be up or down during the outage window.
We will be applying a change to the firewall on most of our servers and possibly rebooting. Services may go down during the outage window, more likely some dropped/denied packets.
A.I. scrapers are hitting koji.fedoraproject.org, making the service unavailable to users. We are attempting to mitiate the issue as much as we can.
This issue has been resolved for now.
We will be applying updates to all our servers and rebooting into newer kernels. Services will be up or down during the outage window.
This outage impacts the Fedora Copr Frontend.
Mass branching happening, new builds in Koji might not happen.
We will be updating and rebooting various servers. Services will be up or down during the outage window.
This outage has started.
The outage has completed, please let us know if you find anything amiss
Our fedora-messaging bus is having issues. We are investigating them, but in the mean time various applications may emit errors or not process correctly.
The messaging cluster should be processing again as normal.
pagure.io is currently unreachable due to network issues. We have reported them and are waiting for a solution/ETA.
The network has returned.
We will switch authentication from OpenID to OIDC (OpenID Connect). There will be a short outage to do this.
We're updating Copr servers to F41
This outage impacts the copr-frontend and the copr-backend.
We will be updating and rebooting servers to pick up the recent RHEL 9.5 release as well as to move a number of instances to Fedora 41
This outage impacts the most maintainer / contributor services for some short windows during the outage.
We will switch authentication from OpenID to OIDC (OpenID Connect). There will be a short outage to do this.
We will be adding disk space to pagure.io. There will be a short outage to do this.
A 10G switch in our main datacenter needs upgrades. Many machines may drop off the network and come back during the outage window.
We plan to switch the community RHSM account to Simple Content Access. Systems should stay available during this period.
This outage could impact the copr-frontend and the copr-backend servers.
We will be upgrading koji to the latest upstream version, 1.35.0 with various bugfixes and enhancements.
During the outage the koji hubs will be down as the database schema is updated, and various builders may restart as their koji version is updated.
Additionally, we will be reinstalling some virthosts with rhel9.
We will be reinstalling some openqa virthost and database hosts as well as reinstalling workers to use a common partitioning and networking setup.
This outage impacts openqa and openqa-labs. During the outage, updates may not go stable; waiting for testing. After the outage is over, openqa will test all pending updates (no need to resubmit).
Networking gear at the datacenter that hosts pagure.io will be upgraded. Network may be up or down during the outage window as routers and switches are rebooted.
This outage impacts pagure.io.
We're updating copr packages to the new versions which will bring new features and bugfixes.
This outage impacts the copr-frontend and the copr-backend.
We will be applying updates to all our servers and rebooting into newer kernels. Services will be up or down during the outage window. As time permits we will also be reinstalling some servers.
The koji buildsytem is under heavy load and not processing requests correctly. We are investigating.
The buildsystem should be back up and going now.
Some networking hardware at our IAD datacenter will be updated. There may be small network outages as devices reboot into new firmware and routes and switch ports reconverge.
The auth cluster is unwell and authentication is currently not working. We are working to try and debug it and bring it back online.
Authentication should be working again.
Mass branching hapenning, new builds in koji might not happen.
We will be applying various updates and rebooting servers. During the outage window various services may be down for short periods of time.
Additionally, we will be upgrading builders to Fedora 40. This will mean that koji buildroot repodata will change to being zstd based (per fedora 40 createrepo_c defaults).
Some fedoraproject.org websites are showing unavailable. We are investigating the issue and hope to restore service soon.
UPDATE: sites should all be back online, sorry for the outage.
We will be moving fedorapeople.org to RHEL9. During the outage, services and content on fedorapeople.org will be unavailable.
Additionally, we will be repointing fedoraplanet.org to a new application that gets contributor rss feeds from the account system. Make sure you have all your rss feeds entered in there.
We will be upgrading the mailman to a newer version. During this outage the mailing lists will be down.
We will be upgrading the wiki to a newer version. During this outage the wiki will be down.
We're updating copr packages to the new versions which will bring new features and bugfixes.
This outage impacts the copr-frontend and the copr-backend.
We will be applying updates to all our servers and rebooting into newer kernels. Services will be up or down during the outage window.
We will be migrating a number of our database servers to RHEL9, newer versions of database software and more resources. During the migration services that use these databases may be offline completely. The small servers ( db-fas01 and db03 ) should move and have service restored sooner than the two larger hosts.
We will be upgrading our production OpenShift cluster that runs many of our applications. Normally, this would just be a 0 downtime event, but in this case we are switching networking models, so we need to completely reboot all the nodes, causing some applications to be unavailable for short time periods.
One of our datacenters primary network link is down. A secondary link is up, but some providers are still trying to route over the down link, resulting in connectivity problems.
Affected fedora resources include:
The provider is looking for the outage cause and networking is working on improving routing to get all traffic to use the secondary link.
Update: The link has been restored and everything should back to normal operation.
The fedora.im / chat.fedoraproject.org and fedoraproject.org matrix servers will be down for 30-45minutes for database maintainance. Messages sent during the outage should arrive after the outage via federation.
We're updating copr packages to the new versions which will bring new features and bugfixes.
This outage impacts the copr-frontend and the copr-backend.
There is an kerberos authentication outage happening right now. Affected is authentication using kinit or fkinit with Fedora kerberos realm.
koji will be upgraded to 1.34.0, which requires a schema update that touches many rows. We estimate this will take about 45minutes to complete and during that time, koji will be completely offline. Package maintainers are advised to not start any long term builds before the outage.
Affected Services:
koji
bodhi
This outage is starting.
The outage is now over.
Koji is unavailable during F40 mass branching. All builds that will be running at that time for the rawhide will be canceled and can be resubmitted by maintainers after the branching.
Once Fedora Linux 40 is branched we will reenable builds in Koji.
F40 is branched, koji is enabled now, should be back up and processing now.
Incoming emails to fedoraproject.org aliases are currently being rejected. Hopefully service will be restored soon.
Service has been restored.
Koji is currently down due a problem with it's sending of tag events to the fedora message bus. Once we have this corrected, we will bring things back up. Sorry for any trouble.
koji should be back up and processing now.
We will be applying updates and rebooting servers. No one service should be down long, but may be up and down in the outage window. Additionally, as time permits we will be doing the following additional work: * resizing disks on database servers * moving some database servers to rhel9 and newer postgresql * applying some firmware updates
21:20 - Unfortunately some updates got applied early to the koji builders, so builds may be affected and it looks like the outage is starting early. Sorry for the trouble.
02:00 - all services should be back to normal with the exception of koji (the fedora buildsystem). We are working to bring it back online.
Koji is still offline, we will bring it back up as soon as we can.
ssh access to fedorapeople.org is currently blocked. We are working with the datacenter to restore access.
ssh access should be restored.
The bodhi web interface is not working properly.
We are working on debugging the issue and bringing the service back online.
The bodhi application should be fully back online and working.
The account system is currently down due to a backend server issue. We are working to bring it back up.
This also impacts the ability to sign in to most Fedora services with Single Sign On (ipsilon), as well as kerberos authentication, and many other services that may use Fedora Accounts.
Authentication and the accounts system should be fully back online. The backend authentication cluster is still not back to normal, but we plan to work on that in the coming days without affecting service.
Thats for your patience.
The ISP related to the Fedora community cage is likely affected by a fiber cut in North Carolina. This situation appears to be causing routing issues from certain locations to the hosted machines, impacting various parts of the Fedora site.
The service is fully restored. The outage was due to a cut fiber that required repair.
This outage will take approximately 4 hours, and will impact the Fedora Copr Frontend and Fedora Copr Backend services.
The ISP related to the Fedora community cage is likely affected by a fiber cut in North Carolina. This situation appears to be causing routing issues from certain locations to the hosted machines, impacting various parts of the Fedora site.
Switches in the datacenter where pagure.io is will be updated and rebooted. There will be a short outage sometime during the outage window and during this time pagure.io will not be reachable.
This upgrade was completed.
We will be updating/rebooting various servers. Services may be up and down in this outage window.
All services should be up and working, please report any problems via the ticketing system.
This outage impacts performance ppc64le tasks in Fedora Copr frontend
We will be updating/rebooting various servers. Services may be up and down in this outage window.
This outage impacts the Fedora Copr Frontend.
The Matrix / libera.chat IRC bridge is unavailable currently. See ticket for more information.
At this point the bridge doesn't seem like it will be back anytime soon, and everything is moving over to matrix.
If any updates are available, we will update the community.
This outage impacts s390x builds in Fedora Coprfrontend
The s390x cluster at the Red Hat Westford location required emergency work on storage which brought down the z15 server which the Fedora and CentOS Stream builders are on. Work is expected to be completed at 19:00 UTC 2023-06-23 but may require further work to bring back builders.
koji builds and composes will be affected with builds waiting until the s390x builders are able to complete the work.
A networking device pagure.io uses to communicate with the world needs an urgent reboot to correct errors. pagure.io will be down for 10-20min as this device is fixed.
We are being hit with a Distributed Denial of Service attack on our dns infrastructure. We are working to mitigate it.
Update 2023-06-14 22:00UTC: The attack seems over, and we have added mitigations to reduce impact if it resumes.
Various network switches and routers will be updated and rebooted. This will result in short times of no connectivity, possibly lasting 20minutes at a time during the outage window.
We will be moving the koji buildsystem database (and the virthost it runs on) to RHEL9 and postgresql 15 (from RHEL8 and postgresql 12). This outage will happen while the outage of s390x builders is occuring to consolidate outages. During the outage window koji will be unavailable and builds will not be possible. After this outage is over, the s390x builder outage may still be ongoing, so archfull builds may still not complete until that outage is over.
The Red Hat Westford location will have more power line work done. This will require the building to be powered down in places which will affect various services like network connections and s390x builders.
koji builds and composes will be affected with builds waiting until the s390x builders are able to complete the work.
We will be upgrading our wiki and it's database server along with the virthosts they are hosted on.
Update: upgrade is taking longer than expected, so we are extending the outage window to complete the upgrade.
We're updating copr packages to the new versions which will bring new features and bugfixes.
This outage impacts the copr-frontend and the copr-backend.
We are seeing instability and issues with various web applications. We are investigating the problem.
Update: The problem database server has been identified and is being fixed.
The database server is back to working normally.
The virthost running the koji database has stopped responding. We are working on restoring service to koji.
The koji database is back up and working
We will be applying updates and rebooting various servers as well as re-installing some. Services may be up and down in the outage window and package maintainers are advised to avoid submmiting builds.
FMN's web interface will be inaccessible during the outage, but the backend service that sends notifications will keep running. No notification will be lost.
Element Matrix services are performing maintenance on chat.fedoraproject.org. During the maintenance window the service will be affected as follows:
Fedora Magazine's WordPress instance is reporting a critical error. This was due to an system error at the provider, which has been resolved.
We're updating copr packages to the new versions which will bring new features and bugfixes.
This outage impacts the copr-frontend and the copr-backend.
https://pagure.io is currently down. There were some issues causing email loops of comments that are being investigated.
Update: Pagure services restarted.
We will be applying updates and rebooting various servers as well as re-installing some. Services may be up and down in the outage window and package maintainers are advised to avoid submmiting builds.
Power is out in the datacenter with Fedora's s390x builders. All builds will queue until they are back online.
update: Power has been restored, but storage is being worked on.
update: Storage access has been fixed, and all builders are up and running.
We're updating copr packages to the new versions which will bring new features and bugfixes.
This outage impacts the copr-frontend and the copr-backend.
We will be upgrading and rebooting various servers. Services may be down during the outage window.
Servers were updated and rebooted. A few issues, but everything should be back up and running now. Thanks for your patience!
We'll just reboot copr-backend into a more powerful machine type.
This outage impacts the copr-backend.
We're updating copr packages to the new versions which will bring new features and bugfixes. We are also upgrading our servers to F37.
This outage impacts the copr-frontend and the copr-backend.
At about 8UTC the koji database server became unresponsive. We have rebooted it, but load is not coming back under control. We are intestigating the loading issues. Until them koji and associated applications (bodhi, etc) will be slow or down.
The database issues seem to have been caused by a large number of expensive database queries coming from a remote ip. We have blocked this ip address and brought everything back up. We will look further for mitigation of this sort of issue moving forward.
The performance is degraded. If possible, please delay your long-running builds and larger rebuilds till Sunday evening.
For more info see the outage schedule.
Fedora Copr RPM/DNF/YUM storage needs a major upgrade, we'll need to stop processing the build queue for about 20 hours, any running builds will be stopped and restarted on Sunday.
Two full short HTTP (2x5 minutes) outages are planned, too (dnf update will
have problems to update anything being hosted in Fedora Copr during these short
periods).
This will affect the Fedora Copr, especially the backend part.
We need to increase the power of one of the Copr VMs. The machine needs to be rebooted, so we expect a several minutes outage.
This will affect the copr-backend machine, so RPM repositories will be down.
The s390x builders are down due to a power issue at the facility they are located in. There are teams onsite working to replace a main breaker. No ETA at this time for completion. During this outage, no 'archfull' package build will complete, but builds started will complete once builders become available again.
The power has been restored and the builders are all back in service.
On 2022-09-27 1gb internal switches will be upgraded. 30min outage is likely during this time period.
On 2022-09-26 internet edge routers and switches will be upgraded. During this time period various Fedora services will be down as switches and routers are rebooted.
We're updating copr packages to the new versions which will bring new features and bugfixes.
This outage impacts the copr-frontend and the copr-backend.
We're updating copr packages to the new versions which will bring new features and bugfixes.
This outage impacts the copr-frontend and the copr-backend.
We will be updating and rebooting various servers to bring them up to date. During the outage window any services may be up and down as proxies and gateways are rebooted. Any fedoraproject services may be affected with the exception of mirrorlists and static web content.
This outage is now finished. Please alert us if there are any issues.
We're updating copr packages to the new versions which will bring new features and bugfixes.
This outage impacts the copr-frontend and the copr-backend.
We're updating copr packages to the new versions which will bring new features and bugfixes.
This outage impacts the copr-frontend and the copr-backend.
We're updating copr packages to the new versions which will bring new features and bugfixes.
This outage impacts the copr-frontend and the copr-backend.
Updating the host for the Fedora Wiki to Fedora 36, which bumps the version of MediaWiki to 1.37.1
We will be deploying a new release of Bodhi with a number of upgrades with openid connect authentication among those.
This outage has now started
This outage has now finished successfully
We will be updating and rebooting various servers to bring them up to date. During the outage window any services may be up and down as proxies and gateways are rebooted. Any fedoraproject services may be affected with the exception of mirrorlists and static web content.
All servers have been rebooted. Please report any issues you may see.
There is an ongoing outage of our main IAD2 datacenter. All servers may be down. We are working on fixing things.
This outage is over and everything should be back online.
We're updating copr packages to the new versions which will bring new features and bugfixes.
This outage impacts the copr-frontend and the copr-backend.
We will be updating and rebooting various servers to bring them up to date. During the outage window any services may be up and down as proxies and gateways are rebooted. Any fedoraproject services may be affected with the exception of mirrorlists and static web content.
This outage has started.
This outage is complete. There's some koji builders still down, but services should all be up.
Serveral virthosts (including the one running pagure.io) are not responding. We are working with datacenter staff to evaluate the issue and bring it back on line. There is no ETA at the moment.
This outage has been cleared up. Seems to have been a network issue.
We're updating copr packages to the new versions which will bring new features and bugfixes. This outage affects the copr-frontend and copr-backend
There is some unspecified network provider issue in the community cage in our lab. One of know issues is that we can not clone from (and thus build packages hosted on) GitHub on the majority of our builders.
We will be updating and rebooting various servers to bring them up to date. During the outage window any services may be up and down as proxies and gateways are rebooted. Any fedoraproject services may be affected with the exception of mirrorlists and static web content.
Copr is going to be migrated from an assigned DigiCert certificate (expires very soon) to the automated Let's Encrypt certificate. We expect that some HTTPD service hiccups can occur, but shouldn't take more then a few minutes.
This outage may affect users who are consuming the content from the Fedora Copr repositories, e.g. enabled through the "dnf copr" command.
copr-backend and the CDN can be affected.
We're updating copr packages to the new versions which will bring new features and bugfixes. This outage affects the copr-frontend and copr-backend
We're updating copr packages to the new versions which will bring new features and bugfixes. This outage impacted the copr-frontend and the copr-backend
We are making some improvements to the performance of the Datanommer database including adding the Timescaledb plugin, a migration to a new database was required as this involved some breaking changes, the migration has already taken place but the required apps will now be required to point to the new database
Datanommer/Datagrepper and any service which interacts with these will be affected
Copr aarch64 builders are temporarily down due to the Amazon AWS outage - https://status.aws.amazon.com/
Our copr-dist-git proxy is down, and overloaded for some reason. Issue is being debugged.
This outage impacts the copr-frontend and the copr-backend.
We're updating Copr servers from Fedora 33 to Fedora 35. The copr-backend storage (Copr build results) will stay mostly online during this outage but some downtime is expected.
This outage impacts the copr-frontend and the copr-backend.
We will be updating and rebooting various servers to bring them up to date. During the outage window any services may be up and down as proxies and gateways are rebooted. Any fedoraproject services may be affected with the exception of mirrorlists and static web content.
We're updating copr packages to the new versions which will bring new features and bugfixes. This outage impacts the copr-frontend and the copr-backend
Bodhi (The fedora updates system) will be upgraded to 5.7.1 (This is a reschedule of a previous outage on 2021-11-08)
Update: s390x builders were not able to be moved for a variety of reasons. We are going to fix those issues and retry next week.
We will be doing several maint tasks during this outage:
All the s390x builders will be moving from the current z13 maintframe to a z15 mainframe.
koji hub and builders will be updated from 1.25.1 to 1.26.1
Updates will be applied to all build servers and reboots done to the latest kernel.
Maintainers are advised to avoid starting builds before the outage that won't complete before the outage is over. Some builds may restart or need to be resubmitted if they are running during the maint window.
We're updating copr packages to the new versions which will bring new features and bugfixes. This outage impacted the copr-frontend and the copr-backend
The retrace server https://retrace.fedoraproject.org/faf/ is currently unreachable. This is being worked on currently We are investigating this issue. Please see the ticket below: https://pagure.io/fedora-infrastructure/issue/10238 Sorry for any trouble.
There was a misconfiguration in network config that was fixed, bringing the box back online.
We're updating copr packages to the new versions which will bring new features and bugfixes. This outage impacts the copr-frontend and the copr-backend
There are ongoing issues at our primary datacenter with networking. Many services are unreachable, or are dropping on and off the network. We are working with networking staff to isolate and fix this issue.
We're updating fedoraproject.org/wiki mediawiki instance. During this outage the wiki will be down.
Sporadically, a process on our authentication servers fails, causing user logins to fedora applications to fail until restarted. We are investigating this issue. Please see the ticket below and in particular: https://pagure.io/fedora-infrastructure/issue/9990#comment-745972 Sorry for any trouble.
We worked with upstream sssd developers to track down this sporadic and difficult to debug issue. Finally they found a reference count issue that might well have been causing this. We updated to a version with a test fix for this issue on 2021-09-30 and haven't seen the problem since then.
Users still seeing any authentication problems should file new tickets and we will assist in tracking those down.
Many thanks to SSSD developers and our users for their patience.
We will be applying updates and rebooting servers into new kernels. During the outage window some services may be up and down, but we will try and keep downtime as minimal as possible.
Most services may be affected for times during the outage window.
We're updating copr packages to the new versions which will bring new features and bugfixes. This outage impacts the copr-frontend and the copr-backend
The facility hosting the fedora s390x builders has a site wide power outage scheduled for June 4th to June 6th. We will be powering off these builders at 22UTC on 2021-06-04. It's likely that they will be back sometime late in the day of 2021-06-05, but it could be they are not back until 2021-06-06. During this time any builds that build against s390x will stall in 'free' state until the builders are back. No rawhide composes will be done in the outage window.
There are ongoing issues using Fedora's authentication system to login to various applications. We are investigating.
The authentication issues have been fixed.
The host for the Blockerbugs web application is being updated from Fedora Linux 32 to Fedora Linux 33. During this time, the Blockerbugs web applcation may not respond.
The Fedora project production OpenShift cluster is not processing new connections. We are investiaging. In the mean time accounts, bodhi and many other applications will be down.
This was due to a expired control plane cert, see ticket for details.
Bodhi (The fedora updates system) will be upgraded to 5.7.0 as well as it's underlaying OS to Fedora 34.
We're updating copr packages to the new versions which will bring new features and bugfixes. This outage impacted the copr-frontend and the copr-backend
A number of updates and patches are pending on our taiga instance teams.fedoraproject.org. During this time the service may be down or unresponsive.
We will be updating and rebooting various servers to bring them up to date and confirm changes from the recent account system migration. During the outage window services may be up or down as various systems reboot. No one service should be affected very long.
Most services will be affected, with the exception of: mirrorlists, docs, hotspot, geoip, and getfedora.
Replacement of FAS2 with the new Fedora Accounts All Fedora Services will be affected.
As this is the authentication system there may be issues logging in to some services. Users may not be able to access people.fedoraproject.org or secondary.fedoraproject.org during the outage. Maintainer test instances are a special case and as such will be in a "frozen" state for access after the outage. This means that no updates will be made to users or ssh keys on these machines.
We're updating copr packages to the new versions which will bring new features and bugfixes. This outage impacts the copr-frontend and the copr-backend
mbs-backend01.iad2.fedoraproject.org keeps reaching disk full so the disk needs to be grown. This involves a brief shutdown of the instance to enlarge the logical volume. This only affects the Fedora instance of Module Build Service.
We will be Moving the project from its temporary server to the final, production hosting.
Affected Services include:
We're updating copr packages to the new versions which will bring new features and bugfixes. This outage impacted the copr-frontend and the copr-backend
Upgrade to a more recent/patched taiga. Only affected service is taiga (ie: https://teams.fedoraproject.org)
We're updating copr packages to the new versions which will bring new features and bugfixes. This outage impacted the copr-frontend and the copr-backend
Service Degredation - OSBS. There will be a service degredation starting at 2020-11-26 09:30 UTC, which will last approximately 1 week.
Configuration of aarch64 cluster for production OSBS
Work is being carried out to add an aarch64 build cluster to production OSBS. While this work is being carried out you may experience some containers or flatpaks build failures.
We're updating copr packages to the new versions which will bring new features and bugfixes. We also moving away from Fedora 31 (soon EOL) to Fedora 33.
The outage will last approximately 3 hours. The copr-backend storage (copr build results) will be offline for a while because we need to migrate the data volume to a new machine and fix-up routing (so things like 'dnf update' will complain for enabled copr projects). We plan to minimize the backend storage outage though (expected "full" downtime is up to 15 minutes).
This outage impacts the copr-frontend and the copr-backend
Apply updates and new kernels for all systems running RHEL7/RHEL8 and Fedora 32.
Most services will see downtime as we run updates and reboot systems.
We are moving pagure.io to a new server running RHEL8 and python3.
Various updates requiring reboots are needed for all systems.
All services will see downtime as we run updates and reboot systems.
The colocation facility that Fedora systems are in needs to do general maintenance, cleanup and resiliency testing. They will be changing some PDU items and wiring during this time frame.
All services will be affected.
We will be upgrading koji to the recent 1.22.0 upstream release, updating builders to the latest kernel and updates, and doing some database vacuuming.
This started happening earlier today. They were off for about an hour, then back on for a while, then off again.
An upgrade to teams.fedoraproject.org will be applied. This has bugfixes and a security issue fixed.
Moving Copr servers from OpenStack to Amazon AWS because OpenStack is going to be shut down soon.
This outage impacts the copr-frontend, copr-backend, copr-distgen, and copr-keygen
we're only updating copr packages to the new versions, to get bugfixes and new features deployed. This outage impacts the copr-frontend and the copr-backend.
The taiga version on teams.fedoraproject.org will be updated to the latest version with various bugfixes and enhancements.
we're only updating copr packages to the new versions, to get bugfixes and new features deployed. This outage impacts the copr-frontend and the copr-backend.
Update all production servers to latest kernels and glibc. Get any associated security fixes pushed into production. All Fedora websites will see short term outages as reboots occur. Other services will be impacted during updates and reboots.
Update build systems to latest kernel and glibc. Get associated security fixes koji and related services in fedoraproject.org will see short term outages as reboots occur.
Update servers and services to newest glibc and kernels. Get any other security fixes onto systems also. All of stg.fedoraproject.org will see short term outages as reboots occur.
Upgrade of copr packages on Copr servers, to get in bug-fixes and new features. This outage impacts the copr-frontend, copr-dist-git and the copr-backend.
We will be updating the koji hub to version 1.19.1, including database schema update. Additionally we will be adding a patch to allow livemedia to compose in rawhide with the latest lorax version. This outage affectskoji
We will be upgrading bodhi to its 5.0 release. This release comes in with few UI changes but also task scheduling system (rabbitmq/celery based) allowing to offload some of the tasks currently performed in the front-end to distributed workers. This should results in the users experiencing a faster bodhi in some requests. This outage affects all services related to or relying on bodhi
We will be updating and rebooting the various servers that make up Fedora Infrastructure. While downtime of any one service should be short, services may go up and down in the outage window. All service may be affected with the exception of mirrorlists, dns and some build system services.
On Demand Compose service will be updated to latest version.
We will be updating and rebooting the various servers that make up the Fedora Build system. During the outage window, koji, src, koschei, bodhi, osbs, container registries, and package signing will be offline or up and down as we reboot servers.
Upgrade of copr packages on Copr servers, to get in bug-fixes and new features. This outage impacts the copr-frontend and the copr-backend.
During this outage window we plan to upgrade koji to the current 1.18.0 release, and additionally add memory to the koji database server to help improve performance. During the outage window new builds may fail to initiate, but in progress builds should complete as expected. This outage affectskoji
During outage window Koschei will be migrated from a set of libvirt-based KVM virtual machines to containerized deployment on OpenShift Container Platform. This outage affects Koschei
During the outage window Module Build Service will be updated to latest version.
Upgrade of copr packages on Copr servers, to get in bug-fixes and new features. This outage impacts the copr-frontend and the copr-backend.
RHEL-7.7 came out and a lot of other updates for various Fedora issues will need updates and reboots of systems. All staging/production and related services will be affected.
RHEL-7.7 came out and a lot of other updates for various Fedora issues will need updates and reboots of systems. All staging and build and related services will be affected.
Upgrade of copr packages on Copr servers, to get in bug-fixes and new features. This outage impacts the copr-frontend and the copr-backend.
We will be moving pagure.io from one datacenter to another. The time to sync the data is expected to be around an hour, but we are leaving time for configuration issues or network problems. We will try and minimise the downtime as much as possible. This outage impacts the pagure.io and docs.pagure.org.
We will be updating the nodes running mediawiki to Fedora 30 and thus also upgrading mediawiki to a newer version. The last scheduled upgrade was aborted due to some issues found in staging. These issues have been fixed and we are ready to finally upgrade. This outage impacts the Fedora Wiki.
There have been a number of kernel and low level library updates which require a system reboot to put into place. We will be rebooting productions servers which do require a full outage and downtime.
There have been a number of kernel and low level library updates which require a system reboot to put into place. We will be rebooting staging and several non-PHX2 servers which do not require a full outage due to redundancy or low service level expectations.
We will be adding additional disk space to src.fedoraproject.org. The outage should be short, but the host will be down while the additional disk is added and resized.
Upgrade of copr packages on Copr servers, to get in bug-fixes and new features. This outage impacts the copr-frontend and the copr-backend.
There have been a number of kernel and low level library updates which require a system reboot to put into place. We will be rebooting productions servers which do require a full outage and downtime.
There have been a number of kernel and low level library updates which require a system reboot to put into place. We will be rebooting staging and several non-PHX2 servers which do not require a full outage due to redundancy or low service level expectations.
Fedora Infrastructure would like to update and reboot all QA and build systems and services. This will update kernels, glibc, and systemd plus many other services on the affected systems.
Fedora Infrastructure would like to update and reboot all core systems and services. This will update kernels, glibc, and systemd plus many other services on the affected systems. All systems under fedoraproject.org will be affected.
Fedora Infrastructure would like to update and reboot all fedora staging and all web proxies. This will update kernels, glibc, and systemd plus many other services on the affected systems.
We will be updating koji for an upcoming CVE. This outage affectskoji
Upgrade of copr packages on Copr servers, to get in bug-fixes and new features. This outage impacts the copr-frontend and the copr-backend.
Various switches at the colocation are needing updates and reboots to get latest firmware working. While the outage should not take the entire 2 hours, it is being blocked out in case there are problems which are not realized and need backing out or other changes.
The facility that houses the Fedora s390 server will have a major power outage starting 2019-01-11 22:00 UTC and ending 2019-01-14 22:00 UTC. During this time the s390 builders will not be available and all builds will be queued up until they are available and will not complete.
Various switches at the colocation are needing updates and reboots to get latest firmware working. While the outage should not take the entire 5 hours, it is being blocked out in case there are problems which are not realized and need backing out or other changes.
We will be upgrading the pagure instance running at src.fedoraproject.org to the latest release of pagure.
We will be applying updates and rebooting servers as well as doing some hardware maint tasks (disk replacement, etc). Most fedoraproject.org services will be affected some in the outage window. Most services should only be down for a short time during the outage window.
Various kernel, library, and general security updates need to be applied to all servers. Due to the kernel, glibc, and other updates.. all systems will be rebooted afterwords. All build, production and related services will be affected.
We will be redeploying our production openshift with more compute nodes, and a number of new features enabled.
Affected Services:
We will be rebuilding registry.fedoraproject.org as Fedora in preparation for Flatpak work.
We will be updating and rebooting staging servers to new kernels as well as giving them new ip addresses. All staging systems will be down for some period during the outage
We will be re-installing our production openshift cluster. This will allow us to resize nodes to the recommended sizes for new openshift releases, upgrade to 3.9 and enable new features we want to try out. Apps will be reloaded as soon as the cluster is back up. Affected Services:
We will be re-installing our openshift cluster to enable new features and test for issues before we reinstall production. Apps will be reloaded via their ansible playbooks after the cluster is back up.
Affected Services:
During outage window Koschei will be reinstalled on Fedora 28.
Two of our 10GB switches are being upgraded. There may be some connectivity issues to koji builders and/or cloud instances while this work completes. Outages are expected to be short, but may occur.
We will be applying critical security updates and rebooting all servers. All services may be affected. We will try to keep any outages short, but expect some things to go up or down in the outage window.
Update of Mailman & Postorius (the admin UI)
During outage window Koschei will be updated to latest upstream version. Underlying operating system will be upgraded to Fedora 27
We need to update nuancier to its latest release which allows some time for the moderators between the end of the submissions and the start of the votes.
On December 4th, 2017 and running until December 8th 2017, Fedora Infrastructure will be moving servers from an existing datacenter location to another new section of the datacenter.
This move will allow us more space and power and to consolidate and rewire existing servers.
We will be upgrading the fedoraproject wiki to a recent version, including database migration to that new version. Additionally we will be moving it to use OpenID Connect for authentication. During the outage window the wiki will be completely unavailable.
The Pagure instance on upstreamfirst.fedorainfracloud.org needs to be upgraded to the latest released version and other regular updates need to be applied.
During outage window Koschei will be updated to latest upstream version. Backend will be reinstalled as Fedora 26. Frontend will need to be stopped for extended time to allow us to run database migration.
Reason for outage: During outage window Koschei will be updated to latest upstream version. Frontend will need to be stopped for short time to allow us to run database migration.
The Pagure instance on upstreamfirst.fedorainfracloud.org needs to be upgraded to the latest released version and other regular updates need to be applied.
The Mailman / Postorius / HyperKitty stack will be upgraded to the latest codebase. The mailing-list service itself should be back up quickly, but the web interfaces will take longer to upgrade.
We will be re-provisioning our jenkins master server on a Fedora 24 instance. All jenkins jobs and builds will be unavailable in the outage window.