Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feature request: Speed up installation of patches #133

Open
Klaas- opened this issue Mar 24, 2022 · 8 comments
Open

Feature request: Speed up installation of patches #133

Klaas- opened this issue Mar 24, 2022 · 8 comments
Assignees
@benank @kjohn-msft @rane-rajasi

Comments

@Klaas-
Copy link

Klaas- commented Mar 24, 2022

Currently you are doing a lot of yum/dnf transactions because you try to update all packages individually. While this makes sense if you conditionally patch servers, we usually update all packages during maintenance windows. An unconditional yum update / dnf update could speed up the update process a lot.
I would very much like the option to just update all packages, maybe with a fallback to individual packages or yum update --skip-broken / dnf update --nobest in case of dependency errors.

Greetings
Klaas
@Klaas-
Copy link
Author

Klaas- commented Mar 24, 2022

Also you can try to get rid of unneeded transactions, I see that you're already trying to look at dependencies, but you're failing to recognize them

example:

        DEBUG: RESOLVING DEPENDENCIES USING COMMAND: LANG=en_US.UTF8 sudo yum install --assumeno cryptsetup.x86_64
        DEBUG: Invoking package manager using: LANG=en_US.UTF8 sudo yum install --assumeno cryptsetup.x86_64
        VERBOSE: ==[SUCCESS]===============================================================
        DEBUG: - Return code from package manager: 1
        DEBUG: - Output from package manager:
        |       Last metadata expiration check: 1:12:31 ago on Thu 24 Mar 2022 07:10:56 AM UTC.
        |       Package cryptsetup-2.3.3-4.el8.x86_64 is already installed.
        |       Dependencies resolved.
        |       =========================================================================================
        |        Package           Arch    Version         Repository                                Size
        |       =========================================================================================
        |       Upgrading:
        |        cryptsetup        x86_64  2.3.3-4.el8_5.1 rhui-rhel-8-for-x86_64-baseos-rhui-rpms  190 k
        |        cryptsetup-libs   x86_64  2.3.3-4.el8_5.1 rhui-rhel-8-for-x86_64-baseos-rhui-rpms  474 k
        |
        |       Transaction Summary
        |       =========================================================================================
        |       Upgrade  2 Packages
        |
        |       Total download size: 664 k
        |       Operation aborted.
        VERBOSE: ==========================================================================
        DEBUG: - Inapplicable line: Last metadata expiration check: 1:12:31 ago on Thu 24 Mar 2022 07:10:56 AM UTC.
        DEBUG: - Inapplicable line: Package cryptsetup-2.3.3-4.el8.x86_64 is already installed.
        DEBUG: - Inapplicable line: Dependencies resolved.
        DEBUG: - Inapplicable line: ========================================================================================
=
        DEBUG: - Inapplicable line:  Package           Arch    Version         Repository                                Siz
e
        DEBUG: - Inapplicable line: ========================================================================================
=
        DEBUG: - Inapplicable line: Upgrading:
        DEBUG: - Inapplicable line:  cryptsetup        x86_64  2.3.3-4.el8_5.1 rhui-rhel-8-for-x86_64-baseos-rhui-rpms  190
k
        DEBUG: - Inapplicable line:  cryptsetup-libs   x86_64  2.3.3-4.el8_5.1 rhui-rhel-8-for-x86_64-baseos-rhui-rpms  474
k
        DEBUG: - Inapplicable line:
        DEBUG: - Inapplicable line: Transaction Summary
        DEBUG: - Inapplicable line: ========================================================================================
=
        DEBUG: - Inapplicable line: Upgrade  2 Packages
        DEBUG: - Inapplicable line:
        DEBUG: - Inapplicable line: Total download size: 664 k
        DEBUG: - Inapplicable line: Operation aborted.
        DEBUG: 0 dependent updates were found for package 'cryptsetup.x86_64'.

You are failing to notice that cryptsetup has cryptsetup-lib as dependency

Also it seems you try to install it twice, even though the first command succeeded:

        DEBUG: UPDATING PACKAGE (WITH DEPENDENCIES) USING COMMAND: sudo yum -y install cryptsetup-2.3.3-4.el8_5.1.x86_64
        DEBUG: <PackageInstallOutput>
        Last metadata expiration check: 1:12:35 ago on Thu 24 Mar 2022 07:10:56 AM UTC.
        Dependencies resolved.
        =========================================================================================
         Package           Arch    Version         Repository                                Size
        =========================================================================================
        Upgrading:
         cryptsetup        x86_64  2.3.3-4.el8_5.1 rhui-rhel-8-for-x86_64-baseos-rhui-rpms  190 k
         cryptsetup-libs   x86_64  2.3.3-4.el8_5.1 rhui-rhel-8-for-x86_64-baseos-rhui-rpms  474 k

        Transaction Summary
        =========================================================================================
        Upgrade  2 Packages

        Total download size: 664 k
        Downloading Packages:
        (1/2): cryptsetup-2.3.3-4.el8_5.1.x86_64.rpm    1.0 MB/s | 190 kB     00:00
        (2/2): cryptsetup-libs-2.3.3-4.el8_5.1.x86_64.r 2.4 MB/s | 474 kB     00:00
        --------------------------------------------------------------------------------
        Total                                           3.3 MB/s | 664 kB     00:00
        Running transaction check
        Transaction check succeeded.
        Running transaction test
        Transaction test succeeded.
        Running transaction
          Preparing        :                                                        1/1
          Upgrading        : cryptsetup-libs-2.3.3-4.el8_5.1.x86_64                 1/4
          Running scriptlet: cryptsetup-libs-2.3.3-4.el8_5.1.x86_64                 1/4
          Upgrading        : cryptsetup-2.3.3-4.el8_5.1.x86_64                      2/4
          Cleanup          : cryptsetup-2.3.3-4.el8.x86_64                          3/4
          Cleanup          : cryptsetup-libs-2.3.3-4.el8.x86_64                     4/4
          Running scriptlet: cryptsetup-libs-2.3.3-4.el8.x86_64                     4/4
          Verifying        : cryptsetup-2.3.3-4.el8_5.1.x86_64                      1/4
          Verifying        : cryptsetup-2.3.3-4.el8.x86_64                          2/4
          Verifying        : cryptsetup-libs-2.3.3-4.el8_5.1.x86_64                 3/4
          Verifying        : cryptsetup-libs-2.3.3-4.el8.x86_64                     4/4
        Installed products updated.

        Upgraded:
          cryptsetup-2.3.3-4.el8_5.1.x86_64    cryptsetup-libs-2.3.3-4.el8_5.1.x86_64

        Complete!

        </PackageInstallOutput>
        DEBUG: CHECKING PACKAGE INSTALL STATUS FOR: cryptsetup.x86_64 (2.3.3-4.el8_5.1)
        DEBUG: Invoking package manager using: sudo yum list installed cryptsetup.x86_64
        VERBOSE: ==[SUCCESS]===============================================================
        DEBUG: - Return code from package manager: 0
        DEBUG: - Output from package manager:
        |       Installed Packages
        |       cryptsetup.x86_64    2.3.3-4.el8_5.1    @rhui-rhel-8-for-x86_64-baseos-rhui-rpms
        VERBOSE: ==========================================================================
        DEBUG: Extracting package and version data...
        DEBUG: - Inapplicable line (0): Installed Packages
        DEBUG: - Installed version match found.
        DEBUG: UPDATING PACKAGE (WITH DEPENDENCIES) USING COMMAND: sudo yum -y install cryptsetup-2.3.3-4.el8_5.1.x86_64
        DEBUG: <PackageInstallOutput>
        Last metadata expiration check: 1:12:41 ago on Thu 24 Mar 2022 07:10:56 AM UTC.
        Package cryptsetup-2.3.3-4.el8_5.1.x86_64 is already installed.
        Dependencies resolved.
        Nothing to do.
        Complete!

        </PackageInstallOutput>

and that means a couple of lines further down in the logs you try to install cryptsetup-lib even though it was already installed as part of cryptsetup installation:

        DEBUG: RESOLVING DEPENDENCIES USING COMMAND: LANG=en_US.UTF8 sudo yum install --assumeno cryptsetup-libs.x86_64
        DEBUG: Invoking package manager using: LANG=en_US.UTF8 sudo yum install --assumeno cryptsetup-libs.x86_64
        VERBOSE: ==[SUCCESS]===============================================================
        DEBUG: - Return code from package manager: 0
        DEBUG: - Output from package manager:
        |       Last metadata expiration check: 1:13:40 ago on Thu 24 Mar 2022 07:10:56 AM UTC.
        |       Package cryptsetup-libs-2.3.3-4.el8_5.1.x86_64 is already installed.
        |       Dependencies resolved.
        |       Nothing to do.
        |       Complete!
        VERBOSE: ==========================================================================
        DEBUG: - Inapplicable line: Last metadata expiration check: 1:13:40 ago on Thu 24 Mar 2022 07:10:56 AM UTC.
        DEBUG: - Inapplicable line: Package cryptsetup-libs-2.3.3-4.el8_5.1.x86_64 is already installed.
        DEBUG: - Inapplicable line: Dependencies resolved.
        DEBUG: - Inapplicable line: Nothing to do.
        DEBUG: - Inapplicable line: Complete!
        DEBUG: 0 dependent updates were found for package 'cryptsetup-libs.x86_64'.
        DEBUG: UPDATING PACKAGE (WITH DEPENDENCIES) USING COMMAND: sudo yum -y install cryptsetup-libs-2.3.3-4.el8_5.1.x86_6
4
        DEBUG: <PackageInstallOutput>
        Last metadata expiration check: 1:13:43 ago on Thu 24 Mar 2022 07:10:56 AM UTC.
        Package cryptsetup-libs-2.3.3-4.el8_5.1.x86_64 is already installed.
        Dependencies resolved.
        Nothing to do.
        Complete!

Even though this whole path is unnecessary because you should notice that it was installed as a dependent package already you also here run yum multiple times, even though the first command already tells you that the version you're trying to install is already installed.

Greetings
Klaas

@Klaas-
Copy link
Author

Klaas- commented Mar 24, 2022 •
edited
Loading

And just as a simple speed comparison:
I start with image:

"imageReference": {
    "value": {
        "offer": "RHEL",
        "publisher": "RedHat",
        "sku": "8-LVM",
        "version": "8.4.2021052301"
    }

Both VMs same hardware B4ms, same region, same VNet etc.

and I update all packages that are in rhui (without reboot)
dnf update -y:

Begin: Thu Mar 24 08:36:13 UTC 2022
End:   Thu Mar 24 08:44:29 UTC 2022

Less than 9 Minutes

Update Extension:
Begin: 2022-03-24T07:20:25Z
Did not finish within the allotted time, but it took more than 3 1/2 hours and it didn't finish half of the packages :)

@kjohn-msft
Copy link
Collaborator

Thanks for the detailed feedback. There are two things here:

  1. Installing all packages at once - we're aware of the reduced transactional overhead of doing one package at a time. There are several design constraints we're working with that require this execution flow, but this will be reviewed once again.

  2. Dependencies not being detected - this is a bug that we will fix as soon as possible.

@Klaas-
Copy link
Author

Klaas- commented Mar 28, 2022

Hi,
also read comment #133 (comment)

the general speed is currently more on the "unacceptable" side of things. Something that could take less than twenty minutes takes 7 hours :)

During a "normal" update window this is not that important, I rarely see more than a couple of packages. But every 6 months there is a minor version update for rhel8, that includes hundreds of packages. So whatever is making updating hundreds of packages really slow needs to be addressed.

I only looked at the log output -- but it felt like there are sleeps between the yum operations, maybe for telemetry? Or is that just because log output gets written after yum operation concludes?

@Klaas-
Copy link
Author

Klaas- commented May 12, 2022

Any updates here @kjohn-msft ? with rhel 8.6 just released that means that you'll have a lot of servers that need updates with hundreds of packages.

@jmapro
Copy link

jmapro commented Nov 23, 2023 •
edited
Loading

Hi @kjohn-msft !
This issue is still alive ? I'm facing the same issue with more than 2 hours to patch the system (RHEL 8.8). This is unacceptable :(

@Klaas-
Copy link
Author

Klaas- commented Nov 23, 2023

@jmapro I gave up, I now do OS Updates with AWX and Ansible ... :) Cut down my usual maintenance duration to like 5 Minutes, and even a minor upgrade (like rhel 8.8 -> 8.9) will only take like 20 minutes. This used to take like 3-4 hours with the azure update management ... :)

@techmantel
Copy link

Facing the same issue on Ubuntu 22.04 and apt. Takes forever (which times out) just to patch a normal monthly package list of maybe 50-80 packages, which takes 5-10 minutes doing it manually being logged on to the box.
This is definitely unacceptable, please fix it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@benank benank

@kjohn-msft kjohn-msft

@rane-rajasi rane-rajasi

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@Klaas- @techmantel @benank @kjohn-msft @jmapro @rane-rajasi