1. Proposed title of this feature request
Add runbook_url to alerts in the OCP UI

2. What is the nature and description of the request?
If an alert includes a runbook_url label, then it should appear in the UI for the alert as a link.

3. Why does the customer need this? (List the business requirements here)
Customer can easily reach the alert runbook and be able to address their issues.

4. List any affected packages or components.

Feature Overview
Provide CSI drivers to replace all the intree cloud provider drivers we currently have. These drivers will probably be released as tech preview versions first before being promoted to GA.

Goals

• Framework for rapid creation of CSI drivers for our cloud providers
• CSI driver for AWS EBS
• CSI driver for AWS EFS
• CSI driver for GCP
• CSI driver for Azure
• CSI driver for VMware vSphere
• CSI Driver for Azure Stack
• CSI Driver for Alicloud
• CSI Driver for IBM Cloud

Requirements

Out of Scope

This work will only cover the drivers themselves, it will not include

• enhancements to the CSI API framework
• the migration to said drivers from the the intree drivers
• work for non-cloud provider storage drivers (FC-SAN, iSCSI) being converted to CSI drivers

Background, and strategic fit
In a future Kubernetes release (currently 1.21) intree cloud provider drivers will be deprecated and replaced with CSI equivalents, we need the drivers created so that we continue to support the ecosystems in an appropriate way.

Assumptions

• Storage SIG won't move out the changeover to a later Kubernetes release

Customer Considerations
Customers will need to be able to use the storage they want.

Documentation Considerations

• Target audience: cluster admins
• Updated content: update storage docs to show how to use these drivers (also better expose the capabilities)

Feature Overview

Add GA support for deploying OpenShift to IBM Public Cloud

Goals

Complete the existing gaps to make OpenShift on IBM Cloud VPC (Next Gen2) General Available

Requirements

• OpenShift can be deployed into an existing network provided by the user (https://issues.redhat.com/browse/CORS-2263)
• OpenShift can be deployed with both Public and Private publish options (https://issues.redhat.com/browse/SPLAT-731)

Optional requirements

• OpenShift can be deployed using Mint mode and STS for cloud provider credentials (future release, tbd)
• OpenShift can be deployed in disconnected mode https://issues.redhat.com/browse/SPLAT-737)
• OpenShift on IBM Cloud supports User Provisioned Infrastructure (UPI) deployment method (future release, 4.14?)

Background and Goal

Currently in OpenShift we do not support distributing hotfix packages to cluster nodes. In time-sensitive situations, a RHEL hotfix package can be the quickest route to resolving an issue. 

Acceptance Criteria

1. Under guidance from Red Hat CEE, customers can deploy RHEL hotfix packages to MachineConfigPools.
2. Customers can easily remove the hotfix when the underlying RHCOS image incorporates the fix.

Why?

• Decouple control and data plane. 
  • Customers do not pay Red Hat more to run HyperShift control planes and supporting infrastructure than Standalone control planes and supporting infrastructure.
• Improve security
  • Shift credentials out of cluster that support the operation of core platform vs workload
• Improve cost
  • Allow a user to toggle what they don’t need.
  • Ensure a smooth path to scale to 0 workers and upgrade with 0 workers.

Assumption

• A customer will be able to associate a cluster as “Infrastructure only”
• E.g. one option: management cluster has role=master, and role=infra nodes only, control planes are packed on role=infra nodes
• OR the entire cluster is labeled infrastructure , and node roles are ignored.

• Anything that runs on a master node by default in Standalone that is present in HyperShift MUST be hosted and not run on a customer worker node.

Doc: https://docs.google.com/document/d/1sXCaRt3PE0iFmq7ei0Yb1svqzY9bygR5IprjgioRkjc/edit 

Feature Overview

We drive OpenShift cross-market customer success and new customer adoption with constant improvements and feature additions to the existing capabilities of our OpenShift Core Networking (SDN and Network Edge). This feature captures that natural progression of the product.

Goals

• Feature enhancements (performance, scale, configuration, UX, ...)
• Modernization (incorporation and productization of new technologies)

Requirements

• Core Networking Stability
• Core Networking Performance and Scale
• Core Neworking Extensibility (Multus CNIs)
• Core Networking UX (Observability)
• Core Networking Security and Compliance

In Scope

• Network Edge (ingress, DNS, LB)
• SDN (CNI plugins, openshift-sdn, OVN, network policy, egressIP, egress Router, ...)
• Networking Observability

Out of Scope

There are definitely grey areas, but in general:

• CNV
• Service Mesh
• CNF

Documentation Considerations

Questions to be addressed:

• What educational or reference material (docs) is required to support this product feature? For users/admins? Other functions (security officers, etc)?
• Does this feature have doc impact?
• New Content, Updates to existing content, Release Note, or No Doc Impact
• If unsure and no Technical Writer is available, please contact Content Strategy.
• What concepts do customers need to understand to be successful in [action]?
• How do we expect customers will use the feature? For what purpose(s)?
• What reference material might a customer want/need to complete [action]?
• Is there source material that can be used as reference for the Technical Writer in writing the content? If yes, please link if available.
• What is the doc impact (New Content, Updates to existing content, or Release Note)?

This is a epic bucket for all activities surrounding the creation of declarative approach to release and maintain OLM catalogs.

We need to continue to maintain specific areas within storage, this is to capture that effort and track it across releases.

Goals

• To allow OCP users and cluster admins to detect problems early and with as little interaction with Red Hat as possible.
• When Red Hat is involved, make sure we have all the information we need from the customer, i.e. in metrics / telemetry / must-gather.
• Reduce storage test flakiness so we can spot real bugs in our CI.

Requirements

Out of Scope

n/a

Background, and strategic fit
With the expected scale of our customer base, we want to keep load of customer tickets / BZs low

Assumptions

Customer Considerations

Documentation Considerations

• Target audience: internal
• Updated content: none at this time.

Notes

In progress:

• CSI certification flakes a lot. We should fix it before we start testing migration.
  • In progress (API server restarts...) https://bugzilla.redhat.com/show_bug.cgi?id=1865857 

• Get local-storage-operator and AWS EBS CSI driver operator logs in must-gather (OLM-managed operators are not included there)
  • In progress for LSO (must-gather script being included in image) https://bugzilla.redhat.com/show_bug.cgi?id=1756096

• CI flakes:
  • Configurable timeouts for e2e tests
    • Azure is slow and times out often
    • Cinder times out formatting volumes
    • AWS resize test times out

High prio:

• Env. check tool for VMware - users often mis-configure permissions there and blame OpenShift. If we had a tool they could run, it might report better errors.
  • Should it be part of the installer?
  • Spike exists

• Add / use cloud API call metrics

• • Helps customers to understand why things are slow
  • Helps build cop to understand a flake
    • With a post-install step that filters data from Prometheus that’s still running in the CI job.
  • Ideas:
    • Cloud is throttling X% of API calls longer than Y seconds
    • Attach / detach / provisioning / deletion / mount / unmount / resize takes longer than X seconds?
  • Capture metrics of operations that are stuck and won’t finish.
    • Sweep operation map from executioner???
    • Report operation metric into the highest bucket after the bucket threshold (i.e. if 10minutes is the last bucket, report an operation into this bucket after 10 minutes and don’t wait for its completion)?
    • Ask the monitoring team?
  • Include in CSI drivers too.
    • With alerts too

• Report events for cloud issues
  • E.g. cloud API reports weird attach/provision error (e.g. due to outage)
• What volume plugins actually users use the most? https://issues.redhat.com/browse/STOR-324

Unsorted

• As the number of storage operators grows, it would be grafana board for storage operators
  • CSI driver metrics (from CSI sidecars + the driver itself  + its operator?)
  • CSI migration?

• Get aggregated logs in cluster
  • They're rotated too soon
  • No logs from dead / restarted pods
  • No tools to combine logs from multiple pods (e.g. 3 controller managers)

• What storage issues customers have? it was 22% of all issues.
  • Insufficient docs?
  • Probably garbage

• Document basic storage troubleshooting for our supports
  • What logs are useful when, what log level to use
  • This has been discussed during the GSS weekly team meeting; however, it would be beneficial to have this documented.

• Common vSphere errors, their debugging and fixing. 

• Document sig-storage flake handling - not all failed [sig-storage] tests are ours

tldr: three basic claims, the rest is explanation and one example

1. We cannot improve long term maintainability solely by fixing bugs.
2. Teams should be asked to produce designs for improving maintainability/debugability.
3. Specific maintenance items (or investigation of maintenance items), should be placed into planning as peer to PM requests and explicitly prioritized against them.

While bugs are an important metric, fixing bugs is different than investing in maintainability and debugability. Investing in fixing bugs will help alleviate immediate problems, but doesn't improve the ability to address future problems. You (may) get a code base with fewer bugs, but when you add a new feature, it will still be hard to debug problems and interactions. This pushes a code base towards stagnation where it gets harder and harder to add features.

One alternative is to ask teams to produce ideas for how they would improve future maintainability and debugability instead of focusing on immediate bugs. This would produce designs that make problem determination, bug resolution, and future feature additions faster over time.

I have a concrete example of one such outcome of focusing on bugs vs quality. We have resolved many bugs about communication failures with ingress by finding problems with point-to-point network communication. We have fixed the individual bugs, but have not improved the code for future debugging. In so doing, we chase many hard to diagnose problem across the stack. The alternative is to create a point-to-point network connectivity capability. this would immediately improve bug resolution and stability (detection) for kuryr, ovs, legacy sdn, network-edge, kube-apiserver, openshift-apiserver, authentication, and console. Bug fixing does not produce the same impact.

We need more investment in our future selves. Saying, "teams should reserve this" doesn't seem to be universally effective. Perhaps an approach that directly asks for designs and impacts and then follows up by placing the items directly in planning and prioritizing against PM feature requests would give teams the confidence to invest in these areas and give broad exposure to systemic problems.

Relevant links:

• Documentation:
  • Edge Diagnostics Scratchpad, our team's internal diagnostic guide.
  • Troubleshooting OCP networking issues - The complete guide, the SDN team's diagnostic guide.
  • Linux Performance, Brendan Gregg's guide to analyzing Linux performance issues.
  • RFC: A proper feedback loop on Alerts.
  • OpenShift Router Reload Technical Overview on Access.
  • Performance Scaling HAProxy with OpenShift on Access.
  • How to collect worker metrics to troubleshoot CPU load, memory pressure and interrupt issues and networking on worker nodes in OCP 4 on Access.
  • OpenShift Performance and Scale Knowledge Base on Mojo, results from OpenShift scalability testing.
  • Scalability and performance, OCP 4.5 documentation about the router's currently known scalability limits.
  • Scaling OpenShift Container Platform HAProxy Router, OCP 3.11 documentation about the manual performance configuration that was possible in OCP 3.
  • Timing web requests with cURL and Chrome from the Cloudflare blog.
  • tcpdump advanced filters, some useful tcpdump commands.
  • OpenShift SDN - Networking, OCP 3.11 documentation on the SDN (useful background reading).
  • Ingress Operator and Controller Status Conditions, design document for improved status condition reporting.
  • Observability tips for HAProxy, a slide deck by Willy Tarreau.
  • Interesting Traces - Out of Order versus Retransmissions, analysis using tshark.
  • The PCP Book: A Complete Documentation of Performance Co-Pilot, by Yogesh Babar.
  • Debugging kernel networking bug, brief guide to using SystemTap on RHCOS.
  • Troubleshooting throughput issues from the OCP 4.5 documentation.
  • Troubleshooting OpenShift Clusters and Workloads.
  • Red Hat Enterprise Linux Network Performance Tuning Guide (PDF).
  • openshift/enhancements#289 stability: point to point network check, a diagnostic built into the kube-apiserver operator.
• Diagnostic tools:
  • dropwatch to watch for packet drops.
  • ethtool to check NIC configuration.
  • iovisor/bcc: BCC - Tools for BPF-based Linux IO analysis, networking, monitoring, and more to trace and diagnose various issues in the networking stack.
  • r-curler to gather timing information about HTTP/HTTPS connections.
  • route-monitor, to monitor routes for reachability.
  • hping(3), a programmable packet generator.
  • OpenTracing / Jaeger in OpenShift.
  • node-problem-detector, a possible integration point for new diagnostics.
  • Using SystemTap by Brendan Gregg.
  • DTrace SystemTap cheatsheet (PDF).
• Visualization and more sophisticated diagnostic tools:
  • eldadru/ksniff, kubectl plugin for tcpdump & Wireshark.
  • ironcladlou/ditm, Dan's "Dan in the Middle" tool.
  • Skydive, network diagnostic and visualization tool.
  • ali, a "load testing tool capable of performing real-time analysis" with visualization.
• Testing tools:
  • stress-ng, a general stress-loading tool (CPU, filesystem, network, ...).
  • mb, the networking benchmarking tool written and used by Jiri Mencak from our Perf+Scale team.
• Case studies:
  • BZ1763206 is an example of diagnosing DNS latency/timeouts.
  • BZ1829779 Investigation details the diagnosis of route latency.
  • BZ1845545 is an example of diagnosing misconfigured DNS for an external LB.
  • Debugging network stalls on Kubernetes, from the GitHub Blog, about diagnosing Kubernetes performance issues related to ksoftirqd.

Feature Overview

• This Section:* High-Level description of the feature ie: Executive Summary

• Note: A Feature is a capability or a well defined set of functionality that delivers business value. Features can include additions or changes to existing functionality. Features can easily span multiple teams, and multiple releases.

Goals

• This Section:* Provide high-level goal statement, providing user context and expected user outcome(s) for this feature

• …

Requirements

• This Section:* A list of specific needs or objectives that a Feature must deliver to satisfy the Feature.. Some requirements will be flagged as MVP. If an MVP gets shifted, the feature shifts. If a non MVP requirement slips, it does not shift the feature.

(Optional) Use Cases

This Section: 

• Main success scenarios - high-level user stories

• Alternate flow/scenarios - high-level user stories

• ...

Questions to answer…

• ...

Out of Scope

• …

Background, and strategic fit

This Section: What does the person writing code, testing, documenting need to know? What context can be provided to frame this feature.

Assumptions

• ...

Customer Considerations

• ...

Documentation Considerations

Questions to be addressed:

• What educational or reference material (docs) is required to support this product feature? For users/admins? Other functions (security officers, etc)?

• Does this feature have doc impact?  

• New Content, Updates to existing content,  Release Note, or No Doc Impact

• If unsure and no Technical Writer is available, please contact Content Strategy.

• What concepts do customers need to understand to be successful in [action]?

• How do we expect customers will use the feature? For what purpose(s)?

• What reference material might a customer want/need to complete [action]?

• Is there source material that can be used as reference for the Technical Writer in writing the content? If yes, please link if available.

• What is the doc impact (New Content, Updates to existing content, or Release Note)?

Feature Overview

• As an infrastructure owner, I want a repeatable method to quickly deploy the initial OpenShift cluster.

• As an infrastructure owner, I want to install the first (management, hub, “cluster 0”) cluster to manage other (standalone, hub, spoke, hub of hubs) clusters.

Goals

• Enable customers and partners to successfully deploy a single “first” cluster in disconnected, on-premises settings

Requirements

4.11 MVP Requirements

• Customers and partners needs to be able to download the installer
• Enable customers and partners to deploy a single “first” cluster (cluster 0) using single node, compact, or highly available topologies in disconnected, on-premises settings
• Installer must support advanced network settings such as static IP assignments, VLANs and NIC bonding for on-premises metal use cases, as well as DHCP and PXE provisioning environments.
• Installer needs to support automation, including integration with third-party deployment tools, as well as user-driven deployments.
• In the MVP automation has higher priority than interactive, user-driven deployments.
• For bare metal deployments, we cannot assume that users will provide us the credentials to manage hosts via their BMCs.
• Installer should prioritize support for platforms None, baremetal, and VMware.
• The installer will focus on a single version of OpenShift, and a different build artifact will be produced for each different version.
• The installer must not depend on a connected registry; however, the installer can optionally use a previously mirrored registry within the disconnected environment.

Use Cases

• As a Telco partner engineer (Site Engineer, Specialist, Field Engineer), I want to deploy an OpenShift cluster in production with limited or no additional hardware and don’t intend to deploy more OpenShift clusters [Isolated edge experience].
• As a Enterprise infrastructure owner, I want to manage the lifecycle of multiple clusters in 1 or more sites by first installing the first  (management, hub, “cluster 0”) cluster to manage other (standalone, hub, spoke, hub of hubs) clusters [Cluster before your cluster].
• As a Partner, I want to package OpenShift for large scale and/or distributed topology with my own software and/or hardware solution.
• As a large enterprise customer or Service Provider, I want to install a “HyperShift Tugboat” OpenShift cluster in order to offer a hosted OpenShift control plane at scale to my consumers (DevOps Engineers, tenants) that allows for fleet-level provisioning for low CAPEX and OPEX, much like AKS or GKE [Hypershift].
• As a new, novice to intermediate user (Enterprise Admin/Consumer, Telco Partner integrator, RH Solution Architect), I want to quickly deploy a small OpenShift cluster for Poc/Demo/Research purposes.

Questions to answer…

Out of Scope

Out of scope use cases (that are part of the Kubeframe/factory project):

• As a Partner (OEMs, ISVs), I want to install and pre-configure OpenShift with my hardware/software in my disconnected factory, while allowing further (minimal) reconfiguration of a subset of capabilities later at a different site by different set of users (end customer) [Embedded OpenShift].
• As an Infrastructure Admin at an Enterprise customer with multiple remote sites, I want to pre-provision OpenShift centrally prior to shipping and activating the clusters in remote sites.

Background, and strategic fit

• This Section: What does the person writing code, testing, documenting need to know? What context can be provided to frame this feature.

Assumptions

1. The user has only access to the target nodes that will form the cluster and will boot them with the image presented locally via a USB stick. This scenario is common in sites with restricted access such as government infra where only users with security clearance can interact with the installation, where software is allowed to enter in the premises (in a USB, DVD, SD card, etc.) but never allowed to come back out. Users can't enter supporting devices such as laptops or phones.
2. The user has access to the target nodes remotely to their BMCs (e.g. iDrac, iLo) and can map an image as virtual media from their computer. This scenario is common in data centers where the customer provides network access to the BMCs of the target nodes.
3. We cannot assume that we will have access to a computer to run an installer or installer helper software.

Customer Considerations

• ...

Documentation Considerations

Questions to be addressed:

• What educational or reference material (docs) is required to support this product feature? For users/admins? Other functions (security officers, etc)?
• Does this feature have doc impact?
• New Content, Updates to existing content, Release Note, or No Doc Impact
• If unsure and no Technical Writer is available, please contact Content Strategy.
• What concepts do customers need to understand to be successful in [action]?
• How do we expect customers will use the feature? For what purpose(s)?
• What reference material might a customer want/need to complete [action]?
• Is there source material that can be used as reference for the Technical Writer in writing the content? If yes, please link if available.
• What is the doc impact (New Content, Updates to existing content, or Release Note)?

References

• Bootable Ephemeral Installer Image for Disconnected Cluster 0 Deployments

Feature Overview

RHEL CoreOS should be updated to RHEL 9.2 sources to take advantage of newer features, hardware support, and performance improvements.

Requirements

• RHEL 9.x sources for RHCOS builds starting with OCP 4.13 and RHEL 9.2.

(Optional) Use Cases

• 9.2 Preview via Layering No longer necessary assuming we stay the course of going all in on 9.2

Assumptions

• ...

Customer Considerations

• ...

Documentation Considerations

Questions to be addressed:

• What educational or reference material (docs) is required to support this product feature? For users/admins? Other functions (security officers, etc)?
• Does this feature have doc impact?
• New Content, Updates to existing content, Release Note, or No Doc Impact
• If unsure and no Technical Writer is available, please contact Content Strategy.
• What concepts do customers need to understand to be successful in [action]?
• How do we expect customers will use the feature? For what purpose(s)?
• What reference material might a customer want/need to complete [action]?
• Is there source material that can be used as reference for the Technical Writer in writing the content? If yes, please link if available.
• What is the doc impact (New Content, Updates to existing content, or Release Note)?

Feature Overview (aka. Goal Summary)  

The MCO should properly report its state in a way that's consistent and able to be understood by customers, troubleshooters, and maintainers alike. 

Some customer cases have revealed scenarios where the MCO state reporting is misleading and therefore could be unreliable to base decisions and automation on.

In addition to correcting some incorrect states, the MCO will be enhanced for a more granular view of update rollouts across machines.

Epic Goal

• Update OpenShift components that are owned by the Builds + Jenkins Team to use Kubernetes 1.25

Why is this important?

• Our components need to be updated to ensure that they are using the latest bug/CVE fixes, features, and that they are API compatible with other OpenShift components.

Acceptance Criteria

• Existing CI/CD tests must be passing

This is epic tracks "business as usual" requirements / enhancements / bug fixing of Insights Operator.

This epic contains all the Dynamic Plugins related stories for OCP release-4.11 

Epic Goal

• Track all the stories under a single epic

Acceptance Criteria

This epic contains all the Dynamic Plugins related stories for OCP release-4.12

Epic Goal

• Track all the stories under a single epic

Acceptance Criteria

This epic contains all the OLM related stories for OCP release-4.12

Epic Goal

• Track all the stories under a single epic

An epic we can duplicate for each release to ensure we have a place to catch things we ought to be doing regularly but can tend to fall by the wayside.

Epic Goal

• Enable OpenShift IPI Installer to deploy OCP to a shared VPC in GCP.
• The host project is where the VPC and subnets are defined. Those networks are shared to one or more service projects.
• Objects created by the installer are created in the service project where possible. Firewall rules may be the only exception.
• Documentation outlines the needed minimal IAM for both the host and service project.

Why is this important?

• Shared VPC's are a feature of GCP to enable granular separation of duties for organizations that centrally manage networking but delegate other functions and separation of billing. This is used more often in larger organizations where separate teams manage subsets of the cloud infrastructure. Enterprises that use this model would also like to create IPI clusters so that they can leverage the features of IPI. Currently organizations that use Shared VPC's must use UPI and implement the features of IPI themselves. This is repetative engineering of little value to the customer and an increased risk of drift from upstream IPI over time. As new features are built into IPI, organizations must become aware of those changes and implement them themselves instead of getting them "for free" during upgrades.

Scenarios

1. Deploy cluster(s) into service project(s) on network(s) shared from a host project.

Acceptance Criteria

• CI - MUST be running successfully with tests automated
• Release Technical Enablement - Provide necessary release enablement details and documents.
• ...

Done Checklist

• CI - CI is running, tests are automated and merged.
• Release Enablement <link to Feature Enablement Presentation>
• DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
• DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
• DEV - Downstream build attached to advisory: <link to errata>
• QE - Test plans in Polarion: <link or reference to Polarion>
• QE - Automated tests merged: <link or reference to automated tests>
• DOC - Downstream documentation merged: <link to meaningful PR>

1. Proposed title of this feature request
Basic authentication for Helm Chart repository in helmchartrepositories.helm.openshift.io CRD.

2. What is the nature and description of the request?
As of v4.6.9, the HelmChartRepository CRD only supports client TLS authentication through spec.connectionConfig.tlsClientConfig.

3. Why do you need this? (List the business requirements here)
Basic authentication is widely used by many chart repositories managers (Nexus OSS, Artifactory, etc.)
Helm CLI also supports them with the helm repo add command.
https://helm.sh/docs/helm/helm_repo_add/

4. How would you like to achieve this? (List the functional requirements here)
Probably by extending the CRD:

spec:
connectionConfig:
username: username
password:
secretName: secret-name

The secret namespace should be openshift-config to align with the tlsClientConfig behavior.

5. For each functional requirement listed in question 4, specify how Red Hat and the customer can test to confirm the requirement is successfully implemented.
Trying to pull helm charts from remote private chart repositories that has disabled anonymous access and offers basic authentication.
E.g.: https://github.com/sonatype/docker-nexus

Epic Goal

• Support manifest lists by image streams and the integrated registry. Clients should be able to pull/push manifests lists from/into the integrated registry. They also should be able to import images via `oc import-image` and them pull them from the internal registry.

Why is this important?

• Manifest lists are becoming more and more popular. Customers want to mirror manifest lists into the registry and be able to pull them by digest.

Scenarios

1. Manifest lists can be pushed into the integrated registry
2. Imported manifests list can be pulled from the integrated registry
3. Image triggers work with manifest lists

Acceptance Criteria

• CI - MUST be running successfully with tests automated
• Release Technical Enablement - Provide necessary release enablement details and documents.
• Existing functionality shouldn't change its behavior

Dependencies (internal and external)

1. ...

Previous Work (Optional)

1. https://github.com/openshift/enhancements/blob/master/enhancements/manifestlist/manifestlist-support.md

Open questions

1. Can we merge creation of images without having the pruner?

Done Checklist

• CI - CI is running, tests are automated and merged.
• Release Enablement <link to Feature Enablement Presentation>
• DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
• DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
• DEV - Downstream build attached to advisory: <link to errata>
• QE - Test plans in Polarion: <link or reference to Polarion>
• QE - Automated tests merged: <link or reference to automated tests>
• DOC - Downstream documentation merged: <link to meaningful PR>

This is a follow up Epic to https://issues.redhat.com/browse/MCO-144, which aimed to get in-place upgrades for Hypershift. This epic aims to capture additional work to focus on using CoreOS/OCP layering into Hypershift, which has benefits such as:

 - removing or reducing the need for ignition

 - maintaining feature parity between self-driving and managed OCP models

 - adding additional functionality such as hotfixes

Epic Goal

• We need the installer to accept a LB type from user and then we could set type of LB in the following object.
  oc get ingress.config.openshift.io/cluster -o yaml
  Then we can fetch info from this object and reconcile the operator to have the NLB changes reflected.

This is an API change and we will consider this as a feature request.

Why is this important?

https://issues.redhat.com/browse/NE-799 Please check this for more details

Scenarios

https://issues.redhat.com/browse/NE-799 Please check this for more details

Acceptance Criteria

• CI - MUST be running successfully with tests automated
• Release Technical Enablement - Provide necessary release enablement details and documents.
• ...

Dependencies (internal and external)

1. installer
2. ingress operator

Previous Work (Optional):

 No

Open questions::

N/A

Done Checklist

• CI - CI is running, tests are automated and merged.
• Release Enablement <link to Feature Enablement Presentation>
• DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
• DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
• DEV - Downstream build attached to advisory: <link to errata>
• QE - Test plans in Polarion: <link or reference to Polarion>
• QE - Automated tests merged: <link or reference to automated tests>
• DOC - Downstream documentation merged: <link to meaningful PR>

Make sure that the CSI driver automatically updates oVirt credentials when they are updated in OpenShift.

We need tests for the ovirt-csi-driver and the cluster-api-provider-ovirt. These tests help us to

• minimize bugs,
• reproduce and fix them faster and
• pin down current behavior of the driver

Also, having dedicated tests on lower levels with a smaller scope (unit, integration, ...) has the following benefits:

• fast feedback cycle (local test execution)
• developer in-code documentation
• easier onboarding for new contributers
• lower resource consumption

Goal

Provide a form driven experience to allow cluster admins to manage the perspectives to meet the ACs below.

Problem:

We have heard the following requests from customers and developer advocates:

• Some admins do not want to provide access to the Developer Perspective from the console
• Some admins do not want to provide non-priv users access to the Admin Perspective from the console

Acceptance criteria:

1. Cluster administrator is able to "hide" the admin perspective for non-priv users
2. Cluster administrator is able to "hide" the developer perspective for all users
3. Be user that User Preferences for individual users behaves appropriately. If only one perspective is available, the perspective switcher is not needed.

Dependencies (External/Internal):

Design Artifacts:

Exploration:

• Miro board: https://miro.com/app/board/uXjVOoSAPbA=/
• Recording: https://drive.google.com/file/d/10JUmaTMdfMH1NceHJW_74Y2cUxqwqmD_/view

Note:

Problem:

Customers don't want their users to have access to some/all of the items which are available in the Developer Catalog.  The request is to change access for the cluster, not per user or persona.

Goal:

Provide a form driven experience to allow cluster admins easily disable the Developer Catalog, or one or more of the sub catalogs in the Developer Catalog.

Why is it important?

Multiple customer requests.

Acceptance criteria:

1. As a cluster admin, I can hide/disable access to the developer catalog for all users across all namespaces.
2. As a cluster admin, I can hide/disable access to a specific sub-catalog in the developer catalog for all users across all namespaces.
   1. Builder Images
   2. Templates
   3. Helm Charts
   4. Devfiles
   5. Operator Backed

Notes

We need to consider how this will work with subcatalogs which are installed by operators: VMs, Event Sources, Event Catalogs, Managed Services, Cloud based services

Dependencies (External/Internal):

Design Artifacts:

Exploration:

Note:

Epic Goal

• Facilitate the transition to for OLM and content to PSA enforcing the `restricted` security profile
• Use the label synch'er to enforce the required security profile
• Current content should work out-of-the-box as is
• Upgrades should not be blocked

Why is this important?

• PSA helps secure the cluster by enforcing certain security restrictions that the pod must meet to be scheduled
• 4.12 will enforce the `restricted` profile, which will affect the deployment of operators in `openshift-*` namespaces 

Scenarios

1. Admin installs operator in an `openshift-*`namespace that is not managed by the label syncher -> label should be applied
2. Admin installs operator in an `openshift-*` namespace that has a label asking the label syncher to not reconcile it -> nothing changes

Acceptance Criteria

• CI - MUST be running successfully with tests automated
• Release Technical Enablement - Provide necessary release enablement details and documents.
• Done only downstream
• Transition documentation written and reviewed

Dependencies (internal and external)

1. label syncher (still searching for the link)

Open questions::

1. Is this only for openshift-* namespaces?

Resources

• notes on label syncer and what's needed for OLM
• PSA Communication - explains all required changes to adapt workloads to PSA under OCP 4.12 (inc. SCC)

Stakeholders

• Daniel S...?

Done Checklist

• CI - CI is running, tests are automated and merged.
• Release Enablement <link to Feature Enablement Presentation>
• DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
• DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
• DEV - Downstream build attached to advisory: <link to errata>
• QE - Test plans in Polarion: <link or reference to Polarion>
• QE - Automated tests merged: <link or reference to automated tests>
• DOC - Downstream documentation merged: <link to meaningful PR>

This epic tracks network tooling improvements for 4.12

New framework and process should be developed to make sharing network tools with devs, support and customers convenient. We are going to add some tools for ovn troubleshooting before ovn-k goes default, also some tools that we got from customer cases, and some more to help analyze and debug collected logs based on stable must-gather/sosreport format we get now thanks to 4.11 Epic.

Our estimation for this Epic is 1 engineer * 2 Sprints

WHY:
This epic is important to help improve the time it takes our customers and our team to understand an issue within the cluster.
A focus of this epic is to develop tools to quickly allow debugging of a problematic cluster. This is crucial for the engineering team to help us scale. We want to provide a tool to our customers to help lower the cognitive burden to get at a root cause of an issue.

OCP/Telco Definition of Done
Epic Template descriptions and documentation.

<--- Cut-n-Paste the entire contents of this description into your new Epic --->

Epic Goal

• Come up with a consistent way to detect node down on OCP and hypershift. Current mechanism for OCP (probe port 9) does not work for hypershift, meaning, hypershift node down detection will be longer (~40 secs). We should aim to have a common mechanism for both. As well, we should consider alternatives to the probing port 9. Perhaps BFD, or other detection.
• Get clarification on node down detection times. Some customers have (apparently) asked for detection on the order of 100ms, recommendation is to use multiple Egress IPs, so this may not be a hard requirement. Need clarification from PM/Customers.

Why is this important?

• …

Scenarios

1. ...

Acceptance Criteria

• CI - MUST be running successfully with tests automated
• Release Technical Enablement - Provide necessary release enablement details and documents.
• ...

Dependencies (internal and external)

1. ...

Previous Work (Optional):

1. …

Open questions::

1. …

Done Checklist

• CI - CI is running, tests are automated and merged.
• Release Enablement <link to Feature Enablement Presentation>
• DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
• DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
• DEV - Downstream build attached to advisory: <link to errata>
• QE - Test plans in Polarion: <link or reference to Polarion>
• QE - Automated tests merged: <link or reference to automated tests>
• DOC - Downstream documentation merged: <link to meaningful PR>

This Epic is here to track the rebase we need to do when kube 1.25 is GA https://www.kubernetes.dev/resources/release/

Keeping this in mind can help us plan our time better. ATTOW GA is planned for August 23

https://docs.google.com/document/d/1h1XsEt1Iug-W9JRheQas7YRsUJ_NQ8ghEMVmOZ4X-0s/edit --> this is the link for rebase help

Changes made in METAL-1 open up opportunities to improve our handling of images by cleaning up redundant code that generates extra work for the user and extra load for the cluster.

Epic Goal

• To improve the reliability of disk cleaning before installation and to provide the user with sufficient warning regarding the consequences of the cleaning

Why is this important?

• Insufficient cleaning can lead to installation failure
• Insufficient warning can lead to complaints of unexpected data loss

Scenarios

Acceptance Criteria

• CI - MUST be running successfully with tests automated
• Release Technical Enablement - Provide necessary release enablement details and documents.
• ...

Dependencies (internal and external)

1. ...

Previous Work (Optional):

1. …

Open questions::

1. …

Done Checklist

• CI - CI is running, tests are automated and merged.
• Release Enablement <link to Feature Enablement Presentation>
• DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
• DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
• DEV - Downstream build attached to advisory: <link to errata>
• QE - Test plans in Polarion: <link or reference to Polarion>
• QE - Automated tests merged: <link or reference to automated tests>
• DOC - Downstream documentation merged: <link to meaningful PR>

Epic Goal

• Increase success-rate of of our CI jobs
• Improve debugability / visibility or tests 

Why is this important?

• Failed presubmit jobs (required or optional) can make an already tested+approved PR to not get in
• Failed periodic jobs interfere our visibility around stability of features

Section 5 of PRD: https://docs.google.com/document/d/1fF-Ajdzc9EDDg687FzTrX577hvY9NdK0/edit#heading=h.gjdgxs

Testing and collaboration with NVIDIA: https://docs.google.com/spreadsheets/d/1LHY-Af-2kQHVwtW4aVdHnmwZLTiatiyf-ySffC8O5NM/edit#gid=0 

Deploying Nvidia Patches:  https://docs.google.com/document/d/1yR4lphjPKd6qZ9sGzZITl0wH1r4ykfMKPjUnlzvWji4/edit# 

This is the continuation of https://issues.redhat.com/browse/NHE-273 but now the focus is on the remainig flows

OCP/Telco Definition of Done
Epic Template descriptions and documentation.

<--- Cut-n-Paste the entire contents of this description into your new Epic --->

Epic Goal

• Run OpenShift builds that do not execute as the "root" user on the host node.

Why is this important?

• OpenShift builds require an elevated set of capabilities to build a container image
• Builds currently run as root to maintain adequate performance
• Container workloads should run as non-root from the host's perspective. Containers running as root are a known security risk.
• Builds currently run as root and require a privileged container. See BUILD-225 for removing the privileged container requirement.

Scenarios

1. Run BuildConfigs in a multi-tenant environment
2. Run BuildConfigs in a heightened security environment/deployment

Acceptance Criteria

• Developers can opt into running builds in a cri-o user namespace by providing an environment variable with a specific value.
• When the correct environment variable is provided, builds run in a cri-o user namespace, and the build pod does not require the "privileged: true" security context.
• User namespace builds can pass basic test scenarios for the Docker and Source strategy build.
• Steps to run unprivileged builds are documented.

Dependencies (internal and external)

1. Buildah supports running inside a non-privileged container
2. CRI-O allows workloads to opt into running containers in user namespaces.

Previous Work (Optional):

1. BUILD-225 - remove privileged requirement for builds.

Open questions::

1. …

Done Checklist

• CI - CI is running, tests are automated and merged.
• Release Enablement <link to Feature Enablement Presentation>
• DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
• DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
• DEV - Downstream build attached to advisory: <link to errata>
• QE - Test plans in Polarion: <link or reference to Polarion>
• QE - Automated tests merged: <link or reference to automated tests>
• DOC - Downstream documentation merged: <link to meaningful PR>

OCP/Telco Definition of Done
Epic Template descriptions and documentation.

<--- Cut-n-Paste the entire contents of this description into your new Epic --->

Epic Goal

• ...

Why is this important?

• …

Scenarios

1. ...

Acceptance Criteria

• CI - MUST be running successfully with tests automated
• Release Technical Enablement - Provide necessary release enablement details and documents.
• ...

Dependencies (internal and external)

1. ...

Previous Work (Optional):

1. …

Open questions::

1. …

Done Checklist

• CI - CI is running, tests are automated and merged.
• Release Enablement <link to Feature Enablement Presentation>
• DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
• DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
• DEV - Downstream build attached to advisory: <link to errata>
• QE - Test plans in Polarion: <link or reference to Polarion>
• QE - Automated tests merged: <link or reference to automated tests>
• DOC - Downstream documentation merged: <link to meaningful PR>

Goal
Provide an indication that advanced features are used

Problem

Today, customers and RH don't have the information on the actual usage of advanced features.

Why is this important?

1. Better focus upsell efforts
2. Compliance information for customers that are not aware they are not using the right subscription

Prioritized Scenarios

In Scope
1. Add a boolean variable in our telemetry to mark if the customer is using advanced features (PV encryption, encryption with KMS, external mode). 

Not in Scope

Integrate with subscription watch - will be done by the subscription watch team with our help.

Customers

All

Customer Facing Story
As a compliance manager, I should be able to easily see if all my clusters are using the right amount of subscriptions

What does success look like?

A clear indication in subscription watch for ODF usage (either essential or advanced). 

We migrated most component as part of https://issues.redhat.com/browse/RHSTOR-2165
We now have a few components remaining roughly 15 to 20%. This epic tragets

Console should be using v1 version of the ConsolePlugin model rather then the old v1alpha1.

CONSOLE-3077 was updating this version, but did not made the cut for the 4.12 release. Based on discussion with Samuel Padgett we should be backporting to 4.12.

The risk should be minimal since we are only updating the model itself + validation + Readme

• https://github.com/openshift/console/pull/12248

This is a clone of issue OCPBUGS-3432. The following is the description of the original issue:
—
Description of problem:

E2E test cases for knative and pipeline packages have been disabled on CI due to respective operator installation issues. 
Tests have to be enabled after new operator version be available or the issue resolves

References:
https://coreos.slack.com/archives/C6A3NV5J9/p1664545970777239

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:

1.
2.
3.

Actual results:

Expected results:

Additional info:

• https://github.com/openshift/console/pull/12307

This is a clone of issue OCPBUGS-6213. The following is the description of the original issue:
—
Please review the following PR: https://github.com/openshift/machine-config-operator/pull/3450

The PR has been automatically opened by ART (#aos-art) team automation and indicates
that the image(s) being used downstream for production builds are not consistent
with the images referenced in this component's github repository.

Differences in upstream and downstream builds impact the fidelity of your CI signal.

If you disagree with the content of this PR, please contact @release-artists
in #aos-art to discuss the discrepancy.

Closing this issue without addressing the difference will cause the issue to
be reopened automatically.

• https://github.com/openshift/machine-config-operator/pull/3535

 Currently on summery logs if there is kube-api issue controller will not upload logs but it should as it has file to read them from

• https://github.com/openshift/assisted-installer/pull/540

Description of problem:

The user mirrored the 4.11.0 release and attempted to use it to generate the the installation ISO in a completely disconnected environment.

When it was the turn for extracting the os image from machine-os-images, the agent based installer ran : oc adm release info --image-for=machine-os-images --insecure=true quay.io/openshift-release-dev/ocp-release@sha256:300bce8246cf880e792e106607925de0a404484637627edf5f517375517d54a4 --registry-config=/tmp/registry-config1141450352

This does not include the --icsp-file, and thus the image reference can be retrieved to perform the extraction.

Version-Release number of selected component (if applicable):

https://github.com/openshift/installer/releases/tag/agent-installer-v4.11.0-dev-preview-2

How reproducible:

100%

Steps to Reproduce:

1. Mirroring the images of 4.11.0 using oc adm mirror command to the local registry.
2. Created install-config.yaml with mirror config
3. Created agent-config.yaml 
4. openshift-install-sep1 agent create image --dir kni-22

Actual results:

INFO[0001] Start configuring static network for 3 hosts  pkg=manifests
INFO[0002] Adding NMConnection file <bond0.nmconnection>  pkg=manifests
INFO[0002] Adding NMConnection file <eno49.nmconnection>  pkg=manifests
INFO[0002] Adding NMConnection file <eno50.nmconnection>  pkg=manifests
INFO[0003] Adding NMConnection file <bond0.nmconnection>  pkg=manifests
INFO[0003] Adding NMConnection file <eno49.nmconnection>  pkg=manifests
INFO[0003] Adding NMConnection file <eno50.nmconnection>  pkg=manifests
INFO[0004] Adding NMConnection file <bond0.nmconnection>  pkg=manifests
INFO[0004] Adding NMConnection file <eno49.nmconnection>  pkg=manifests
INFO[0004] Adding NMConnection file <eno50.nmconnection>  pkg=manifests
DEBUG   Fetching BaseIso Image...
DEBUG     Fetching Agent Manifests...
DEBUG     Reusing previously-fetched Agent Manifests
DEBUG     Fetching Install Config...
DEBUG     Reusing previously-fetched Install Config
DEBUG     Fetching Mirror Registries Config...
DEBUG     Reusing previously-fetched Mirror Registries Config
DEBUG   Generating BaseIso Image...
INFO[0004] Extracting base ISO from release payload
ERRO[0014] command 'oc adm release info --image-for=machine-os-images --insecure=true quay.io/openshift-release-dev/ocp-release@sha256:300bce8246cf880e792e106607925de0a404484637627edf5f517375517d54a4 --registry-config=/tmp/registry-config1141450352' exited with non-zero exit code 1:
error: unable to read image quay.io/openshift-release-dev/ocp-release@sha256:300bce8246cf880e792e106607925de0a404484637627edf5f517375517d54a4: Get "http://quay.io/v2/": dial tcp: lookup quay.io on 10.92.86.56:53: server misbehaving
WARN[0014] Failed to extract base ISO from release payload - check registry configuration
INFO[0014] Downloading base ISO
DEBUG Obtaining RHCOS image file from 'https://rhcos.mirror.openshift.com/art/storage/releases/rhcos-4.11/411.86.202207150124-0/x86_64/rhcos-411.86.202207150124-0-live.x86_64.iso'
ERROR failed to write asset (Agent Installer ISO) to disk: image reader not available
FATAL failed to fetch Agent Installer ISO: failed to fetch dependency of "Agent Installer ISO": failed to generate asset "BaseIso Image": failed to get base ISO image: command 'oc adm release info --image-for=machine-os-images --insecure=true quay.io/openshift-release-dev/ocp-release@sha256:300bce8246cf880e792e106607925de0a404484637627edf5f517375517d54a4 --registry-config=/tmp/registry-config1141450352' exited with non-zero exit code 1:
FATAL error: unable to read image quay.io/openshift-release-dev/ocp-release@sha256:300bce8246cf880e792e106607925de0a404484637627edf5f517375517d54a4: Get "http://quay.io/v2/": dial tcp: lookup quay.io on 10.92.86.56:53: server misbehaving
FATAL

Expected results:

Image correctly generated

Additional info:

Host OS: RHEL 8.4
NMstate version: nmstate-1.0.2-5.el8.noarch

• https://github.com/openshift/installer/pull/6339

This is a clone of issue OCPBUGS-5505. The following is the description of the original issue:
—

Description of problem:

The upgradeability check in CVO is throttled (essentially cached) for a nondeterministic period of time, same as the minimal sync period computed at runtime. The period can be up to 4 minutes, determined at CVO start time as 2minutes * (0..1 + 1). We agreed with Trevor that such throttling is unnecessarily aggressive (the check is not that expensive). It also causes CI flakes, because the matching test only has 3 minutes timeout. Additionally, the non-determinism and longer throttling results makes UX worse by actions done in the cluster may have their observable effect delayed.

Version-Release number of selected component (if applicable):

discovered in 4.10 -> 4.11 upgrade jobs

How reproducible:

The test seems to flake ~10% of 4.10->4.11 Azure jobs (sippy). There does not seem to be that much impact on non-Azure jobs though which is a bit weird.

Steps to Reproduce:

Inspect the CVO log and E2E logs from failing jobs with the provided [^check-cvo.py] helper:

$ ./check-cvo.py cvo.log && echo PASS || echo FAIL

Preferably, inspect CVO logs of clusters that just underwent an upgrade (upgrades makes the original problematic behavior more likely to surface)

Actual results:

$ ./check-cvo.py openshift-cluster-version_cluster-version-operator-5b6966c474-g4kwk_cluster-version-operator.log && echo PASS || echo FAIL
FAIL: Cache hit at 11:59:55.332339 0:03:13.665006 after check at 11:56:41.667333
FAIL: Cache hit at 12:06:22.663215 0:03:13.664964 after check at 12:03:08.998251
FAIL: Cache hit at 12:12:49.997119 0:03:13.665598 after check at 12:09:36.331521
FAIL: Cache hit at 12:19:17.328510 0:03:13.664906 after check at 12:16:03.663604
FAIL: Cache hit at 12:25:44.662290 0:03:13.666759 after check at 12:22:30.995531
Upgradeability checks:           5
Upgradeability check cache hits: 12
FAIL

Note that the bug is probabilistic, so not all unfixed clusters will exhibit the behavior. My guess of the incidence rate is about 30-40%.

Expected result

$ ./check-cvo.py openshift-cluster-version_cluster-version-operator-7b8f85d455-mk9fs_cluster-version-operator.log && echo PASS || echo FAIL
Upgradeability checks:           12
Upgradeability check cache hits: 11
PASS

The actual numbers are not relevant (unless the upgradeabilily check count is zero, which means the test is not conclusive, the script warns about that), lack of failure is.

Additional info:

$ curl --silent https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.11-upgrade-from-stable-4.10-e2e-azure-upgrade/1607602927633960960/artifacts/e2e-azure-upgrade/gather-extra/artifacts/pods/openshift-cluster-version_cluster-version-operator-7b7d4b5bbd-zjqdt_cluster-version-operator.log | grep upgradeable.go
...
I1227 06:50:59.023190       1 upgradeable.go:122] Cluster current version=4.10.46
I1227 06:50:59.042735       1 upgradeable.go:42] Upgradeable conditions were recently checked, will try later.
I1227 06:51:14.024345       1 upgradeable.go:42] Upgradeable conditions were recently checked, will try later.
I1227 06:53:23.080768       1 upgradeable.go:42] Upgradeable conditions were recently checked, will try later.
I1227 06:56:59.366010       1 upgradeable.go:122] Cluster current version=4.11.0-0.ci-2022-12-26-193640

$ curl --silent https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.11-upgrade-from-stable-4.10-e2e-azure-upgrade/1607602927633960960/artifacts/e2e-azure-upgrade/openshift-e2e-test/artifacts/e2e.log | grep 'Kubernetes 1.25 and therefore OpenShift 4.12'
Dec 27 06:51:15.319: INFO: Waiting for Upgradeable to be AdminAckRequired for "Kubernetes 1.25 and therefore OpenShift 4.12 remove several APIs which require admin consideration. Please see the knowledge article https://access.redhat.com/articles/6955381 for details and instructions." ...
Dec 27 06:54:15.413: FAIL: Error while waiting for Upgradeable to complain about AdminAckRequired with message "Kubernetes 1.25 and therefore OpenShift 4.12 remove several APIs which require admin consideration. Please see the knowledge article https://access.redhat.com/articles/6955381 for details and instructions.": timed out waiting for the condition

The test passes. Also, the "Upgradeable conditions were recently checked, will try later." messages in CVO logs should never occur after a deterministic, short amount of time (I propose 1 minute) after upgradeability was checked.

I tested the throttling period in https://github.com/openshift/cluster-version-operator/pull/880. With the period of 15m, the test passrate was 4 of 9. Wiht the period of 1m, the test did not fail at all.

Some context in Slack thread

• https://github.com/openshift/cluster-version-operator/pull/884

Description of problem:

Since openenshift/cluster-ingress-operator#817 merged, the e2e-aws-operator CI job has been failing for multiple PRs in the cluster-ingress-operator repository.  In particular, the TestScopeChange test has been consistently failing. Example failures:

• https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_cluster-ingress-operator/837/pull-ci-openshift-cluster-ingress-operator-master-e2e-aws-operator/1580571113170145280
• https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_cluster-ingress-operator/779/pull-ci-openshift-cluster-ingress-operator-master-e2e-aws-operator/1580597541211213824
• https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_cluster-ingress-operator/836/pull-ci-openshift-cluster-ingress-operator-master-e2e-aws-operator/1580641046826586112

The operator is repeatedly logging errors like the following in those failing CI jobs:

ERROR    operator.dns_controller    controller/controller.go:121    failed to delete dnsrecord; will retry    \{"dnsrecord": {"metadata":{"name":"scope-wildcard","namespace":"openshift-ingress-operator","uid":"2cb9936f-d6a0-4377-b3ed-c5167c5e9e4d","resourceVersion":"42217","generation":2,"creationTimestamp":"2022-10-13T16:19:23Z","deletionTimestamp":"2022-10-13T16:20:27Z","deletionGracePeriodSeconds":0,"labels":{"ingresscontroller.operator.openshift.io/owning-ingresscontroller":"scope"},"ownerReferences":[\{"apiVersion":"operator.openshift.io/v1","kind":"IngressController","name":"scope","uid":"713ac1c5-451b-42d1-89fd-c3910eb80fe3","controller":true,"blockOwnerDeletion":true}],"finalizers":["operator.openshift.io/ingress-dns"],"managedFields":[\{"manager":"ingress-operator","operation":"Update","apiVersion":"ingress.operator.openshift.io/v1","time":"2022-10-13T16:19:23Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:finalizers":{".":{},"v:\"operator.openshift.io/ingress-dns\"":{}},"f:labels":\{".":{},"f:ingresscontroller.operator.openshift.io/owning-ingresscontroller":{}},"f:ownerReferences":\{".":{},"k:\{\"uid\":\"713ac1c5-451b-42d1-89fd-c3910eb80fe3\"}":{}}},"f:spec":\{".":{},"f:dnsManagementPolicy":{},"f:dnsName":{},"f:recordTTL":{},"f:recordType":{},"f:targets":{}}}},\{"manager":"ingress-operator","operation":"Update","apiVersion":"ingress.operator.openshift.io/v1","time":"2022-10-13T16:19:24Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{".":{},"f:observedGeneration":{},"f:zones":{}}},"subresource":"status"}]},"spec":\{"dnsName":"*.scope.ci-op-x1j7dsgt-43abb.origin-ci-int-aws.dev.rhcloud.com.","targets":["af6e309caa14c41eabe69f3f9eb15cf1-1656133782.us-west-2.elb.amazonaws.com"],"recordType":"CNAME","recordTTL":30,"dnsManagementPolicy":"Managed"},"status":\{"zones":[{"dnsZone":{"tags":{"Name":"ci-op-x1j7dsgt-43abb-45zhd-int","kubernetes.io/cluster/ci-op-x1j7dsgt-43abb-45zhd":"owned"}},"conditions":[\{"type":"Published","status":"True","lastTransitionTime":"2022-10-13T16:19:23Z","reason":"ProviderSuccess","message":"The DNS provider succeeded in ensuring the record"}]},\{"dnsZone":{"id":"Z2GYOLTZHS5VK"},"conditions":[\{"type":"Published","status":"True","lastTransitionTime":"2022-10-13T16:19:24Z","reason":"ProviderSuccess","message":"The DNS provider succeeded in ensuring the record"}]}],"observedGeneration":1}}, "error": "failed to get hosted zone for load balancer target \"af6e309caa14c41eabe69f3f9eb15cf1-1656133782.us-west-2.elb.amazonaws.com\": couldn't find hosted zone ID of ELB af6e309caa14c41eabe69f3f9eb15cf1-1656133782.us-west-2.elb.amazonaws.com", "errorCauses": [\{"error": "failed to get hosted zone for load balancer target \"af6e309caa14c41eabe69f3f9eb15cf1-1656133782.us-west-2.elb.amazonaws.com\": couldn't find hosted zone ID of ELB af6e309caa14c41eabe69f3f9eb15cf1-1656133782.us-west-2.elb.amazonaws.com"}, \{"error": "failed to get hosted zone for load balancer target \"af6e309caa14c41eabe69f3f9eb15cf1-1656133782.us-west-2.elb.amazonaws.com\": couldn't find hosted zone ID of ELB af6e309caa14c41eabe69f3f9eb15cf1-1656133782.us-west-2.elb.amazonaws.com"}]}}}

The scope-wildcard dnsrecord is created for the TestScopeChange test.

Using search.ci, it seems that the failures occurred many times on #817 before it merged and then started occurring for the other PRs after #817 merged.

I filed a PR, openshift/cluster-ingress-operator#838, that reverts #817. I have run the e2e-aws-operator CI job on this PR twice. While the job has failed both times, the TestScopeChange test did not fail either time.

At this point, we have strong evidence that #817 is causing TestScopeChange to fail.

Grant Spence did some testing and determined that there is some interaction between TestAllowedSourceRangesStatus and TestScopeChange. It may suffice to serialize some tests (TestScopeChanged is currently a parallel test, as is TestAllowedSourceRangesStatus and two other tests that #817 adds).

If the problem cannot be resolved by serializing tests, it may be necessary to revert #817 to unblock CI.

Note that this issue is blocking NE-942, NE-1072, and NE-682, as well as any bugfix PRs for the master branch in openshift/cluster-ingress-operator.

Version-Release number of selected component (if applicable):

4.12

How reproducible:

Consistently.

Steps to Reproduce:

1. Run CI on a PR against the master branch of cluster-ingress-operator.

Actual results:

The TestScopeChange test fails as described.

Expected results:

TestScopeChange should not fail.

• https://github.com/openshift/cluster-ingress-operator/pull/839

This is a clone of issue OCPBUGS-3214. The following is the description of the original issue:
—
Description of problem:

The installer has logic that avoids adding the router CAs to the kubeconfig if the console is not available.  It's not clear why it does this, but it means that the router CAs don't get added when the console is deliberately disabled (it is now an optional capability in 4.12).

Version-Release number of selected component (if applicable):

Seen in 4.12+4.13

How reproducible:

Always, when starting a cluster w/o the Console capability

Steps to Reproduce:

1. Edit the install-config to set:
capabilities:
  baselineCapabilitySet: None
2. install the cluster
3. check the CAs in the kubeconfig, the wildcard route CA will be missing (compare it w/ a normal cluster)

Actual results:

router CAs missing

Expected results:

router CAs should be present

Additional info:

This needs to be backported to 4.12.

• https://github.com/openshift/installer/pull/6578

This is a clone of issue OCPBUGS-4955. The following is the description of the original issue:
—
Description of problem:

Customer needs "IfNotPresent" ImagePullPolicy set for bundle unpacker images which reference iamges by digest. Currently, policy is set to "Always" no matter what.

Version-Release number of selected component (if applicable):

How reproducible:

Always

Steps to Reproduce:

1.Install an operator via bundle referencing an image by digest
2.Check the bundle unpacker pod

Actual results:

Image pull policy will be set to "Always"

Expected results:

Image pull policy will be set to "IfNotPresent" when pulling via digest

Additional info:

• https://github.com/openshift/operator-framework-olm/pull/439

This is a clone of https://bugzilla.redhat.com/show_bug.cgi?id=2083087 (OCPBUGSM-44070) to backport this issue.

Description of problem:
"Delete dependent objects of this resource" is a bit of confusing for some users because when creating the Application in Dev console not only the deployment but also IS, route, svc, secret objects will be created as well. When deleting the Application (in fact it is deployment), there is an option called "Delete dependent objects of this resource" and some users might think this means the IS, route, svc and any other objects which are created alongside with the deployment will be deleted as well

Version-Release number of selected component (if applicable):
4.8

How reproducible:
Always

Steps to Reproduce:
1. Create Application in Dev console
2. Delete the deployment
3. Check "Delete dependent objects of this resource"

Actual results:
Only deployment will be deleted and IS, svc, route will not be deleted

Expected results:
We either change the description of this option, or we really delete IS, svc, route and any other objects created under this Application.

Additional info:

• https://github.com/openshift/console/pull/12491

Description of problem:

Provisioning interface on master node not getting ipv4 dhcp ip address from bootstrap dhcp server on OCP 4.10.16 IPI BareMetal install.

Customer is performing an OCP 4.10.16 IPI BareMetal install and bootstrap node provisions just fine, but when master nodes are booted for provisioning, they are not getting an ipv4 address via dhcp. As such, the install is not moving forward at this point.

Version-Release number of selected component (if applicable):

OCP 4.10.16

How reproducible:

Perform OCP 4.10.16 IPI BareMetal install.

Actual results:

provisioning interface comes up (as evidenced by ipv6 address) but is not getting an ipv4 address via dhcp. OCP install / provisioning fails at this point.

Expected results:

provisioning interface successfully received an ipv4 ip address and successfully provisioned master nodes (and subsequently worker nodes as well.)

Additional info:

As a troubleshooting measure, manually adding an ipv4 ip address did allow the coreos image on the bootstrap node to be reached via curl.

Further, the kernel boot line for the first master node was updated for a static ip addresss assignment for further confirmation that the master node would successfully image this way which further confirming that the issue is the provisioning interface not receiving an ipv4 ip address from the dhcp server.

• https://github.com/openshift/cluster-baremetal-operator/pull/286

OVS 2.17+ introduced an optimization of "weak references" to substantially speed up database snapshots. in some cases weak references may leak memory; to aforementioned commit fixes that and has been pulled into ovs2.17-62 and later.

• https://github.com/openshift/ovn-kubernetes/pull/1409
• https://github.com/openshift/ovn-kubernetes/pull/1387

This is a clone of issue OCPBUGS-1428. The following is the description of the original issue:
—
Description of problem:

When using an OperatorGroup attached to a service account, AND if there is a secret present in the namespace, the operator installation will fail with the message:
the service account does not have any API secret sa=testx-ns/testx-sa
This issue seems similar to https://bugzilla.redhat.com/show_bug.cgi?id=2094303 - which was resolved in 4.11.0 - however, the new element now, is that the presence of a secret in the namespace  is causing the issue.
The name of the secret seems significant - suggesting something somewhere is depending on the order that secrets are listed in. For example, If the secret in the namespace is called "asecret", the problem does not occur. If it is called "zsecret", the problem always occurs.

"zsecret" is not a "kubernetes.io/service-account-token". The issue I have raised here relates to Opaque secrets - zsecret is an Opaque secret. The issue may apply to other types of secrets, but specifically my issue is that when there is an opaque secret present in the namespace, the operator install fails as described. I aught to be allowed to have an opaque secret present in the namespace where I am installing the operator.

Version-Release number of selected component (if applicable):

4.11.0 & 4.11.1

How reproducible:

100% reproducible

Steps to Reproduce:

1.Create namespace: oc new-project testx-ns
2. oc apply -f api-secret-issue.yaml

Actual results:

Expected results:

Additional info:

API YAML:

cat api-secret-issue.yaml 
apiVersion: v1
kind: Secret
metadata:
  name: zsecret
  namespace: testx-ns
  annotations:
   kubernetes.io/service-account.name: testx-sa
type: Opaque
stringData:
  mykey: mypass

—

apiVersion: v1
kind: ServiceAccount
metadata:
  name: testx-sa
  namespace: testx-ns

—

kind: OperatorGroup
apiVersion: operators.coreos.com/v1
metadata:
  name: testx-og
  namespace: testx-ns
spec:
  serviceAccountName: "testx-sa"
  targetNamespaces:
  - testx-ns

—

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: testx-role
  namespace: testx-ns
rules:

• apiGroups: ["*"]
    resources: ["*"]
    verbs: ["*"] 
    
  —

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: testx-rolebinding
  namespace: testx-ns
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: testx-role
subjects:

• kind: ServiceAccount
    name: testx-sa
    namespace: testx-ns

—

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: etcd-operator
  namespace: testx-ns
spec:
  channel: singlenamespace-alpha
  installPlanApproval: Automatic
  name: etcd
  source: community-operators
  sourceNamespace: openshift-marketplace

• https://github.com/openshift/operator-framework-olm/pull/412

• https://github.com/openshift/machine-config-operator/pull/3425

• https://github.com/openshift/cluster-etcd-operator/pull/984

This is a clone of issue OCPBUGS-3032. The following is the description of the original issue:
—
If installation fails at an early stage (e.g. pulling release images, configuring hosts, waiting for agents to come up) there is no indication that anything has gone wrong, and the installer binary may not even be able to connect.
We should at least display what is happening on the console so that users have some avenue to figure out for themselves what is going on.

• https://github.com/openshift/installer/pull/6680

• https://github.com/openshift/console/pull/12276

Description of problem:
Users on a disconnected cluster with a proxy could not import a Devfile (from GitHub).

The API call /api/devfile/ takes 30 seconds until it fails with 504 Gateway timeout.

Version-Release number of selected component (if applicable):
This might happen since 4.8

Tested this yet only on 4.12.0-0.nightly-2022-09-07-112008

How reproducible:
Always

Steps to Reproduce:

1. Start a disconnected cluster with a proxy
2. Open the browser network inspector and filter for /api/devfile
3. Switch to Developer perspective
4. Navigate to Add > Developer Catalog (All Services) > Devfiles
5. Select a Devfile like Basic Go (https://github.com/devfile-samples/devfile-sample-go-basic.git)
6. Press Create

Actual results:

• Network call fails after 30 seconds
• Import doesn't work

Expected results:

• Import should create a Deployment and switch to topology view

Additional info:
The console Pod log contains this error:

E0909 10:28:18.448680 1 devfile-handler.go:74] Failed to parse devfile: failed to populateAndParseDevfile: Get "https://registry.devfile.io/devfiles/go": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

• https://github.com/openshift/console/pull/12043

https://github.com/openshift/origin/pull/27444 was intended to move the scaling test out of serial to it's own test suite, but it added it to parallel – meaning it's running in all our normal upgrade jobs, causing them to frequently fail with repeating pathological events as well as greatly increasing their run time.

See https://github.com/openshift/origin/pull/27444#discussion_r991296925 for more info

• https://github.com/openshift/origin/pull/27463

Since 4.11 OCP comes with OperatorHub definition which declares a capability
and enables all catalog sources. For OKD we want to enable just community-operators
as users may not have Red Hat pull secret set.
This commit would ensure that OKD version of marketplace operator gets
its own OperatorHub manifest with a custom set of operator catalogs enabled

• https://github.com/operator-framework/operator-marketplace/pull/481

Description of problem:

When spot instances with taints are added to the cluster on AWS, machine-api-termination-handler daemonset pods do not launch on these instances because of the taints. 

machine-api-termination-handler is used for checking the notification of  intance termination, so if it doesn't launch properly, application pods on spot instances could stop without normal shut down procedures. 

It is common to use taint-toleration to specify workloads on spot instances, because it does not require changing application manifests of other workloads. 

Version-Release number of selected component (if applicable):

How reproducible:

100%

Steps to Reproduce:

1. Creating ROSA cluster
2. Adding spot instances with taints on OCM
3. oc get daemonset machine-api-termination-handler -n openshift-machine-api

Actual results:

machine-api-termination-handler pods do not launch on spot instances

Expected results:

machine-api-termination-handler pods launch on spot instances

Additional info:

Adding followings to machine-api-termination-handler daemonset could resolve the problem.
---  
tolerations:        
- operator: Exists

• https://github.com/openshift/machine-api-operator/pull/1072

Description of problem:

If using ingresscontroller.spec.routeSelector.matchExpressions or ingresscontroller.spec.namespaceSelector.matchExpressions, the route will not count in the new route_metrics_controller_routes_per_shard prometheus metric.

This is due to the logic only using "matchLabels". The logic needs to be updated to also use "matchExpressions".

Version-Release number of selected component (if applicable):

4.12

How reproducible:

100%

Steps to Reproduce:

1. Create IC with matchExpressions:
oc apply -f - <<EOF
apiVersion: operator.openshift.io/v1
kind: IngressController
metadata:
  name: sharded
  namespace: openshift-ingress-operator
spec:
  domain: reproducer.$domain
  routeSelector:
    matchExpressions:
    - key: type
      operator: In
      values:
      - shard
  replicas: 1
  nodePlacement:
    nodeSelector:
      matchLabels:
        node-role.kubernetes.io/worker: ""
EOF

2. Create the route:
oc apply -f - <<EOF
apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: route-shard
  labels:
    type: shard
spec:
  to:
    kind: Service
    name: router-shard
EOF

 3. Check route_metrics_controller_routes_per_shard{name="sharded"} in prometheus, it's 0 

Actual results:

route_metrics_controller_routes_per_shard{name="sharded"} has 0 routes

Expected results:

route_metrics_controller_routes_per_shard{name="sharded"} should have 1 route

Additional info:

• https://github.com/openshift/cluster-ingress-operator/pull/850

Description of problem:

OCPBUGS-3499 and OCPBUGS-3501 both require a more recent version of openshift/library-go containing the shared validation and host-assignment logic.

• https://github.com/openshift/kubernetes/pull/1423
• https://github.com/openshift/kubernetes/pull/1414