)]}'
{"/COMMIT_MSG":[{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"20d4ca3f3e9dbdddbcd76244e2e1c40d024d67f6","unresolved":false,"context_lines":[{"line_number":6,"context_line":""},{"line_number":7,"context_line":"Resource Management Daemon - Base Enablement"},{"line_number":8,"context_line":""},{"line_number":9,"context_line":"We propose a solution for OpenStack Nova to leverage the RMD to manage a subset of host resources traditionally not visible to Nova. These are resources that shall allow more fine grain resource control for more predictable performance. This will require running RMD on the compute nodes along with RMD plugins to interface each resource it supports and modifying Nova Compute."},{"line_number":10,"context_line":""},{"line_number":11,"context_line":"Implements: blueprint rmd-base-enablement"},{"line_number":12,"context_line":""}],"source_content_type":"text/x-gerrit-commit-message","patch_set":5,"id":"ffb9cba7_ea205c4c","line":9,"updated":"2019-04-21 22:56:51.000000000","message":"please wrap lines.","commit_id":"28b06cb71b84a2f2cee5b8c041c3e2781321c423"}],"specs/train/approved/rmd-base-enablement.rst":[{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"20d4ca3f3e9dbdddbcd76244e2e1c40d024d67f6","unresolved":false,"context_lines":[{"line_number":37,"context_line":"compute node. As the density of the software that runs on the platform"},{"line_number":38,"context_line":"increases, software components like VNFs, containers or infrastructure"},{"line_number":39,"context_line":"components like vSwitch/vRouter often compete for hardware resources and this"},{"line_number":40,"context_line":"can have significant impact on performance and service availability."},{"line_number":41,"context_line":""},{"line_number":42,"context_line":"To prevent such negative impact, one solution is to leave some resource"},{"line_number":43,"context_line":"headroom while scheduling tasks to a machine. Even this can prove inadequate"}],"source_content_type":"text/x-rst","patch_set":5,"id":"ffb9cba7_aa1a6414","line":40,"updated":"2019-04-21 22:56:51.000000000","message":"Every performance-critical VNF workload I have ever seen in production has been on compute nodes that have no overcommit, have been pinned to specific host CPUs, have dedicated passthrough PCI devices or SR-IOV VFs that have been tweaked by the admin/operator w.r.t. QoS, and nearly all of these workloads have had some amount of memory pages dedicated to them.\n\nWhat additional benefit is RMD going to provide to these VNF workloads since these workloads already are pinned to dedicated host resources?","commit_id":"28b06cb71b84a2f2cee5b8c041c3e2781321c423"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"caeec2d1a3a83edd4a394bc066e6092c61d74c07","unresolved":false,"context_lines":[{"line_number":37,"context_line":"compute node. As the density of the software that runs on the platform"},{"line_number":38,"context_line":"increases, software components like VNFs, containers or infrastructure"},{"line_number":39,"context_line":"components like vSwitch/vRouter often compete for hardware resources and this"},{"line_number":40,"context_line":"can have significant impact on performance and service availability."},{"line_number":41,"context_line":""},{"line_number":42,"context_line":"To prevent such negative impact, one solution is to leave some resource"},{"line_number":43,"context_line":"headroom while scheduling tasks to a machine. Even this can prove inadequate"}],"source_content_type":"text/x-rst","patch_set":5,"id":"ffb9cba7_a7ffc0ff","line":40,"in_reply_to":"ffb9cba7_723e70db","updated":"2019-04-24 13:28:29.000000000","message":"actully they can.\nyou can track available cache ways and memory bandwith\nand you could allocate it staticaly via libvirt without rmd\nat least when it comes to cache and memory\n\nhttps://libvirt.org/formatdomain.html#elementsCPUTuning\n\nyou can do it via the cache and memory tune subelements\nof the cputune element.","commit_id":"28b06cb71b84a2f2cee5b8c041c3e2781321c423"},{"author":{"_account_id":29334,"name":"Dakshina Ilangovan","email":"dakshina.ilangovan@intel.com","username":"dakshinai"},"change_message_id":"65af0255ba1ab001e71899a0351fa06a0cca46f1","unresolved":false,"context_lines":[{"line_number":37,"context_line":"compute node. As the density of the software that runs on the platform"},{"line_number":38,"context_line":"increases, software components like VNFs, containers or infrastructure"},{"line_number":39,"context_line":"components like vSwitch/vRouter often compete for hardware resources and this"},{"line_number":40,"context_line":"can have significant impact on performance and service availability."},{"line_number":41,"context_line":""},{"line_number":42,"context_line":"To prevent such negative impact, one solution is to leave some resource"},{"line_number":43,"context_line":"headroom while scheduling tasks to a machine. Even this can prove inadequate"}],"source_content_type":"text/x-rst","patch_set":5,"id":"ffb9cba7_6a593333","line":40,"in_reply_to":"ffb9cba7_9637f8ff","updated":"2019-04-30 07:03:38.000000000","message":"Agree to first para.\n\nWith CAT we can allocate cache ways to a set of process ids running on floating CPUs or a set of dedicated CPUs. \n\nThe issue with specifying cache ways for floating CPUs is that we have to repeat the cache allocation on each LLC (today per NUMA). This is to ensure that floating CPUs from multiple NUMA can still access their respective LLC and get the dedicated cache. With OpenStack, we can not account this in the allocation table. If user requests 3 cache ways on 2 socket system, then we end up allocating 6 cache ways (3 on each).\n\nTo keep things simple, we allocate cache ways to dedicated CPUs and ensure CPU pinning information is available from OpenStack","commit_id":"28b06cb71b84a2f2cee5b8c041c3e2781321c423"},{"author":{"_account_id":5754,"name":"Alex Xu","email":"hejie.xu@intel.com","username":"xuhj"},"change_message_id":"50015fc9112b5664c0daf5595bf77892d60cc454","unresolved":false,"context_lines":[{"line_number":37,"context_line":"compute node. As the density of the software that runs on the platform"},{"line_number":38,"context_line":"increases, software components like VNFs, containers or infrastructure"},{"line_number":39,"context_line":"components like vSwitch/vRouter often compete for hardware resources and this"},{"line_number":40,"context_line":"can have significant impact on performance and service availability."},{"line_number":41,"context_line":""},{"line_number":42,"context_line":"To prevent such negative impact, one solution is to leave some resource"},{"line_number":43,"context_line":"headroom while scheduling tasks to a machine. Even this can prove inadequate"}],"source_content_type":"text/x-rst","patch_set":5,"id":"ffb9cba7_9637f8ff","line":40,"in_reply_to":"ffb9cba7_a7ffc0ff","updated":"2019-04-29 22:24:24.000000000","message":"I thought more than statically allocate cache to the vCPU. The RMD can do something dynamically, like you specify a minimal cache and a maximal cache for a VCPU. The minimal cache will be guaranteed. And if there is more free cache, RMD can tuning the cache to the maximal value.\n\nOn cache case, I thought the RMD is only for dedicated vCPU? since the underlayer interface for the CAT is about allocated a specific number of cache for specific pCPU. It doesn\u0027t make sense for shared vCPU, since it will float on a set of pCPU.","commit_id":"28b06cb71b84a2f2cee5b8c041c3e2781321c423"},{"author":{"_account_id":29334,"name":"Dakshina Ilangovan","email":"dakshina.ilangovan@intel.com","username":"dakshinai"},"change_message_id":"65af0255ba1ab001e71899a0351fa06a0cca46f1","unresolved":false,"context_lines":[{"line_number":37,"context_line":"compute node. As the density of the software that runs on the platform"},{"line_number":38,"context_line":"increases, software components like VNFs, containers or infrastructure"},{"line_number":39,"context_line":"components like vSwitch/vRouter often compete for hardware resources and this"},{"line_number":40,"context_line":"can have significant impact on performance and service availability."},{"line_number":41,"context_line":""},{"line_number":42,"context_line":"To prevent such negative impact, one solution is to leave some resource"},{"line_number":43,"context_line":"headroom while scheduling tasks to a machine. Even this can prove inadequate"}],"source_content_type":"text/x-rst","patch_set":5,"id":"ffb9cba7_ff0e4739","line":40,"in_reply_to":"ffb9cba7_a7ffc0ff","updated":"2019-04-30 07:03:38.000000000","message":"Yes we can use libvirt integration for static allocation. But a resource like LLC need not be a one-time config unless we have LLC size to spare. With static configuration, the LLC is under-utilized. We could use the cluster for more workloads otherwise.\n\nI\u0027m not aware of CPU tuning features, any idea on what conditions they fine-tune?\n\nRMD is not a vehicle for LLC but resources that can be fine-tuned dynamically and not be restricted to static configuration. Secondly, since we want to react fast to changes in workload performance, we expect RMD to run locally. This can also avoid scenarios of control plane deciding to migrate the VM due to change of requirements.","commit_id":"28b06cb71b84a2f2cee5b8c041c3e2781321c423"},{"author":{"_account_id":29334,"name":"Dakshina Ilangovan","email":"dakshina.ilangovan@intel.com","username":"dakshinai"},"change_message_id":"b3066faf83f8b0bac6a3562e5d2039cd667053f6","unresolved":false,"context_lines":[{"line_number":37,"context_line":"compute node. As the density of the software that runs on the platform"},{"line_number":38,"context_line":"increases, software components like VNFs, containers or infrastructure"},{"line_number":39,"context_line":"components like vSwitch/vRouter often compete for hardware resources and this"},{"line_number":40,"context_line":"can have significant impact on performance and service availability."},{"line_number":41,"context_line":""},{"line_number":42,"context_line":"To prevent such negative impact, one solution is to leave some resource"},{"line_number":43,"context_line":"headroom while scheduling tasks to a machine. Even this can prove inadequate"}],"source_content_type":"text/x-rst","patch_set":5,"id":"ffb9cba7_723e70db","line":40,"in_reply_to":"ffb9cba7_aa1a6414","updated":"2019-04-24 06:09:41.000000000","message":"We cannot guarantee predictable performance during the entire lifecycle of the VNF as it is sharing certain latency sensitive resources on the platform. Even with core-pinning, the cores will share a resource like LLC that now provides a capability to be fine-tuned. Similarly with power, under-utilized cores can be throttled with frequency for power savings.\n\nThe set of resources we offer to fine-tune with RMD, can not serve the model of allocate and forget and can be optimized for two reasons\n\n1. Predictable performance avoiding jitter from contending workloads.\n2. Give room for more workloads if platform is being under-utilized.","commit_id":"28b06cb71b84a2f2cee5b8c041c3e2781321c423"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"20d4ca3f3e9dbdddbcd76244e2e1c40d024d67f6","unresolved":false,"context_lines":[{"line_number":52,"context_line":"each compute node. For instance, hardware resources like Last Level Cache"},{"line_number":53,"context_line":"(LLC), Memory bandwidth, Power management, etc., while being time sensitive"},{"line_number":54,"context_line":"and shared across multiple VMs, require dynamic and time-sensitive adjustments"},{"line_number":55,"context_line":"at node level leading to the requirement of having a node agent like RMD."},{"line_number":56,"context_line":""},{"line_number":57,"context_line":"Secondly, since VNFs use a shared platform, hardware partitioning/isolation"},{"line_number":58,"context_line":"can be critical.  Such partitioning enables each application to get the"}],"source_content_type":"text/x-rst","patch_set":5,"id":"ffb9cba7_4ae168fe","line":55,"updated":"2019-04-21 22:56:51.000000000","message":"See above. I\u0027ve yet to see a latency-sensitive or realtime workload deployed to anything other than a compute node with no overcommit and they always consume dedicated CPU, NUMA/memory page and networking resources. Always.\n\nThe physical resource-constrained environment you are describing above is hard for me to grasp, really.\n\nI just wonder if this level of dynamically-adjusted micro-resources is appropriate for a compute abstraction layer like Nova. Where is the \"abstraction\" any more?","commit_id":"28b06cb71b84a2f2cee5b8c041c3e2781321c423"},{"author":{"_account_id":29334,"name":"Dakshina Ilangovan","email":"dakshina.ilangovan@intel.com","username":"dakshinai"},"change_message_id":"65af0255ba1ab001e71899a0351fa06a0cca46f1","unresolved":false,"context_lines":[{"line_number":52,"context_line":"each compute node. For instance, hardware resources like Last Level Cache"},{"line_number":53,"context_line":"(LLC), Memory bandwidth, Power management, etc., while being time sensitive"},{"line_number":54,"context_line":"and shared across multiple VMs, require dynamic and time-sensitive adjustments"},{"line_number":55,"context_line":"at node level leading to the requirement of having a node agent like RMD."},{"line_number":56,"context_line":""},{"line_number":57,"context_line":"Secondly, since VNFs use a shared platform, hardware partitioning/isolation"},{"line_number":58,"context_line":"can be critical.  Such partitioning enables each application to get the"}],"source_content_type":"text/x-rst","patch_set":5,"id":"ffb9cba7_0ac75798","line":55,"in_reply_to":"ffb9cba7_07500cd4","updated":"2019-04-30 07:03:38.000000000","message":"Hi Sean, About the paper, it has multiple parts.\n\nFor your first question, they do show the throughput and latency even without interference from a noisy neighbour. Secondly, they dedicate cache ways to all components in the NFV setup - vswitch, OS and VMs.\n\nNoisy neighbor tries to mimic a situation when there are multiple VMs on a single node competing for resources.\n\nYes, it is advised that we carefully slice LLC to different processes sharing the system. Performance will take a hit if processes are not profiled correctly. We expect the admin to correctly configure OS and Infrastructure pools before any VMs are scheduled.\n\nRMD encourages over-subscription through best-effort policies where it can let the VM move from max to min.\n\nWe do not expect all flavors in a cluster to carry CAT requirements, but any flavor with CAT information will land on a RMD/RDT enabled node.","commit_id":"28b06cb71b84a2f2cee5b8c041c3e2781321c423"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"caeec2d1a3a83edd4a394bc066e6092c61d74c07","unresolved":false,"context_lines":[{"line_number":52,"context_line":"each compute node. For instance, hardware resources like Last Level Cache"},{"line_number":53,"context_line":"(LLC), Memory bandwidth, Power management, etc., while being time sensitive"},{"line_number":54,"context_line":"and shared across multiple VMs, require dynamic and time-sensitive adjustments"},{"line_number":55,"context_line":"at node level leading to the requirement of having a node agent like RMD."},{"line_number":56,"context_line":""},{"line_number":57,"context_line":"Secondly, since VNFs use a shared platform, hardware partitioning/isolation"},{"line_number":58,"context_line":"can be critical.  Such partitioning enables each application to get the"}],"source_content_type":"text/x-rst","patch_set":5,"id":"ffb9cba7_07500cd4","line":55,"in_reply_to":"ffb9cba7_12b29495","updated":"2019-04-24 13:28:29.000000000","message":"yes althogh the bigest flaw with that work while got is the aply cat to the noisy neigbour and not to the workload undertest. as such it is not a vaild reference  point for how CAT woudl work in openstack.\n\n\nfor that work to be relevent to they would have needed to apply CAT to the vswtich and vhost vm that are the latency sensitive / realtime workload standins in that experiment not\nto the memtester vm.\n\n\nin a generic cloud env the memtester/noisy neighober will likely be a different tenant then the latency sensitive workload.\n\nas such to make the results of that paper valid in an open stack env you need to start form a point where all flavor have cat applied not just the super optimized vms. that is not to say any of the infomation in that document is wrong but it is missleading to assert that it will translate directly to openstack without also asserting that the noisy neighbour was spawned with a rmd/cat enabled flaovr that preemtivly prevented it form trashing the cache.\n\nthe same is true of memory bandwith allocation by the way not just cache alloctaiton.\n\nhow these technolgies work at the hardware level it does not acatully allocate cache or memory bandiwith to a proces it limits a process to a set of cache or an amount of memory bandwithd.  if all other prcoess are not also limited it does not work as you are effectivly trying to provide a minium quantiy of a resouce when the hardware can only do max limits.\n\nyou can implement a min bandwith policy using max bandwith but only if there  is no over-subsciption and only if all consomer of the bandwith have a policy applied.\n\nRMD tries to work around this by creating an OS pool which it confines all proces too for cache leave the remainder free for allocation to vms that request cache. this works but it also reduces the perfomace of all vms that cannot acess the remainder of the cache. so there are trade offs.","commit_id":"28b06cb71b84a2f2cee5b8c041c3e2781321c423"},{"author":{"_account_id":29334,"name":"Dakshina Ilangovan","email":"dakshina.ilangovan@intel.com","username":"dakshinai"},"change_message_id":"b3066faf83f8b0bac6a3562e5d2039cd667053f6","unresolved":false,"context_lines":[{"line_number":52,"context_line":"each compute node. For instance, hardware resources like Last Level Cache"},{"line_number":53,"context_line":"(LLC), Memory bandwidth, Power management, etc., while being time sensitive"},{"line_number":54,"context_line":"and shared across multiple VMs, require dynamic and time-sensitive adjustments"},{"line_number":55,"context_line":"at node level leading to the requirement of having a node agent like RMD."},{"line_number":56,"context_line":""},{"line_number":57,"context_line":"Secondly, since VNFs use a shared platform, hardware partitioning/isolation"},{"line_number":58,"context_line":"can be critical.  Such partitioning enables each application to get the"}],"source_content_type":"text/x-rst","patch_set":5,"id":"ffb9cba7_12b29495","line":55,"in_reply_to":"ffb9cba7_4ae168fe","updated":"2019-04-24 06:09:41.000000000","message":"Hi Jay pipes,\n\nPlease have a look at this paper - https://builders.intel.com/docs/networkbuilders/deterministic_network_functions_virtualization_with_Intel_Resource_Director_Technology.pdf\n\nIn this paper they study the effect of noisy neighbor to the VM running from hyper-visor or another VM. Here for a NFV use-case, they first determine cache footprint of contending applications and choose a cache layout that gives predictable performance (least variation) - latency/throughput. Next they prove that with CAT layout, how the performance of a regular VM is improved even with a noisy neighbor in play.\n\nThe abstraction comes from RMD. The end-user is just expected to provide an expectation of min and max resource requirements for his workloads at the time of provisioning using OpenStack and RMD should do the fine-tuning under the cover.","commit_id":"28b06cb71b84a2f2cee5b8c041c3e2781321c423"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"20d4ca3f3e9dbdddbcd76244e2e1c40d024d67f6","unresolved":false,"context_lines":[{"line_number":75,"context_line":"   requests."},{"line_number":76,"context_line":""},{"line_number":77,"context_line":"4. Provide hooks into platform to address different interfaces for the same or"},{"line_number":78,"context_line":"   multiple resources.  This results in vendor or platform independence from"},{"line_number":79,"context_line":"   OpenStack."},{"line_number":80,"context_line":""},{"line_number":81,"context_line":"Currently Nova only provides static allocation of hardware resources when an"},{"line_number":82,"context_line":"instance like VM is scheduled with no concept of local node monitoring and"}],"source_content_type":"text/x-rst","patch_set":5,"id":"ffb9cba7_6abb0ce5","line":79,"range":{"start_line":78,"start_character":24,"end_line":79,"end_character":13},"updated":"2019-04-21 22:56:51.000000000","message":"I have no idea how this is the case.","commit_id":"28b06cb71b84a2f2cee5b8c041c3e2781321c423"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"caeec2d1a3a83edd4a394bc066e6092c61d74c07","unresolved":false,"context_lines":[{"line_number":75,"context_line":"   requests."},{"line_number":76,"context_line":""},{"line_number":77,"context_line":"4. Provide hooks into platform to address different interfaces for the same or"},{"line_number":78,"context_line":"   multiple resources.  This results in vendor or platform independence from"},{"line_number":79,"context_line":"   OpenStack."},{"line_number":80,"context_line":""},{"line_number":81,"context_line":"Currently Nova only provides static allocation of hardware resources when an"},{"line_number":82,"context_line":"instance like VM is scheduled with no concept of local node monitoring and"}],"source_content_type":"text/x-rst","patch_set":5,"id":"ffb9cba7_87387c3d","line":79,"range":{"start_line":78,"start_character":24,"end_line":79,"end_character":13},"in_reply_to":"ffb9cba7_09c60c69","updated":"2019-04-24 13:28:29.000000000","message":"the north bound api my stay syntactically the same but\nthe semantics will change. on a system without rmd vms with a standard flavor will not have cache ro memory bandwidth limits and will have best effort un restricted access to both.\n\non a system with rmd it will have restricted access to both and in the future if dynamic power management of cores is added the speed select policy of the cores could be adjusted to two a lower tdp/performance level as they do not express a policy.\n\nthat means on the same hardware over time the performance of an app using a non rmd flavor could be reduced over time if rmd is added and enhanced with new capablites as it will need to restict these vms in order to be able to provide cache/mem/power management to the vms that requrest it.\n\nthe simple solution is to say just use host aggrates to seperate RMD hosts form non RMD hosts and that is vailid but\ni dont think if entirely true to say the apis will remain the same.\n\ni personally would consider any application of a qos policy to a resource consumed by a vm, that is not specifically requested by the vm (vai a flavor,image,neutron port qos policy) to be a semantic change to the api and one that we should not enable by default.","commit_id":"28b06cb71b84a2f2cee5b8c041c3e2781321c423"},{"author":{"_account_id":29334,"name":"Dakshina Ilangovan","email":"dakshina.ilangovan@intel.com","username":"dakshinai"},"change_message_id":"b3066faf83f8b0bac6a3562e5d2039cd667053f6","unresolved":false,"context_lines":[{"line_number":75,"context_line":"   requests."},{"line_number":76,"context_line":""},{"line_number":77,"context_line":"4. Provide hooks into platform to address different interfaces for the same or"},{"line_number":78,"context_line":"   multiple resources.  This results in vendor or platform independence from"},{"line_number":79,"context_line":"   OpenStack."},{"line_number":80,"context_line":""},{"line_number":81,"context_line":"Currently Nova only provides static allocation of hardware resources when an"},{"line_number":82,"context_line":"instance like VM is scheduled with no concept of local node monitoring and"}],"source_content_type":"text/x-rst","patch_set":5,"id":"ffb9cba7_09c60c69","line":79,"range":{"start_line":78,"start_character":24,"end_line":79,"end_character":13},"in_reply_to":"ffb9cba7_6abb0ce5","updated":"2019-04-24 06:09:41.000000000","message":"The statement here explains that, for each resource (that requires time-sensitive control or finer-grain slicing like LLC or power), with direct integration in RMD, the dependency with OpenStack is minimized since only high level details (to assist in provisioning) is exposed to OpenStack. For every change in platform like interface changes - msr to kernel or vendor upgrades, changes can be integrated to RMD without impacting OpenStack since northbound APIs will remain the same.","commit_id":"28b06cb71b84a2f2cee5b8c041c3e2781321c423"},{"author":{"_account_id":29334,"name":"Dakshina Ilangovan","email":"dakshina.ilangovan@intel.com","username":"dakshinai"},"change_message_id":"65af0255ba1ab001e71899a0351fa06a0cca46f1","unresolved":false,"context_lines":[{"line_number":75,"context_line":"   requests."},{"line_number":76,"context_line":""},{"line_number":77,"context_line":"4. Provide hooks into platform to address different interfaces for the same or"},{"line_number":78,"context_line":"   multiple resources.  This results in vendor or platform independence from"},{"line_number":79,"context_line":"   OpenStack."},{"line_number":80,"context_line":""},{"line_number":81,"context_line":"Currently Nova only provides static allocation of hardware resources when an"},{"line_number":82,"context_line":"instance like VM is scheduled with no concept of local node monitoring and"}],"source_content_type":"text/x-rst","patch_set":5,"id":"ffb9cba7_054c2840","line":79,"range":{"start_line":78,"start_character":24,"end_line":79,"end_character":13},"in_reply_to":"ffb9cba7_87387c3d","updated":"2019-04-30 07:03:38.000000000","message":"Yes aggregates could help.\n\nWe expect static provisioning requests to RMD from flavor. All additional policy information can be complementary but optional.","commit_id":"28b06cb71b84a2f2cee5b8c041c3e2781321c423"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"20d4ca3f3e9dbdddbcd76244e2e1c40d024d67f6","unresolved":false,"context_lines":[{"line_number":94,"context_line":""},{"line_number":95,"context_line":"We propose a solution for OpenStack Nova to leverage the Resource Management"},{"line_number":96,"context_line":"Daemon (RMD) to manage a subset of host resources traditionally not visible to"},{"line_number":97,"context_line":"Nova. These are resources that shall allow more fine grain resource control"},{"line_number":98,"context_line":"for more predictable performance. This will require running RMD on the compute"},{"line_number":99,"context_line":"nodes along with RMD plugins to interface each resource it supports and"},{"line_number":100,"context_line":"modifying Nova Compute."},{"line_number":101,"context_line":""}],"source_content_type":"text/x-rst","patch_set":5,"id":"ffb9cba7_eace1c82","line":98,"range":{"start_line":97,"start_character":6,"end_line":98,"end_character":33},"updated":"2019-04-21 22:56:51.000000000","message":"Predictable performance comes from pinning VMs to host physical processors. You have noisy neighbour issues when you don\u0027t pin CPU and memory pages.\n\nSo, you\u0027re saying that RMD addresses the noisy neighbour problem *only* for compute nodes that are servicing latency-sensitive VNF workloads when the compute node has been set up for overcommit.\n\nSeems like a very narrow use case IMHO.","commit_id":"28b06cb71b84a2f2cee5b8c041c3e2781321c423"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"caeec2d1a3a83edd4a394bc066e6092c61d74c07","unresolved":false,"context_lines":[{"line_number":94,"context_line":""},{"line_number":95,"context_line":"We propose a solution for OpenStack Nova to leverage the Resource Management"},{"line_number":96,"context_line":"Daemon (RMD) to manage a subset of host resources traditionally not visible to"},{"line_number":97,"context_line":"Nova. These are resources that shall allow more fine grain resource control"},{"line_number":98,"context_line":"for more predictable performance. This will require running RMD on the compute"},{"line_number":99,"context_line":"nodes along with RMD plugins to interface each resource it supports and"},{"line_number":100,"context_line":"modifying Nova Compute."},{"line_number":101,"context_line":""}],"source_content_type":"text/x-rst","patch_set":5,"id":"ffb9cba7_e74ff8ba","line":98,"range":{"start_line":97,"start_character":6,"end_line":98,"end_character":33},"in_reply_to":"ffb9cba7_5509d629","updated":"2019-04-24 13:28:29.000000000","message":"RMD can only adress noisy neighbor issue if you apply the LLC constriatns to the noisy neighbor. that can be done in two ways first you can apply a a policy to all vms by defualt that dont request a cache assignment or second you can change all your flavors to that they all use CAT.\n\nthe first option is a semanitc change to the mean of the original flavors that cannot be discoverd via the api and therefore cannot be reliaed on as a tenant. \n\nthe other approach works but for most clouds updating all flavors would not be a viabel option.\n\nthe semi sainer way to take option to is to define a host aggreate and use the aggreate type filter to restict what flaor can be schduled to that hsot aggrate to only those that use cat or other rmd features.\n\nif i was to deploy RMD i would go the host aggrate route.","commit_id":"28b06cb71b84a2f2cee5b8c041c3e2781321c423"},{"author":{"_account_id":29334,"name":"Dakshina Ilangovan","email":"dakshina.ilangovan@intel.com","username":"dakshinai"},"change_message_id":"65af0255ba1ab001e71899a0351fa06a0cca46f1","unresolved":false,"context_lines":[{"line_number":94,"context_line":""},{"line_number":95,"context_line":"We propose a solution for OpenStack Nova to leverage the Resource Management"},{"line_number":96,"context_line":"Daemon (RMD) to manage a subset of host resources traditionally not visible to"},{"line_number":97,"context_line":"Nova. These are resources that shall allow more fine grain resource control"},{"line_number":98,"context_line":"for more predictable performance. This will require running RMD on the compute"},{"line_number":99,"context_line":"nodes along with RMD plugins to interface each resource it supports and"},{"line_number":100,"context_line":"modifying Nova Compute."},{"line_number":101,"context_line":""}],"source_content_type":"text/x-rst","patch_set":5,"id":"ffb9cba7_8ad4a721","line":98,"range":{"start_line":97,"start_character":6,"end_line":98,"end_character":33},"in_reply_to":"ffb9cba7_e74ff8ba","updated":"2019-04-30 07:03:38.000000000","message":"LLC is one such example for RMD, and noisy neighbour is introduced just to mimic multiple other workloads competing with workload under test. RMD is useful for 1. resources that need fine-tuning during the life-cycle of the VM, 2. resources that need fast turn-around for changes at local node level.\n\nAs you said we do not always expect a noisy neighbour. Noisy neighbour can also be a workload that did not have sufficient resources configured.\n\nTalking about LLC, there are two ways to use it\n\n1. Let all the VMs and other system/infrastructure processes share the entire LLC.\n\n2. Dedicate LLC to VMs.\n\nMinus RMD, if we have Libvirt helping with initial static allocation for dedicated LLC on VM, it will be on a node that supports LLC slicing using CAT. On such a node, we expect only VMs requesting LLC cache ways to land.\n\nSo this is not typical to RMD but it is implied that all VMs on RDT node, can either share entire LLC or each request a portion of LLC. Note we can also let some VMs default to using remaining LLC but it defeats the advantage of dedicated cache.\n\nOS, infrastructure process can still default to using entire LLC.\n\nNow with RMD in the mix, there are 2 improvements\n\n1. We have an option to configure cache ways for OS and infra processes, assuming sys-admin has profiled and is aware. Infrastructure process can be flexibly overlapped with all VMs for LLC for better performance. This is still optional even with RMD. \n\n2. RMD does dynamic fine-tuning of LLC assignments or P-states and other resources. This is useful with or without over-commit.","commit_id":"28b06cb71b84a2f2cee5b8c041c3e2781321c423"},{"author":{"_account_id":29334,"name":"Dakshina Ilangovan","email":"dakshina.ilangovan@intel.com","username":"dakshinai"},"change_message_id":"b3066faf83f8b0bac6a3562e5d2039cd667053f6","unresolved":false,"context_lines":[{"line_number":94,"context_line":""},{"line_number":95,"context_line":"We propose a solution for OpenStack Nova to leverage the Resource Management"},{"line_number":96,"context_line":"Daemon (RMD) to manage a subset of host resources traditionally not visible to"},{"line_number":97,"context_line":"Nova. These are resources that shall allow more fine grain resource control"},{"line_number":98,"context_line":"for more predictable performance. This will require running RMD on the compute"},{"line_number":99,"context_line":"nodes along with RMD plugins to interface each resource it supports and"},{"line_number":100,"context_line":"modifying Nova Compute."},{"line_number":101,"context_line":""}],"source_content_type":"text/x-rst","patch_set":5,"id":"ffb9cba7_5509d629","line":98,"range":{"start_line":97,"start_character":6,"end_line":98,"end_character":33},"in_reply_to":"ffb9cba7_eace1c82","updated":"2019-04-24 06:09:41.000000000","message":"We imply predictable performance when using certain shared resources - like LLC.","commit_id":"28b06cb71b84a2f2cee5b8c041c3e2781321c423"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"20d4ca3f3e9dbdddbcd76244e2e1c40d024d67f6","unresolved":false,"context_lines":[{"line_number":103,"context_line":"Nova Compute:"},{"line_number":104,"context_line":""},{"line_number":105,"context_line":"* Enabling a path between Nova Compute and RMD that allows Nova Compute to"},{"line_number":106,"context_line":"  query the inventory of resources managed by RMD, and to delegate resource"},{"line_number":107,"context_line":"  allocation requests to RMD.  The OpenStack flavor determines as always"},{"line_number":108,"context_line":"  eligible hosts and the amount and types of the various resources being"},{"line_number":109,"context_line":"  requested for an instance."},{"line_number":110,"context_line":""}],"source_content_type":"text/x-rst","patch_set":5,"id":"ffb9cba7_6aa0eca6","line":107,"range":{"start_line":106,"start_character":58,"end_line":107,"end_character":28},"updated":"2019-04-21 22:56:51.000000000","message":"It would be better to use a different term than \"resource allocation requests\". The reason is because Placement already does the resource allocation. By the time the instance request reaches the destination host, the resources have already been claimed and allocated in placement.\n\nWhat you are really referring to is dynamic CPU and cache-line/cache-ways *assignment* to specific operating system processes.","commit_id":"28b06cb71b84a2f2cee5b8c041c3e2781321c423"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"caeec2d1a3a83edd4a394bc066e6092c61d74c07","unresolved":false,"context_lines":[{"line_number":103,"context_line":"Nova Compute:"},{"line_number":104,"context_line":""},{"line_number":105,"context_line":"* Enabling a path between Nova Compute and RMD that allows Nova Compute to"},{"line_number":106,"context_line":"  query the inventory of resources managed by RMD, and to delegate resource"},{"line_number":107,"context_line":"  allocation requests to RMD.  The OpenStack flavor determines as always"},{"line_number":108,"context_line":"  eligible hosts and the amount and types of the various resources being"},{"line_number":109,"context_line":"  requested for an instance."},{"line_number":110,"context_line":""}],"source_content_type":"text/x-rst","patch_set":5,"id":"ffb9cba7_59d6ecc5","line":107,"range":{"start_line":106,"start_character":58,"end_line":107,"end_character":28},"in_reply_to":"ffb9cba7_156a7eb2","updated":"2019-04-24 13:28:29.000000000","message":"any dyncmic contol shoud only happen if a policy defiend by openstack says it valid to do so.\n\nfor examle if a flavor requse 10 cacheways it is not valide for rmd to reduce that to 8 or grow it to 12 if the flavor/image does not expressly allow that.\n\nl3 cache in skylake and later generation is an exclucive not inclucive cache meaning that data and instruction stored in l3 cache are not stored in l1 or l2. if a running application needs to acess data that is stored in l3 cache, and rmd srinks the cache becasue a new vm was spawned t would intoduce a l3 cache miss resulting in a dram fetch.\nthat will cause the vm under dynamic contol to stall\nfor hundreds of cycles.  there is a similar problem when growing the cache.\n\ndynamic contol is cool in general but if i care about latency at all my vm will have to use a static allcoation.\nif im using the dynmic asignment its to imporve throughput by reducing latency but not to gurantee any specifc latency reductions or sla.","commit_id":"28b06cb71b84a2f2cee5b8c041c3e2781321c423"},{"author":{"_account_id":29334,"name":"Dakshina Ilangovan","email":"dakshina.ilangovan@intel.com","username":"dakshinai"},"change_message_id":"65af0255ba1ab001e71899a0351fa06a0cca46f1","unresolved":false,"context_lines":[{"line_number":103,"context_line":"Nova Compute:"},{"line_number":104,"context_line":""},{"line_number":105,"context_line":"* Enabling a path between Nova Compute and RMD that allows Nova Compute to"},{"line_number":106,"context_line":"  query the inventory of resources managed by RMD, and to delegate resource"},{"line_number":107,"context_line":"  allocation requests to RMD.  The OpenStack flavor determines as always"},{"line_number":108,"context_line":"  eligible hosts and the amount and types of the various resources being"},{"line_number":109,"context_line":"  requested for an instance."},{"line_number":110,"context_line":""}],"source_content_type":"text/x-rst","patch_set":5,"id":"ffb9cba7_4aa8ef8a","line":107,"range":{"start_line":106,"start_character":58,"end_line":107,"end_character":28},"in_reply_to":"ffb9cba7_59d6ecc5","updated":"2019-04-30 07:03:38.000000000","message":"About dynamic control, yes, it should happen with the knowledge of OpenStack, by which we mean, OpenStack flavors will provide min and max requirements for the resources and RMD works within those boundaries.\n\nWe are not introducing dynamic control for resources that require slicing like LLC at the moment, because, if placement has accounted for a min value of LLC and if RMD increases this min to max, this information is not available to placement. We know at this point there is no way to synchronize updates to placement to ensure any new scheduling works with updated allocation information.\n\nAbout LLC dynamic changes, we should deploy a good algorithm to increase or decrease LLC only if it observes that LLC has not been used recently to indicate under-utilization. This could ensure that we are not freeing valid cache data and introducing latency from cache warmup.","commit_id":"28b06cb71b84a2f2cee5b8c041c3e2781321c423"},{"author":{"_account_id":29334,"name":"Dakshina Ilangovan","email":"dakshina.ilangovan@intel.com","username":"dakshinai"},"change_message_id":"b3066faf83f8b0bac6a3562e5d2039cd667053f6","unresolved":false,"context_lines":[{"line_number":103,"context_line":"Nova Compute:"},{"line_number":104,"context_line":""},{"line_number":105,"context_line":"* Enabling a path between Nova Compute and RMD that allows Nova Compute to"},{"line_number":106,"context_line":"  query the inventory of resources managed by RMD, and to delegate resource"},{"line_number":107,"context_line":"  allocation requests to RMD.  The OpenStack flavor determines as always"},{"line_number":108,"context_line":"  eligible hosts and the amount and types of the various resources being"},{"line_number":109,"context_line":"  requested for an instance."},{"line_number":110,"context_line":""}],"source_content_type":"text/x-rst","patch_set":5,"id":"ffb9cba7_156a7eb2","line":107,"range":{"start_line":106,"start_character":58,"end_line":107,"end_character":28},"in_reply_to":"ffb9cba7_6aa0eca6","updated":"2019-04-24 06:09:41.000000000","message":"We have two parts\n\n1. Provisioning step - RMD resources will be described in the placement if resource quantities will be sliced and allocated. E.g. LLC. They can be exposed as traits if we need to expose certain compute node features - E.g. CPU power control states, OpenStack can deliver workloads to such compute nodes and RMD can later work on any fine-tuning for them.\n\n2. Dynamic control - This will be covered on a per resource basis as long as what is reported in placement remains the same. E.g. Initial LLC allocation request, so that all fine-tuning happens under the cover.\n\nIs it ok if we change this as \"delegate resource allocation requests on resources supported by RMD\"?","commit_id":"28b06cb71b84a2f2cee5b8c041c3e2781321c423"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"20d4ca3f3e9dbdddbcd76244e2e1c40d024d67f6","unresolved":false,"context_lines":[{"line_number":108,"context_line":"  eligible hosts and the amount and types of the various resources being"},{"line_number":109,"context_line":"  requested for an instance."},{"line_number":110,"context_line":""},{"line_number":111,"context_line":"* Enabling a path from Nova Compute to Placement API to update resource"},{"line_number":112,"context_line":"  inventory information obtained from RMD."},{"line_number":113,"context_line":""},{"line_number":114,"context_line":"The changes are illustrated below:"},{"line_number":115,"context_line":""}],"source_content_type":"text/x-rst","patch_set":5,"id":"ffb9cba7_4aaf2894","line":112,"range":{"start_line":111,"start_character":0,"end_line":112,"end_character":42},"updated":"2019-04-21 22:56:51.000000000","message":"Sounds like we\u0027re going to get into another situation like we have with the PCI manager module in Nova. \"Normal\" resources like VCPU, MEMORY_MB, DISK_GB, etc are reported and managed by the virt driver. But there\u0027s a whole set of \"special\" resources that are \"managed\" (but not properly IMHO) by the PCI module.","commit_id":"28b06cb71b84a2f2cee5b8c041c3e2781321c423"},{"author":{"_account_id":29334,"name":"Dakshina Ilangovan","email":"dakshina.ilangovan@intel.com","username":"dakshinai"},"change_message_id":"b3066faf83f8b0bac6a3562e5d2039cd667053f6","unresolved":false,"context_lines":[{"line_number":108,"context_line":"  eligible hosts and the amount and types of the various resources being"},{"line_number":109,"context_line":"  requested for an instance."},{"line_number":110,"context_line":""},{"line_number":111,"context_line":"* Enabling a path from Nova Compute to Placement API to update resource"},{"line_number":112,"context_line":"  inventory information obtained from RMD."},{"line_number":113,"context_line":""},{"line_number":114,"context_line":"The changes are illustrated below:"},{"line_number":115,"context_line":""}],"source_content_type":"text/x-rst","patch_set":5,"id":"ffb9cba7_d5b8662b","line":112,"range":{"start_line":111,"start_character":0,"end_line":112,"end_character":42},"in_reply_to":"ffb9cba7_4aaf2894","updated":"2019-04-24 06:09:41.000000000","message":"After our last discussion, we have changed the proposal to follow the synchronized approach of letting virt-driver report and manage resources by talking to RMD. Can you confirm if this is fine.","commit_id":"28b06cb71b84a2f2cee5b8c041c3e2781321c423"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"20d4ca3f3e9dbdddbcd76244e2e1c40d024d67f6","unresolved":false,"context_lines":[{"line_number":173,"context_line":"new vendor resource that has to be supported, today there is no centralized"},{"line_number":174,"context_line":"approach to provide a user interface with high-level abstraction of allocation"},{"line_number":175,"context_line":"requests and with additional Quality of Service (QoS) metadata."},{"line_number":176,"context_line":"The Resource Management Daemon (RMD) solution will allow vendors to easily"},{"line_number":177,"context_line":"integrate their resources into a local node daemon instead of having to"},{"line_number":178,"context_line":"upstream code for different interfaces for multiple resources and their"},{"line_number":179,"context_line":"multiple versions.  This will eliminate vendor/platform dependent changes in"},{"line_number":180,"context_line":"OpenStack."},{"line_number":181,"context_line":""}],"source_content_type":"text/x-rst","patch_set":5,"id":"ffb9cba7_6a89cc24","line":178,"range":{"start_line":176,"start_character":51,"end_line":178,"end_character":13},"updated":"2019-04-21 22:56:51.000000000","message":"this is the primary use case, IMHO.","commit_id":"28b06cb71b84a2f2cee5b8c041c3e2781321c423"},{"author":{"_account_id":29334,"name":"Dakshina Ilangovan","email":"dakshina.ilangovan@intel.com","username":"dakshinai"},"change_message_id":"65af0255ba1ab001e71899a0351fa06a0cca46f1","unresolved":false,"context_lines":[{"line_number":173,"context_line":"new vendor resource that has to be supported, today there is no centralized"},{"line_number":174,"context_line":"approach to provide a user interface with high-level abstraction of allocation"},{"line_number":175,"context_line":"requests and with additional Quality of Service (QoS) metadata."},{"line_number":176,"context_line":"The Resource Management Daemon (RMD) solution will allow vendors to easily"},{"line_number":177,"context_line":"integrate their resources into a local node daemon instead of having to"},{"line_number":178,"context_line":"upstream code for different interfaces for multiple resources and their"},{"line_number":179,"context_line":"multiple versions.  This will eliminate vendor/platform dependent changes in"},{"line_number":180,"context_line":"OpenStack."},{"line_number":181,"context_line":""}],"source_content_type":"text/x-rst","patch_set":5,"id":"ffb9cba7_ea71a345","line":178,"range":{"start_line":176,"start_character":51,"end_line":178,"end_character":13},"in_reply_to":"ffb9cba7_1c186273","updated":"2019-04-30 07:03:38.000000000","message":"Hi Sean, RMD is a growing community. We started with Intel CAT and had contributions from QCT for MBA. RMD plugin architecture is designed to support multiple vendors but we need the support.\n\nWe have intel teams that also work on Libvirt integration for x86 features but the goal here is to complement those efforts with RMD\u0027s goals. RMD is vehicle for only those resources that benefit from dynamic management. We do not want to re-invent the wheel for static configuration that can be better done using OpenStack api\u0027s","commit_id":"28b06cb71b84a2f2cee5b8c041c3e2781321c423"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"caeec2d1a3a83edd4a394bc066e6092c61d74c07","unresolved":false,"context_lines":[{"line_number":173,"context_line":"new vendor resource that has to be supported, today there is no centralized"},{"line_number":174,"context_line":"approach to provide a user interface with high-level abstraction of allocation"},{"line_number":175,"context_line":"requests and with additional Quality of Service (QoS) metadata."},{"line_number":176,"context_line":"The Resource Management Daemon (RMD) solution will allow vendors to easily"},{"line_number":177,"context_line":"integrate their resources into a local node daemon instead of having to"},{"line_number":178,"context_line":"upstream code for different interfaces for multiple resources and their"},{"line_number":179,"context_line":"multiple versions.  This will eliminate vendor/platform dependent changes in"},{"line_number":180,"context_line":"OpenStack."},{"line_number":181,"context_line":""}],"source_content_type":"text/x-rst","patch_set":5,"id":"ffb9cba7_1c186273","line":178,"range":{"start_line":176,"start_character":51,"end_line":178,"end_character":13},"in_reply_to":"ffb9cba7_55f3160b","updated":"2019-04-24 13:28:29.000000000","message":"RMD is not vendor nutral however.\nintel has opensouce it yes but it does not have support for non intel plathforms.\n\nAMD have similar plathform QOS features in soem case and other achitecutres like power or arm have similar capablitys.\n\nunlike dpdk which ligitmetly has a multi vendor, multi architecture community RMD does not.\n\nwithout contibutions form other vendor this really jsut enables intel to integrate functionality into a local daemon instead fo upstream the code to openstack which is a project to intergrate multi vendor solutions under a common api.","commit_id":"28b06cb71b84a2f2cee5b8c041c3e2781321c423"},{"author":{"_account_id":29334,"name":"Dakshina Ilangovan","email":"dakshina.ilangovan@intel.com","username":"dakshinai"},"change_message_id":"b3066faf83f8b0bac6a3562e5d2039cd667053f6","unresolved":false,"context_lines":[{"line_number":173,"context_line":"new vendor resource that has to be supported, today there is no centralized"},{"line_number":174,"context_line":"approach to provide a user interface with high-level abstraction of allocation"},{"line_number":175,"context_line":"requests and with additional Quality of Service (QoS) metadata."},{"line_number":176,"context_line":"The Resource Management Daemon (RMD) solution will allow vendors to easily"},{"line_number":177,"context_line":"integrate their resources into a local node daemon instead of having to"},{"line_number":178,"context_line":"upstream code for different interfaces for multiple resources and their"},{"line_number":179,"context_line":"multiple versions.  This will eliminate vendor/platform dependent changes in"},{"line_number":180,"context_line":"OpenStack."},{"line_number":181,"context_line":""}],"source_content_type":"text/x-rst","patch_set":5,"id":"ffb9cba7_55f3160b","line":178,"range":{"start_line":176,"start_character":51,"end_line":178,"end_character":13},"in_reply_to":"ffb9cba7_6a89cc24","updated":"2019-04-24 06:09:41.000000000","message":"This is not the primary goal. Since RMD needs control and view over a subset of host resources for dynamic control, we get the added benefit of having all the changes done in one place independent of OpenStack.","commit_id":"28b06cb71b84a2f2cee5b8c041c3e2781321c423"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"20d4ca3f3e9dbdddbcd76244e2e1c40d024d67f6","unresolved":false,"context_lines":[{"line_number":176,"context_line":"The Resource Management Daemon (RMD) solution will allow vendors to easily"},{"line_number":177,"context_line":"integrate their resources into a local node daemon instead of having to"},{"line_number":178,"context_line":"upstream code for different interfaces for multiple resources and their"},{"line_number":179,"context_line":"multiple versions.  This will eliminate vendor/platform dependent changes in"},{"line_number":180,"context_line":"OpenStack."},{"line_number":181,"context_line":""},{"line_number":182,"context_line":"Use Cases"},{"line_number":183,"context_line":"---------"}],"source_content_type":"text/x-rst","patch_set":5,"id":"ffb9cba7_2a83d406","line":180,"range":{"start_line":179,"start_character":19,"end_line":180,"end_character":10},"updated":"2019-04-21 22:56:51.000000000","message":"Hardly","commit_id":"28b06cb71b84a2f2cee5b8c041c3e2781321c423"},{"author":{"_account_id":29334,"name":"Dakshina Ilangovan","email":"dakshina.ilangovan@intel.com","username":"dakshinai"},"change_message_id":"b3066faf83f8b0bac6a3562e5d2039cd667053f6","unresolved":false,"context_lines":[{"line_number":176,"context_line":"The Resource Management Daemon (RMD) solution will allow vendors to easily"},{"line_number":177,"context_line":"integrate their resources into a local node daemon instead of having to"},{"line_number":178,"context_line":"upstream code for different interfaces for multiple resources and their"},{"line_number":179,"context_line":"multiple versions.  This will eliminate vendor/platform dependent changes in"},{"line_number":180,"context_line":"OpenStack."},{"line_number":181,"context_line":""},{"line_number":182,"context_line":"Use Cases"},{"line_number":183,"context_line":"---------"}],"source_content_type":"text/x-rst","patch_set":5,"id":"ffb9cba7_75f65afc","line":180,"range":{"start_line":179,"start_character":19,"end_line":180,"end_character":10},"in_reply_to":"ffb9cba7_2a83d406","updated":"2019-04-24 06:09:41.000000000","message":"As explained above","commit_id":"28b06cb71b84a2f2cee5b8c041c3e2781321c423"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"caeec2d1a3a83edd4a394bc066e6092c61d74c07","unresolved":false,"context_lines":[{"line_number":176,"context_line":"The Resource Management Daemon (RMD) solution will allow vendors to easily"},{"line_number":177,"context_line":"integrate their resources into a local node daemon instead of having to"},{"line_number":178,"context_line":"upstream code for different interfaces for multiple resources and their"},{"line_number":179,"context_line":"multiple versions.  This will eliminate vendor/platform dependent changes in"},{"line_number":180,"context_line":"OpenStack."},{"line_number":181,"context_line":""},{"line_number":182,"context_line":"Use Cases"},{"line_number":183,"context_line":"---------"}],"source_content_type":"text/x-rst","patch_set":5,"id":"ffb9cba7_dc796a93","line":180,"range":{"start_line":179,"start_character":19,"end_line":180,"end_character":10},"in_reply_to":"ffb9cba7_75f65afc","updated":"2019-04-24 13:28:29.000000000","message":"i agree it does not remove the need for vendor/palthform specific code in openstack. it does not even remove it in nova or the livbrit driver.\n\ni also dont think that should be a goal.\none of the roles of openstack is to be an integration point to expose vendor specifc implementation thorugh a common abstrated api that is vendor neutral(where possible).","commit_id":"28b06cb71b84a2f2cee5b8c041c3e2781321c423"},{"author":{"_account_id":29334,"name":"Dakshina Ilangovan","email":"dakshina.ilangovan@intel.com","username":"dakshinai"},"change_message_id":"65af0255ba1ab001e71899a0351fa06a0cca46f1","unresolved":false,"context_lines":[{"line_number":176,"context_line":"The Resource Management Daemon (RMD) solution will allow vendors to easily"},{"line_number":177,"context_line":"integrate their resources into a local node daemon instead of having to"},{"line_number":178,"context_line":"upstream code for different interfaces for multiple resources and their"},{"line_number":179,"context_line":"multiple versions.  This will eliminate vendor/platform dependent changes in"},{"line_number":180,"context_line":"OpenStack."},{"line_number":181,"context_line":""},{"line_number":182,"context_line":"Use Cases"},{"line_number":183,"context_line":"---------"}],"source_content_type":"text/x-rst","patch_set":5,"id":"ffb9cba7_eaa683ab","line":180,"range":{"start_line":179,"start_character":19,"end_line":180,"end_character":10},"in_reply_to":"ffb9cba7_dc796a93","updated":"2019-04-30 07:03:38.000000000","message":"RMD vendor/platform independence is not the goal but ends up as an added advantage when resources that need dynamic control are integrated into RMD.","commit_id":"28b06cb71b84a2f2cee5b8c041c3e2781321c423"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"caeec2d1a3a83edd4a394bc066e6092c61d74c07","unresolved":false,"context_lines":[{"line_number":187,"context_line":""},{"line_number":188,"context_line":"The users and deployers of workloads that are sensitive to hardware resources"},{"line_number":189,"context_line":"may require dynamic management of the resources for optimal performance in a"},{"line_number":190,"context_line":"virtualized environment.  Hardware resources like Last Level Cache (LLC) and"},{"line_number":191,"context_line":"memory bandwidth can benefit from such dynamic adjustments [2]_. Since RMD has"},{"line_number":192,"context_line":"a holistic view of the set of resources on the local node under its umbrella"},{"line_number":193,"context_line":"and delivers a model to monitor these resources using collectors like"},{"line_number":194,"context_line":"collectd, it can make dynamic adjustments to resource allocations, which form"}],"source_content_type":"text/x-rst","patch_set":5,"id":"ffb9cba7_3c98c6be","line":191,"range":{"start_line":190,"start_character":27,"end_line":191,"end_character":64},"updated":"2019-04-24 13:28:29.000000000","message":"we could do that via openstack too with relitive ease at least in the case fo the libvirt driver.\n\nyou are usign dynamic in to contexts here.\n\nopenstack could dyncmially assing static allocation of cache or memory bandiwth (note we recently added minium bandididth based schduling via neutron memory bandwith would be pretty much identical)\n\nwhat openenstack cannot provide simply is realtime redistibution fo bandwith or cahce but i woudl argue at least in the cache case that you do not wnat that if you care about latency as everytime the cache or cpu pstate is adjusted there will be a cpu stall and a coresponding latency spike.","commit_id":"28b06cb71b84a2f2cee5b8c041c3e2781321c423"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"caeec2d1a3a83edd4a394bc066e6092c61d74c07","unresolved":false,"context_lines":[{"line_number":194,"context_line":"collectd, it can make dynamic adjustments to resource allocations, which form"},{"line_number":195,"context_line":"a local closed loop feedback.  The allocation will be based on the resource"},{"line_number":196,"context_line":"policy provided by the orchestrator. Of course, such changes would have to be"},{"line_number":197,"context_line":"made within the boundaries specified by the algorithm or workload owner.  In"},{"line_number":198,"context_line":"future, OpenStack can also interact with RMD via the control plane, with an"},{"line_number":199,"context_line":"updated policy to make dynamic adjustments based on long-term monitoring"},{"line_number":200,"context_line":"results obtained from all nodes."},{"line_number":201,"context_line":""},{"line_number":202,"context_line":"Use case #2: Vendor Independence for OpenStack Resource Management"},{"line_number":203,"context_line":"Actors: Users, Vendors, Developers"}],"source_content_type":"text/x-rst","patch_set":5,"id":"ffb9cba7_fce6ae44","line":200,"range":{"start_line":197,"start_character":74,"end_line":200,"end_character":31},"updated":"2019-04-24 13:28:29.000000000","message":"how would openstack do this.\nwhat service and under  what policy would such adjustments be made.","commit_id":"28b06cb71b84a2f2cee5b8c041c3e2781321c423"},{"author":{"_account_id":29334,"name":"Dakshina Ilangovan","email":"dakshina.ilangovan@intel.com","username":"dakshinai"},"change_message_id":"65af0255ba1ab001e71899a0351fa06a0cca46f1","unresolved":false,"context_lines":[{"line_number":194,"context_line":"collectd, it can make dynamic adjustments to resource allocations, which form"},{"line_number":195,"context_line":"a local closed loop feedback.  The allocation will be based on the resource"},{"line_number":196,"context_line":"policy provided by the orchestrator. Of course, such changes would have to be"},{"line_number":197,"context_line":"made within the boundaries specified by the algorithm or workload owner.  In"},{"line_number":198,"context_line":"future, OpenStack can also interact with RMD via the control plane, with an"},{"line_number":199,"context_line":"updated policy to make dynamic adjustments based on long-term monitoring"},{"line_number":200,"context_line":"results obtained from all nodes."},{"line_number":201,"context_line":""},{"line_number":202,"context_line":"Use case #2: Vendor Independence for OpenStack Resource Management"},{"line_number":203,"context_line":"Actors: Users, Vendors, Developers"}],"source_content_type":"text/x-rst","patch_set":5,"id":"ffb9cba7_4ae46fe0","line":200,"range":{"start_line":197,"start_character":74,"end_line":200,"end_character":31},"in_reply_to":"ffb9cba7_fce6ae44","updated":"2019-04-30 07:03:38.000000000","message":"As clarified above, we do not have a way to synchronize LLC updates made by RMD back to placement today.","commit_id":"28b06cb71b84a2f2cee5b8c041c3e2781321c423"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"caeec2d1a3a83edd4a394bc066e6092c61d74c07","unresolved":false,"context_lines":[{"line_number":199,"context_line":"updated policy to make dynamic adjustments based on long-term monitoring"},{"line_number":200,"context_line":"results obtained from all nodes."},{"line_number":201,"context_line":""},{"line_number":202,"context_line":"Use case #2: Vendor Independence for OpenStack Resource Management"},{"line_number":203,"context_line":"Actors: Users, Vendors, Developers"},{"line_number":204,"context_line":""},{"line_number":205,"context_line":"Users, vendors and developers need to control resources in easy to understand"}],"source_content_type":"text/x-rst","patch_set":5,"id":"ffb9cba7_bcd2765b","line":202,"range":{"start_line":202,"start_character":13,"end_line":202,"end_character":66},"updated":"2019-04-24 13:28:29.000000000","message":"openstack already has this in the form of the placment service with standard resocues defiend in os-resouce class and qulative aspects defiend in os-traits.\n\ni dont see how RMD helps here infact i think it would be a regression unless the suggestion is that RMD would be come a deliverable of the placement serice and a part of openstack under the governace of opendev foundation?","commit_id":"28b06cb71b84a2f2cee5b8c041c3e2781321c423"},{"author":{"_account_id":29334,"name":"Dakshina Ilangovan","email":"dakshina.ilangovan@intel.com","username":"dakshinai"},"change_message_id":"65af0255ba1ab001e71899a0351fa06a0cca46f1","unresolved":false,"context_lines":[{"line_number":199,"context_line":"updated policy to make dynamic adjustments based on long-term monitoring"},{"line_number":200,"context_line":"results obtained from all nodes."},{"line_number":201,"context_line":""},{"line_number":202,"context_line":"Use case #2: Vendor Independence for OpenStack Resource Management"},{"line_number":203,"context_line":"Actors: Users, Vendors, Developers"},{"line_number":204,"context_line":""},{"line_number":205,"context_line":"Users, vendors and developers need to control resources in easy to understand"}],"source_content_type":"text/x-rst","patch_set":5,"id":"ffb9cba7_0a40b7a4","line":202,"range":{"start_line":202,"start_character":13,"end_line":202,"end_character":66},"in_reply_to":"ffb9cba7_bcd2765b","updated":"2019-04-30 07:03:38.000000000","message":"I think this should not be listed under RMD\u0027s goal. This is more of side-effect to resources that require dynamic control from RMD.\n\nRMD is designed to be independent of OpenStack, K8 or other services.\n\nBut we are open to suggestions if a specific part of placement should interact with RMD.","commit_id":"28b06cb71b84a2f2cee5b8c041c3e2781321c423"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"caeec2d1a3a83edd4a394bc066e6092c61d74c07","unresolved":false,"context_lines":[{"line_number":225,"context_line":"control L3 cache to OpenStack, based on the vendor and platform used on the"},{"line_number":226,"context_line":"compute node."},{"line_number":227,"context_line":""},{"line_number":228,"context_line":"Use case #3: Platform Independence for OpenStack Resource Management"},{"line_number":229,"context_line":"Actors: Developers, Deployers"},{"line_number":230,"context_line":""},{"line_number":231,"context_line":"In order to manage platform-specific resources such as Intel RDT technology,"},{"line_number":232,"context_line":"developers should take into account the CPU SKU model, kernel version, kernel"},{"line_number":233,"context_line":"boot parameters and other kernel options like mount parameters for file"},{"line_number":234,"context_line":"systems in use. RMD helps to abstract these checks, and updates OpenStack"},{"line_number":235,"context_line":"inventory (via the Placement API) to let deployers know what resources are"},{"line_number":236,"context_line":"available on each OpenStack compute node."},{"line_number":237,"context_line":""},{"line_number":238,"context_line":"Proposed change"},{"line_number":239,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":5,"id":"ffb9cba7_9c121209","line":236,"range":{"start_line":228,"start_character":0,"end_line":236,"end_character":41},"updated":"2019-04-24 13:28:29.000000000","message":"this is distinctly not a cloud use case.\n\ndeveloper/end users can express requirement like \ntrait:avx\u003drequired via an image or operators can express it in a flavor in a vendor neutral way today.\n\nusing RMD create vendor locking not indepdence.\n\nby the way all discussion i have had with people proposing integration RMD with openstack have revovled around integration with the libvirt driver. libvirt already provides and abstract api which we use to discover cpu features.\nwhere they are not expose via libvirt we can use the abstraction provided via sysfs.\n\nthinkg slike the kernel version and kernel boot paramtare violate the cloud abstration and should never be exposted \nto end users and may infact be a security risk.\n\nover the last few year openstack partically the nova and placement project have worked to build an abstration where\n\nusers can declare there requirement with resouce request and tratis and openstack will either fufill that requrest or fail to boot the vm. exposing host specific infomation via rmd to endusers would regress that.\n\ni could see the discoverablity aspect maybe being useful in the ironic project for there inspector ram disk but that is not a nova usecase and should not be in this spec.","commit_id":"28b06cb71b84a2f2cee5b8c041c3e2781321c423"},{"author":{"_account_id":29334,"name":"Dakshina Ilangovan","email":"dakshina.ilangovan@intel.com","username":"dakshinai"},"change_message_id":"65af0255ba1ab001e71899a0351fa06a0cca46f1","unresolved":false,"context_lines":[{"line_number":225,"context_line":"control L3 cache to OpenStack, based on the vendor and platform used on the"},{"line_number":226,"context_line":"compute node."},{"line_number":227,"context_line":""},{"line_number":228,"context_line":"Use case #3: Platform Independence for OpenStack Resource Management"},{"line_number":229,"context_line":"Actors: Developers, Deployers"},{"line_number":230,"context_line":""},{"line_number":231,"context_line":"In order to manage platform-specific resources such as Intel RDT technology,"},{"line_number":232,"context_line":"developers should take into account the CPU SKU model, kernel version, kernel"},{"line_number":233,"context_line":"boot parameters and other kernel options like mount parameters for file"},{"line_number":234,"context_line":"systems in use. RMD helps to abstract these checks, and updates OpenStack"},{"line_number":235,"context_line":"inventory (via the Placement API) to let deployers know what resources are"},{"line_number":236,"context_line":"available on each OpenStack compute node."},{"line_number":237,"context_line":""},{"line_number":238,"context_line":"Proposed change"},{"line_number":239,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":5,"id":"ffb9cba7_0a051746","line":236,"range":{"start_line":228,"start_character":0,"end_line":236,"end_character":41},"in_reply_to":"ffb9cba7_9c121209","updated":"2019-04-30 07:03:38.000000000","message":"Hi Sean, you are correct. Traits and flavors are the right points and we propose to use them even for resources managed under RMD (for any dynamic control). It is incorrect to position vendor or platform independence as RMD\u0027s goal here. What we imply here is that if any resource is reported to placement via RMD (via virt-driver), all these resources are abstracted on resource type. If it is RDT, then irrespective of kernel version or vendor (in this case only 1), we report it as a single resource class. We still allow virt-driver to report resource provider traits/classes and it is upto virt-driver to call RMD for response to update_provider_tree or spawn(). RMD does not expose host specific information to users.","commit_id":"28b06cb71b84a2f2cee5b8c041c3e2781321c423"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"caeec2d1a3a83edd4a394bc066e6092c61d74c07","unresolved":false,"context_lines":[{"line_number":245,"context_line":""},{"line_number":246,"context_line":"1. Modifications to Nova Compute"},{"line_number":247,"context_line":"--------------------------------"},{"line_number":248,"context_line":"The virt-driver in Nova compute leverages RMD when RMD is deployed"},{"line_number":249,"context_line":"on a compute node. The virt-driver validates the existence of RMD on the"},{"line_number":250,"context_line":"compute node when RMD flags are enabled from the Nova configuration. All"},{"line_number":251,"context_line":"interactions to RMD HTTPS REST interface are mutually authenticated using"}],"source_content_type":"text/x-rst","patch_set":5,"id":"ffb9cba7_5ccb3a71","line":248,"range":{"start_line":248,"start_character":4,"end_line":248,"end_character":15},"updated":"2019-04-24 13:28:29.000000000","message":"libvirt virt-driver","commit_id":"28b06cb71b84a2f2cee5b8c041c3e2781321c423"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"caeec2d1a3a83edd4a394bc066e6092c61d74c07","unresolved":false,"context_lines":[{"line_number":245,"context_line":""},{"line_number":246,"context_line":"1. Modifications to Nova Compute"},{"line_number":247,"context_line":"--------------------------------"},{"line_number":248,"context_line":"The virt-driver in Nova compute leverages RMD when RMD is deployed"},{"line_number":249,"context_line":"on a compute node. The virt-driver validates the existence of RMD on the"},{"line_number":250,"context_line":"compute node when RMD flags are enabled from the Nova configuration. All"},{"line_number":251,"context_line":"interactions to RMD HTTPS REST interface are mutually authenticated using"}],"source_content_type":"text/x-rst","patch_set":5,"id":"ffb9cba7_fcc9ce77","line":248,"range":{"start_line":248,"start_character":32,"end_line":248,"end_character":41},"updated":"2019-04-24 13:28:29.000000000","message":"will leverage","commit_id":"28b06cb71b84a2f2cee5b8c041c3e2781321c423"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"caeec2d1a3a83edd4a394bc066e6092c61d74c07","unresolved":false,"context_lines":[{"line_number":247,"context_line":"--------------------------------"},{"line_number":248,"context_line":"The virt-driver in Nova compute leverages RMD when RMD is deployed"},{"line_number":249,"context_line":"on a compute node. The virt-driver validates the existence of RMD on the"},{"line_number":250,"context_line":"compute node when RMD flags are enabled from the Nova configuration. All"},{"line_number":251,"context_line":"interactions to RMD HTTPS REST interface are mutually authenticated using"},{"line_number":252,"context_line":"certificates enabled from Nova configuration. The Nova configuration will also"},{"line_number":253,"context_line":"carry configuration for the polling interval to request information from RMD."},{"line_number":254,"context_line":""},{"line_number":255,"context_line":"The responsibilities of the virt-driver to support RMD are outlined"}],"source_content_type":"text/x-rst","patch_set":5,"id":"ffb9cba7_dcf3aa72","line":252,"range":{"start_line":250,"start_character":69,"end_line":252,"end_character":44},"updated":"2019-04-24 13:28:29.000000000","message":"this will work but what is the advantage of usign ssl encrypted tcp transport to a local service when you could just connect to rm via a unix socket?\n\nthe rmd config does appear to supprot that \n\nhttps://github.com/intel/rmd/blob/18d1ae32a4ea331b5573ca75336002f2226b82a6/app/config/config.go#L39\n\nand looking at the code \nhttps://github.com/intel/rmd/blob/48190af0e3c09644841ea506c0aae2dd801c736a/app/server.go#L176-L212\n\nit serves the same rest api over the unix socket.","commit_id":"28b06cb71b84a2f2cee5b8c041c3e2781321c423"},{"author":{"_account_id":29334,"name":"Dakshina Ilangovan","email":"dakshina.ilangovan@intel.com","username":"dakshinai"},"change_message_id":"65af0255ba1ab001e71899a0351fa06a0cca46f1","unresolved":false,"context_lines":[{"line_number":247,"context_line":"--------------------------------"},{"line_number":248,"context_line":"The virt-driver in Nova compute leverages RMD when RMD is deployed"},{"line_number":249,"context_line":"on a compute node. The virt-driver validates the existence of RMD on the"},{"line_number":250,"context_line":"compute node when RMD flags are enabled from the Nova configuration. All"},{"line_number":251,"context_line":"interactions to RMD HTTPS REST interface are mutually authenticated using"},{"line_number":252,"context_line":"certificates enabled from Nova configuration. The Nova configuration will also"},{"line_number":253,"context_line":"carry configuration for the polling interval to request information from RMD."},{"line_number":254,"context_line":""},{"line_number":255,"context_line":"The responsibilities of the virt-driver to support RMD are outlined"}],"source_content_type":"text/x-rst","patch_set":5,"id":"ffb9cba7_0a2677cf","line":252,"range":{"start_line":250,"start_character":69,"end_line":252,"end_character":44},"in_reply_to":"ffb9cba7_dcf3aa72","updated":"2019-04-30 07:03:38.000000000","message":"Even with socket support, RMD expects to validate users talking to it. This is primarily because we do not want unauthorized users to use RMD\u0027s privileged access to the host. With unix socket, RMD validates users using Linux PAM.\n\nWe need some advice here with regards to integration with OpenStack. OpenStack does manage its users in Keystone. The same privileged users can talk to RMD. Do we need an wrapper that can let RMD use Keystone authentication or can OpenStack work with certificates?","commit_id":"28b06cb71b84a2f2cee5b8c041c3e2781321c423"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"caeec2d1a3a83edd4a394bc066e6092c61d74c07","unresolved":false,"context_lines":[{"line_number":249,"context_line":"on a compute node. The virt-driver validates the existence of RMD on the"},{"line_number":250,"context_line":"compute node when RMD flags are enabled from the Nova configuration. All"},{"line_number":251,"context_line":"interactions to RMD HTTPS REST interface are mutually authenticated using"},{"line_number":252,"context_line":"certificates enabled from Nova configuration. The Nova configuration will also"},{"line_number":253,"context_line":"carry configuration for the polling interval to request information from RMD."},{"line_number":254,"context_line":""},{"line_number":255,"context_line":"The responsibilities of the virt-driver to support RMD are outlined"},{"line_number":256,"context_line":"below:"}],"source_content_type":"text/x-rst","patch_set":5,"id":"ffb9cba7_7c463e7f","line":253,"range":{"start_line":252,"start_character":47,"end_line":253,"end_character":77},"updated":"2019-04-24 13:28:29.000000000","message":"well we could just use the same update interval that we use to do the normal periodic sync. we should minimise the number of rmd config options","commit_id":"28b06cb71b84a2f2cee5b8c041c3e2781321c423"},{"author":{"_account_id":29334,"name":"Dakshina Ilangovan","email":"dakshina.ilangovan@intel.com","username":"dakshinai"},"change_message_id":"65af0255ba1ab001e71899a0351fa06a0cca46f1","unresolved":false,"context_lines":[{"line_number":249,"context_line":"on a compute node. The virt-driver validates the existence of RMD on the"},{"line_number":250,"context_line":"compute node when RMD flags are enabled from the Nova configuration. All"},{"line_number":251,"context_line":"interactions to RMD HTTPS REST interface are mutually authenticated using"},{"line_number":252,"context_line":"certificates enabled from Nova configuration. The Nova configuration will also"},{"line_number":253,"context_line":"carry configuration for the polling interval to request information from RMD."},{"line_number":254,"context_line":""},{"line_number":255,"context_line":"The responsibilities of the virt-driver to support RMD are outlined"},{"line_number":256,"context_line":"below:"}],"source_content_type":"text/x-rst","patch_set":5,"id":"ffb9cba7_e5e0f428","line":253,"range":{"start_line":252,"start_character":47,"end_line":253,"end_character":77},"in_reply_to":"ffb9cba7_7c463e7f","updated":"2019-04-30 07:03:38.000000000","message":"Correct, this line should be removed.","commit_id":"28b06cb71b84a2f2cee5b8c041c3e2781321c423"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"caeec2d1a3a83edd4a394bc066e6092c61d74c07","unresolved":false,"context_lines":[{"line_number":257,"context_line":""},{"line_number":258,"context_line":"* Query RMD to get the inventory of resources it manages by polling"},{"line_number":259,"context_line":""},{"line_number":260,"context_line":"  OpenStack should be aware of the resources reported and managed by RMD"},{"line_number":261,"context_line":"  across all compute nodes that deploy RMD, to let users request and schedule"},{"line_number":262,"context_line":"  for those resources. The virt-driver will query resource inventory"},{"line_number":263,"context_line":"  information from RMD\u0027s REST interface as part of its \"update_provider_tree\""}],"source_content_type":"text/x-rst","patch_set":5,"id":"ffb9cba7_dc1dea73","line":260,"range":{"start_line":260,"start_character":2,"end_line":260,"end_character":72},"updated":"2019-04-24 13:28:29.000000000","message":"so technically openstack is still the owner of all there resouce on the plathform. it can chosse to delegate contol of some of the resocue to RMD but if there is ever a conflcit openstack should thake prescedence.","commit_id":"28b06cb71b84a2f2cee5b8c041c3e2781321c423"},{"author":{"_account_id":29334,"name":"Dakshina Ilangovan","email":"dakshina.ilangovan@intel.com","username":"dakshinai"},"change_message_id":"65af0255ba1ab001e71899a0351fa06a0cca46f1","unresolved":false,"context_lines":[{"line_number":257,"context_line":""},{"line_number":258,"context_line":"* Query RMD to get the inventory of resources it manages by polling"},{"line_number":259,"context_line":""},{"line_number":260,"context_line":"  OpenStack should be aware of the resources reported and managed by RMD"},{"line_number":261,"context_line":"  across all compute nodes that deploy RMD, to let users request and schedule"},{"line_number":262,"context_line":"  for those resources. The virt-driver will query resource inventory"},{"line_number":263,"context_line":"  information from RMD\u0027s REST interface as part of its \"update_provider_tree\""}],"source_content_type":"text/x-rst","patch_set":5,"id":"ffb9cba7_2a233bbf","line":260,"range":{"start_line":260,"start_character":2,"end_line":260,"end_character":72},"in_reply_to":"ffb9cba7_dc1dea73","updated":"2019-04-30 07:03:38.000000000","message":"Correct.","commit_id":"28b06cb71b84a2f2cee5b8c041c3e2781321c423"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"caeec2d1a3a83edd4a394bc066e6092c61d74c07","unresolved":false,"context_lines":[{"line_number":262,"context_line":"  for those resources. The virt-driver will query resource inventory"},{"line_number":263,"context_line":"  information from RMD\u0027s REST interface as part of its \"update_provider_tree\""},{"line_number":264,"context_line":"  method. The query should also reflect new inventory information on startup"},{"line_number":265,"context_line":"  of RMD, after any hot-plug in the OpenStack compute node or when resources"},{"line_number":266,"context_line":"  are used by other non Openstack processes on the compute node."},{"line_number":267,"context_line":""},{"line_number":268,"context_line":"* Update the resource inventory information, obtained from RMD, to the"}],"source_content_type":"text/x-rst","patch_set":5,"id":"ffb9cba7_afff96ff","line":265,"range":{"start_line":265,"start_character":10,"end_line":265,"end_character":58},"updated":"2019-04-24 13:28:29.000000000","message":"this would be handeled via the periodic task","commit_id":"28b06cb71b84a2f2cee5b8c041c3e2781321c423"},{"author":{"_account_id":29334,"name":"Dakshina Ilangovan","email":"dakshina.ilangovan@intel.com","username":"dakshinai"},"change_message_id":"65af0255ba1ab001e71899a0351fa06a0cca46f1","unresolved":false,"context_lines":[{"line_number":262,"context_line":"  for those resources. The virt-driver will query resource inventory"},{"line_number":263,"context_line":"  information from RMD\u0027s REST interface as part of its \"update_provider_tree\""},{"line_number":264,"context_line":"  method. The query should also reflect new inventory information on startup"},{"line_number":265,"context_line":"  of RMD, after any hot-plug in the OpenStack compute node or when resources"},{"line_number":266,"context_line":"  are used by other non Openstack processes on the compute node."},{"line_number":267,"context_line":""},{"line_number":268,"context_line":"* Update the resource inventory information, obtained from RMD, to the"}],"source_content_type":"text/x-rst","patch_set":5,"id":"ffb9cba7_ea28c397","line":265,"range":{"start_line":265,"start_character":10,"end_line":265,"end_character":58},"in_reply_to":"ffb9cba7_afff96ff","updated":"2019-04-30 07:03:38.000000000","message":"Yes, we expect that this is covered.","commit_id":"28b06cb71b84a2f2cee5b8c041c3e2781321c423"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"caeec2d1a3a83edd4a394bc066e6092c61d74c07","unresolved":false,"context_lines":[{"line_number":262,"context_line":"  for those resources. The virt-driver will query resource inventory"},{"line_number":263,"context_line":"  information from RMD\u0027s REST interface as part of its \"update_provider_tree\""},{"line_number":264,"context_line":"  method. The query should also reflect new inventory information on startup"},{"line_number":265,"context_line":"  of RMD, after any hot-plug in the OpenStack compute node or when resources"},{"line_number":266,"context_line":"  are used by other non Openstack processes on the compute node."},{"line_number":267,"context_line":""},{"line_number":268,"context_line":"* Update the resource inventory information, obtained from RMD, to the"},{"line_number":269,"context_line":"  Placement API"}],"source_content_type":"text/x-rst","patch_set":5,"id":"ffb9cba7_4f37fae4","line":266,"range":{"start_line":265,"start_character":62,"end_line":266,"end_character":64},"updated":"2019-04-24 13:28:29.000000000","message":"this is expressly not allowed on an openstack host.\n\nyou may reserve a portion of resource for host use via openstack config but the remaining resouce are owned by openstack and may not be consumed by other process or they will result in over subsription beyond the policy defiend in nova configs.","commit_id":"28b06cb71b84a2f2cee5b8c041c3e2781321c423"},{"author":{"_account_id":29334,"name":"Dakshina Ilangovan","email":"dakshina.ilangovan@intel.com","username":"dakshinai"},"change_message_id":"65af0255ba1ab001e71899a0351fa06a0cca46f1","unresolved":false,"context_lines":[{"line_number":262,"context_line":"  for those resources. The virt-driver will query resource inventory"},{"line_number":263,"context_line":"  information from RMD\u0027s REST interface as part of its \"update_provider_tree\""},{"line_number":264,"context_line":"  method. The query should also reflect new inventory information on startup"},{"line_number":265,"context_line":"  of RMD, after any hot-plug in the OpenStack compute node or when resources"},{"line_number":266,"context_line":"  are used by other non Openstack processes on the compute node."},{"line_number":267,"context_line":""},{"line_number":268,"context_line":"* Update the resource inventory information, obtained from RMD, to the"},{"line_number":269,"context_line":"  Placement API"}],"source_content_type":"text/x-rst","patch_set":5,"id":"ffb9cba7_ea01a307","line":266,"range":{"start_line":265,"start_character":62,"end_line":266,"end_character":64},"in_reply_to":"ffb9cba7_4f37fae4","updated":"2019-04-30 07:03:38.000000000","message":"Yes, this is corner case but just to ensure that we have it covered. \n\nI believe we cannot prevent someone from using host resources without OpenStack on a compute node but virt-driver periodic task does keep a check on inventory.","commit_id":"28b06cb71b84a2f2cee5b8c041c3e2781321c423"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"caeec2d1a3a83edd4a394bc066e6092c61d74c07","unresolved":false,"context_lines":[{"line_number":284,"context_line":"  allocations."},{"line_number":285,"context_line":""},{"line_number":286,"context_line":"  With the introduction and QoS and other meta-data into the flavor that can"},{"line_number":287,"context_line":"  be used by RMD, Nova should implement an RMD filter on the flavor to extract"},{"line_number":288,"context_line":"  min value requested for resources that can be used by the scheduler. Based"},{"line_number":289,"context_line":"  on the request for a resource managed by RMD through custom flavors and"},{"line_number":290,"context_line":"  inventory information in the Placement API, the scheduler can select the"},{"line_number":291,"context_line":"  appropriate compute node reporting the respective RMD resource. The"}],"source_content_type":"text/x-rst","patch_set":5,"id":"ffb9cba7_2faf26e4","line":288,"range":{"start_line":287,"start_character":18,"end_line":288,"end_character":70},"updated":"2019-04-24 13:28:29.000000000","message":"for the first two usecase of cache and memory bandwith a new filter should not be required as we will do the accounting of both in placemetn although the numa toplogy filter man need to be modifed to ensure the cache allsotion and memroy bandwith allotions are from teh correct numa node.","commit_id":"28b06cb71b84a2f2cee5b8c041c3e2781321c423"},{"author":{"_account_id":29334,"name":"Dakshina Ilangovan","email":"dakshina.ilangovan@intel.com","username":"dakshinai"},"change_message_id":"65af0255ba1ab001e71899a0351fa06a0cca46f1","unresolved":false,"context_lines":[{"line_number":284,"context_line":"  allocations."},{"line_number":285,"context_line":""},{"line_number":286,"context_line":"  With the introduction and QoS and other meta-data into the flavor that can"},{"line_number":287,"context_line":"  be used by RMD, Nova should implement an RMD filter on the flavor to extract"},{"line_number":288,"context_line":"  min value requested for resources that can be used by the scheduler. Based"},{"line_number":289,"context_line":"  on the request for a resource managed by RMD through custom flavors and"},{"line_number":290,"context_line":"  inventory information in the Placement API, the scheduler can select the"},{"line_number":291,"context_line":"  appropriate compute node reporting the respective RMD resource. The"}],"source_content_type":"text/x-rst","patch_set":5,"id":"ffb9cba7_652404b0","line":288,"range":{"start_line":287,"start_character":18,"end_line":288,"end_character":70},"in_reply_to":"ffb9cba7_2faf26e4","updated":"2019-04-30 07:03:38.000000000","message":"Yes this is not required for cache and memory bandwidth. I\u0027m not sure if we can represent LLC per NUMA node in the placement yet. Any suggestions?","commit_id":"28b06cb71b84a2f2cee5b8c041c3e2781321c423"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"caeec2d1a3a83edd4a394bc066e6092c61d74c07","unresolved":false,"context_lines":[{"line_number":306,"context_line":"  scheduler, the following scenarios will occur:"},{"line_number":307,"context_line":""},{"line_number":308,"context_line":"  a. If RMD allocation succeeds, no further interactions are needed."},{"line_number":309,"context_line":"  b. If RMD allocation fails, an exception is returned to Nova Compute,"},{"line_number":310,"context_line":"     allocation entries are removed from the allocation table and the compute"},{"line_number":311,"context_line":"     node is blacklisted for that request to avoid any retries by the"},{"line_number":312,"context_line":"     scheduler. Note than an exception is returned from RMD even if one of the"},{"line_number":313,"context_line":"     resource allocations requested in the policy fails."},{"line_number":314,"context_line":"  c. RMD persists policies it receives in a lightweight no-sql file. When a"},{"line_number":315,"context_line":"     workload (e.g., VM) crashes or restarts or RMD is restarted, RMD clears"},{"line_number":316,"context_line":"     the policies and allocations if the workload no longer exists or restores"}],"source_content_type":"text/x-rst","patch_set":5,"id":"ffb9cba7_0fe26230","line":313,"range":{"start_line":309,"start_character":3,"end_line":313,"end_character":56},"updated":"2019-04-24 13:28:29.000000000","message":"this shoudl never happen. if it does it means the infoamation report by rmd that was forwarded to placmenet was incorrect.\n\nplacement should be the singel source of truth when it comes to the quantitative capacity of resources in open stack.","commit_id":"28b06cb71b84a2f2cee5b8c041c3e2781321c423"},{"author":{"_account_id":29334,"name":"Dakshina Ilangovan","email":"dakshina.ilangovan@intel.com","username":"dakshinai"},"change_message_id":"65af0255ba1ab001e71899a0351fa06a0cca46f1","unresolved":false,"context_lines":[{"line_number":306,"context_line":"  scheduler, the following scenarios will occur:"},{"line_number":307,"context_line":""},{"line_number":308,"context_line":"  a. If RMD allocation succeeds, no further interactions are needed."},{"line_number":309,"context_line":"  b. If RMD allocation fails, an exception is returned to Nova Compute,"},{"line_number":310,"context_line":"     allocation entries are removed from the allocation table and the compute"},{"line_number":311,"context_line":"     node is blacklisted for that request to avoid any retries by the"},{"line_number":312,"context_line":"     scheduler. Note than an exception is returned from RMD even if one of the"},{"line_number":313,"context_line":"     resource allocations requested in the policy fails."},{"line_number":314,"context_line":"  c. RMD persists policies it receives in a lightweight no-sql file. When a"},{"line_number":315,"context_line":"     workload (e.g., VM) crashes or restarts or RMD is restarted, RMD clears"},{"line_number":316,"context_line":"     the policies and allocations if the workload no longer exists or restores"}],"source_content_type":"text/x-rst","patch_set":5,"id":"ffb9cba7_854ed86c","line":313,"range":{"start_line":309,"start_character":3,"end_line":313,"end_character":56},"in_reply_to":"ffb9cba7_0fe26230","updated":"2019-04-30 07:03:38.000000000","message":"Correct, we mean exceptions from RMD not being available.\n\nAlso, there is a special case with LLC. Today we can allocate cache ways only contiguously on hardware, which means we will end up with fragmentation as and when workloads exit and LLC is freed. So, if we have 2 empty buckets of LLC of size 3 and 4, we cannot allocate LLC of size 6, even though placement thinks free LLC of size 7 (3+4) exists.\n\nWe do not want to do defragmentation from software as it could introduce latency due to cache warm-up.","commit_id":"28b06cb71b84a2f2cee5b8c041c3e2781321c423"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"caeec2d1a3a83edd4a394bc066e6092c61d74c07","unresolved":false,"context_lines":[{"line_number":314,"context_line":"  c. RMD persists policies it receives in a lightweight no-sql file. When a"},{"line_number":315,"context_line":"     workload (e.g., VM) crashes or restarts or RMD is restarted, RMD clears"},{"line_number":316,"context_line":"     the policies and allocations if the workload no longer exists or restores"},{"line_number":317,"context_line":"     the allocations from polices if the workload exists. Nova Compute takes"},{"line_number":318,"context_line":"     care of updating the Placement API in OpenStack for workloads that do not"},{"line_number":319,"context_line":"     exist."},{"line_number":320,"context_line":"  d. Any dynamic re-allocations to workload resources done by RMD is handled"},{"line_number":321,"context_line":"     in the range of min and max allocation values requested for those"},{"line_number":322,"context_line":"     resources."}],"source_content_type":"text/x-rst","patch_set":5,"id":"ffb9cba7_af60f6a6","line":319,"range":{"start_line":317,"start_character":58,"end_line":319,"end_character":11},"updated":"2019-04-24 13:28:29.000000000","message":"nova will only delete the allocation in placement for non existent workload if the instance is deltete or move to a different host.\n\nno update should be performed if the vm crashes,reboots or is just shutdown.\n\nthose resouces are still reserved for that vm when it is started again and should not requrie a palcement update.\n\nrestarting the vm shoudl also not results in scenario b under any circumstances.","commit_id":"28b06cb71b84a2f2cee5b8c041c3e2781321c423"},{"author":{"_account_id":29334,"name":"Dakshina Ilangovan","email":"dakshina.ilangovan@intel.com","username":"dakshinai"},"change_message_id":"65af0255ba1ab001e71899a0351fa06a0cca46f1","unresolved":false,"context_lines":[{"line_number":314,"context_line":"  c. RMD persists policies it receives in a lightweight no-sql file. When a"},{"line_number":315,"context_line":"     workload (e.g., VM) crashes or restarts or RMD is restarted, RMD clears"},{"line_number":316,"context_line":"     the policies and allocations if the workload no longer exists or restores"},{"line_number":317,"context_line":"     the allocations from polices if the workload exists. Nova Compute takes"},{"line_number":318,"context_line":"     care of updating the Placement API in OpenStack for workloads that do not"},{"line_number":319,"context_line":"     exist."},{"line_number":320,"context_line":"  d. Any dynamic re-allocations to workload resources done by RMD is handled"},{"line_number":321,"context_line":"     in the range of min and max allocation values requested for those"},{"line_number":322,"context_line":"     resources."}],"source_content_type":"text/x-rst","patch_set":5,"id":"ffb9cba7_45a220fe","line":319,"range":{"start_line":317,"start_character":58,"end_line":319,"end_character":11},"in_reply_to":"ffb9cba7_af60f6a6","updated":"2019-04-30 07:03:38.000000000","message":"Question here,\n\nVM crashes - does OpenStack have a wait period before it decides to delete the VM and free up resources or is manual intervention needed.\n\nVM reboot/shutdown - upon restart, the pid will change? If so, how does OpenStack re-associate allocations (if they were not freed)","commit_id":"28b06cb71b84a2f2cee5b8c041c3e2781321c423"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"caeec2d1a3a83edd4a394bc066e6092c61d74c07","unresolved":false,"context_lines":[{"line_number":317,"context_line":"     the allocations from polices if the workload exists. Nova Compute takes"},{"line_number":318,"context_line":"     care of updating the Placement API in OpenStack for workloads that do not"},{"line_number":319,"context_line":"     exist."},{"line_number":320,"context_line":"  d. Any dynamic re-allocations to workload resources done by RMD is handled"},{"line_number":321,"context_line":"     in the range of min and max allocation values requested for those"},{"line_number":322,"context_line":"     resources."},{"line_number":323,"context_line":""},{"line_number":324,"context_line":"  Nova Compute polls for these changes to resource allocations of the"},{"line_number":325,"context_line":"  workloads done by RMD and sends updates to Allocations API in the Placement"}],"source_content_type":"text/x-rst","patch_set":5,"id":"ffb9cba7_af3796b0","line":322,"range":{"start_line":320,"start_character":5,"end_line":322,"end_character":15},"updated":"2019-04-24 13:28:29.000000000","message":"this reallocation should never lead to a failure to spawn a vm as descibed in senario b","commit_id":"28b06cb71b84a2f2cee5b8c041c3e2781321c423"},{"author":{"_account_id":29334,"name":"Dakshina Ilangovan","email":"dakshina.ilangovan@intel.com","username":"dakshinai"},"change_message_id":"65af0255ba1ab001e71899a0351fa06a0cca46f1","unresolved":false,"context_lines":[{"line_number":317,"context_line":"     the allocations from polices if the workload exists. Nova Compute takes"},{"line_number":318,"context_line":"     care of updating the Placement API in OpenStack for workloads that do not"},{"line_number":319,"context_line":"     exist."},{"line_number":320,"context_line":"  d. Any dynamic re-allocations to workload resources done by RMD is handled"},{"line_number":321,"context_line":"     in the range of min and max allocation values requested for those"},{"line_number":322,"context_line":"     resources."},{"line_number":323,"context_line":""},{"line_number":324,"context_line":"  Nova Compute polls for these changes to resource allocations of the"},{"line_number":325,"context_line":"  workloads done by RMD and sends updates to Allocations API in the Placement"}],"source_content_type":"text/x-rst","patch_set":5,"id":"ffb9cba7_c5b5b033","line":322,"range":{"start_line":320,"start_character":5,"end_line":322,"end_character":15},"in_reply_to":"ffb9cba7_af3796b0","updated":"2019-04-30 07:03:38.000000000","message":"I should have removed this point since we do not want to do any dynamic allocations under the hood without the knowledge of placement.\n\nScenario b is expected to happen only during spawn() and delegation to RMD\n\nDynamic changes should recover back to previous state and not happen if failure occurs","commit_id":"28b06cb71b84a2f2cee5b8c041c3e2781321c423"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"caeec2d1a3a83edd4a394bc066e6092c61d74c07","unresolved":false,"context_lines":[{"line_number":321,"context_line":"     in the range of min and max allocation values requested for those"},{"line_number":322,"context_line":"     resources."},{"line_number":323,"context_line":""},{"line_number":324,"context_line":"  Nova Compute polls for these changes to resource allocations of the"},{"line_number":325,"context_line":"  workloads done by RMD and sends updates to Allocations API in the Placement"},{"line_number":326,"context_line":"  API for each workload."},{"line_number":327,"context_line":""}],"source_content_type":"text/x-rst","patch_set":5,"id":"ffb9cba7_af0556c4","line":324,"range":{"start_line":324,"start_character":51,"end_line":324,"end_character":62},"updated":"2019-04-24 13:28:29.000000000","message":"we should proably use the term resource assignments to refer to the rmd management of the resouces and keep allocations for placement.","commit_id":"28b06cb71b84a2f2cee5b8c041c3e2781321c423"},{"author":{"_account_id":29334,"name":"Dakshina Ilangovan","email":"dakshina.ilangovan@intel.com","username":"dakshinai"},"change_message_id":"65af0255ba1ab001e71899a0351fa06a0cca46f1","unresolved":false,"context_lines":[{"line_number":321,"context_line":"     in the range of min and max allocation values requested for those"},{"line_number":322,"context_line":"     resources."},{"line_number":323,"context_line":""},{"line_number":324,"context_line":"  Nova Compute polls for these changes to resource allocations of the"},{"line_number":325,"context_line":"  workloads done by RMD and sends updates to Allocations API in the Placement"},{"line_number":326,"context_line":"  API for each workload."},{"line_number":327,"context_line":""}],"source_content_type":"text/x-rst","patch_set":5,"id":"ffb9cba7_457b007a","line":324,"range":{"start_line":324,"start_character":51,"end_line":324,"end_character":62},"in_reply_to":"ffb9cba7_af0556c4","updated":"2019-04-30 07:03:38.000000000","message":"yes will change the term and remove point d as described above.","commit_id":"28b06cb71b84a2f2cee5b8c041c3e2781321c423"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"caeec2d1a3a83edd4a394bc066e6092c61d74c07","unresolved":false,"context_lines":[{"line_number":337,"context_line":""},{"line_number":338,"context_line":"The privilege division between the two processes is by design to prevent any"},{"line_number":339,"context_line":"misuse of super user permissions that RMD may have to access the hardware"},{"line_number":340,"context_line":"resources. All requests to RMD are placed via HTTPS based REST API. The"},{"line_number":341,"context_line":"authority of a user is verified using HTTPS based mutual authentication or by"},{"line_number":342,"context_line":"using credentials supplied over HTTPS which are validated using PAM on Linux"},{"line_number":343,"context_line":"systems [5]_. The REST API consumes a policy that can contain requests for"}],"source_content_type":"text/x-rst","patch_set":5,"id":"ffb9cba7_0f17c2ef","line":340,"range":{"start_line":340,"start_character":11,"end_line":340,"end_character":67},"updated":"2019-04-24 13:28:29.000000000","message":"or you can use the unix socket as a transprot instead and use simple filesystem permissions to contol acess. this is more secure as it does not enable remote acess and is more efficent in terms fo memory and cpu utilisation as we can avoid the need for encryptiuon and muluial exchange of ssl cert for auth.","commit_id":"28b06cb71b84a2f2cee5b8c041c3e2781321c423"},{"author":{"_account_id":29334,"name":"Dakshina Ilangovan","email":"dakshina.ilangovan@intel.com","username":"dakshinai"},"change_message_id":"65af0255ba1ab001e71899a0351fa06a0cca46f1","unresolved":false,"context_lines":[{"line_number":337,"context_line":""},{"line_number":338,"context_line":"The privilege division between the two processes is by design to prevent any"},{"line_number":339,"context_line":"misuse of super user permissions that RMD may have to access the hardware"},{"line_number":340,"context_line":"resources. All requests to RMD are placed via HTTPS based REST API. The"},{"line_number":341,"context_line":"authority of a user is verified using HTTPS based mutual authentication or by"},{"line_number":342,"context_line":"using credentials supplied over HTTPS which are validated using PAM on Linux"},{"line_number":343,"context_line":"systems [5]_. The REST API consumes a policy that can contain requests for"}],"source_content_type":"text/x-rst","patch_set":5,"id":"ffb9cba7_059988c2","line":340,"range":{"start_line":340,"start_character":11,"end_line":340,"end_character":67},"in_reply_to":"ffb9cba7_0f17c2ef","updated":"2019-04-30 07:03:38.000000000","message":"Back to old question, can we use Linux PAM and have virt-driver pass credentials? Not all resource interfaces are file-system based, so we instead authenticate on user.","commit_id":"28b06cb71b84a2f2cee5b8c041c3e2781321c423"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"caeec2d1a3a83edd4a394bc066e6092c61d74c07","unresolved":false,"context_lines":[{"line_number":354,"context_line":"vendor/platform specific resources. However, Libvirt is only interested in"},{"line_number":355,"context_line":"static provisioning of a workload with the following limitations:"},{"line_number":356,"context_line":""},{"line_number":357,"context_line":"* Libvirt assigns a specific quantity of requested resources to a VM while"},{"line_number":358,"context_line":"  these resource allocation do not change for the life a VM without an"},{"line_number":359,"context_line":"  explicit update from the control plane."},{"line_number":360,"context_line":"* Libvirt does not track or modify the shared resources of a CPU at runtime"},{"line_number":361,"context_line":"  that are allocated across multiple VMs to avoid noisy neighbor situations or"},{"line_number":362,"context_line":"  shared resource contention problems."}],"source_content_type":"text/x-rst","patch_set":5,"id":"ffb9cba7_efc90e78","line":359,"range":{"start_line":357,"start_character":1,"end_line":359,"end_character":41},"updated":"2019-04-24 13:28:29.000000000","message":"yep this is semanticaly and synaticly consistent with the abstract api exposed by nova via flavors.\n\ni honestly thing we shoudl support both models.\nmake rmd optional and do the basic static allocation via libvirt.  that siad i wont have time to work on that so im not going to block on that.","commit_id":"28b06cb71b84a2f2cee5b8c041c3e2781321c423"},{"author":{"_account_id":29334,"name":"Dakshina Ilangovan","email":"dakshina.ilangovan@intel.com","username":"dakshinai"},"change_message_id":"65af0255ba1ab001e71899a0351fa06a0cca46f1","unresolved":false,"context_lines":[{"line_number":354,"context_line":"vendor/platform specific resources. However, Libvirt is only interested in"},{"line_number":355,"context_line":"static provisioning of a workload with the following limitations:"},{"line_number":356,"context_line":""},{"line_number":357,"context_line":"* Libvirt assigns a specific quantity of requested resources to a VM while"},{"line_number":358,"context_line":"  these resource allocation do not change for the life a VM without an"},{"line_number":359,"context_line":"  explicit update from the control plane."},{"line_number":360,"context_line":"* Libvirt does not track or modify the shared resources of a CPU at runtime"},{"line_number":361,"context_line":"  that are allocated across multiple VMs to avoid noisy neighbor situations or"},{"line_number":362,"context_line":"  shared resource contention problems."}],"source_content_type":"text/x-rst","patch_set":5,"id":"ffb9cba7_e58b1465","line":359,"range":{"start_line":357,"start_character":1,"end_line":359,"end_character":41},"in_reply_to":"ffb9cba7_efc90e78","updated":"2019-04-30 07:03:38.000000000","message":"Correct. RMD complements Libvirt.","commit_id":"28b06cb71b84a2f2cee5b8c041c3e2781321c423"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"caeec2d1a3a83edd4a394bc066e6092c61d74c07","unresolved":false,"context_lines":[{"line_number":357,"context_line":"* Libvirt assigns a specific quantity of requested resources to a VM while"},{"line_number":358,"context_line":"  these resource allocation do not change for the life a VM without an"},{"line_number":359,"context_line":"  explicit update from the control plane."},{"line_number":360,"context_line":"* Libvirt does not track or modify the shared resources of a CPU at runtime"},{"line_number":361,"context_line":"  that are allocated across multiple VMs to avoid noisy neighbor situations or"},{"line_number":362,"context_line":"  shared resource contention problems."},{"line_number":363,"context_line":"* The only run time changes made by Libvirt are when external entities like"},{"line_number":364,"context_line":"  Vitrage, Monasca or analytics node initiate resource changes via live"},{"line_number":365,"context_line":"  migration etc."}],"source_content_type":"text/x-rst","patch_set":5,"id":"ffb9cba7_0feca2f1","line":362,"range":{"start_line":360,"start_character":2,"end_line":362,"end_character":38},"updated":"2019-04-24 13:28:29.000000000","message":"libvirt does expose cache and nova can track the utilisation by multiple vms","commit_id":"28b06cb71b84a2f2cee5b8c041c3e2781321c423"},{"author":{"_account_id":29334,"name":"Dakshina Ilangovan","email":"dakshina.ilangovan@intel.com","username":"dakshinai"},"change_message_id":"65af0255ba1ab001e71899a0351fa06a0cca46f1","unresolved":false,"context_lines":[{"line_number":357,"context_line":"* Libvirt assigns a specific quantity of requested resources to a VM while"},{"line_number":358,"context_line":"  these resource allocation do not change for the life a VM without an"},{"line_number":359,"context_line":"  explicit update from the control plane."},{"line_number":360,"context_line":"* Libvirt does not track or modify the shared resources of a CPU at runtime"},{"line_number":361,"context_line":"  that are allocated across multiple VMs to avoid noisy neighbor situations or"},{"line_number":362,"context_line":"  shared resource contention problems."},{"line_number":363,"context_line":"* The only run time changes made by Libvirt are when external entities like"},{"line_number":364,"context_line":"  Vitrage, Monasca or analytics node initiate resource changes via live"},{"line_number":365,"context_line":"  migration etc."}],"source_content_type":"text/x-rst","patch_set":5,"id":"ffb9cba7_85ea580c","line":362,"range":{"start_line":360,"start_character":2,"end_line":362,"end_character":38},"in_reply_to":"ffb9cba7_0feca2f1","updated":"2019-04-30 07:03:38.000000000","message":"Shall reword this. Libvirt does not monitor (telemetry) and make dynamic changes on local compute node.","commit_id":"28b06cb71b84a2f2cee5b8c041c3e2781321c423"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"caeec2d1a3a83edd4a394bc066e6092c61d74c07","unresolved":false,"context_lines":[{"line_number":407,"context_line":""},{"line_number":408,"context_line":"We expect the polling agent in Nova Compute to be fine-tuned using Nova"},{"line_number":409,"context_line":"configuration to avoid frequent polling and reduce any subsequent resource"},{"line_number":410,"context_line":"inventory updates to the Placement API. RMD allocations will occur after the"},{"line_number":411,"context_line":"creation of the workload (e.g. VM) and the time to allocate resources is an"},{"line_number":412,"context_line":"extension of the time taken by a hypervisor like Libvirt to provision"},{"line_number":413,"context_line":"resources for the VM."},{"line_number":414,"context_line":""},{"line_number":415,"context_line":"Tthe nova virt-driver will perform an extra step to communicate with RMD to"},{"line_number":416,"context_line":"request for resource inventory information and deliver the RMD policy for"}],"source_content_type":"text/x-rst","patch_set":5,"id":"ffb9cba7_af8a3618","line":413,"range":{"start_line":410,"start_character":40,"end_line":413,"end_character":21},"updated":"2019-04-24 13:28:29.000000000","message":"again i think you mean that RMD will only assign the cache alocation to the vm after it create and this is not refering to placmeent allocation which will happen before the vm is spawned.\n\ncan you use assignments instead.","commit_id":"28b06cb71b84a2f2cee5b8c041c3e2781321c423"},{"author":{"_account_id":29334,"name":"Dakshina Ilangovan","email":"dakshina.ilangovan@intel.com","username":"dakshinai"},"change_message_id":"65af0255ba1ab001e71899a0351fa06a0cca46f1","unresolved":false,"context_lines":[{"line_number":407,"context_line":""},{"line_number":408,"context_line":"We expect the polling agent in Nova Compute to be fine-tuned using Nova"},{"line_number":409,"context_line":"configuration to avoid frequent polling and reduce any subsequent resource"},{"line_number":410,"context_line":"inventory updates to the Placement API. RMD allocations will occur after the"},{"line_number":411,"context_line":"creation of the workload (e.g. VM) and the time to allocate resources is an"},{"line_number":412,"context_line":"extension of the time taken by a hypervisor like Libvirt to provision"},{"line_number":413,"context_line":"resources for the VM."},{"line_number":414,"context_line":""},{"line_number":415,"context_line":"Tthe nova virt-driver will perform an extra step to communicate with RMD to"},{"line_number":416,"context_line":"request for resource inventory information and deliver the RMD policy for"}],"source_content_type":"text/x-rst","patch_set":5,"id":"ffb9cba7_45f0e0fd","line":413,"range":{"start_line":410,"start_character":40,"end_line":413,"end_character":21},"in_reply_to":"ffb9cba7_af8a3618","updated":"2019-04-30 07:03:38.000000000","message":"Sure Sean, will fix this.","commit_id":"28b06cb71b84a2f2cee5b8c041c3e2781321c423"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"caeec2d1a3a83edd4a394bc066e6092c61d74c07","unresolved":false,"context_lines":[{"line_number":412,"context_line":"extension of the time taken by a hypervisor like Libvirt to provision"},{"line_number":413,"context_line":"resources for the VM."},{"line_number":414,"context_line":""},{"line_number":415,"context_line":"Tthe nova virt-driver will perform an extra step to communicate with RMD to"},{"line_number":416,"context_line":"request for resource inventory information and deliver the RMD policy for"},{"line_number":417,"context_line":"resource allocation to the VM. This will introduce a locking period to wait"},{"line_number":418,"context_line":"for the verification of the RMD’s policy and then for the reconfiguration of"}],"source_content_type":"text/x-rst","patch_set":5,"id":"ffb9cba7_6f843e0d","line":415,"range":{"start_line":415,"start_character":1,"end_line":415,"end_character":2},"updated":"2019-04-24 13:28:29.000000000","message":"delete","commit_id":"28b06cb71b84a2f2cee5b8c041c3e2781321c423"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"caeec2d1a3a83edd4a394bc066e6092c61d74c07","unresolved":false,"context_lines":[{"line_number":412,"context_line":"extension of the time taken by a hypervisor like Libvirt to provision"},{"line_number":413,"context_line":"resources for the VM."},{"line_number":414,"context_line":""},{"line_number":415,"context_line":"Tthe nova virt-driver will perform an extra step to communicate with RMD to"},{"line_number":416,"context_line":"request for resource inventory information and deliver the RMD policy for"},{"line_number":417,"context_line":"resource allocation to the VM. This will introduce a locking period to wait"},{"line_number":418,"context_line":"for the verification of the RMD’s policy and then for the reconfiguration of"}],"source_content_type":"text/x-rst","patch_set":5,"id":"ffb9cba7_4f793a2d","line":415,"range":{"start_line":415,"start_character":9,"end_line":415,"end_character":21},"updated":"2019-04-24 13:28:29.000000000","message":"libvirt virt dirver\n\nthere are many nova virt direvers and this spec only addresses integration with libvirt not the others.","commit_id":"28b06cb71b84a2f2cee5b8c041c3e2781321c423"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"caeec2d1a3a83edd4a394bc066e6092c61d74c07","unresolved":false,"context_lines":[{"line_number":432,"context_line":"Developer impact"},{"line_number":433,"context_line":"----------------"},{"line_number":434,"context_line":""},{"line_number":435,"context_line":"For every new resource that RMD supports, OpenStack developers can leverage it"},{"line_number":436,"context_line":"by making appropriate changes to Nova Compute to expose and consume those"},{"line_number":437,"context_line":"resources."},{"line_number":438,"context_line":""},{"line_number":439,"context_line":"Upgrade impact"},{"line_number":440,"context_line":"--------------"}],"source_content_type":"text/x-rst","patch_set":5,"id":"ffb9cba7_4f9e1a55","line":437,"range":{"start_line":435,"start_character":0,"end_line":437,"end_character":10},"updated":"2019-04-24 13:28:29.000000000","message":"this is an example of why the statement that RMD removes the need for plathform specific resouces on line 179\nis incorrect","commit_id":"28b06cb71b84a2f2cee5b8c041c3e2781321c423"},{"author":{"_account_id":29334,"name":"Dakshina Ilangovan","email":"dakshina.ilangovan@intel.com","username":"dakshinai"},"change_message_id":"65af0255ba1ab001e71899a0351fa06a0cca46f1","unresolved":false,"context_lines":[{"line_number":432,"context_line":"Developer impact"},{"line_number":433,"context_line":"----------------"},{"line_number":434,"context_line":""},{"line_number":435,"context_line":"For every new resource that RMD supports, OpenStack developers can leverage it"},{"line_number":436,"context_line":"by making appropriate changes to Nova Compute to expose and consume those"},{"line_number":437,"context_line":"resources."},{"line_number":438,"context_line":""},{"line_number":439,"context_line":"Upgrade impact"},{"line_number":440,"context_line":"--------------"}],"source_content_type":"text/x-rst","patch_set":5,"id":"ffb9cba7_05de686c","line":437,"range":{"start_line":435,"start_character":0,"end_line":437,"end_character":10},"in_reply_to":"ffb9cba7_4f9e1a55","updated":"2019-04-30 07:03:38.000000000","message":"By changes we mean, changes to report the resource as traits or resource classes by resource type. The resource type will not change for vendor or platform changes.","commit_id":"28b06cb71b84a2f2cee5b8c041c3e2781321c423"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"caeec2d1a3a83edd4a394bc066e6092c61d74c07","unresolved":false,"context_lines":[{"line_number":469,"context_line":"Testing"},{"line_number":470,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":471,"context_line":""},{"line_number":472,"context_line":"Functional testing has to be done for the following flows:"},{"line_number":473,"context_line":""},{"line_number":474,"context_line":"1. virt-driver queries RMD for resource inventory"},{"line_number":475,"context_line":""},{"line_number":476,"context_line":"   * When multiple resource inventories from multiple requests are returned"},{"line_number":477,"context_line":"     from RMD."},{"line_number":478,"context_line":"   * When a single resource inventory is returned from RMD."},{"line_number":479,"context_line":"   * When no resource inventory is returned from RMD."},{"line_number":480,"context_line":"   * When HTTP error is returned from RMD."},{"line_number":481,"context_line":""},{"line_number":482,"context_line":"2. virt-driver to update the RMD resource inventory Placement API"},{"line_number":483,"context_line":""},{"line_number":484,"context_line":"   * Successful inventory update to Placement API for n resources."},{"line_number":485,"context_line":"   * Successful inventory update to Placement API for 1 resource."},{"line_number":486,"context_line":"   * Error on inventory update to Placement API for a single resource among n"},{"line_number":487,"context_line":"     resources."},{"line_number":488,"context_line":"   * Error on inventory update to Placement API for a single resource."},{"line_number":489,"context_line":""},{"line_number":490,"context_line":"3. virt-driver to send resource allocation requests to RMD"},{"line_number":491,"context_line":""},{"line_number":492,"context_line":"   * Successful allocation of resources for a policy to RMD."},{"line_number":493,"context_line":"   * Error on allocation request for a policy from RMD."},{"line_number":494,"context_line":""},{"line_number":495,"context_line":"Documentation Impact"},{"line_number":496,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":5,"id":"ffb9cba7_efc58e3d","line":493,"range":{"start_line":472,"start_character":0,"end_line":493,"end_character":55},"updated":"2019-04-24 13:28:29.000000000","message":"will this be enabled with a test fixture that fakes out RMDs rest api or do you intend to intoduce a new tox env that reuqires RMD execute the tests.\n\ni assume a test fixture will be used the same way we do for placement  but please make this clear.","commit_id":"28b06cb71b84a2f2cee5b8c041c3e2781321c423"}]}