diff --git a/sig-support/meetings/24-2023-12-20.md b/sig-support/meetings/24-2023-12-20.md new file mode 100644 index 00000000..7d1b3643 --- /dev/null +++ b/sig-support/meetings/24-2023-12-20.md @@ -0,0 +1,292 @@ +# Akash Network - Support Special Interest Group (SIG) - Meeting #24 + +## Agenda + +- Participants will look at issues in [Akash Support Repo](https://github.com/akash-network/support/issues). +- Scott Carruthers will lead triaging issues, and discussion of implementation for issue fixes. +- Scott will discuss Support from the community and ask for any community feedback. +- Participants can have an open discussion of any other issues or support related items on Akash Network. + +## Meeting Details + +- Date: Wednesday, December 20, 2023 +- Time: 07:00 AM PT (Pacific Time) +- [Recording](https://kesic5sqastbfceerxf7d5zcqcumbjpipfw45q6oxxmcguazpotq.arweave.net/USSBdlAEphKIhI3L8fcigKjApeh5bc7Dzr3YI1AZe6c) +- [Transcript](#transcript) + +## Participants +- Andrey Arapov +- Max +- Rodrigo Rochin +- Scott Carruthers +- Tyler Wright + +# Meeting Notes +#### Tyler Wright +- Mention of bounties available for those contributing to solving issues in the support repo. +- Discussed desire for increased community contributions on open support issues in 2024. +- Biweekly meetings will continue, providing an opportunity for participants to ask questions to the core team and refer to Scott's engineering notes for deeper understanding. +- Participants urged to review Scott's engineering notes for a better understanding of Akash Network's technical aspects. +- Recognition of Scott Carruthers' work on testing, especially for those building on top or setting up their environments. +- Anticipation of increased testing opportunities in early 2024 for local environment setup, simulating core team work. +- Confirmation of three planned network upgrades in the first quarter of 2024. +- Mention of action items related to labeling in the support repo. +- Reminder of further investigation needed for issue #156, to be discussed in the January 10th meeting. + +### Issues Triage Session +- Andrey Arapov explains the [issue #166](https://github.com/akash-network/support/issues/166) related to Wild Card use and GPU models in bid prices. +- Issue arises from bid price script selecting the highest value without knowing GPU availability on the cluster. +- Andrey proposes a solution involving Akash provider recognizing the scheduler's deployment allocation for unspecified GPUs. +- Scott Carruthers assigns the issue to Arthur for further review and possibly assigns subtasks to Andrey. +- Discusses the second unassigned issue created by Arthur related to migrating client go code away from the node repository, planning for work in 2024. +- Mentions a third simple issue about HTTP options in go code with a proposal to change from uint32 to uint64. +- Decision to assign the second issue (client go code migration) to Arthur for clarity +#### GPU Reporting, Issue #137 +- Max raises a concern about a GPU reporting issue causing a crash when indexing providers and their resources. +- Scott acknowledges the existence of an issue related to wrong GPU reporting with an extremely high value. +- Andrey Arapov provides details on how the issue can be reproduced by restarting the device plugin and suggests a potential solution involving a simple check in the Akash provider to handle wrong or incomplete information. +- Confirmation from Andrey Arapov that the count returns to normal after a while. +- Andrey Arapov suggests testing scenarios with multiple worker nodes, some with GPUs and some without, to observe how the device plugin handles this configuration. +- Scott Carruthers recalls past testing scenarios and agrees to revisit and conduct further testing to understand how the device plugin operates in different node configurations. + +### Support +- Tyler Wright emphasizes the importance of the Discord channel for technical conversations and support within the community. +- Acknowledges the efforts of insiders and vanguards who make themselves available 24/7 for user queries. +- Tyler Wright gives a shout-out to Max for his responsiveness in addressing Cloudmos-related issues and encourages users to post such issues in the Cloudmos repo. + +#### Other matters discuused +##### Kernel Version Issue in Providers +- Andrey Arapov shares information about a potential issue related to providers losing leases, pods terminating, and worker nodes falling off. +- Mentions that the issue may be caused by running a newer kernel (above 5.15 ) on the host, leading to CPU performance problems. +- Recommends downgrading the kernel to 5.15 or adjusting liveness probe timings as workarounds. +- Highlights the importance of not over-provisioning resources and suggests managing resources dynamically. +- Tyler Wright expresses gratitude for the information and suggests posting a written format or announcement in the providers' announcements channel. +### Action Items +- Further investigation of issue #156 to be discussed in the January 10th meeting. +- Group to do more investigation to understand the persistence of the high count in GPU reporting, Issue #137. +- Andrey Arapov to create a written document or announcement regarding the kernel version issue and share it in the providers announcements channel. + +# **Transcript** + +_This editable transcript was computer generated and might contain errors. People can also change the text after it was created._ + +Tyler Wright: All right, welcome everybody to six support biweekly meeting number 24. The special interest group for support meets every two weeks to go over triashing any issues in the support repo where all issues related to core product live on the Akash Network. In addition to triashing any open issues or any issues waiting triage on the group goes over and goes over any open issues that need for their discussion further insights Etc. And looks for folks that want to be assigned those specific issues. Again, anything related to the cash core product? Goes inside the support repo inside the Akash organization. + +Tyler Wright: One thing I also want to call out is we move into 2024 is the desire for this is something we've talked about more Community contributions on open support issues over the last year Scott Carruthers from the core engineering team has been putting together documentation that is more engineering focused. So again, I know that this is the last six support biweekly of 2023. We'll take the next couple of weeks off. + +Tyler Wright: Certainly, there are bounties available for those that want to contribute to solving issues and support repo again. There will be these biweekly meetings. We're folks that want to take on solving issues can ask any specific questions to members of the core team as well as again have a chance to take a look at these costs engineering notes that Scott has put together as reference points as reading materials to dive deeper and into the technicals of the Akash Network and how it works. + +Tyler Wright: So that's something that will be more of an action item for 2024 for folks that are joining these six support meetings. The goal is to have more participation from technical go developers in the community that can help with again support issues and can participate more in the development of the core covids. + +Tyler Wright: part of me cool out of the last six support biweekly meeting number 23 the group went over Maine at nine Archer led the group and talked about some of the issues waiting Triads as well as described the network mainnet upgrade schedule. For 2024 again to reiterate there's a three-planned network upgrades happening in the first quarter of 2024. But the goal is whether there's more or less to have smaller Network upgrades so that again testing can be much more efficient and faster. one thing I want to call out on the + +Tyler Wright: project board is the work that Scott Carruthers who's on this call has been doing around and testing again for those that are looking to build on top or looking to set up their own environment or against solve some of these support issues in the support repo. There's going to be a lot more and testing in early 2024 for folks to set up on their local environments. It's kind of simulates a lot of the work that the core team is doing so look out for that as well. I know that there are some action items that have last ly. With the holidays here. And again, we'll be skipping. I believe the next couple of six support time slots. So the next six support biweekly meeting is planned for January 10th. + +Tyler Wright: part of me with that being said there's a couple of action items from last session around labeling and adding some more labels to support repo as well as further investigation of 156 that we'll talk about We'll talk about January 10th. + +Tyler Wright: I know archers out of office this week, but again another key member of the core team is here to lead us through triaging of issues all hand over to Scott Carruthers to lead us through the triashing of any issues of waiting for us. + +00:05:00 + +Scott Carruthers: All right. Thanks Tyler. Let me share my screen. + +Scott Carruthers: All right, so I was doing a review of the open issues and unassigned prior to the call. So I think we might have three to go through so probably will be pretty brief. So the first one and as a issues that you opened a couple days ago regarding Wild Card use and a GPU model and in the bit price that we're responding with do you want to In your own words go through this issue. + +Andrey Arapov: Yeah, so basically when someone requests GPU and a bit price script basically takes the highest view from the list but not the one that is available on the cluster because bit price script cannot know this information prior and bit price script and provider doesn't receive the information about which the appear available it is going to Allocate the deployment for so because of this some providers they will be basically bidding for lower and with the highest end GPU pricing. It's because of the workaround specified there you can click this automatically sets underlying under underscore. + +Andrey Arapov: Yeah, this one is the issues. I was creating prior and this is a consistency. This one is So on the partial workaround had to be done well because of similar issues but vice versa so basically to prevent someone when someone requests and deployment without the particular GPU set + +Andrey Arapov: so the trolls back to some value which is the value of the highest usual about it because for instance provider can have several worker in Authority in a single worker note and can have low end and high-end gpus and providers will want to make sure that they beat on the deployment request that record do not specify the individual GPU if they don't know what they want to see it, but the problem with that, the recently implemented on the cloud most the availability of the GPU in bid itself so we can see which the few but it happens after it's like the + +Andrey Arapov: positive events action. So the core issues that it's impossible to know which the cash provider will be allocating for particular requests, which doesn't specify the GPU explicitly basically, so it's something that should be done on the cash provider level itself or on the bit price script Which is less likely because then it would take probably longer time to respond and would probably require more permission to query the cluster which is Not probably the best decision. I would say Arthur would be the best person to decide here. But my take is that + +Andrey Arapov: provider should be able to recognize to which this scheduler is going to deployment to when the GPU is not explicitly specified in the islipment request. So even that's clear. + +Scott Carruthers: Yeah, no, it's clear. And yeah, it was clear to me. I'm even just reading through the description. But yeah, I wanted to let you go through the issues as well. So yeah, that's even more thorough description see a couple it makes sense that we need to have that information before the bid. So I'm just going to assign the starter and possibly. He'll want me to take a look at it as well. But I'll just sign it to Archer current + +Andrey Arapov: Yeah, yeah because the whole thing it happens the way that it creates. a bit + +Andrey Arapov: and reserves the space but doesn't know which exact GPU is there maybe it knows post action, but I think it's possible To know beforehand at the same time because when provider issues the beat and when broadcasting the network there is information about which GPU so it's possible. I think it's already kind of does it but somehow it needs to pass this information to the bit price script and what it currently passes is everything but this I think so. I think that maybe it's not even difficult to implement. It just needs to specify this information for the big price screen, basically. + +00:10:00 + +Scott Carruthers: And if he wants me to take a look at it, we can make that decision took the away in treehouse are off and set it as sub to Andy. Are you okay with sub 2 designation? + +Andrey Arapov: Yeah, definitely. It doesn't really break things much just offers the lower end GPS in some particular cases for the price of the highest and GPU, but it's not It's suboptimal, but it's definitely not so one but something like have to be fine. + +Scott Carruthers: Yeah, okay. Yeah, I definitely definitely understand the issue providers with multiple GPU models, maybe bidding very high prices when essentially a lower end GPU this extended so yeah, definitely. understand the severity + +Scott Carruthers: Okay, so we now have assignment for that issues so that leaves us two more another one that was unassigned is one that er. created to migrate the client go code away from the notary pository and into the Akash API Repository. Not entirely sure why he didn't assign this when? He created this issues. So this is just obviously some additional item and they roadmap in the backlog that we will be working on the engineering team. I'm probably in the very part of 2024. So not exactly sure why Archer didn't Assign this I think I might show you going to leave. This is on a sign just so that when we go through these issues. I'm gonna assign us Archer just because he created the issue. + +Tyler Wright: Yeah, I think we should have signed to Archer might just be oversight and I'm sure he'll do it sometime very soon. + +Scott Carruthers: Yeah. + +Scott Carruthers: and then again, I was kind of reading through some of these unassigned before our call and then next one is a pretty simple issue at least from their request perspective. I haven't reviewed. + +Scott Carruthers: possible impact or so essentially the issues here is in the go code for HTTP options. They + +Scott Carruthers: Variables are set to un32 so that prohibits. Change the HTTP options Max body size when this user is trying to go above the maximum lineup limit for a un32 So He suggests. I know we changed it from un32 and the go code to un64 so obviously pretty simple change within the go code Might need some further investigation to make sure that there's no issues with that, but theoretically should be for other reason. So something right Repository? I'm just going to assign both Archer and myself to this one. + +Scott Carruthers: type + +Scott Carruthers: Any thoughts are? comments on this issue + +Andrey Arapov: Yep. + +Scott Carruthers: that completes any of the issues that were created in the last two weeks since we last meant that had not been assigned. So obviously we can go into additional conversation about ones that I just went through are issues that have been assigned but we just want to devote some additional conversation live here together. I guess I'll kind of open it up to the floor any. For their conversation needed on the ones that we just covered or any of the issues that were assigned prior that we haven't covered this morning. + +00:15:00 + +Scott Carruthers: I heard a hand raise. I'm just looking at the issues So whoever Rings their hand, please go ahead. + +Tyler Wright: Go ahead Max. + +Max: Hey Scott, I'll show if you talked about it, but I wanted to raise the issues about the GPU. Account that is a huge number and the one that we talked about on this card. + +Scott Carruthers: Yeah. There is a current issues for that. + +Max: Is there an issues good for this? + +Scott Carruthers: What would that I'm trying to think of the title for it definitely does have a issue. + +Scott Carruthers: but yeah this short answer to your question is they're definitely is a issue that's assigned to the issue that we've always had with this and this continued after + +Scott Carruthers: the latest experience last Friday is I not been able to reproduce the issue. Open but we don't know what causes the issues that when this initially came up. They provider that encountered it had recently. moments before this issue happened. They removed a GPU card from a running server and there was some faucet possibly that could have caused the issues. I don't have a physical server with a GPU card that I can test that but yeah, there are definitely as a issues for it. And now we're trying to determine how we can reproduce the issue. + +Max: Okay, cool. I think I see it at the bottom there wrong GP reporting extremely high value. + +Max: The one just below that the highlighted one. 137 + +Scott Carruthers: 137 okay. + +Tyler Wright: Okay. I said yeah. + +Max: because yeah Andrew, I don't know if you talk to him the privately that he was having that issues and it made it so that when we + +Max: when we index the providers and their resources it crashes because the numbers too big and makes it so that his provider is a displayed as offline on cloud moves and + +Max: And obviously Butters in because you can't get bids from others. + +Andrey Arapov: Yeah, I was able to reproduce it. + +Scott Carruthers: Yes. + +Andrey Arapov: I recommended it before. + +Scott Carruthers: I'm sorry Did you say you were able to reproduce it? And did you make a comment in the session? + +Andrey Arapov: Yeah, yeah, that's correct. There is a comment in the bottom of it. + +Scott Carruthers: Are you talking about this or maybe could you just review? So how are you able to reproduce the issue? + +Andrey Arapov: So basically if you bump the common delete pods all for NVIDIA device plugin namespace, it makes them restart and while they're restarting the cash provider, it fits based on the information from this plugin that gets this having for about how many available dupus it has and probably when doesn't get the proper information, I believe it falls back or maybe in video device plug-in gives this wrong number or not even digit at all. And so a cash provider it's the wrong information. That's my presentation. And when you query status endpoint, it's for three slash status from the cash provider you get this very high years number as you can see in the GPU column. So, I think it's a matter of some simple check in the car provider just if you + +Andrey Arapov: Get some wrong number don't display it or convert it into something maybe wait in the device plugin to initialize for fully something like this. So it's kind of issues quite clear for me. + +Scott Carruthers: Yeah, so all that makes sense. so when you were doing your testing there and Max I believe in our conversation on Friday. We talked about this as well. So there we have definitely seeing an issue when they device plug-in is restarting that it represents that very high count, so I think it would do that 100% of the time but in Normal Practice or the normal experience it would then normalize after the device plugin at normal operation. + +Andrey Arapov: Yeah. + +Scott Carruthers: So in your testing did that number eventually return to normal or did it stick at the + +Andrey Arapov: Yeah. Yeah, it returns always as you can see on the screen the very same screenshot the second time around the provider info which queries the states and point. It returns back to the normal value. So it doesn't get stuck in this High number and large numbers. + +00:20:00 + +Scott Carruthers: yeah, yeah and so possibly they're related one known until we dig into the issue more but The issues with that we've experienced now twice one with inter Milos provider and then the original DC Norse experience with this is the number never returns to normal so last week when enter mellow was experiencing this issues that very high count persisted. So we do know that we can reproduce the issues by restarting the device plug-in, but they think that we're not able to reproduce as the circumstance where for some reason that very high count persist. + +Scott Carruthers: Does that make sense? + +Andrey Arapov: Yeah, so there is another issues probably wanted to proceed when this High count number persistent. It's something else probably yeah. + +Scott Carruthers: Right. Yeah. + +Scott Carruthers: Okay, so yeah, Max are definitely as an issue. and yeah, that's basically we're trying to come up. So we do know that we can reproduce the temporary high count and that probably needs to be addressed as well or unable to reproduce the issue where for some reason that I count persists. Andrew Miller was actually able to resolve the issue and he believed that it was attributable to mislabeling. of his nodes, but it's kind of strange because the inventory operator if a node is mislabeled It seems very strange that that would manifest itself into a very high count. I don't know why miscellane the node would have that issue and IFA tested this when DC Norse originally had the issues as well and Miss labeling, no definitely did not reproduce So that obviously would be a very + +Scott Carruthers: severe issues something that's very easily reproducible by line. a node, but that's certainly didn't reproduce the issues for me. But Andrew Melo claimed that when he adjusted the labels at somehow rectified the issues. Yeah. I heard another hand go up. + +Andrey Arapov: Yeah, I a guess there are two ways to two types of labels one label where you just specify which capability which kind of GPU or you're working on has our range of the peers and the other ways relatively new one type of label where you can say Hey, listen media device plugin run only on this particular working nose and the GPU it's a selector know it's a recent one. And with this you can have actually the device plug-in running, but I mean, it won't actually run on the specific notes maybe relation or correlation of related to this kind of label was causing another trouble,… + +Scott Carruthers: Yeah. + +Andrey Arapov: but It All Leads to basically I think the way how because in the end of the day the provider it's the information from the plugin I believe or from something in between from the conversation. And then it presents this High count value. I think it's still on the provider side where it needs to filter this information and either present zero when it cannot read it or when there is super high count which is not thermal and wait until slices and then once it's initialized then display normal account, which is about want to 10 20 something like that my opinion + +Scott Carruthers: Yeah, I definitely makes sense of the account should be represented as 0 as the plugin is initialized so I'm definitely familiar with the label that you're referring to so that we can control the device plugin only. Spawning on worker knows that have GPU resources and we can control that. I thought that Andrew Melo was insinuating that it wasn't that labeling, but it was actually they per worker no labeling of specific GPU models not the device plugin label. But I have to go back and review the conversation. + +Andrey Arapov: Yeah, I think it's good point and another Point like we think have never tested having more than one working out and having a video device plugin running on those notes that do not have GPU and so that to have the mixed in the configuration where you have health or part of the working also with GPU and media plugin running and half of the notes or one of three doesn't matter these without GPU. But with the India the best plugin, I'm not sure how it handles. This thing could have maybe someone tried it. I don't know. It's another good point to test probably + +00:25:00 + +Scott Carruthers: Yeah, we definitely could do some additional testing on that. I think I had that scenario that you're talking about before we introduce the label for the device plugin. So in other words and just simple provider testing where you have + +Scott Carruthers: A master and a worker node, and the master is also a worker node. So before we introduced that label, the device plugin would have run both on the master worker node, and the worker note with the GPU and in that scenario. I don't believe we encounter issues. But yeah, definitely worth testing again possibly that could be related. + +Andrey Arapov: That don't remember the phone with the device plugin actually runs on the control plane node alone because control play No, the usually have the special condition and tolerance I think is for system plugins only and I don't remember if in this kind of configuration that would be actually running on the control plane. But yeah something that we could test along with. + +Scott Carruthers: Yeah. Yeah, I'm in a control plane node. That was also a worker node. So in + +Andrey Arapov: Yeah. Yeah in this case you need to do ok. + +Scott Carruthers: yeah, so we would just say the Envy DP pod spawn on the master node again, which is also a worker node. I'm and it wouldn't start properly because there's no GPU resources. So yeah under that scenario I tested but yeah, we can definitely go back and Tessa further. + +Scott Carruthers: Okay, I'm going to stop sharing my screen. Do I hear another? And up as well. No, I'm good after the screen once over good. + +Tyler Wright: No. + +Scott Carruthers: Okay, so yeah, a loss anyone else has so max. Thanks for bringing up that issue. Definitely an issue is created and again we're trying to reproduce our Pinpoint where that issue is. So if there's no other questions or comments about any other issues, I'm Tyler. I think we can move to the next stage. + +Tyler Wright: Scott. Thank you Andre and Max for participation. Cool, next up on the agenda is a check-in on support channels again Discord telegram, but mostly Discord for technical conversation is a channel that's used by the community. There are a number of insiders and vanguards that make themselves available 24 hours a day by seven. So there's always somebody ideally online to answer any questions from users that are trying to either deploy or become a provider or just have general questions across Network so much appreciate the inside and vanguards. But I want to see. + +Tyler Wright: If rodri Max or maybe Andre who are again quite active in Discord have seen anything of note. And before I get my ticket off and hand it over to that group to see if there's anything of note. I also want to shout out Max. I know that if there's any issues related to Cloud most again, if you go inside the cloud most repo in the cock organization and the issues area, you can feel free to drop any Cod most specific issues in that repo and I know that Max and the rest of the cloud most team are very responsive and again also very responsive inside the ecosystem So again any Cloud most related feel free to drop in the cloud most repo go ahead Andre. + +Andrey Arapov: Yeah, I just wanted to highlight one thing. It's probably not related much for six support, but maybe partially it's about providers. There's less provider owner contacted me yesterday. They've been digging some issues. Basically there is a possibility of issues where it could be that provider is losing some pleases or some general issues or provide report running or not starting or some posts are terminating too long or some of the worker notes like falling off because they'll say literally Calico mode deployment which creates the internal Network on the conversation can be calling off and people don't know what's going on. mean, there is a responsibility like only one instance of this issues. But if you ever see this make sure you'd like to check what Colonel are they running because some people are deciding to go with the greatest kernel and they have a great let's say the + +00:30:00 + +Andrey Arapov: Kernel 515 in Ubuntu 22 up to 6.2. And in this case, what happens is that when your more core CPU gets 100% visualized. + +Andrey Arapov: One of the VMS on that host and in the VM itself there it doesn't matter. What kind of Kernel it can be 515 but on the first it's important if it's 6.2 newer kernel and then what happens is essentially it doesn't really share the CPU Cycles properly and when you're running Calico node, which creates this internal Network and the kubernetes it normally requests 150 Milli cores, like 4.15 and up to 300 mini course and what happens is that this is not enough for it to pass the liveness broke threshold. So there are two ways to fix it. Basically the best one which we did initially is let him downgrade the kernel back to 5:15 on the first not only Williams doesn't matter again. + +Andrey Arapov: It's halted and the other workaround there is lightness probe. You can double it up. So it normally Waits 70 seconds for the Calico note both the up and running but you can double it up up to 140 seconds and then it will be running but still it's not a perfect solution because then other deployments which requesting normally a thousand minute course CPU we should or one week or CPU. They don't want the Performing as good as they supposed to because of this kind of weird conditional kernel there's a thread open also in the proxmox Forum, but this is just for awareness if you ever see any + +Andrey Arapov: Weather or anyone running on the new of the provider some of their deployments complaining about the CPU performance, number one reason is usually over provisioning. It's when people who use VMware or proxmox virtual machine, they say that hey VMS have more CPUs than they should so whenever it's getting really busy then there is because of the CPU cycle of sharing and interrupts They will receive less cycles for the CPU and it will perform much less while they're seeing the same number. So it's number one or provisioning provided should not over provision anything ideally another thing kind of manager dynamically, which is another case. Second thing is make sure that they don't run to new kernel or if they are sure that it's part of this issue. The best is to use the default official Colonel, which is 5:15 or even 20 to 4. So just for information something I wanted to share but + +Andrey Arapov: Just remember that? + +Andrey Arapov: You… + +Tyler Wright: much appreciated + +Andrey Arapov: worries guys. If you forget what I said, you can always ping me. I will try to put it in a rapid into a written format. I think I already wrote it in my notes, but just in case. + +Tyler Wright: Yeah, maybe that's something I know that Andrey do a great job of. What you see things in Breaking time you throw announcements in the providers announcements Channel and pin those items so that people can kind of track on their own but that might be a good place to put it and then again we can make sure that the Insiders are up to date and anybody else that's like providing support. + +Andrey Arapov: Yeah, good point. I'll put some reminder for myself because I'm now signing off and I'll try to remember to do it tomorrow. + +Tyler Wright: rodri max. Is there anything that you all have seen in terms of support that you think is worth sharing or discussing that? + +Rodrigo Rochin: I'm on this part all good. Nothing out of the ordinary just people asking about deployments and Issues with providers but like you said everyone's been answering there. + +Tyler Wright: excellent + +Tyler Wright: excellent All Is there anything else worth anybody wants to discuss right now? + +Tyler Wright: Cool was actually items will continue to track issue number 137 and I believe number issues number one 60 and again, our next support biweekly will be in the new year January 10th 2024. If anybody has any questions in between this meeting in our next meeting, feel free to drop those in the six support Discord Channel or feel free to use and GitHub support repo to leave comments and specific issues or create new issues. And then again we can talk about those once we come back in the new year. + +00:35:00 + +Tyler Wright: thank you Scott for leading us through triaging of open issues and issues awaiting three hours or issues that haven't been assigned notes will be following up shortly much appreciate everybody's time and efforts including all the Insiders at vanguards that are again active in. obviously shoutout Andrey for all the work that he does in Discord helping providers, but helping everybody that Again support babe will be a little bit lighter over the next couple of weeks. I'm sure inside of vanguards and members of the core team will be taking some time with families. But again folks like myself and others will be available just to monitor Discord and be around and again if anything needs escalation, Again, we can make sure it gets escalated to the right parties. + +Tyler Wright: Hope everyone has a great holiday season very happy New Year. And again if anybody wants to look into the Project board to see what's going on the product and Engineering side as well as the marketing Community boards. There will be some bounties that'll be available to the community over the break for folks that we're looking to build onto the top of the cost Network solve some issues and support repo and Beyond so look after those bounties that live in the community and marketing project board in the Akash organization. Again, hope everyone has a great holiday season the happy New Year, and we will talk January 10th 2024. appreciate you + +Andrey Arapov: Thank everyone. Have a good one. + +Tyler Wright: All right. + +Scott Carruthers: Thanks, everyone. + +Rodrigo Rochin: the business + +Meeting ended after 00:37:00 👋 +