Hi, this is Wayne again with a topic “What $1,000,000 of water cooling looks like – Building Shadow’s Power Upgrade”.
Hey there well been a little while I came all the way to France to check out ovh Cloud, not actually for float plane Reasons I’m here, because Shadow is sponsoring this video, so that we can check out their power. Upgrade servers. They’Re super powerful, they’re super cool. They have eight gpus, all this fun stuff, we’re gon na build one. We’Re gon na see how they build 80 000 computers a year and then we’re gon na go to their Data Center and see how it’s all implemented, powered cooled everything it’s going to be awesome. There’S a lot to see come along with me. You may have heard of Shadow before, but just in case you haven’t they’re a super easy to access high performance, personal computer in the cloud.
What does that mean for you? It means that you can extend the lifespan of your various devices if you have some junky old, laptop or desktop, or whatever you can move the heavy compute that you might need to do in your day-to-day to the cloud and Shadow can handle it. Shadow can also be used in a business environment such as deploying multiple Shadow, PCS for those working remote or configuring Shadow PCS, with up to an RTX 6000 GPU for high workloads. It’S not just performance tasks either in regards to like standard old school research stuff, you can play games too. You can do the fun things on your dad’s old, Smart TV or your phone or your VR device, or your Raspberry Pi or whatever the heck you want. You can run 1080p games at 240 FPS and 4K games at 60fps. Let’S go see how it’s done.
Every server for ovh’s European market and Asian Pacific Market is made here. Shadow or not, and every single component for ovh worldwide is made here as well, including ovh’s version of an all-in-one water cooler, which can apparently cool an entire row of racks of servers. I don’t know the exact capacity, but it’s safe to say it’s solar.
This facility processes, through several hundred million dollars worth of small components every year, including this huge pile of trays and trays and trays of ram. They also have gpus CPUs and lots of copper piping, because ovh uses water cooling in all of their facilities from here. It goes into the rest of the facility where they build everything test everything do r d, etc, etc.
Once it’s done, it comes back here, actually just across the way where we can see all these cool signs where they line everything up. It goes onto trucks and, in this case, gets sent to the main data center that we use at flowplane. Now we get to the fun stuff. This is the area where they’re, actually building the power upgrade servers and the first part of that is to strip all of the standard stock, cooling components off of these Quadro gpus from Nvidia in the future they’re going to support Radeon as well. Right now we’re just dealing with Nvidia. As we move down the line, we can see that they start putting them into sandwiches, where they’re putting these custom made heat sinks and heat pipes and everything onto the gpus and then sandwiching them together with a dual Riser.
It’S actually like super cool. I uh, I don’t know if they would have. Let me do this if I was Linus because he’d drop it, but you know I won’t do that they’re able to keep these at a reasonable temperature. I washed it under load 100 load. Both of these sitting at around 45 Celsius, which was pretty intense speaking of 100 load, this station is where they test them. So once they’ve made two of these GPU sandwiches, I’m going to call them, they rack them up here. They have a power supply and everything they need to test. There’S a motherboard sitting on the lower part of this server they test here.
This is where they control I’m not going to press any buttons, because this is where they control being able to turn on the water and make sure everything is cooling properly up here. They they load over the network, a Linux system, they put a load on it. They test it make sure everything’s good if it’s good, we move on further down the line, so ovh and Shadow are always chasing efficiencies. It makes sense at the scale, because if you could save a small amount of material on something it’s going to become a lot over all of the systems that you’re building and they design their own water blocks for all of their systems, the ones we saw earlier. The big chunky ones was an earlier version. They have a very new one that looks a little bit like this you’d. Be surprised that this setup can cool a Quadro, but it totally can they’ve done a bunch of research.
I’M going to try to install this right. Now, and not completely fail, we’ll see how that goes. You have to kind of slide it in like that. You’Ve engaged the cooling at this point and then these little arms here have to touch some chipsets on the board. There we go it’s a little slippery, but once they mount it, it’s now ready to be kind of sandwiched together. The first thing that you need to do is screw these standoffs onto the very long bolts and then once that’s done, which they have kind of cool screwdrivers, for they don’t have one right here. So I can’t show you lttstore.com. You can get a cool screwdriver too, but once that’s done once the standoffs are planted just put another GPU on top and do the same thing once you get the heatsink in place, you’re ready to prep it for adding another GPU. First off.
You need to add this support, so it gets tucked under one of the heat pipes here and then I’m not going to do it because it’s going to take a bunch of time. Actually, no I’m doing it fast. Let’S go nice. Okay, then, you feed the bolts through put these risers on each one of the posts, and then you can add another GPU. On top of that, so they’re not touching, there’s no metal on metal and then ovh’s own marked PCB custom. Riser gets plugged into both gpus at the same time and your GPU sandwich is ready to go.
These are the tested, good gpus, which they are then going to put onto actual servers that are theoretically going to go into actual production after the gpus are mounted. We then have to mount the motherboard, the ram, the CPU, all that stuff you’re used to seeing and all of the plugins for the Riser cables, because I mean well there’s eight of them one for every GPU after that, we’re pretty much good to go for this Stage, there’s a little bit more testing that has to be done a few other behind the scenes things, but it gets held here because, as far as building the computer is concerned other than a little bit more chassis, it’s essentially done my favorite last bit that I Have to point out: is these pipes up here water distribution, things they call them clarinets? I just had to include that once each rack has been proved good for at least a day. It is then sent to an ovh data center like this one upon arrival. They do re-testing again all of the same previous phases that they did in the previous location. All of it is done again just to make sure that nothing went wrong in transport. Then once they’re done and everything is ready – it’s installed here and is able to be used by users like you, for whatever high performance applications, they might need other than obviously the one we’re selling right now I don’t want to be ready in a few minutes.
All of the servers at every single ovh data center is water cooled and that takes some serious infrastructure to be able to accomplish, along with a bunch of r d. Some of that infrastructure has actually been right in front of us this whole time. It’S this gigantic metal plate below that huge metal plate is a massive Reservoir. This Reservoir was actually built by a separate company, a bottling company that owned this building before ovh, but they’ve reused that infrastructure to use it as a backup Reservoir, just in case they need it below each one of these hatches is just a massive amount of water And that water is constantly being pH tested and there’s even these really Advanced, looking pool skimmers moving around on top of it that are just making sure that none of the loops are going to Gunk up, because that would be a really really big problem. Speaking of big problems moving all this water around is quite a difficult task and you’ll find these pumps all throughout the facility, you’ll always find them in pairs. The reason for that is because they’ll run one at a time they switch every work day.
That’S to make sure that each one has a very similar amount of worked hours and by using them all the time, they’re actually quite sure that they work, which is good. All of these Advanced systems synchronize together, including a few things that I can’t really show you or talk about on camera, result in quite a dramatic difference. This pipe, for instance, is the water coming into the building. When I touch it, it’s actually quite cold.
This pipe is the water going out of the building and when I touch it, oh hey it’s around room temperature. The difference they say on average is about 15 to 20 degrees, and you can see quite quickly why this would actually be worth doing instead of running air conditioning for this entire gigantic building. To make sure that all of this keeps running smoothly, they have the monitoring room. I I like to see it as a control center. I think they could probably launch a satellite from here, but we’ll see time will tell over on the right side. They have their fuel monitoring to make sure that their generators, which are their like third backup for power that they very rarely ever, would need to use actually still have fuel, make sure they’re going to run in the case that they do actually need it moving forward.
We have their incidence, dashboard or notifications, everything’s green, because everything’s going well right now, these red on the far right hand, side, are actually just information that they’re receiving from different parts of the data center, not not anything that really needs to be addressed right now. Moving up here, they have their server monitoring the a what haha one is potentially down or might potentially have an issue. That’S actually kind of cool that we saw that happen live so that will now be seen by the team and be able to be addressed. Uh and the green zeros, on the other two mean that there’s no problems in those zones.
Other parts of the screen like in the middle here that’s checking if different benchmarks have failed. So maybe a system needs to be worked on down here. It’S things that need to be repaired or upgraded, Etc: they’re, always tracking everything so that things can be handled very quickly. I actually saw people run around in the room behind the camera to address that as it happened, which is super cool.
You want that type of stuff in a data center, because things are going to go wrong, so being able to address them quickly is is, is key speaking of address them quickly. They also monitor the weather. Just in case something happens if there’s a storm, if there’s a heat wave in the case of a storm, they might want to go around and just make sure that everything’s, okay, as the storm’s going on after the storm passes, make sure everything is all hunky-dory.
They haven’t necessarily had a problem, but it’s good to know coming outside it’s more of the same, but even bigger. They have some massive reservoirs back there that I can’t show you. They also have some really cool heat exchangers, just off camera in this direction that I can’t show you, but what I will say is standing amongst those outside in the cold is actually warmer than standing in the data center inside so clearly they’re working pretty well in The background, if you haven’t noticed, there’s some big power lines, power lines that would power.
You know like a city, but the 220 000 volts that they’re delivering to the ovh building is just for ovh. That’S their main intake just in case there’s something wrong with that across the building on the other side underground, they have more additional power lines, bringing in 20 000 volts, and if all of that isn’t working, they have some huge generators. Each one gives one megawatt of power when it’s running and can run for up to 14 hours, but again they they very rarely ever have to use those.
That’S it for the tour of ovh and seeing how they build the servers and all that kind of stuff. Now I’m back in Canada at the studio, and I have to show Linus how this kind of all comes together. So I have a Raspberry Pi system with Shadow installed, an iOS device with Shadow installed and a non-gaming laptop, an LG gram with Shadow installed on the shadow Windows system.
We installed some various games and we’re gon na. Have him check them out? I get that you want to show that it’s a thin client we’re using the raspberry pi. Did you have to use the raspie mouse? Yes, Shadow does not claim to improve your peripherals at all. So this is this is what it came with. Also, we had some issues getting the Raspberry Pi to work with a better monitor than this. No I’m fine, but not scepter is King tier yeah.
It’S right in the name, all right, let’s go sick, really, no one! No one drove you. What are you guys idiots? So playing Halo, which is notoriously a little bit rough to run and playing it on a Raspberry Pi. It was pretty cool and the fact that he’s more focused on the game than anything else means I think it’s working pretty yeah. I got a kill already, never mind focused on the game.
I think you mean winning the game. Oh Shadow’s, always running on another already running on another device. There we found out what happens if you try to connect a second device that was pretty simple. It prompts you and you press a button and then hahaha okay, here we go hold on a second. Is this gon na work no way where’s? My uh? Okay, I mean to be clear. This is uh, not an ideal controller setup, but you totally could that’s sweet.
Oh yeah uh these on-screen joystick. As someone who doesn’t mobile game, oh right come on come on come on. Oh, I almost like popped him come on. I got him.
I got him ah, first fight too, he never died pretty good point to end it on. Could you see yeah thanks for watching? I don’t know what else to watch after this they’ll put something on screen: watch the old Shadow one or something I don’t know yeah. That sounds good. Let us know what you think in the comments down below and yeah see: ya.
.