> Starting in the first week of 2024, the FreeBSD boot process suddenly got about 3x slower. I started bisecting commits, and tracked it down to... a commit which increased the root disk size from 5 GB to 6 GB. Why? Well, I reached out to some of my friends at Amazon, and it turned out that the answer was somewhere between "magic" and "you really don't want to know"; but the important part for me was that increasing the root disk size to 8 GB restored performance to earlier levels.
My understanding is that EBS has some heuristics for deciding whether to keep data cached; an AMI which has a cached snapshot as its root disk will boot much faster than an AMI where all the data needs to be pulled from S3.
What's the smallest size for which those heuristics keep the snapshot cached?
(I'm currently using 1GB snapshots, because my actual disk image is a tiny fraction of that size. But if bumping that to 2GB or 4GB would make it faster, that's a small price to pay.)
Do you have any other wisdom regarding mysterious reasons for fast or slow booting? EC2's boot process is deeply opaque, and any insight at all is better than nothing.
Yeah, I am constantly curious about how the sausage that is cloud services like AWS is made. It seems generally slick on the surface, but what’s holding it all together? I imagine it as a tangled ball of tools like Puppet, Chef, etc. and custom glue.
I can't remember exactly but it was a few hours. I already knew which week the issue arose (from comparing weekly snapshots) so that gave me a head start.
But yes, I built a lot of AMIs. And launched new EC2 instances for each of them -- it wasn't just a matter of rebooting since the first time an AMI launches there's different behaviour (both from FreeBSD, e.g. growing the root disk, and from EC2, e.g. disk caching).
Sweet! By the way we just added FreeBSD to the download page on ziglang.org (as of today), so FreeBSD users can grab master branch builds automatically built by the CI.
It's also now a first-class supported cross-compilation target, including when linking libc, so you can do stuff like `zig cc -o hello hello.c -target riscv64-freebsd`.
And then of course if you have any C/C++ dependencies, you can fetch and build them with the zig build system, so it should be possible to easily cross-compile even quite complex projects for FreeBSD now.
Hopefully that helps more projects decide to add FreeBSD support and respective testing to their CI!
It turns out that at a certain point, money can buy time. Do I fix the leaky tap myself, or hire a plumber? After electricians rip up my basement drywall (perfectly reasonably -- I was getting solar panels installed and the electrical panel needed to be upgraded) do I fix it myself or do I hire a professional drywaller?
To be fair, some of the time I spent on this came away from Tarsnap. But less than you might imagine.
It’s really not hard. I added onto a house I owned back in 2009 after returning from Iraq and saving a bunch of money. IRS also gave me a huge refund that year. Something with GWOT vets and taxes with combat pay. Anyhow, converted a covered porch to a room, had permits, raised the floor, added electric, energy efficient Argon gas windows, etc. You give 84 lumber the blue print plans from the architect and they tell you how much material you need. After I got to the stage to add furring strips for the concrete block part and insulation (which is easy too), I was ready to add the dry wall. Just lift it up to the wall then screw it in. Dry wall tape, corner bead, and mud to cover the seams. Orange peel can hide imperfections. Then paint. Pretty straightforward
I know how to do drywall as well as a pro - it just takes me 6x as long. It is easy to do, but you can't do what the pros do without a lot of practice. By planning on 6x longer you can slow down, do thiner coats and such (pros do 3 coats of mud, I do 6 for the same thickness). Which falls into the do or hire.
True. In this case it was drywall in a poorly lit basement, so I wasn't all that concerned about it turning out perfectly -- but it absolutely did turn out much better than it would have if I tried to do it myself.
Dry wallers are amazing. As a diyer, it’s one of the few things I can never seem to get right. I’m happy to put holes in it, but seeing professionals patch it is another level.
The other day I had the opportunity to get a 10% discount on a fridge if I could pay the whole thing in one payment. If I didn't have the money I wouldn't get the discount, so in a way being poor means everything is more expensive.
"The reason that the rich were so rich, Vimes reasoned, was because they managed to spend less money. Take boots, for example. ... A really good pair of leather boots cost fifty dollars. But an affordable pair of boots, which were sort of OK for a season or two and then leaked like hell when the cardboard gave out, cost about ten dollars. ... But the thing was that good boots lasted for years and years. A man who could afford fifty dollars had a pair of boots that'd still be keeping his feet dry in ten years' time, while a poor man who could only afford cheap boots would have spent a hundred dollars on boots in the same time and would still have wet feet.
This was the Captain Samuel Vimes 'Boots' theory of socio-economic unfairness."
I was rather hoping Amazon would spend and contribute more. But it seems they basically only want to pay for the minimum FreeBSD support.
Amazon isn't even on FreeBSD sponsors [1]. And Google only sponsored $9K last year. Apple isn't there. Edit: And Credit to Microsoft being at least on the list! And forgot to mention Meta / Facebook missing from it as well.
I would have expect them to sponsor FreeBSD and OpenBSD annually by default given they use and continue to benefits the work out of both.
I'd love to see Amazon contribute more, of course; but the fact they don't show up as donors to the FreeBSD Foundation doesn't mean they're not supporting FreeBSD. The money they paid me didn't flow through the Foundation, for example; I'd guess that Foundation-funded development is maybe 10% of all corporate-funded FreeBSD development. (It's an important 10%, especially because it can be focused on "what does FreeBSD need" rather than "what does company X need" -- but it's still a small minority.)
I wonder for what reason Microsoft funds them. Their Hyper-V extensions are not as complete as Linux. There's no Microsoft-supported port of .NET. I can't think of any services that run on *BSD from Microsoft, cloud or otherwise.
My thought process was that it is such a small amount of money for Advertisement or marketing to have your logo on some prominent Open source project to at least try and redeem / make themselves look good.
I wonder if this is a very, very, very long-term holdover from the Hotmail team having some expertise, and Microsoft wanting to be able to offer something a bit nix-y that wasn't Linux
I wanted to use FreeBSD for my home gateway/firewall/dns/dhcp server, but unfortunately my 10 GbE NICs didn't seem to have drivers, so I ended up going with Nix instead. I used FreeBSD many years ago as a workstation, and found the experience to be quite memorable. It's nice to see that it's still chugging along.
I use FreeBSD for all my (and my companies) infrastructure. I only use hardware with Intel NICs because they are 100% reliable on FreeBSD. Anything with Realtek seem to crap out under load despite the hard work of the FreeBSD engineers who maintain the drivers (I'm not complaining and I respect their efforts).
It's a small price to pay and it stops me having to install less stable operating systems.
I remember the time around version 7 or 8 when FreeBSD had better drivers than Linux for stuff like Atheros wifi cards.
I favored FreeBSD until around 2021 when computers with different CPU mixed together started to become common. I first bought a RockPro64 with 2 big and 4 little cores and then an Intel Alder lake. As far as I understand FreeBSD scheduler to this day don't know how to properly play with these so it brings the system to the lowest denominator of the slower cores.
I can tell you some financial services I have worked for do use FreeBSD on EC2 as well as on the metal in data centers to do millions of transactions a month.
I like the OS, thanks for your work.
Can someone who uses FreeBSD fill me in on the niche that it fills in the Unix space? Why not use OpenBSD or NetBSD, which are far simpler and coherent? If the answer is support for stuff like ZFS, Nvidia drivers, ELF, etc. why not Linux? I'm well aware of the problems with GNU, but do you have problems even with something like Musl Void?
I'm genuinely actually curious. FreeBSD exists in kind of a shadow realm for me where I've never been quite able to pin down the soul that keeps it chugging, but I know it exists somewhere in there.
I have worked for financial services companies that used FreeBSD both in EC2 and on the metal in data centers (self managed).
The two features we used all the time were zfs and jails.
Each service ran in its own jail for isolation. One (not even beefy) server could run all the services which was insanely cost efficient.
A cloud migration was undertaken at some point to have a hybrid setup, using a mix of Linux (k8s) and FreeBSD, and costs skyrocketed. It’s a trade off because in the data center we had to buy and replace our own disks, react to fires taking place, being only in one country etc.
AWS gives you multi region, and tons of good stuff, and that has a price.
ZFS was not leveraged that much but it saved our beacon once when a table in the production database was accidentally dropped and we could instantly rollback to the previous zfs snapshot (there was a tiny bit of data loss as a result but this did not matter too much for this application - uptime was more important). ZFS was also used for backups I believe.
A few times I used dtrace in production to troubleshoot.
When we introduced Linux to our fleet of FreeBSD servers, every team picked a different distro organically so it was a bit of a zoo. With FreeBSD on the server you only have the one variant.
I still use and like both, but I must say I really like that FreeBSD is a kernel+OS integrated together.
> Can someone who uses FreeBSD fill me in on the niche that it fills in the Unix space? Why not use OpenBSD or NetBSD, which are far simpler and coherent? If the answer is support for stuff like ZFS, Nvidia drivers, ELF, etc. why not Linux?
My experience with FreeBSD is that it provides a nice balance of the concerns OpenBSD and NetBSD specifically address. Historically, FreeBSD prioritized Intel CPU's (where NetBSD had greater portability) and had solid security (where OpenBSD had more of a focus on it).
The FreeBSD ZFS support really is a game changer. I believe Nvidia only recently has had native FreeBSD drivers - for a long time FreeBSD's kernel Linux support was required.
> I'm genuinely actually curious. FreeBSD exists in kind of a shadow realm for me where I've never been quite able to pin down the soul that keeps it chugging, but I know it exists somewhere in there.
Again, for me, FreeBSD has proven to be a nice blend of the features other BSD's provide as well as being incredibly stable on the h/w platforms I tend to use.
I figured the goldilocks metric might factor in. Are you dealing with non-x86 platforms? I've always been disappointed by the ARM experience on Linux. It always feels second-class.
FreeBSD is throughput oriented in a way that OpenBSD certainly isn't, and I don't think NetBSD is either (although, I haven't really looked, I feel like NetBSD competes on portability and doesn't spend a lot of time making sure networking throughput is high).
All of the BSDs tend to have a lot less churn, for better and worse; so IMHO, they make a nicer platform to integrate on.
If you want a high profile example, look at what Netflix CDN does.
Could you do that work with Linux? Probably --- but nobody who does is talking about it as much.
This kind of high throughput service has been a FreeBSD niche since forever too. Walnut Creek CDROM, Inc ran what was reportedly the world's busiest ftp site, ftp.cdrom.com on FreeBSD in the early days of the internet.
Yahoo ran on FreeBSD (I worked there 2004-2011) WhatsApp ran on FreeBSD (I worked there 2011-2019) Both were leaving FreeBSD when I left, but sadly, I didn't leave to work somewhere else with FreeBSD :p
My impression is that FreeBSD is Apple's shadow in FOSS, they hold a lot of soft power over it. I know the kernels are different and obviously only part of the userspace is the same, but is FreeBSD actually far enough away from Apple to say it's not bogged down by corporate interests? I don't imagine it's the same as Linux at all, but it exists in a non-trivial way, no?
It's not. Apple (or rather NeXT) took some of the userland for macOS but it's not contributing back and it doesn't have much influence. It's more like a fork a long time ago.
A few companies do. Skype and Netflix did but hardly use it now (at least Skype left it, not sure about Netflix but I never hear about it from bsd devs). Ix systems and netgate do but they're tiny.. No, it's not influenced in a trivial way and certainly not by apple.
This is a huge difference to Linux where the vast majority of kernel commits come from big tech and have nothing to do with things end users care about. Also there's nothing in the FreeBSD world like the Linux Foundation which is basically a corporate lobby group.
> but it's not contributing back and it doesn't have much influence.
I understand the former. But with how Apple operates, it's really hard to believe they'd pull downstream from something they don't have some kind of soft power over. They do still pull downstream AFAIK? Maybe that's changed?
>Ix systems
I did some reading and saw a FreeBSD contributor ended up going to Apple until 2013 before he founded this company.
https://www.ixsystems.com/clients/
Apple is listed here. Six degrees of separation and all, but probably not a coincidence.
Nothing wrong with that, business is a social structure. This is how they work. We make and keep friends, even if only professionally. Backchannels are where real deals are made.
But this to me is not nothing. No corporate influence means there's a lot of nice things you don't get. You just can't afford the manpower. It looks more like 9 Front than a BSD that has some serious billion-dollar problems under its belt.
That sounds harsh, not a judgement. Just very deep skepticism of the assertion of no influence. I'm realizing there's not a lot that can be done to sway that intentionally.
> This is a huge difference to Linux
This I'm well aware of. I just like having a perspective across the fence. These days they're starting to get a little too aggressive for my tastes. FreeBSD seems fine in comparison.
> But with how Apple operates, it's really hard to believe they'd pull downstream from something they don't have some kind of soft power over. They do still pull downstream AFAIK? Maybe that's changed?
Apple doesn't merge often. They basically haven't merged kernel tcp since 2002. When I started using OSX in 2011, they hadn't merged userland for several years, and when I stopped in 2019, they had only merged once.
They famously stopped picking up bash when upstream changed the license, and most of the FreeBSD userland doesn't change that frequently, so most things you wouldn't notice a difference. cal(1) started highlighting the current day at some point, tar probably grew new compresion arguments, etc.
Apple certainly was a major contributor/driving force/etc of LLVM for a while, not sure if they still are? And LLVM was adopted by FreeBSD, so maybe that's where this idea is coming from?
> And LLVM was adopted by FreeBSD, so maybe that's where this idea is coming from?
Partially, but after seeing the Jordan Hubbard connection, there's a lot of layers to this. May have reinforced my biases, but it's definitely non-trivial according to my hippie-tier anarchist baseline. Oops. Worst case scenario of answering your own question.
But your reply does give me actually contradicting evidence. It wouldn't surprise me that distance has grown to the point of total atrophy, given the general trajectory Apple has been on since 2012 or so. This is why I ask these questions, because the people on the ground give the most informative answers.
As Ptahhotep advises circa ~2300BCE:
> Fine words are more sought after than greenstone, but can be found with the women at the grindstone.
> May have reinforced my biases, but it's definitely non-trivial according to my hippie-tier anarchist baseline.
The definition of 'trivial' would come into play yes. I would only consider it non-trivial if a commercial party can (and does) influence the direction of development. I don't think Apple does so. Even Netflix. In the Linux world there's billions of investment and many contributors are directly employed by big business. The waters are much murkier there.
Again, I'm not saying it's a bad thing. It's just not something I want which is one of the reasons I picked FreeBSD. Other reasons were the great ports collection, the division between OS and apps (you can have rolling apps but a stable OS), the traditionalism (only change things if it's really needed) and the single main flavour of the OS which makes support much easier. Also the excellent documentation.
> No corporate influence means there's a lot of nice things you don't get.
Yes that is the flipside. But I don't mind that. If you choose your hardware carefully it works fine.
Note that this is not too different from using Windows or Mac. Your hardware is also chosen carefully to work with those, just not by you but by the vendor. With FreeBSD you're more involved with the nuts & bolts and this is exactly what I want. I don't want my OS to be a black box I don't understand.
> My impression is that FreeBSD is Apple's shadow in FOSS, they hold a lot of soft power over it.
Apple has no influence over the FreeBSD project.
> I know the kernels are different and obviously only part of the userspace is the same, but is FreeBSD actually far enough away from Apple to say it's not bogged down by corporate interests?
Yes.
OS-X (now macOS) is based on XNU[0], which itself has roots in the Mach[1] microkernel. The Unix user-space programs distributed with OS-X/macOS are those found in FreeBSD distributions AFAIK. This is also conformant with FreeBSD licenses for same.
So there is no "soft power" Apple has over FreeBSD. And FreeBSD is not "Apple's shadow in FOSS".
> I don't imagine it's the same as Linux at all, but it exists in a non-trivial way, no?
No. It does not.
EDIT: Just in case you'd like to verify any of the above yourself, see here[2].
I'm not sure where you're getting the "Apple holds soft power over FreeBSD" thing from. Netflix is probably at the top of the list given all their performance and stability work -- and, you know, the fact they push a large chunk of all Internet traffic using FreeBSD -- and NetApp and Juniper are somewhere up there, but I'm not convinced Apple would even be in the top 10.
> I'm not sure where you're getting the "Apple holds soft power over FreeBSD" thing from.
The only thing I've ever heard from FreeBSD-land, not paying attention to users, but the maintainers and the tools. Apple comes up. In the same manner that RedHat and others come up for Linux. How to explain? It's an abstract pattern. Transparent, understandable.
I mentioned somewhere about the connection through ix systems. And honestly to project, if I was a maintainer of something used between Netflix and Apple, I'd prioritize Apple. Apple has outlived IBM. If you know your history, you know how serious that is. If you've got authority over something as large as FreeBSD? Yeah, you don't ignore that kind of actual power especially when it's personal. Like I say, all based on guesses. But some things are hard to mistake.
Apple did very important work making LLVM happen, but that was a long time ago. At this point there are lots of companies involved in that project.
As far as "power" is concerned... speaking as release engineer, I don't give special treatment to anyone; nor have I even been asked to. If anyone has a special relationship it's Netflix but if anything that's the opposite way around: "Can you please throw 10% of all Internet traffic at this TCP stack patch and let us know if anything breaks" is a thing. They're incredibly helpful with Q/A.
Consider me convinced. Like I say, it was never anything but empirical skepticism. Neither for or against until sufficient evidence has been collected, as painful as that can be. Thank you for your work on scrypt.
I know some maintainers of the userland and apple never comes up. Most maintainers have no commercial ties to anyone so they don't really care abour corporate influence anyway. They just maintain the software because they like using it. This is exactly the kind of thing I like, in Linux there are too many companies putting money into it because they want to make money back (usually not from normal users but from cloud instances, steering the project in a direction away from its grassroots origins).
I'm sure if Apple wants something it would be considered but there would be a strong validation of "what's in it for us" on the freebsd side. There's also some pretty bad experiences with corporate influence and this is reviewed a lot more independently since the netgate wireguard disaster. https://arstechnica.com/gadgets/2021/03/buffer-overruns-lice...
Unlike in the Linux world where RedHat and canonical are so embedded due to most of the devs working for them that there will be a lot less questions. And not just those two, also companies like Huawei are heavy kernel contributors.
I'm not saying it's bad to have such commercial influence. But it's not what I want for the OS I run.
For me, who's daily driven FreeBSD in the past, and switched back to it again recently. FreeBSD serves as a refuge from systemd, and the only BSD that is a fairly drop in replacement for linux in terms of software compatibility, as well as the only game in BSD town in terms of support for modern hardware.(though it does significantly lag Linux in this regard still, so YMMV).
As to why I use it over the various systemd free linux distros? Well, there's a couple things. First lot of those distros, like Artix linux say, actually have smaller communities than FreeBSD(I'm guesstimating based on the activity level in their irc channels). The Linux community might be much, much larger than the FreeBSD community, but it's also extremely fragmented.
triggerwarning, hyperbole incoming. Don't bother correcting me, it's a polemic, not a scientific paper
Secondly, for someone like me, who's been using various unix like OSes for two decades, FreeBSD is just a nice, batteries included, well integrated system. Things like jails, Dtrace, ZFS, Bhyve, pf etc. All being in the base install means they're just better integrated with the kernel, and eachother. Most of those things exist for linux, or have equivalents, but they're not all part of the same project. Obviously Dtrace and ZFS originated in Solaris, but they've been made first-class citizens. There's a harmony to FreeBSD that Linux distros lack. Documentation is also very good, all accessible via manpages(no GNU INFO...). And, as I mentioned briefly before. It doesn't have a lot of the cruft that's been added to linux distros over the years(though some of it is available in ports if you want it). In FreeBSD, my experience is actually useful. Things I remember how to do from 5 years ago, 10 years ago, 15 years ago, still work. If I'm on some modern, plug and play linux distro, I have no idea what's going on under the hood any more. All I know is it's not what was going on 5 years ago, which isn't what was going on 10 years ago, which isn't what was going on 15 years ago. The amount of pointless churn going on in the linux space is ridiculous. When I started using linux, what I loved about it was that it was transparent. I could change anything. The system was easy to understand. Yes, it was janky, but it was understandable jank, whereas Windows was janky in an opaque way. 20 years later, Linux is still janky, but nothing is understandable, at least not to my greybeard brain. Systemd takes over a new daemon every distro upgrade. DNS resolving now involves 4 different daemons with 15 different configuration files, there's two display protocols, both broken in different ways, /etc is full of long files written in strange, alien languages, and every file has its own bespoke language. There seems to be 54 different ways to make any change to your system, and all of them are somehow unsatisfactory in a unique way. I just can't, anymore. Enough already.
This does not paint a good picture for FreeBSD development. An operating system of that size and complexity surely needs someone company to sponsor the release manager full time, not part time for a year.
I don't say this to besmirch FreeBSD, FWIW. I think it's very important that Linux is not the only game in town.
I may be wrong about OPs intention, but AFAICT, because no encoding is specified, the client gets to choose. For someone not using a default encoding that's a superset of ASCII (like ISO-2022-KR) the page appears as a �.
Current practice is to put a meta tag with your encoding, use a Unicode BOM, or less favorably, send the charset attribute in the Content-type header.
There are some hilarious tidbits in here
> Starting in the first week of 2024, the FreeBSD boot process suddenly got about 3x slower. I started bisecting commits, and tracked it down to... a commit which increased the root disk size from 5 GB to 6 GB. Why? Well, I reached out to some of my friends at Amazon, and it turned out that the answer was somewhere between "magic" and "you really don't want to know"; but the important part for me was that increasing the root disk size to 8 GB restored performance to earlier levels.
The original object size limit for S3 was 5 GB, as noted in my 2006 blog post:
https://aws.amazon.com/blogs/aws/amazon_s3/
I do not know if this has anything to do with the cliff that you saw.
Pretty sure that's not related. For one thing I don't think EBS snapshots are stored in S3 as 5 GB segments.
Now I really want to know though.
My understanding is that EBS has some heuristics for deciding whether to keep data cached; an AMI which has a cached snapshot as its root disk will boot much faster than an AMI where all the data needs to be pulled from S3.
Some huge customer chunked their data into 5GB pieces so now there's a "if size == 5GB" in the cache code.
Maybe, but I don't think that would explain 8 GB also being fast while 6 GB is slow?
Yeah, I found that pretty unintuitive when I read it. How did you find 8GB worked? Trial and error?
Customer started using 8GB chunks /s
What's the smallest size for which those heuristics keep the snapshot cached?
(I'm currently using 1GB snapshots, because my actual disk image is a tiny fraction of that size. But if bumping that to 2GB or 4GB would make it faster, that's a small price to pay.)
I believe 1 GB is also fast.
Thanks, that helps to hear!
Do you have any other wisdom regarding mysterious reasons for fast or slow booting? EC2's boot process is deeply opaque, and any insight at all is better than nothing.
Nothing comes to mind, but if you want to drop me an email I can walk you through some benchmarking.
Yeah, I am constantly curious about how the sausage that is cloud services like AWS is made. It seems generally slick on the surface, but what’s holding it all together? I imagine it as a tangled ball of tools like Puppet, Chef, etc. and custom glue.
A lot of AWS services are built on other AWS services. Like Lambda, SQS, and other such "core services" are used by others under the hood.
At Amazon scale mostly everything is custom
Less puppet/chef
Yeah, I would imagine they maybe started with off-the-shelf tools that were then gradually replaced as the system grew and matured.
I wonder how long did it take to bisect such issue. Build image every time and reboot a vm?
I can't remember exactly but it was a few hours. I already knew which week the issue arose (from comparing weekly snapshots) so that gave me a head start.
But yes, I built a lot of AMIs. And launched new EC2 instances for each of them -- it wasn't just a matter of rebooting since the first time an AMI launches there's different behaviour (both from FreeBSD, e.g. growing the root disk, and from EC2, e.g. disk caching).
Thanks for the additional information, a few hours sounds great, I was expecting multiple days to narrow it down, given a lengthy feedback loop.
Sweet! By the way we just added FreeBSD to the download page on ziglang.org (as of today), so FreeBSD users can grab master branch builds automatically built by the CI.
It's also now a first-class supported cross-compilation target, including when linking libc, so you can do stuff like `zig cc -o hello hello.c -target riscv64-freebsd`.
And then of course if you have any C/C++ dependencies, you can fetch and build them with the zig build system, so it should be possible to easily cross-compile even quite complex projects for FreeBSD now.
Hopefully that helps more projects decide to add FreeBSD support and respective testing to their CI!
Zig's cross compilation is awesome, and it's nice to see FreeBSD on the supported target list.
There is also a lot of work on the laptop front, I read that the BSD foundation invested $750k for this implementing: (S0ix Sleep State, etc )
you can find the project laptop here https://github.com/FreeBSDFoundation/proj-laptop
Yep there's a lot of work going on. I was just writing about the work I was doing. ;-)
Lots of respect for cperciva.
Don’t know how he manages all of this + Tarsnap.
It turns out that at a certain point, money can buy time. Do I fix the leaky tap myself, or hire a plumber? After electricians rip up my basement drywall (perfectly reasonably -- I was getting solar panels installed and the electrical panel needed to be upgraded) do I fix it myself or do I hire a professional drywaller?
To be fair, some of the time I spent on this came away from Tarsnap. But less than you might imagine.
> or do I hire a professional drywaller
When it comes to drywall, always hire a professional. Learn from other's mistakes... it's not as easy as you think and it won't turn out well.
It’s really not hard. I added onto a house I owned back in 2009 after returning from Iraq and saving a bunch of money. IRS also gave me a huge refund that year. Something with GWOT vets and taxes with combat pay. Anyhow, converted a covered porch to a room, had permits, raised the floor, added electric, energy efficient Argon gas windows, etc. You give 84 lumber the blue print plans from the architect and they tell you how much material you need. After I got to the stage to add furring strips for the concrete block part and insulation (which is easy too), I was ready to add the dry wall. Just lift it up to the wall then screw it in. Dry wall tape, corner bead, and mud to cover the seams. Orange peel can hide imperfections. Then paint. Pretty straightforward
I know how to do drywall as well as a pro - it just takes me 6x as long. It is easy to do, but you can't do what the pros do without a lot of practice. By planning on 6x longer you can slow down, do thiner coats and such (pros do 3 coats of mud, I do 6 for the same thickness). Which falls into the do or hire.
True. In this case it was drywall in a poorly lit basement, so I wasn't all that concerned about it turning out perfectly -- but it absolutely did turn out much better than it would have if I tried to do it myself.
Dry wallers are amazing. As a diyer, it’s one of the few things I can never seem to get right. I’m happy to put holes in it, but seeing professionals patch it is another level.
My wall has orange peel on it so I just spray it with orange peel after getting the spackle smooth and sanded. Never know I patched it.
At a certain point money can even buy money.
The other day I had the opportunity to get a 10% discount on a fridge if I could pay the whole thing in one payment. If I didn't have the money I wouldn't get the discount, so in a way being poor means everything is more expensive.
"The reason that the rich were so rich, Vimes reasoned, was because they managed to spend less money. Take boots, for example. ... A really good pair of leather boots cost fifty dollars. But an affordable pair of boots, which were sort of OK for a season or two and then leaked like hell when the cardboard gave out, cost about ten dollars. ... But the thing was that good boots lasted for years and years. A man who could afford fifty dollars had a pair of boots that'd still be keeping his feet dry in ten years' time, while a poor man who could only afford cheap boots would have spent a hundred dollars on boots in the same time and would still have wet feet. This was the Captain Samuel Vimes 'Boots' theory of socio-economic unfairness."
https://en.wikipedia.org/wiki/Boots_theory
Being more poor is always more expensive than being less poor.
All poor people know this in their bones because they face this every day of their lives.
I was rather hoping Amazon would spend and contribute more. But it seems they basically only want to pay for the minimum FreeBSD support.
Amazon isn't even on FreeBSD sponsors [1]. And Google only sponsored $9K last year. Apple isn't there. Edit: And Credit to Microsoft being at least on the list! And forgot to mention Meta / Facebook missing from it as well.
I would have expect them to sponsor FreeBSD and OpenBSD annually by default given they use and continue to benefits the work out of both.
[1] https://freebsdfoundation.org/our-donors/donors/?donationYea...
I'd love to see Amazon contribute more, of course; but the fact they don't show up as donors to the FreeBSD Foundation doesn't mean they're not supporting FreeBSD. The money they paid me didn't flow through the Foundation, for example; I'd guess that Foundation-funded development is maybe 10% of all corporate-funded FreeBSD development. (It's an important 10%, especially because it can be focused on "what does FreeBSD need" rather than "what does company X need" -- but it's still a small minority.)
This comment does not present the full picture.
First, it presents the snapshot of donations within a given year to the Foundation. The history of donations is not represented by definition.
Second, it does not present contributed development. Those are typically summarily available on the release notes of each release [1].
[1] https://www.freebsd.org/releases/
Has Amazon donated more to FreeBSD than Notch the Minecraft guy?
I wonder for what reason Microsoft funds them. Their Hyper-V extensions are not as complete as Linux. There's no Microsoft-supported port of .NET. I can't think of any services that run on *BSD from Microsoft, cloud or otherwise.
There are customers that run FreeBSD on Azure and Microsoft officially supports it: https://learn.microsoft.com/en-us/azure/virtual-machines/lin...
I don't know, but Microsoft has some developers working on Hyper-V for FreeBSD. They've even come to FreeBSD developer summits.
My thought process was that it is such a small amount of money for Advertisement or marketing to have your logo on some prominent Open source project to at least try and redeem / make themselves look good.
I wonder if this is a very, very, very long-term holdover from the Hotmail team having some expertise, and Microsoft wanting to be able to offer something a bit nix-y that wasn't Linux
Amazon does the least for FOSS out of the FAANGs.
I wanted to use FreeBSD for my home gateway/firewall/dns/dhcp server, but unfortunately my 10 GbE NICs didn't seem to have drivers, so I ended up going with Nix instead. I used FreeBSD many years ago as a workstation, and found the experience to be quite memorable. It's nice to see that it's still chugging along.
I use FreeBSD for all my (and my companies) infrastructure. I only use hardware with Intel NICs because they are 100% reliable on FreeBSD. Anything with Realtek seem to crap out under load despite the hard work of the FreeBSD engineers who maintain the drivers (I'm not complaining and I respect their efforts).
It's a small price to pay and it stops me having to install less stable operating systems.
I remember the time around version 7 or 8 when FreeBSD had better drivers than Linux for stuff like Atheros wifi cards.
I favored FreeBSD until around 2021 when computers with different CPU mixed together started to become common. I first bought a RockPro64 with 2 big and 4 little cores and then an Intel Alder lake. As far as I understand FreeBSD scheduler to this day don't know how to properly play with these so it brings the system to the lowest denominator of the slower cores.
Out of curiosity, who are the top users of FreeBSD/EC2?
I have no clue. Seriously, the users who talk to me are maybe 0.1% of the total FreeBSD/EC2 user base.
I would love to know who is using FreeBSD in EC2.
I can tell you some financial services I have worked for do use FreeBSD on EC2 as well as on the metal in data centers to do millions of transactions a month. I like the OS, thanks for your work.
Does netflix only use it on their edge boxes?
Yes, their CDN is FreeBSD but last I heard all of their cloud operations are Linux.
Can someone who uses FreeBSD fill me in on the niche that it fills in the Unix space? Why not use OpenBSD or NetBSD, which are far simpler and coherent? If the answer is support for stuff like ZFS, Nvidia drivers, ELF, etc. why not Linux? I'm well aware of the problems with GNU, but do you have problems even with something like Musl Void?
I'm genuinely actually curious. FreeBSD exists in kind of a shadow realm for me where I've never been quite able to pin down the soul that keeps it chugging, but I know it exists somewhere in there.
I have worked for financial services companies that used FreeBSD both in EC2 and on the metal in data centers (self managed). The two features we used all the time were zfs and jails. Each service ran in its own jail for isolation. One (not even beefy) server could run all the services which was insanely cost efficient. A cloud migration was undertaken at some point to have a hybrid setup, using a mix of Linux (k8s) and FreeBSD, and costs skyrocketed. It’s a trade off because in the data center we had to buy and replace our own disks, react to fires taking place, being only in one country etc. AWS gives you multi region, and tons of good stuff, and that has a price.
ZFS was not leveraged that much but it saved our beacon once when a table in the production database was accidentally dropped and we could instantly rollback to the previous zfs snapshot (there was a tiny bit of data loss as a result but this did not matter too much for this application - uptime was more important). ZFS was also used for backups I believe.
A few times I used dtrace in production to troubleshoot.
When we introduced Linux to our fleet of FreeBSD servers, every team picked a different distro organically so it was a bit of a zoo. With FreeBSD on the server you only have the one variant.
I still use and like both, but I must say I really like that FreeBSD is a kernel+OS integrated together.
> Can someone who uses FreeBSD fill me in on the niche that it fills in the Unix space? Why not use OpenBSD or NetBSD, which are far simpler and coherent? If the answer is support for stuff like ZFS, Nvidia drivers, ELF, etc. why not Linux?
My experience with FreeBSD is that it provides a nice balance of the concerns OpenBSD and NetBSD specifically address. Historically, FreeBSD prioritized Intel CPU's (where NetBSD had greater portability) and had solid security (where OpenBSD had more of a focus on it).
The FreeBSD ZFS support really is a game changer. I believe Nvidia only recently has had native FreeBSD drivers - for a long time FreeBSD's kernel Linux support was required.
> I'm genuinely actually curious. FreeBSD exists in kind of a shadow realm for me where I've never been quite able to pin down the soul that keeps it chugging, but I know it exists somewhere in there.
Again, for me, FreeBSD has proven to be a nice blend of the features other BSD's provide as well as being incredibly stable on the h/w platforms I tend to use.
>>I believe Nvidia only recently has had native FreeBSD drivers
When I first started using FreeBSD, in 2004, Nvidia had native FreeBSD drivers for all their boards.
I figured the goldilocks metric might factor in. Are you dealing with non-x86 platforms? I've always been disappointed by the ARM experience on Linux. It always feels second-class.
FreeBSD is throughput oriented in a way that OpenBSD certainly isn't, and I don't think NetBSD is either (although, I haven't really looked, I feel like NetBSD competes on portability and doesn't spend a lot of time making sure networking throughput is high).
All of the BSDs tend to have a lot less churn, for better and worse; so IMHO, they make a nicer platform to integrate on.
Very interesting. Doing more reading on FreeBSD's presence in the HPC space. Thank you.
If you want a high profile example, look at what Netflix CDN does.
Could you do that work with Linux? Probably --- but nobody who does is talking about it as much.
This kind of high throughput service has been a FreeBSD niche since forever too. Walnut Creek CDROM, Inc ran what was reportedly the world's busiest ftp site, ftp.cdrom.com on FreeBSD in the early days of the internet.
Yahoo ran on FreeBSD (I worked there 2004-2011) WhatsApp ran on FreeBSD (I worked there 2011-2019) Both were leaving FreeBSD when I left, but sadly, I didn't leave to work somewhere else with FreeBSD :p
FreeBSD has a much bigger userbase than openbsd or netbsd. Not even comparable.
Its software catalog is also much bigger. It's a viable modern desktop daily driver and I can't say that for the other two.
As to why not Linux? I don't want Linux. It's too bogged down by corporate interests.
My impression is that FreeBSD is Apple's shadow in FOSS, they hold a lot of soft power over it. I know the kernels are different and obviously only part of the userspace is the same, but is FreeBSD actually far enough away from Apple to say it's not bogged down by corporate interests? I don't imagine it's the same as Linux at all, but it exists in a non-trivial way, no?
It's not. Apple (or rather NeXT) took some of the userland for macOS but it's not contributing back and it doesn't have much influence. It's more like a fork a long time ago.
A few companies do. Skype and Netflix did but hardly use it now (at least Skype left it, not sure about Netflix but I never hear about it from bsd devs). Ix systems and netgate do but they're tiny.. No, it's not influenced in a trivial way and certainly not by apple.
This is a huge difference to Linux where the vast majority of kernel commits come from big tech and have nothing to do with things end users care about. Also there's nothing in the FreeBSD world like the Linux Foundation which is basically a corporate lobby group.
> but it's not contributing back and it doesn't have much influence.
I understand the former. But with how Apple operates, it's really hard to believe they'd pull downstream from something they don't have some kind of soft power over. They do still pull downstream AFAIK? Maybe that's changed?
>Ix systems
I did some reading and saw a FreeBSD contributor ended up going to Apple until 2013 before he founded this company. https://www.ixsystems.com/clients/ Apple is listed here. Six degrees of separation and all, but probably not a coincidence. Nothing wrong with that, business is a social structure. This is how they work. We make and keep friends, even if only professionally. Backchannels are where real deals are made. But this to me is not nothing. No corporate influence means there's a lot of nice things you don't get. You just can't afford the manpower. It looks more like 9 Front than a BSD that has some serious billion-dollar problems under its belt.
That sounds harsh, not a judgement. Just very deep skepticism of the assertion of no influence. I'm realizing there's not a lot that can be done to sway that intentionally.
> This is a huge difference to Linux
This I'm well aware of. I just like having a perspective across the fence. These days they're starting to get a little too aggressive for my tastes. FreeBSD seems fine in comparison.
> But with how Apple operates, it's really hard to believe they'd pull downstream from something they don't have some kind of soft power over. They do still pull downstream AFAIK? Maybe that's changed?
Apple doesn't merge often. They basically haven't merged kernel tcp since 2002. When I started using OSX in 2011, they hadn't merged userland for several years, and when I stopped in 2019, they had only merged once.
They famously stopped picking up bash when upstream changed the license, and most of the FreeBSD userland doesn't change that frequently, so most things you wouldn't notice a difference. cal(1) started highlighting the current day at some point, tar probably grew new compresion arguments, etc.
Apple certainly was a major contributor/driving force/etc of LLVM for a while, not sure if they still are? And LLVM was adopted by FreeBSD, so maybe that's where this idea is coming from?
> And LLVM was adopted by FreeBSD, so maybe that's where this idea is coming from?
Partially, but after seeing the Jordan Hubbard connection, there's a lot of layers to this. May have reinforced my biases, but it's definitely non-trivial according to my hippie-tier anarchist baseline. Oops. Worst case scenario of answering your own question.
But your reply does give me actually contradicting evidence. It wouldn't surprise me that distance has grown to the point of total atrophy, given the general trajectory Apple has been on since 2012 or so. This is why I ask these questions, because the people on the ground give the most informative answers.
As Ptahhotep advises circa ~2300BCE:
> Fine words are more sought after than greenstone, but can be found with the women at the grindstone.
> May have reinforced my biases, but it's definitely non-trivial according to my hippie-tier anarchist baseline.
The definition of 'trivial' would come into play yes. I would only consider it non-trivial if a commercial party can (and does) influence the direction of development. I don't think Apple does so. Even Netflix. In the Linux world there's billions of investment and many contributors are directly employed by big business. The waters are much murkier there.
Again, I'm not saying it's a bad thing. It's just not something I want which is one of the reasons I picked FreeBSD. Other reasons were the great ports collection, the division between OS and apps (you can have rolling apps but a stable OS), the traditionalism (only change things if it's really needed) and the single main flavour of the OS which makes support much easier. Also the excellent documentation.
> No corporate influence means there's a lot of nice things you don't get.
Yes that is the flipside. But I don't mind that. If you choose your hardware carefully it works fine.
Note that this is not too different from using Windows or Mac. Your hardware is also chosen carefully to work with those, just not by you but by the vendor. With FreeBSD you're more involved with the nuts & bolts and this is exactly what I want. I don't want my OS to be a black box I don't understand.
>>Netflix did but hardly use it now
Unless I misunderstood you, Netflix delivers all video content via FreeBSD and contributes code back and money to the foundation.
I had read they were migrating to Linux but I could be wrong. I just never hear much about Netflix in FreeBSD circles.
If they still use it they are more of a user than an influencer in its development.
> My impression is that FreeBSD is Apple's shadow in FOSS, they hold a lot of soft power over it.
Apple has no influence over the FreeBSD project.
> I know the kernels are different and obviously only part of the userspace is the same, but is FreeBSD actually far enough away from Apple to say it's not bogged down by corporate interests?
Yes.
OS-X (now macOS) is based on XNU[0], which itself has roots in the Mach[1] microkernel. The Unix user-space programs distributed with OS-X/macOS are those found in FreeBSD distributions AFAIK. This is also conformant with FreeBSD licenses for same.
So there is no "soft power" Apple has over FreeBSD. And FreeBSD is not "Apple's shadow in FOSS".
> I don't imagine it's the same as Linux at all, but it exists in a non-trivial way, no?
No. It does not.
EDIT: Just in case you'd like to verify any of the above yourself, see here[2].
0 - https://en.wikipedia.org/wiki/XNU
1 - https://en.wikipedia.org/wiki/Mach_(kernel)
2 - https://github.com/apple-oss-distributions/xnu
I'm not sure where you're getting the "Apple holds soft power over FreeBSD" thing from. Netflix is probably at the top of the list given all their performance and stability work -- and, you know, the fact they push a large chunk of all Internet traffic using FreeBSD -- and NetApp and Juniper are somewhere up there, but I'm not convinced Apple would even be in the top 10.
> I'm not sure where you're getting the "Apple holds soft power over FreeBSD" thing from.
The only thing I've ever heard from FreeBSD-land, not paying attention to users, but the maintainers and the tools. Apple comes up. In the same manner that RedHat and others come up for Linux. How to explain? It's an abstract pattern. Transparent, understandable.
I mentioned somewhere about the connection through ix systems. And honestly to project, if I was a maintainer of something used between Netflix and Apple, I'd prioritize Apple. Apple has outlived IBM. If you know your history, you know how serious that is. If you've got authority over something as large as FreeBSD? Yeah, you don't ignore that kind of actual power especially when it's personal. Like I say, all based on guesses. But some things are hard to mistake.
Apple did very important work making LLVM happen, but that was a long time ago. At this point there are lots of companies involved in that project.
As far as "power" is concerned... speaking as release engineer, I don't give special treatment to anyone; nor have I even been asked to. If anyone has a special relationship it's Netflix but if anything that's the opposite way around: "Can you please throw 10% of all Internet traffic at this TCP stack patch and let us know if anything breaks" is a thing. They're incredibly helpful with Q/A.
Consider me convinced. Like I say, it was never anything but empirical skepticism. Neither for or against until sufficient evidence has been collected, as painful as that can be. Thank you for your work on scrypt.
I know some maintainers of the userland and apple never comes up. Most maintainers have no commercial ties to anyone so they don't really care abour corporate influence anyway. They just maintain the software because they like using it. This is exactly the kind of thing I like, in Linux there are too many companies putting money into it because they want to make money back (usually not from normal users but from cloud instances, steering the project in a direction away from its grassroots origins).
I'm sure if Apple wants something it would be considered but there would be a strong validation of "what's in it for us" on the freebsd side. There's also some pretty bad experiences with corporate influence and this is reviewed a lot more independently since the netgate wireguard disaster. https://arstechnica.com/gadgets/2021/03/buffer-overruns-lice...
Unlike in the Linux world where RedHat and canonical are so embedded due to most of the devs working for them that there will be a lot less questions. And not just those two, also companies like Huawei are heavy kernel contributors.
I'm not saying it's bad to have such commercial influence. But it's not what I want for the OS I run.
> If the answer is support for stuff like ZFS, Nvidia drivers, ELF, etc. why not Linux?
FreeBSD has better ZFS support than Linux, because it doesn't have the licensing issues.
For me, who's daily driven FreeBSD in the past, and switched back to it again recently. FreeBSD serves as a refuge from systemd, and the only BSD that is a fairly drop in replacement for linux in terms of software compatibility, as well as the only game in BSD town in terms of support for modern hardware.(though it does significantly lag Linux in this regard still, so YMMV).
As to why I use it over the various systemd free linux distros? Well, there's a couple things. First lot of those distros, like Artix linux say, actually have smaller communities than FreeBSD(I'm guesstimating based on the activity level in their irc channels). The Linux community might be much, much larger than the FreeBSD community, but it's also extremely fragmented.
triggerwarning, hyperbole incoming. Don't bother correcting me, it's a polemic, not a scientific paper
Secondly, for someone like me, who's been using various unix like OSes for two decades, FreeBSD is just a nice, batteries included, well integrated system. Things like jails, Dtrace, ZFS, Bhyve, pf etc. All being in the base install means they're just better integrated with the kernel, and eachother. Most of those things exist for linux, or have equivalents, but they're not all part of the same project. Obviously Dtrace and ZFS originated in Solaris, but they've been made first-class citizens. There's a harmony to FreeBSD that Linux distros lack. Documentation is also very good, all accessible via manpages(no GNU INFO...). And, as I mentioned briefly before. It doesn't have a lot of the cruft that's been added to linux distros over the years(though some of it is available in ports if you want it). In FreeBSD, my experience is actually useful. Things I remember how to do from 5 years ago, 10 years ago, 15 years ago, still work. If I'm on some modern, plug and play linux distro, I have no idea what's going on under the hood any more. All I know is it's not what was going on 5 years ago, which isn't what was going on 10 years ago, which isn't what was going on 15 years ago. The amount of pointless churn going on in the linux space is ridiculous. When I started using linux, what I loved about it was that it was transparent. I could change anything. The system was easy to understand. Yes, it was janky, but it was understandable jank, whereas Windows was janky in an opaque way. 20 years later, Linux is still janky, but nothing is understandable, at least not to my greybeard brain. Systemd takes over a new daemon every distro upgrade. DNS resolving now involves 4 different daemons with 15 different configuration files, there's two display protocols, both broken in different ways, /etc is full of long files written in strange, alien languages, and every file has its own bespoke language. There seems to be 54 different ways to make any change to your system, and all of them are somehow unsatisfactory in a unique way. I just can't, anymore. Enough already.
This does not paint a good picture for FreeBSD development. An operating system of that size and complexity surely needs someone company to sponsor the release manager full time, not part time for a year.
I don't say this to besmirch FreeBSD, FWIW. I think it's very important that Linux is not the only game in town.
40 hours/month. Not week.
colin, fix your encoding
Err, I'm happy to fix things, but can you elaborate on what is broken?
I may be wrong about OPs intention, but AFAICT, because no encoding is specified, the client gets to choose. For someone not using a default encoding that's a superset of ASCII (like ISO-2022-KR) the page appears as a �.
Current practice is to put a meta tag with your encoding, use a Unicode BOM, or less favorably, send the charset attribute in the Content-type header.
The em-dashes were mojibaked. They seem fine now. Cheers.