Proxmox VE: Import Wizard for Migrating VMware ESXi VMs (opens in new tab)

I run macOS/OSX Sonoma in Proxmox. It does pcie/gpu passthrough (AMD rx580 8gb). The proxmox host is running a AMD Ryzen 7 5700G cpu and has 64gb memory. The mac vm disks are on a zfs fs on a wd black nvme ssd.

It's the fastest mac I've ever owned and it's virtual, executed on a machine running a chip that apple never supported, and you'd never be able to tell it was a vm unless you were told so. kvm and vfio are amazing pieces of software.

A good place to start: https://github.com/luchina-gabriel/OSX-PROXMOX

The lack of 3D paravirtual devices is a real sore spot in kvm. To my knowledge, virgl still isn't quite there but is all there is so far in this space. VMware has the best implementation IMO and everything else is a step down.

Note that macOS guest support ended with ESXi 7.0: https://kb.vmware.com/s/article/88698

Running macOS is only supported/licensing-compliant on Apple-branded hardware anyway, and with the supported Intel Macs getting pretty old this was inevitable anyway.

The SPICE backend has decent OpenGL 3D support with software rendering, it's slow but it works for simple graphics. It's intended for 2D so the desktop's pretty fast IMO. That only works for Linux and Windows guests though, not Apple ones.

MacOS VMs do work in Proxmox with a Hackintosh setup but you pretty much have to passthrough a GPU to the VM if you're using the GUI. Otherwise you're stuck with the Apple VNC remote desktop and that's unbearably slow.

You can run macos vms under proxmox.

You can also accelerate video (though it's uncertain what "pure software" means?)

this guy: https://www.nicksherlock.com

has been writing up proxox vms running macos for a while, the latest being:

https://www.nicksherlock.com/2022/10/installing-macos-13-ven...

yes, it uses opencore.

I haven't really bothered with macos GPU acceleration, It is possible but a little fiddly with an nvidia card. I mostly rely on screen sharing to display the remote vm and that's really good (rich text copy/paste, drag and drop files, etc)

As to general guest 3d, you can do this with GPU passthrough, and although it's technical the first time, each vm is then easy.

basically I add this line to each VM for my graphics card:

  hostpci0: 01:00,pcie=1,x-vga=1

and this passthroughs a USB controller for my USB dac:

  hostpci1: 03:00,pcie=1

(this is specific to my system device tree, it will usually be different for each specific machine)

The passthough has worked wonderfully for Windows, ubuntu and arch linux guests, giving a full accelerated desktop, and smooth gaming with non-choppy sound.

two things I wish proxmox did better:

- passthrough USB into containers

usb is pretty fiddly to propagate into a container. It is marginally better with VMs but sometimes devices still don't show up.

- docker/podmain containers

proxmox has lxc containers, but using them is a little like being a sysadmin. Docker/podmain containers that you can build from a dockerfile would be a much better experience.

Nested virtualization also works great on ESXi for macOS guests ( so you can run docker desktop if so inclined ). I believe this is possible with proxmox as well with CPU=host but have not tried it.

For graphics, another cool thing is intel iGPU pci passthrough - I have had success with this when running esxi on my 2018 mac mini https://williamlam.com/2020/06/passthrough-of-integrated-gpu...

* Proxmox can emulate VMWare adapters. https://pve.proxmox.com/wiki/QEMU/KVM_Virtual_Machines

* If you don't mind dedicating the video card to a VM, you can do PCI-passthrough. https://pve.proxmox.com/wiki/PCI_Passthrough

Apple has been adding a lot of virt and SPICE things IIRC. Some of it isn't supported in VMware (it lacks a bunch of standard virt support), but the facilities are growing instead of shrinking which is a good sign.

On Proxmox you can do the same. You're going to need OpenCore if you're not on a Mac indeed. But if you're not on a Mac you're breaking the EULA anyway.

Does this work with VMWare Workstation as well? I'd love to run macOS in a VM on my Linux desktop for the few apps I have to use on macOS...

With a million dollar per year to play with, you should ultimately be able to replace all of these. Especially since it's not like Proxmox is lacking its own third-party support options (but it being built on FLOSS tech still leaves you with a lot more flexibility).

I'll agree in theory but to play devil's advocate:

* I did the VCP4 and 5 courses. It's entirely a sales certification. I mean it's a technical certification, but I've never run into anyone who certified for the purpose of running an organisation's tech. Rather, you certify for the purpose of your company being able to sell the product. Note also much of VMware's training focus lately has been on things outside their main virtualisation, like Horizon or their MDM product. * Accurate. But I don't think it'll be far off. * Proxmox does Ceph out of the box. I'll also add that it's very easy to manage, unlike vSAN. I'll further add that none of the VMware training and certifications I've ever done covered vSAN, all the courses assume someone bought a SAN. * All the "hybrid cloud" pushed at least by Microsoft completely assumes you're in Hyper-V and is irrelevant to VMware * I've consulted to an awful lot of VMware organisations and I've never seen servicenow integration in place. I'm sure it's relevant for some peopel.

All of those points would also assume:

* You are big enough to need that and actually implement it

* You have the budget to do so

* You actually have the need to do that in-house

If you are at that scale but you don't have the internal knowledge, you were going to get bitten anyway. If you are not at that scale, you were already bitten and you shouldn't have been doing it anyway.

> It's the ecosystem.

Definitively, and situations like the Broadcom one IMO just underline that as a company you should never ever get your core infra locked into proprietary vendors' ecosystem, as that is just waiting for getting squeezed out, which they can for the reasons you laid out.

> Your outsourced VMware-certified experts don't actually know that much about virtualization (somehow).

That should be a wake-up call to have some in-house expertise for any core infra you run, at least as a mid-sized, or bigger, company. Most projects targeting the enterprise, like Proxmox VE, provide official trainings for exactly that reason.

https://proxmox.com/en/services/training

> * Your backup software provider is just now researching adding Proxmox support (https://www.theregister.com/2024/01/22/veeam_proxmox_oracle_...)

Yeah, that's understandable, one wants to avoid switching both, the hyper-visors that hosts core-infrastructure and the backup solution that holds all data, often even from the whole period a company needs to legally save that.

But as you saw, even the biggest backup player sees enough reason to hedge their offerings and takes Proxmox VE very seriously as alternative, the rest is a matter of time.

> A few years ago you 'reduced storage cost and complexity' by moving to VMware vSAN, now you have a SAN purchase and data migration on your task list

No, you should rather evaluate Proxmox's Ceph integration instead of getting yet another overly expensive SAN box. As ceph allows you to also run a powerful and near indestructible HCI storage, but avoids any lock-in as Ceph is FLOSS and there are many companies providing support for it and other hyper-visors that can use it.

> * The hybrid cloud solution that was implemented isn't compatible with Proxmox. > * The ServiceNow integration for VMware works great and is saving you tons of time and money. You want to give that up?

That certainly needs more work and is part of the chicken and egg problem that backup support is (or well, was) facing, but also somewhat underlines how lock-in works.

> * Can you live without logging, reporting, and dashboards until your team gets some free time?

Proxmox VE has some integrated logging and metrics, and provides native support to send to external metrics server, we use that for all of our infra (that runs on a dozen PVE servers in various datacenters around the world) with great success and not much initial implementation effort.

So yeah, it's the ecosystem, but there are alternatives for most things and just throwing up your hands only signals to those companies that they can squeeze you much tighter.

Your outsourced experts are actually just people with google.

Proxmox on zfs means zfs snapshot send/receive, simple. I made my own immutable zfs backup system for £5

Unprivileged LXCs? Interesting, I thought containers would require privileged LXC. At least, that it my takeaway from trying to run Podman in a nesting enabled, but unprivileged LXC under non-root user. I kept running into

> newuidmap: write to uid_map failed: Operation not permitted

I tried googling it, tried some of the solutions, but reached the conclusion that it's happening because the LXC is not privileged.

Tons of products like this have existed for years. Virt-v2v (which I wrote) since 2007/8, Platespin, Xen XCP, AWS's tooling, Azure Migrate etc.

VMware has a track record of pretty great reliability across a vast array of hardware. Yes, the APIs suck, but they're a case study on tech debt: vSphere is basically the Windows equivalent of datacenter APIs. They chose the best technology at the time (2009, which meant SOAP, Powershell, XML, etc) and had too much inertia to rework it.

The VMWare APIs are indeed pretty bad, even the ones on their modern products for some reason (i.e. NSX etc.) where they did adopt more modern methods but still managed to pull a Microsoft with 'one API for you, a different API for us'.

Being pretty bad doesn't mean they don't work of course, but when the best a product has to offer is clickops, they have missed the boat about 15 years ago.

I really hope that the price increase creates a business opportunity for new technology. This space has been plagued by subpar "free" alternatives (Openstack, Kubernetes) for a decade.

I can't concur. VMware was the leader in virtualization technology for a long time, and honestly nothing is quite as simple to start with as ESXi if you've never used a type 1 hypervisor before. I'm not so familiar with the APIs, so perhaps you're correct in that sense.

By application do you mean vCenter? It's in an entirely different league than proxmox.

https://i0.wp.com/williamlam.com/wp-content/uploads/2023/04/...

https://www.reddit.com/media?url=https%3A%2F%2Fpreview.redd....

i would disagree with you there, especially because there is very little on the sdn front which matches NSX-T in terms of SDN capabilities, this is something in which vmware has been ahead, the only other people with the same capabilities seem to be hyperscalers.

We went from $66 last year to $3600 this year.

There won’t be another year.

Proxmox documentation does mention OVF support https://pve.proxmox.com/wiki/Migrate_to_Proxmox_VE#Import_OV...

Seems a bit barebones, as in no support for a nice OVF properties launch UI, but one should be able to extract an OVA to an OVF and VMDK an manually edit the OVF with appropriate properties.

I actually had plans this week to try exactly that...

Do you have any more info on this KVM back end for Virtualbox? I love Virtualbox (I know, I know) but the one annoying thing is the dependency on out of tree kernel modules (at least they’re open source though).

From the post in case you missed it:

Q: Will other import sources be supported in the future?

A: We plan to integrate our OVF/OVA import tools into this new stack in the future. Currently, integrating additional import sources is not on our roadmap, but will be re-evaluated periodically.

It never takes as little as you (or others, myself included) think it should. Big companies have a lot of inertia and changing anything which is working today, even if it saves a lot, attached your name to the risk it will fail horribly, so you'd be reluctant to suggest it, esp. since it's usually not your own money (your budget maybe, but not your own money).

Broadcom knows this very well and likely turned the price screw exactly right - just before the breaking point for the critical mass of their customers.

What I think will lead to the eventual implosion of VMware's market share, on a longer timescale, is the removal of free ESXi. Many people acquire familiarity with small scale/home/demo labs or PoC prototypes, then they recommend going with what they're familiar with. This led Microsoft where they are now, by always giving big discounts to students and never going too hard on those running cracked copies. They saw it as an investment and they were bloody right. If the product had been better it would completely dominate now, but even as shoddy as it is, it's a huge cash cow.

All of AVGO/Broadcom's moves with VMware have been to keep revenue somewhat steady by focusing on their biggest customers locked into their ecosystem, while drastically cutting back everything else to lower expenses. This should produce excellent short-term financial results which the market will very likely reward with a higher stock price over the next year or two. The board and C-suites know what they are doing.

Of course, destroying the trust they had with their customers means the long-term prospects of the VMware are not so good.

I'd exercise caution, in my experience, it'll take years for companies to transition from VMware to somewhere else. In the interim, their revenue will most likely pop as they're squeezing the shit out of these unlucky souls.

It’s not about whether you think their revenue will crater (or any other fundamentals).

It’s about answering the question: Why is the current price of puts wrong?

All of the most popular Linux distros tend have the virtio drivers installed by default.

Oh, that would be a smart move. :)

If it doesn't, any idea if it's something they could automate easily?

proxmox is using LVM for direct attached raw volumes. LVM is just a logical volume manager for linux, which gives you more features than using old fashioned disk partitioning. I guess they chose this path for windows virtual machine migration because windows running on vmware before, does usually not have the required virtio drivers installed to support the qemu hypervisors virtio solution for disk bus virtualization out of the box. It would mean the hypervisor has to simulate IDE or SCSI bus which comes with great overhead perfomance wise (in the case of migration)

So an direct attached lvm volume is the best solution performance wise. In the vmware world this would be an direct attached raw device either from local disk or SAN.

For fresh install on proxmox its better to chose qcow as disk image format with virtio-scsi bus (comparable to vhdx, vmdk, qemus disk format) and add virtio drivers during windows setup.

i ran into the same sort of documentation desert.

off the top of my head:

- keep in mind there is LVM and LVM2, and proxmox now uses lvm2

- I don't understand the thinpool allocation. You don't have to use lvm-thin if you don't want to deal with oversubscribed volumes, or don't care about snapshots or cloning storage.

- get to know "pvesm". A lot of things you can do in the gui

- when making linux VMs, I found it easier to use separate devices for the efi partition and the linux partition, such as:

  efidisk0: local-lvm:vm-205-disk-0,size=4M
  virtio0: local-lvm:vm-205-disk-1,iothread=1,size=1G
  virtio1: local-lvm:vm-205-disk-2,cache=writeback,discard=on,iothread=1,size=32G

(virtio0 = efi, virtio1 = /)

and I can mount/expand/resize /dev/mapper/big2-vm--205--disk--2 without having to deal with disk partitions

If you're open to alternatives, Proxmox does HA.

It also has some decent clustering capabilities enabling online VM migration between hosts (equivalent to VMotion), which can go a long way towards solving related use cases. :)

I see plans with access to the Enterprise repos starting at €110/yr, and plans with 3 support tickets starting at €340. €1020 is the starting price for a plan with a 2hr SLA.

https://shop.proxmox.com/index.php?rp=/store/proxmox-ve-comm...

For non production purposes, you're probably fine to use the the "No subscription" repositories. They're an official thing, although their website seems to go out of it's way to not mention it.

You don't actually need a subscription to run Proxmox, it's FOSS software after all.

I think "adventure" is how I'd put it, too. Perhaps that which I found most surprising was the difference in defaults between the two. ESXi gave me what I considered pretty good defaults, where Proxmox were more conservative or generic (struggling to find the right word). For example, I was surprised that I had to pick an option for CPU type instead of it defaulting to host, which I would have expected. Saying that, I never checked on ESXi, but I never had reason to look in to performance disparities there.

Once I found the surface, I have really grown to like it, expanding my footprint to use their backup server, too. Proxmox makes you work for it, but is worth it.

I’ve been doing that for almost two years now, including ARM nodes (via a fork). It’s been awesome, and even though I am fully aware Proxmox does not match the entire VMware feature set (I used to migrate VMware stuff to Azure), it has been a game changer in multiple ways.

Case in point: just this weekend a drive started to die on one of my hosts (I still use HDDs on older machines). I backed up the VMs on it to my NAS (you can do that by just having a Samba storage entry defined across the entire cluster), replaced the disk, restored, done.

Your experience is very relatable. My first Proxmox adventure began with installing Proxmox 8 on 2 hetzner boxes: one with CPU, one with GPU. Spent two straight weekends on the CPU box and just when I was about to give up on proxmox completely I had a good night’s sleep and things finally ‘clicked’. Now I’m drinking the proxmox koolaid 100% and making it my go-to OS.

For the GPU box I completely abandoned the install after attempting to do the gymnastics around GPU passthrough. I like Proxmox but I’m not a masochist - Looking forward to the day when that just works.

I appreciate projects like Proxmox but it must be also said that you can achieve same functionality sans the UI with tools available on most Linux distributions: libvirt, lx{c,d}, podman etc.