Got me down a rabbit hole of trying to find the ROI of doing something like this. This is what every GM at AWS thinks about. So, a back of the napkin calculation of running at FP8 and approx 35GB. You can fit that on a g6e.xlarge @ 48GB for about $900/year at reserved capacity.
Assuming you are netting at $5.5 (post stripe fees), and maybe $100/month for API/auth box for egress, you probably need about 170 paying customers to breakeven. This is assuming light/bursty usage.
Kudos! This is a a win win for both you and customer using it as it removes the undifferentiated heavy lifting.
What type of hardware, GPU are you utilizing? I have the same model self hosted at home
Got me down a rabbit hole of trying to find the ROI of doing something like this. This is what every GM at AWS thinks about. So, a back of the napkin calculation of running at FP8 and approx 35GB. You can fit that on a g6e.xlarge @ 48GB for about $900/year at reserved capacity.
Assuming you are netting at $5.5 (post stripe fees), and maybe $100/month for API/auth box for egress, you probably need about 170 paying customers to breakeven. This is assuming light/bursty usage.
Kudos! This is a a win win for both you and customer using it as it removes the undifferentiated heavy lifting.
[dead]
[dead]