©XSIBackup-Free: Free Backup Software for ©VMWare ©ESXi

Forum ©XSIBackup: ©VMWare ©ESXi Backup Software


You are not logged in.

#1 2022-03-08 05:20:39

jybleau
Member
Registered: 2019-07-08
Posts: 7

Repo on VMFS6 and block size related to file system capacity.

Hello,

Regarding the number of files in a repo vs the capacity of a file system.

is it ok to use the default 1MB block size if a repo is stored on a VMFS6 volumes?
The total size of the VMs to backup is arround 300GB.

Or should I set the block size to 50MB like it was XSIBackup-pro-classic?


Second related question:

If I started backing up in a repo with XSIBackup-DC using a file size of 50MB, can I change it to a smaller file size and still backup into this repo without problems?

Thanks

Offline

#2 2022-03-08 11:22:42

admin
Administrator
Registered: 2017-04-21
Posts: 2,055

Re: Repo on VMFS6 and block size related to file system capacity.

Don't use VMFS volumes to store deduplicated repositories (*), that is suicidal.
Use XFS or ext4 on NFS volumes or over IP directly to any Linux server.
Do not use big block sizes unless you have no other choice, 99% of the times there is no justification to do so.
You can't change the block size of a repo once you have created it, why would you want to do so?

(*) Please, note that you can replicate data to a VMFS volume without problems, as replicas are just a set of a few files.

Offline

#3 2022-03-08 16:26:49

jybleau
Member
Registered: 2019-07-08
Posts: 7

Re: Repo on VMFS6 and block size related to file system capacity.

Damn, I did that for the last 2 years using the classic version cool
It was not a big repo, fortunately.

I'll start using the replica action for storing directly on VMFS!


Ok, so, asking for another advise:
If my backup server is ESXI, with plenty of storage, is it viable to think I can put a deduplicated backup repo within a dedicated linux VM using ext4?

I don't have access to a dedicated NAS for backups, yet.

Thanks

Offline

#4 2022-03-09 12:17:01

admin
Administrator
Registered: 2017-04-21
Posts: 2,055

Re: Repo on VMFS6 and block size related to file system capacity.

With the Classic version it was a different thing, as it used a bigger block size.
There are two main concerns when using some FS to store deduplicated content in the FS as a database:

- The FS must be fast
- The FS must be able to handle the data

VMFS is slow, as it was not designed to handle millions of small files, but on the contrary to handle a few big files: the virtual disks.

VMFS is terribly slow when it comes to handle many small files, although VMFS-6 is indeed able to handle millions of files, it is very slow. On the other hand, apart from also being slow, VMFS-5 has an inode limit of aproximately 130.000 files and folders. When it reaches that limit it just blows up.

When using (c)XSIBackup Classic to accumulate data in old XSITools 1.0 repositories all these problems are minimized: first of all cause you are using a big block size (50MB), which reduces the effect of data access slowness by a factor of 50. Also the VMFS-5 inode limit is relative, as 100.000 inodes can store 5.0TB of real data, which may in turn host 15-30 TB in VMs.

(c)XSIBackup-DC is more optimized for speed and data compression. Classic version is a toy when compared to it. It uses a 1MB default block size, which you shouldn't change and which yields awesome compression figures, apart from offering CBT compatibility and a lot of extra features when compared to Classic version.

Still, it uses a block size which is 50 times smaller and this will make VMFS slow behaviour to become obvious and be much slower than expected processing data. In regards to VMFS-5, don't even try to use it for obvious reasons. Just 100GB of real data will make it stop responding.

Offline

#5 2022-03-09 17:08:48

jybleau
Member
Registered: 2019-07-08
Posts: 7

Re: Repo on VMFS6 and block size related to file system capacity.

Ok. Thank you for taking the time to provide a more detailed explanation. smile

I understand better now why it really matters not to store the repo directly on a VMFS.

Offline

#6 2022-03-25 13:10:18

admin
Administrator
Registered: 2017-04-21
Posts: 2,055

Re: Repo on VMFS6 and block size related to file system capacity.

Just an update for anybody reading this.
If you have no other choice, namely: you "really" do not have any other choice than using some VMFS file system to store your deduplicated backups.
(This is literally impossible, as you always can use some VM on top of VMFS and share it via NFS or IP).

So, for the above statement: "NO OTHER CHOICE" to be plausible you would need to be in a hurry and not have time or the knowledge to devise a proper solution, or be pointed at with some gun by some furious guy that wants you to do things wrongly.

If any of the above premises is true and you will still use some VMFS file system against our recommendation and all logic, at least use a big block size of 50MB (--block-size=50M) to minimize the effect of a slow FS and eventually a limited inode number in VMFS-5

Offline

#7 2023-08-11 20:22:56

ccadev
Member
Registered: 2023-08-11
Posts: 1

Re: Repo on VMFS6 and block size related to file system capacity.

I was testing it out on VMFS6 under ESXi6.7 - it was going to run out of inodes really, really quickly.
VMFS-6      18.2T   3.1T     15.1T  17% /vmfs/volumes/datastoresata
vmkfstools -P -v 10 /vmfs/volumes/61df1a4a-37b2d8e1-9015-b083fed151bb
Files (max/free): 326608/6823
And it was going down quickly - caught it before it caused a problem.
I'm going to see if I can get it working on a 28TB Ubuntu22.04 server with 20 cores and 320GB RAM.

Offline

#8 2023-08-15 21:09:09

admin
Administrator
Registered: 2017-04-21
Posts: 2,055

Re: Repo on VMFS6 and block size related to file system capacity.

It is encouraged all over our documentation not to use VMFS as the target for deduplicated backups but some XFS or ext4 file system instead.

You don't need 20 cores nor 320GB of RAM. If you use 20 cores 18 will be iddle. (c)XSIBackup uses two cores. You should better use some low number of cores powerful CPUs, such as some double core Intel Gen above 10th than a CPU with a lot of weak cores such as AMD Epyc series.

In regards to the RAM if you add 320GB you will be wasting 318GB, unless you need to do some really intensive pruning.

Offline

Board footer