Page History

3.

Create Cloud Profile

• Use profiles → New to create a cloud profile that provides relevant information for accessing the cloud storage from application.  To start with the configuration you would need access ID  key and secret key generated by your cloud provider

• from their console settings. It is also possible to skip providing this information, if the compute engine was

• already authorized to perform

• S3 object access operations through IAM settings.

  Tip
  title AWS EC2
  For AWS EC2 instance created with IAM role that authorizes S3 operations it may not be required to configure the accessId and accessKey within the software. Instead use the option --accessEC2IAM=ROLE option with objbaker. This will acquire S3 credentials from EC2 machine

• Create access file with credentials

Create file v:\etc\.s3wasabi_passwd using notepad and add the access information in this format accessId:accessKey. The accessId could be account ID, and the accessKey is base64 encoded secret.

• Location of SSL certficate bundle
  For SSL to work the curl library needs to know the location of its CA CERT bundle, which is installed in c:\program files\crossmeta.  When you run the objbaker this information is provided with --cacert=ca-bundle.crt option . While running interactively since the startup directory is c:\program files\crossmeta this would work. But when the program is running as NT service the startup directory would be c:\windows\syswow64. In that case either copy ca-bundle.crt to c:\windows\syswow64 or specify full pathname to --cacert option.
  

  List buckets with objbaker

  Before configuring object disk with Windows, make sure objbaker can list the buckets from your storage account.  The baseURL is the most important parameter and should match with access URL of the cloud provider.  You should use HTTPS for secure SSL access.

  Info
  title Info
  For Azure baseURL will contain the <storage-account-name>.blob.core.windows.net The baseURL should end with slash character / The baseURL option can be skipped for Amazon AWS alone

  Azure

  Code Block
  language	text
  theme	RDark
  title	instance-data via http://169.254.169.254/latest/meta-data/iam/security-credentials/ROLE every five minutes.

  
  Use profiles → New to launch the following dialogs.

  
  Image Added
  
  
  Image Added
  
  
  The following dialog shows how to configure cloud profile for Wasabi Technologies which comes under Other S3 generic category. In this configuration BaseURL is required to match their service URLs
  
  Image Added
  
  
  Once configured it will provide a list of bucket/container names that is currently present.
  
  Image Added
  
  

4. Configure your first object storage disk

Select the cloud profile to use for creating object storage disk.  You can select an existing Bucket from the list or enter one manually which will create a new bucket for you.  Logical size of VHDX up to 64TB allowed by Windows. Fill in the following information for the new Disk

1. Enter Disk name, which maybe same as bucket name or different.
2. If you selected existing bucket the Bucket will be autofilled, otherwise enter a new bucket name to be created.
3. Enter Size for the Disk representing the Bucket.  This is only a logical size while the bucket storage size is physical. Logical size of VHDX up to 64TB allowed by Windows.
4. Select the Blocksize of the physical blocks in the cloud bucket.  This would normally match the application blocksize for optimal performance.  For NTFS it would match the cluster size of the file system.
5. Click Create to proceed with the configuration.

Image Added

5. Register the object storage disk as service

• Once cloud disk was setup Register it as service program so that it is available automatically on system startup.
  
  Right-click on Disk and select Register from popup menu.
  
  Image Added
  
  
  
• In the Register cloud disk service you may accept the default values or change accordingly.
  
  
  Image Added
  
  
  

• To start the service Right-click on Disk and select Start from popup menu.

  Warning
  title Reset-mounted flag
  If starting the service after unclean shutdown of earlier successful mount it is required to clear the mounted flag by selecting Reset-Mounted menu option and then attempt to start the service.

  
  

  After the above command finishes the object storage is accessible as V:\mnt\file.vhdx as one giant file and in this example it shows logical capacity as 10TB.
  
  

  Code Block
  title qemu-img
  C:\Program Files\crossmeta>objbacker.exe --cacert=ca-bundle.crt --ssl --accessFile=/etc/.s3azure_passwd --s3Type=azure --baseURL=https://zfstest.blob.core.windows.net/ --bucketCommand=list

  Code Block
  language text theme Confluence title AWS
  C:\Program Files\crossmeta>objbacker.exe --cacert=ca-bundle.crt --ssl --accessFile=/etc/.s3amazon_passwd  --authVersion=aws2 --baseURL=https://s3.us-west-1.wasabisys.com/ --bucketCommand=list

  Code Block
  language text theme RDark title Google Cloud
  C:\Program Files\crossmeta>objbacker.exe --cacert=ca-bundle.crt --ssl --accessFile=/etc/.s3google_passwd --s3Type=google_s3 --baseURL=https://storage.googleapis.com/ --bucketCommand=list

  Code Block
  language text theme Confluence title Wasabi
  C:\Program Files\crossmeta>objbacker.exe --cacert=ca-bundle.crt --ssl --accessFile=/etc/.s3wasabi_passwd  --baseURL=https://s3.us-west-1.wasabisys.com/ --bucketCommand=list

  For troubleshooting add --debug-http to the above command to get detailed REST API output.
  

  Configure object storage as VHD

  Before configuring object disk with Windows make sure the Crossmeta Fuse driver has started. If required start the driver as follows
  

  Code Block
  language text theme RDark title FUSE
  C:\Program Files\crossmeta>net start cxfuse The CROSSMETA FUSE kernel mode driver service was started successfully.

  Options to objbaker
  

--blockCacheSize=100 

azure

google_s3

amazon_s3

Access URL for the object storage.  Must be specified for all S3 object provider other than Amazon

Azure blobs contain the storage account name in the URL.

--baseURL=https://zfstest.blob.core.windows.net/

• Mount object storage bucket
  For Windows to make use of  object storage bucket  we have to intialize the object storage as VHDX image.  VHDX is preferred over VHD because of inherent journal feature to protect against unsafe shutdown and also it can support up to 64TB for a single image.  This step is performed in foreground interactively to make sure it is working before configuring it as service later.  In the following example object storage bucket smbtest1 is being mounted. Please make note of the -f option for foreground.  Also for filename we chose as extension .vhdx

  Code Block
  title using bucket smbtest1 from Wasabi
  C:\Program Files\crossmeta>objbacker.exe --cacert=ca-bundle.crt --accessFile=/etc/.s3wasabi_passwd --authVersion=aws2 --baseURL=https://s3.us-west-1.wasabisys.com/ --directIO --noAutoDetect --size=10T --timeout=30 --blockCacheSize=100 --blockCacheThreads=8 --minWriteDelay=5000 --md5CacheSize=1000000 --ssl --blockSize=1024K -o no_pagecache --filename=file.vhdx  smbtest1 /mnt -f objbacker.exe: auto-detection disabled; using configured block size 1m and file size 10t INFO: s3backer process 5312 for /mnt started INFO: mounting /mnt

  After the above command finishes the object storage is accessible as V:\mnt\file.vhdx as one giant file with logical capacity as 10TB.

  Initialize VHDX on the object storage bucket using qemu-img program available in c:\program files\qemu

  Code Block
  title qemu-img
  cd c:\program files\qemu C:\Program Files\qemu> qemu-img create -f vhdx v:\mnt\file.vhdx 10T

  Verify VHDX information
  

  Code Block
  title qemu-img
  c:\Program Files\qemu> qemu-img info v:\mnt\file.vhdx

  Activate VHDX file as virtual disk in Windows from Administrator PowerShell window
  

  Code Block
  language powershell title Attach VHDX
  c:\users\Administrator> mount-diskimage -imagepath v:\mnt\file.vhdx

  Create a file system NTFS or ReFS

  qemu>dir v:\mnt Volume in drive V is vfsroot Volume Serial Number is 0122-B5C1 Directory of v:\mnt 05/22/2019 07:48 PM <DIR> . 05/22/2019 07:48 PM <DIR> .. 05/22/2019 07:48 PM10,995,116,277,760 file.vhdx 05/22/2019 07:48 PM 1,422 stats 2 File(s) 10,995,116,279,182 bytes 2 Dir(s) 5,242,880 bytes free

6. Configure the object storage disk as VHDX

For Windows to make use of  object storage bucket  we have to initialize the object storage as VHDX image.  VHDX is preferred over VHD because of inherent journal feature to protect against unsafe shutdown and also it can support up to 64TB for a single image

• To create new VHDX image click mount/umount from the treeview pane.
• Select the Disk and Right-Click for the popup-menu and choose Create VHD menu option

  
  Image Added

7. Mount the VHDX image of object storage disk

• Select Disk and Right-Click to choose Mount VHD menu option to attach the VHDX image as Disk to Windows.
• Only for newly created VHDX image
  Launch diskmgmt.msc to initialize disk for the first time and assign a drive letter.  You may create a file system within Disk Management GUI or from Command Prompt.
  
  Image Added
  
  
  

  To Create a file system NTFS or ReFS from command prompt. Use the actual drive letter that was assigned from the above command and allocation size of 1MB is chosen to match with the block size used while configuring cloud disk.

  Code Block
  language powershell title Format
  c:\users\Administrator> format /fs:ntfs /A:1M /q F:

  Warning
  title NTFS allocation size
  Windows 2019 supports allocation size of 1MB for NTFS file systems while earlier version limited to 64KB only.  This means NTFS created with 1MB allocation size may not be recognized by older Windows.

  Detach the virtual disk
  

  Code Block
  language powershell title Detach VHDX
  c:\users\Administrator> dismount-diskimage -imagepath v:\mnt\file.vhdx

  Stop the objbaker program to proceed configuration as service

  From Desktop → Crossmeta command shell running as administrator

  Code Block
  language powershell title Stop objbaker
  V:\>mount /dev/loop0 on / type ufs (local, read-only) devfs on /dev type devfs (local) \fuse.chan3596 on /mnt type fuse V:\> umount /mnt

  The above command would stop the objbacker program that is running inside another Command Prompt Window.
  

4. Register as NT service

• Create service using sc.exe

  If everything worked out fine so far we are ready to make this object disk as service now.  Pick a service name to start this object disk as NT service program automatically from now on. In this example we register objdisk1 as NT service program.

  In the following command line for sc.exe please note that binPath=  and displayName= has no space character before = but space character after =

  Code Block
  language powershell title Create service
  C:\Program Files\crossmeta>sc create objdisk1 start= delayed-auto binpath="\"c:\program files\crossmeta\objbacker.exe\" --cacert=\"c:\program files\crossmeta\ca-bundle.crt\" --accessFile=/etc/.s3wasabi_passwd --baseURL=https://s3.us-west-1.wasabisys.com/ --directIO --noAutoDetect --size=10T --timeout=30 --blockCacheSize=100 --blockCacheThreads=8 --minWriteDelay=5000 --md5CacheSize=1000000 --ssl --blockSize=1024K -o no_pagecache --filename=file.vhdx smbtest1 /mnt" displayName= "My first object disk smbtest1 from Wasabi" depend= cxfuse [SC] CreateService SUCCESS C:\Program Files\crossmeta>sc qc objdisk1 [SC] QueryServiceConfig SUCCESS SERVICE_NAME: objdisk1 TYPE : 10 WIN32_OWN_PROCESS START_TYPE : 2 AUTO_START (DELAYED) ERROR_CONTROL : 1 NORMAL BINARY_PATH_NAME : "c:\program files\crossmeta\objbacker.exe" --cacert="c:\program files\crossmeta\ca-bundle.crt" --accessFile=/etc/.s3wasabi_passwd --baseURL=https://s3.us-west-1.wasabisys.com/ --directIO --noAutoDetect --size=10T --timeout=30 --blockCacheSize=100 --blockCacheThreads=8 --minWriteDelay=5000 --md5CacheSize=1000000 --ssl --blockSize=1024K -o no_pagecache --filename=file.vhdx smbtest1 /mnt LOAD_ORDER_GROUP : TAG : 0 DISPLAY_NAME : My first object disk smbtest1 from Wasabi DEPENDENCIES : cxfuse SERVICE_START_NAME : LocalSystem

sc start objdisk1
powershell  mount-diskimage  -imagepath v:\mnt\file.vhdx

Stopping the objdisk service

From Desktop → Crossmeta command shell running as administrator

v:\> umount /mnt

v:\> sc stop objdisk1

5. Automatic Startup

Create startup script

net start objdisk1
powershell mount-vhdimage -imagepath v:\mnt

Schedule the above as task

schtasks.exe /create /tn "Start Cloud Disks" /ru SYSTEM /Sc ONSTART /tr
"C:\program files\s3disks.cmd"

8. Detach the  VHDX image

To stop the objbaker service program for the disk the VHDX image as to be detached first.

• Select Disk and Right-Click to choose Umount VHD menu option

9. Stop the service of object storage disk 

After detaching the VHDX image it is safe to proceed with stopping the service if needed. 

• Select services → Disk and Right-Click to choose Stop menu option

Related pages