📦 deps(thirdparty): update snapshots
This commit is contained in:
ci[bot]
+932
@@ -0,0 +1,932 @@
|
||||
---
|
||||
title: "Oracle Object Storage"
|
||||
description: "Rclone docs for Oracle Object Storage"
|
||||
type: page
|
||||
versionIntroduced: "v1.60"
|
||||
---
|
||||
|
||||
> **Official documentation:** [https://rclone.org/oracleobjectstorage/](https://rclone.org/oracleobjectstorage/)
|
||||
# Oracle Object Storage
|
||||
|
||||
Object Storage provided by the Oracle Cloud Infrastructure (OCI).
|
||||
Read more at <oracle.com>:
|
||||
|
||||
- [Oracle Object Storage Overview](https://docs.oracle.com/iaas/Content/Object/Concepts/objectstorageoverview.htm)
|
||||
- [Oracle Object Storage FAQ](https://www.oracle.com/cloud/storage/object-storage/faq/)
|
||||
|
||||
Paths are specified as `remote:bucket` (or `remote:` for the `lsd` command).
|
||||
You may put subdirectories in too, e.g. `remote:bucket/path/to/dir`.
|
||||
|
||||
Sample command to transfer local artifacts to remote:bucket in oracle object storage:
|
||||
|
||||
```console
|
||||
rclone -vvv --progress --stats-one-line --max-stats-groups 10 --log-format date,time,UTC,longfile --fast-list --buffer-size 256Mi --oos-no-check-bucket --oos-upload-cutoff 10Mi --multi-thread-cutoff 16Mi --multi-thread-streams 3000 --transfers 3000 --checkers 64 --retries 2 --oos-chunk-size 10Mi --oos-upload-concurrency 10000 --oos-attempt-resume-upload --oos-leave-parts-on-error sync ./artifacts remote:bucket -vv
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
Here is an example of making an oracle object storage configuration. `rclone config`
|
||||
walks you through it.
|
||||
|
||||
Here is an example of how to make a remote called `remote`. First run:
|
||||
|
||||
```console
|
||||
rclone config
|
||||
```
|
||||
|
||||
This will guide you through an interactive setup process:
|
||||
|
||||
```text
|
||||
n) New remote
|
||||
d) Delete remote
|
||||
r) Rename remote
|
||||
c) Copy remote
|
||||
s) Set configuration password
|
||||
q) Quit config
|
||||
e/n/d/r/c/s/q> n
|
||||
|
||||
Enter name for new remote.
|
||||
name> remote
|
||||
|
||||
Option Storage.
|
||||
Type of storage to configure.
|
||||
Choose a number from below, or type in your own value.
|
||||
[snip]
|
||||
XX / Oracle Cloud Infrastructure Object Storage
|
||||
\ (oracleobjectstorage)
|
||||
Storage> oracleobjectstorage
|
||||
|
||||
Option provider.
|
||||
Choose your Auth Provider
|
||||
Choose a number from below, or type in your own string value.
|
||||
Press Enter for the default (env_auth).
|
||||
1 / automatically pickup the credentials from runtime(env), first one to provide auth wins
|
||||
\ (env_auth)
|
||||
/ use an OCI user and an API key for authentication.
|
||||
2 | you’ll need to put in a config file your tenancy OCID, user OCID, region, the path, fingerprint to an API key.
|
||||
| https://docs.oracle.com/en-us/iaas/Content/API/Concepts/sdkconfig.htm
|
||||
\ (user_principal_auth)
|
||||
/ use instance principals to authorize an instance to make API calls.
|
||||
3 | each instance has its own identity, and authenticates using the certificates that are read from instance metadata.
|
||||
| https://docs.oracle.com/en-us/iaas/Content/Identity/Tasks/callingservicesfrominstances.htm
|
||||
\ (instance_principal_auth)
|
||||
/ use workload identity to grant Kubernetes pods policy-driven access to Oracle Cloud
|
||||
4 | Infrastructure (OCI) resources using OCI Identity and Access Management (IAM).
|
||||
| https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contenggrantingworkloadaccesstoresources.htm
|
||||
\ (workload_identity_auth)
|
||||
5 / use resource principals to make API calls
|
||||
\ (resource_principal_auth)
|
||||
6 / no credentials needed, this is typically for reading public buckets
|
||||
\ (no_auth)
|
||||
provider> 2
|
||||
|
||||
Option namespace.
|
||||
Object storage namespace
|
||||
Enter a value.
|
||||
namespace> idbamagbg734
|
||||
|
||||
Option compartment.
|
||||
Object storage compartment OCID
|
||||
Enter a value.
|
||||
compartment> ocid1.compartment.oc1..aaaaaaaapufkxc7ame3sthry5i7ujrwfc7ejnthhu6bhanm5oqfjpyasjkba
|
||||
|
||||
Option region.
|
||||
Object storage Region
|
||||
Enter a value.
|
||||
region> us-ashburn-1
|
||||
|
||||
Option endpoint.
|
||||
Endpoint for Object storage API.
|
||||
Leave blank to use the default endpoint for the region.
|
||||
Enter a value. Press Enter to leave empty.
|
||||
endpoint>
|
||||
|
||||
Option config_file.
|
||||
Full Path to OCI config file
|
||||
Choose a number from below, or type in your own string value.
|
||||
Press Enter for the default (~/.oci/config).
|
||||
1 / oci configuration file location
|
||||
\ (~/.oci/config)
|
||||
config_file> /etc/oci/dev.conf
|
||||
|
||||
Option config_profile.
|
||||
Profile name inside OCI config file
|
||||
Choose a number from below, or type in your own string value.
|
||||
Press Enter for the default (Default).
|
||||
1 / Use the default profile
|
||||
\ (Default)
|
||||
config_profile> Test
|
||||
|
||||
Edit advanced config?
|
||||
y) Yes
|
||||
n) No (default)
|
||||
y/n> n
|
||||
|
||||
Configuration complete.
|
||||
Options:
|
||||
- type: oracleobjectstorage
|
||||
- namespace: idbamagbg734
|
||||
- compartment: ocid1.compartment.oc1..aaaaaaaapufkxc7ame3sthry5i7ujrwfc7ejnthhu6bhanm5oqfjpyasjkba
|
||||
- region: us-ashburn-1
|
||||
- provider: user_principal_auth
|
||||
- config_file: /etc/oci/dev.conf
|
||||
- config_profile: Test
|
||||
Keep this "remote" remote?
|
||||
y) Yes this is OK (default)
|
||||
e) Edit this remote
|
||||
d) Delete this remote
|
||||
y/e/d> y
|
||||
```
|
||||
|
||||
See all buckets
|
||||
|
||||
```console
|
||||
rclone lsd remote:
|
||||
```
|
||||
|
||||
Create a new bucket
|
||||
|
||||
```console
|
||||
rclone mkdir remote:bucket
|
||||
```
|
||||
|
||||
List the contents of a bucket
|
||||
|
||||
```console
|
||||
rclone ls remote:bucket
|
||||
rclone ls remote:bucket --max-depth 1
|
||||
```
|
||||
|
||||
## Authentication Providers
|
||||
|
||||
OCI has various authentication methods. To learn more about authentication methods
|
||||
please refer [oci authentication methods](https://docs.oracle.com/en-us/iaas/Content/API/Concepts/sdk_authentication_methods.htm)
|
||||
These choices can be specified in the rclone config file.
|
||||
|
||||
Rclone supports the following OCI authentication provider.
|
||||
|
||||
```text
|
||||
User Principal
|
||||
Instance Principal
|
||||
Resource Principal
|
||||
Workload Identity
|
||||
No authentication
|
||||
```
|
||||
|
||||
### User Principal
|
||||
|
||||
Sample rclone config file for Authentication Provider User Principal:
|
||||
|
||||
```ini
|
||||
[oos]
|
||||
type = oracleobjectstorage
|
||||
namespace = id<redacted>34
|
||||
compartment = ocid1.compartment.oc1..aa<redacted>ba
|
||||
region = us-ashburn-1
|
||||
provider = user_principal_auth
|
||||
config_file = /home/opc/.oci/config
|
||||
config_profile = Default
|
||||
```
|
||||
|
||||
Advantages:
|
||||
|
||||
- One can use this method from any server within OCI or on-premises or from
|
||||
other cloud provider.
|
||||
|
||||
Considerations:
|
||||
|
||||
- you need to configure user’s privileges / policy to allow access to object
|
||||
storage
|
||||
- Overhead of managing users and keys.
|
||||
- If the user is deleted, the config file will no longer work and may cause
|
||||
automation regressions that use the user's credentials.
|
||||
|
||||
### Instance Principal
|
||||
|
||||
An OCI compute instance can be authorized to use rclone by using it's identity
|
||||
and certificates as an instance principal. With this approach no credentials
|
||||
have to be stored and managed.
|
||||
|
||||
Sample rclone configuration file for Authentication Provider Instance Principal:
|
||||
|
||||
```console
|
||||
[opc@rclone ~]$ cat ~/.config/rclone/rclone.conf
|
||||
[oos]
|
||||
type = oracleobjectstorage
|
||||
namespace = id<redacted>fn
|
||||
compartment = ocid1.compartment.oc1..aa<redacted>k7a
|
||||
region = us-ashburn-1
|
||||
provider = instance_principal_auth
|
||||
```
|
||||
|
||||
Advantages:
|
||||
|
||||
- With instance principals, you don't need to configure user credentials and
|
||||
transfer/ save it to disk in your compute instances or rotate the credentials.
|
||||
- You don’t need to deal with users and keys.
|
||||
- Greatly helps in automation as you don't have to manage access keys, user
|
||||
private keys, storing them in vault, using kms etc.
|
||||
|
||||
Considerations:
|
||||
|
||||
- You need to configure a dynamic group having this instance as member and add
|
||||
policy to read object storage to that dynamic group.
|
||||
- Everyone who has access to this machine can execute the CLI commands.
|
||||
- It is applicable for oci compute instances only. It cannot be used on external
|
||||
instance or resources.
|
||||
|
||||
### Resource Principal
|
||||
|
||||
Resource principal auth is very similar to instance principal auth but used for
|
||||
resources that are not compute instances such as
|
||||
[serverless functions](https://docs.oracle.com/en-us/iaas/Content/Functions/Concepts/functionsoverview.htm).
|
||||
To use resource principal ensure Rclone process is started with these environment
|
||||
variables set in its process.
|
||||
|
||||
```console
|
||||
export OCI_RESOURCE_PRINCIPAL_VERSION=2.2
|
||||
export OCI_RESOURCE_PRINCIPAL_REGION=us-ashburn-1
|
||||
export OCI_RESOURCE_PRINCIPAL_PRIVATE_PEM=/usr/share/model-server/key.pem
|
||||
export OCI_RESOURCE_PRINCIPAL_RPST=/usr/share/model-server/security_token
|
||||
```
|
||||
|
||||
Sample rclone configuration file for Authentication Provider Resource Principal:
|
||||
|
||||
```ini
|
||||
[oos]
|
||||
type = oracleobjectstorage
|
||||
namespace = id<redacted>34
|
||||
compartment = ocid1.compartment.oc1..aa<redacted>ba
|
||||
region = us-ashburn-1
|
||||
provider = resource_principal_auth
|
||||
```
|
||||
|
||||
### Workload Identity
|
||||
|
||||
Workload Identity auth may be used when running Rclone from Kubernetes pod on
|
||||
a Container Engine for Kubernetes (OKE) cluster. For more details on configuring
|
||||
Workload Identity, see [Granting Workloads Access to OCI Resources](https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contenggrantingworkloadaccesstoresources.htm).
|
||||
To use workload identity, ensure Rclone is started with these environment
|
||||
variables set in its process.
|
||||
|
||||
```console
|
||||
export OCI_RESOURCE_PRINCIPAL_VERSION=2.2
|
||||
export OCI_RESOURCE_PRINCIPAL_REGION=us-ashburn-1
|
||||
```
|
||||
|
||||
### No authentication
|
||||
|
||||
Public buckets do not require any authentication mechanism to read objects.
|
||||
Sample rclone configuration file for No authentication:
|
||||
|
||||
```ini
|
||||
[oos]
|
||||
type = oracleobjectstorage
|
||||
namespace = id<redacted>34
|
||||
compartment = ocid1.compartment.oc1..aa<redacted>ba
|
||||
region = us-ashburn-1
|
||||
provider = no_auth
|
||||
```
|
||||
|
||||
### Modification times and hashes
|
||||
|
||||
The modification time is stored as metadata on the object as
|
||||
`opc-meta-mtime` as floating point since the epoch, accurate to 1 ns.
|
||||
|
||||
If the modification time needs to be updated rclone will attempt to perform a server
|
||||
side copy to update the modification if the object can be copied in a single part.
|
||||
In the case the object is larger than 5Gb, the object will be uploaded rather than
|
||||
copied.
|
||||
|
||||
Note that reading this from the object takes an additional `HEAD` request as the
|
||||
metadata isn't returned in object listings.
|
||||
|
||||
The MD5 hash algorithm is supported.
|
||||
|
||||
### Multipart uploads
|
||||
|
||||
rclone supports multipart uploads with OOS which means that it can
|
||||
upload files bigger than 5 GiB.
|
||||
|
||||
Note that files uploaded *both* with multipart upload *and* through
|
||||
crypt remotes do not have MD5 sums.
|
||||
|
||||
rclone switches from single part uploads to multipart uploads at the
|
||||
point specified by `--oos-upload-cutoff`. This can be a maximum of 5 GiB
|
||||
and a minimum of 0 (ie always upload multipart files).
|
||||
|
||||
The chunk sizes used in the multipart upload are specified by
|
||||
`--oos-chunk-size` and the number of chunks uploaded concurrently is
|
||||
specified by `--oos-upload-concurrency`.
|
||||
|
||||
Multipart uploads will use `--transfers` * `--oos-upload-concurrency` *
|
||||
`--oos-chunk-size` extra memory. Single part uploads to not use extra
|
||||
memory.
|
||||
|
||||
Single part transfers can be faster than multipart transfers or slower
|
||||
depending on your latency from oos - the more latency, the more likely
|
||||
single part transfers will be faster.
|
||||
|
||||
Increasing `--oos-upload-concurrency` will increase throughput (8 would
|
||||
be a sensible value) and increasing `--oos-chunk-size` also increases
|
||||
throughput (16M would be sensible). Increasing either of these will
|
||||
use more memory. The default values are high enough to gain most of
|
||||
the possible performance without using too much memory.
|
||||
|
||||
<!-- autogenerated options start - DO NOT EDIT - instead edit fs.RegInfo in backend/oracleobjectstorage/oracleobjectstorage.go and run make backenddocs to verify --> <!-- markdownlint-disable-line line-length -->
|
||||
### Standard options
|
||||
|
||||
Here are the Standard options specific to oracleobjectstorage (Oracle Cloud Infrastructure Object Storage).
|
||||
|
||||
#### --oos-provider
|
||||
|
||||
Choose your Auth Provider
|
||||
|
||||
Properties:
|
||||
|
||||
- Config: provider
|
||||
- Env Var: RCLONE_OOS_PROVIDER
|
||||
- Type: string
|
||||
- Default: "env_auth"
|
||||
- Examples:
|
||||
- "env_auth"
|
||||
- automatically pickup the credentials from runtime(env), first one to provide auth wins
|
||||
- "user_principal_auth"
|
||||
- use an OCI user and an API key for authentication.
|
||||
- you’ll need to put in a config file your tenancy OCID, user OCID, region, the path, fingerprint to an API key.
|
||||
- https://docs.oracle.com/en-us/iaas/Content/API/Concepts/sdkconfig.htm
|
||||
- "instance_principal_auth"
|
||||
- use instance principals to authorize an instance to make API calls.
|
||||
- each instance has its own identity, and authenticates using the certificates that are read from instance metadata.
|
||||
- https://docs.oracle.com/en-us/iaas/Content/Identity/Tasks/callingservicesfrominstances.htm
|
||||
- "workload_identity_auth"
|
||||
- use workload identity to grant OCI Container Engine for Kubernetes workloads policy-driven access to OCI resources using OCI Identity and Access Management (IAM).
|
||||
- https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contenggrantingworkloadaccesstoresources.htm
|
||||
- "resource_principal_auth"
|
||||
- use resource principals to make API calls
|
||||
- "no_auth"
|
||||
- no credentials needed, this is typically for reading public buckets
|
||||
|
||||
#### --oos-namespace
|
||||
|
||||
Object storage namespace
|
||||
|
||||
Properties:
|
||||
|
||||
- Config: namespace
|
||||
- Env Var: RCLONE_OOS_NAMESPACE
|
||||
- Type: string
|
||||
- Required: true
|
||||
|
||||
#### --oos-compartment
|
||||
|
||||
Specify compartment OCID, if you need to list buckets.
|
||||
|
||||
List objects works without compartment OCID.
|
||||
|
||||
Properties:
|
||||
|
||||
- Config: compartment
|
||||
- Env Var: RCLONE_OOS_COMPARTMENT
|
||||
- Provider: !no_auth
|
||||
- Type: string
|
||||
- Required: false
|
||||
|
||||
#### --oos-region
|
||||
|
||||
Object storage Region
|
||||
|
||||
Properties:
|
||||
|
||||
- Config: region
|
||||
- Env Var: RCLONE_OOS_REGION
|
||||
- Type: string
|
||||
- Required: true
|
||||
|
||||
#### --oos-endpoint
|
||||
|
||||
Endpoint for Object storage API.
|
||||
|
||||
Leave blank to use the default endpoint for the region.
|
||||
|
||||
Properties:
|
||||
|
||||
- Config: endpoint
|
||||
- Env Var: RCLONE_OOS_ENDPOINT
|
||||
- Type: string
|
||||
- Required: false
|
||||
|
||||
#### --oos-config-file
|
||||
|
||||
Path to OCI config file
|
||||
|
||||
Properties:
|
||||
|
||||
- Config: config_file
|
||||
- Env Var: RCLONE_OOS_CONFIG_FILE
|
||||
- Provider: user_principal_auth
|
||||
- Type: string
|
||||
- Default: "~/.oci/config"
|
||||
- Examples:
|
||||
- "~/.oci/config"
|
||||
- oci configuration file location
|
||||
|
||||
#### --oos-config-profile
|
||||
|
||||
Profile name inside the oci config file
|
||||
|
||||
Properties:
|
||||
|
||||
- Config: config_profile
|
||||
- Env Var: RCLONE_OOS_CONFIG_PROFILE
|
||||
- Provider: user_principal_auth
|
||||
- Type: string
|
||||
- Default: "Default"
|
||||
- Examples:
|
||||
- "Default"
|
||||
- Use the default profile
|
||||
|
||||
### Advanced options
|
||||
|
||||
Here are the Advanced options specific to oracleobjectstorage (Oracle Cloud Infrastructure Object Storage).
|
||||
|
||||
#### --oos-storage-tier
|
||||
|
||||
The storage class to use when storing new objects in storage. https://docs.oracle.com/en-us/iaas/Content/Object/Concepts/understandingstoragetiers.htm
|
||||
|
||||
Properties:
|
||||
|
||||
- Config: storage_tier
|
||||
- Env Var: RCLONE_OOS_STORAGE_TIER
|
||||
- Type: string
|
||||
- Default: "Standard"
|
||||
- Examples:
|
||||
- "Standard"
|
||||
- Standard storage tier, this is the default tier
|
||||
- "InfrequentAccess"
|
||||
- InfrequentAccess storage tier
|
||||
- "Archive"
|
||||
- Archive storage tier
|
||||
|
||||
#### --oos-upload-cutoff
|
||||
|
||||
Cutoff for switching to chunked upload.
|
||||
|
||||
Any files larger than this will be uploaded in chunks of chunk_size.
|
||||
The minimum is 0 and the maximum is 5 GiB.
|
||||
|
||||
Properties:
|
||||
|
||||
- Config: upload_cutoff
|
||||
- Env Var: RCLONE_OOS_UPLOAD_CUTOFF
|
||||
- Type: SizeSuffix
|
||||
- Default: 200Mi
|
||||
|
||||
#### --oos-chunk-size
|
||||
|
||||
Chunk size to use for uploading.
|
||||
|
||||
When uploading files larger than upload_cutoff or files with unknown
|
||||
size (e.g. from "rclone rcat" or uploaded with "rclone mount" they will be uploaded
|
||||
as multipart uploads using this chunk size.
|
||||
|
||||
Note that "upload_concurrency" chunks of this size are buffered
|
||||
in memory per transfer.
|
||||
|
||||
If you are transferring large files over high-speed links and you have
|
||||
enough memory, then increasing this will speed up the transfers.
|
||||
|
||||
Rclone will automatically increase the chunk size when uploading a
|
||||
large file of known size to stay below the 10,000 chunks limit.
|
||||
|
||||
Files of unknown size are uploaded with the configured
|
||||
chunk_size. Since the default chunk size is 5 MiB and there can be at
|
||||
most 10,000 chunks, this means that by default the maximum size of
|
||||
a file you can stream upload is 48 GiB. If you wish to stream upload
|
||||
larger files then you will need to increase chunk_size.
|
||||
|
||||
Increasing the chunk size decreases the accuracy of the progress
|
||||
statistics displayed with "-P" flag.
|
||||
|
||||
|
||||
Properties:
|
||||
|
||||
- Config: chunk_size
|
||||
- Env Var: RCLONE_OOS_CHUNK_SIZE
|
||||
- Type: SizeSuffix
|
||||
- Default: 5Mi
|
||||
|
||||
#### --oos-max-upload-parts
|
||||
|
||||
Maximum number of parts in a multipart upload.
|
||||
|
||||
This option defines the maximum number of multipart chunks to use
|
||||
when doing a multipart upload.
|
||||
|
||||
OCI has max parts limit of 10,000 chunks.
|
||||
|
||||
Rclone will automatically increase the chunk size when uploading a
|
||||
large file of a known size to stay below this number of chunks limit.
|
||||
|
||||
|
||||
Properties:
|
||||
|
||||
- Config: max_upload_parts
|
||||
- Env Var: RCLONE_OOS_MAX_UPLOAD_PARTS
|
||||
- Type: int
|
||||
- Default: 10000
|
||||
|
||||
#### --oos-upload-concurrency
|
||||
|
||||
Concurrency for multipart uploads.
|
||||
|
||||
This is the number of chunks of the same file that are uploaded
|
||||
concurrently.
|
||||
|
||||
If you are uploading small numbers of large files over high-speed links
|
||||
and these uploads do not fully utilize your bandwidth, then increasing
|
||||
this may help to speed up the transfers.
|
||||
|
||||
Properties:
|
||||
|
||||
- Config: upload_concurrency
|
||||
- Env Var: RCLONE_OOS_UPLOAD_CONCURRENCY
|
||||
- Type: int
|
||||
- Default: 10
|
||||
|
||||
#### --oos-copy-cutoff
|
||||
|
||||
Cutoff for switching to multipart copy.
|
||||
|
||||
Any files larger than this that need to be server-side copied will be
|
||||
copied in chunks of this size.
|
||||
|
||||
The minimum is 0 and the maximum is 5 GiB.
|
||||
|
||||
Properties:
|
||||
|
||||
- Config: copy_cutoff
|
||||
- Env Var: RCLONE_OOS_COPY_CUTOFF
|
||||
- Type: SizeSuffix
|
||||
- Default: 4.656Gi
|
||||
|
||||
#### --oos-copy-timeout
|
||||
|
||||
Timeout for copy.
|
||||
|
||||
Copy is an asynchronous operation, specify timeout to wait for copy to succeed
|
||||
|
||||
|
||||
Properties:
|
||||
|
||||
- Config: copy_timeout
|
||||
- Env Var: RCLONE_OOS_COPY_TIMEOUT
|
||||
- Type: Duration
|
||||
- Default: 1m0s
|
||||
|
||||
#### --oos-disable-checksum
|
||||
|
||||
Don't store MD5 checksum with object metadata.
|
||||
|
||||
Normally rclone will calculate the MD5 checksum of the input before
|
||||
uploading it so it can add it to metadata on the object. This is great
|
||||
for data integrity checking but can cause long delays for large files
|
||||
to start uploading.
|
||||
|
||||
Properties:
|
||||
|
||||
- Config: disable_checksum
|
||||
- Env Var: RCLONE_OOS_DISABLE_CHECKSUM
|
||||
- Type: bool
|
||||
- Default: false
|
||||
|
||||
#### --oos-encoding
|
||||
|
||||
The encoding for the backend.
|
||||
|
||||
See the [encoding section in the overview](/overview/#encoding) for more info.
|
||||
|
||||
Properties:
|
||||
|
||||
- Config: encoding
|
||||
- Env Var: RCLONE_OOS_ENCODING
|
||||
- Type: Encoding
|
||||
- Default: Slash,InvalidUtf8,Dot
|
||||
|
||||
#### --oos-leave-parts-on-error
|
||||
|
||||
If true avoid calling abort upload on a failure, leaving all successfully uploaded parts for manual recovery.
|
||||
|
||||
It should be set to true for resuming uploads across different sessions.
|
||||
|
||||
WARNING: Storing parts of an incomplete multipart upload counts towards space usage on object storage and will add
|
||||
additional costs if not cleaned up.
|
||||
|
||||
|
||||
Properties:
|
||||
|
||||
- Config: leave_parts_on_error
|
||||
- Env Var: RCLONE_OOS_LEAVE_PARTS_ON_ERROR
|
||||
- Type: bool
|
||||
- Default: false
|
||||
|
||||
#### --oos-attempt-resume-upload
|
||||
|
||||
If true attempt to resume previously started multipart upload for the object.
|
||||
This will be helpful to speed up multipart transfers by resuming uploads from past session.
|
||||
|
||||
WARNING: If chunk size differs in resumed session from past incomplete session, then the resumed multipart upload is
|
||||
aborted and a new multipart upload is started with the new chunk size.
|
||||
|
||||
The flag leave_parts_on_error must be true to resume and optimize to skip parts that were already uploaded successfully.
|
||||
|
||||
|
||||
Properties:
|
||||
|
||||
- Config: attempt_resume_upload
|
||||
- Env Var: RCLONE_OOS_ATTEMPT_RESUME_UPLOAD
|
||||
- Type: bool
|
||||
- Default: false
|
||||
|
||||
#### --oos-no-check-bucket
|
||||
|
||||
If set, don't attempt to check the bucket exists or create it.
|
||||
|
||||
This can be useful when trying to minimise the number of transactions
|
||||
rclone does if you know the bucket exists already.
|
||||
|
||||
It can also be needed if the user you are using does not have bucket
|
||||
creation permissions.
|
||||
|
||||
|
||||
Properties:
|
||||
|
||||
- Config: no_check_bucket
|
||||
- Env Var: RCLONE_OOS_NO_CHECK_BUCKET
|
||||
- Type: bool
|
||||
- Default: false
|
||||
|
||||
#### --oos-sse-customer-key-file
|
||||
|
||||
To use SSE-C, a file containing the base64-encoded string of the AES-256 encryption key associated
|
||||
with the object. Please note only one of sse_customer_key_file|sse_customer_key|sse_kms_key_id is needed.'
|
||||
|
||||
Properties:
|
||||
|
||||
- Config: sse_customer_key_file
|
||||
- Env Var: RCLONE_OOS_SSE_CUSTOMER_KEY_FILE
|
||||
- Type: string
|
||||
- Required: false
|
||||
- Examples:
|
||||
- ""
|
||||
- None
|
||||
|
||||
#### --oos-sse-customer-key
|
||||
|
||||
To use SSE-C, the optional header that specifies the base64-encoded 256-bit encryption key to use to
|
||||
encrypt or decrypt the data. Please note only one of sse_customer_key_file|sse_customer_key|sse_kms_key_id is
|
||||
needed. For more information, see Using Your Own Keys for Server-Side Encryption
|
||||
(https://docs.cloud.oracle.com/Content/Object/Tasks/usingyourencryptionkeys.htm)
|
||||
|
||||
Properties:
|
||||
|
||||
- Config: sse_customer_key
|
||||
- Env Var: RCLONE_OOS_SSE_CUSTOMER_KEY
|
||||
- Type: string
|
||||
- Required: false
|
||||
- Examples:
|
||||
- ""
|
||||
- None
|
||||
|
||||
#### --oos-sse-customer-key-sha256
|
||||
|
||||
If using SSE-C, The optional header that specifies the base64-encoded SHA256 hash of the encryption
|
||||
key. This value is used to check the integrity of the encryption key. see Using Your Own Keys for
|
||||
Server-Side Encryption (https://docs.cloud.oracle.com/Content/Object/Tasks/usingyourencryptionkeys.htm).
|
||||
|
||||
Properties:
|
||||
|
||||
- Config: sse_customer_key_sha256
|
||||
- Env Var: RCLONE_OOS_SSE_CUSTOMER_KEY_SHA256
|
||||
- Type: string
|
||||
- Required: false
|
||||
- Examples:
|
||||
- ""
|
||||
- None
|
||||
|
||||
#### --oos-sse-kms-key-id
|
||||
|
||||
if using your own master key in vault, this header specifies the
|
||||
OCID (https://docs.cloud.oracle.com/Content/General/Concepts/identifiers.htm) of a master encryption key used to call
|
||||
the Key Management service to generate a data encryption key or to encrypt or decrypt a data encryption key.
|
||||
Please note only one of sse_customer_key_file|sse_customer_key|sse_kms_key_id is needed.
|
||||
|
||||
Properties:
|
||||
|
||||
- Config: sse_kms_key_id
|
||||
- Env Var: RCLONE_OOS_SSE_KMS_KEY_ID
|
||||
- Type: string
|
||||
- Required: false
|
||||
- Examples:
|
||||
- ""
|
||||
- None
|
||||
|
||||
#### --oos-sse-customer-algorithm
|
||||
|
||||
If using SSE-C, the optional header that specifies "AES256" as the encryption algorithm.
|
||||
Object Storage supports "AES256" as the encryption algorithm. For more information, see
|
||||
Using Your Own Keys for Server-Side Encryption (https://docs.cloud.oracle.com/Content/Object/Tasks/usingyourencryptionkeys.htm).
|
||||
|
||||
Properties:
|
||||
|
||||
- Config: sse_customer_algorithm
|
||||
- Env Var: RCLONE_OOS_SSE_CUSTOMER_ALGORITHM
|
||||
- Type: string
|
||||
- Required: false
|
||||
- Examples:
|
||||
- ""
|
||||
- None
|
||||
- "AES256"
|
||||
- AES256
|
||||
|
||||
#### --oos-description
|
||||
|
||||
Description of the remote.
|
||||
|
||||
Properties:
|
||||
|
||||
- Config: description
|
||||
- Env Var: RCLONE_OOS_DESCRIPTION
|
||||
- Type: string
|
||||
- Required: false
|
||||
|
||||
### Metadata
|
||||
|
||||
User metadata is stored as opc-meta- keys.
|
||||
|
||||
Here are the possible system metadata items for the oracleobjectstorage backend.
|
||||
|
||||
| Name | Help | Type | Example | Read Only |
|
||||
|------|------|------|---------|-----------|
|
||||
| opc-meta-atime | Time of last access | ISO 8601 | 2025-06-30T22:27:43-04:00 | N |
|
||||
| opc-meta-btime | Time of file birth (creation) | ISO 8601 | 2025-06-30T22:27:43-04:00 | N |
|
||||
| opc-meta-gid | Group ID of owner | decimal number | 500 | N |
|
||||
| opc-meta-mode | File type and mode | octal, unix style | 0100664 | N |
|
||||
| opc-meta-mtime | Time of last modification | ISO 8601 | 2025-06-30T22:27:43-04:00 | N |
|
||||
| opc-meta-uid | User ID of owner | decimal number | 500 | N |
|
||||
|
||||
See the [metadata](/docs/#metadata) docs for more info.
|
||||
|
||||
## Backend commands
|
||||
|
||||
Here are the commands specific to the oracleobjectstorage backend.
|
||||
|
||||
Run them with:
|
||||
|
||||
```console
|
||||
rclone backend COMMAND remote:
|
||||
```
|
||||
|
||||
The help below will explain what arguments each command takes.
|
||||
|
||||
See the [backend](/commands/rclone_backend/) command for more
|
||||
info on how to pass options and arguments.
|
||||
|
||||
These can be run on a running backend using the rc command
|
||||
[backend/command](/rc/#backend-command).
|
||||
|
||||
### rename
|
||||
|
||||
change the name of an object.
|
||||
|
||||
```console
|
||||
rclone backend rename remote: [options] [<arguments>+]
|
||||
```
|
||||
|
||||
This command can be used to rename a object.
|
||||
|
||||
Usage example:
|
||||
|
||||
```console
|
||||
rclone backend rename oos:bucket relative-object-path-under-bucket object-new-name
|
||||
```
|
||||
|
||||
### list-multipart-uploads
|
||||
|
||||
List the unfinished multipart uploads.
|
||||
|
||||
```console
|
||||
rclone backend list-multipart-uploads remote: [options] [<arguments>+]
|
||||
```
|
||||
|
||||
This command lists the unfinished multipart uploads in JSON format.
|
||||
|
||||
Usage example:
|
||||
|
||||
```console
|
||||
rclone backend list-multipart-uploads oos:bucket/path/to/object
|
||||
```
|
||||
|
||||
It returns a dictionary of buckets with values as lists of unfinished
|
||||
multipart uploads.
|
||||
|
||||
You can call it with no bucket in which case it lists all bucket, with
|
||||
a bucket or with a bucket and path.
|
||||
|
||||
```json
|
||||
{
|
||||
"test-bucket": [
|
||||
{
|
||||
"namespace": "test-namespace",
|
||||
"bucket": "test-bucket",
|
||||
"object": "600m.bin",
|
||||
"uploadId": "51dd8114-52a4-b2f2-c42f-5291f05eb3c8",
|
||||
"timeCreated": "2022-07-29T06:21:16.595Z",
|
||||
"storageTier": "Standard"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
### cleanup
|
||||
|
||||
Remove unfinished multipart uploads.
|
||||
|
||||
```console
|
||||
rclone backend cleanup remote: [options] [<arguments>+]
|
||||
```
|
||||
|
||||
This command removes unfinished multipart uploads of age greater than
|
||||
max-age which defaults to 24 hours.
|
||||
|
||||
Note that you can use --interactive/-i or --dry-run with this command to see
|
||||
what it would do.
|
||||
|
||||
Usage examples:
|
||||
|
||||
```console
|
||||
rclone backend cleanup oos:bucket/path/to/object
|
||||
rclone backend cleanup -o max-age=7w oos:bucket/path/to/object
|
||||
```
|
||||
|
||||
Durations are parsed as per the rest of rclone, 2h, 7d, 7w etc.
|
||||
|
||||
Options:
|
||||
|
||||
- "max-age": Max age of upload to delete.
|
||||
|
||||
### restore
|
||||
|
||||
Restore objects from Archive to Standard storage.
|
||||
|
||||
```console
|
||||
rclone backend restore remote: [options] [<arguments>+]
|
||||
```
|
||||
|
||||
This command can be used to restore one or more objects from Archive to
|
||||
Standard storage.
|
||||
|
||||
Usage examples:
|
||||
|
||||
```console
|
||||
rclone backend restore oos:bucket/path/to/directory -o hours=HOURS
|
||||
rclone backend restore oos:bucket -o hours=HOURS
|
||||
```
|
||||
|
||||
This flag also obeys the filters. Test first with --interactive/-i or --dry-run flags
|
||||
|
||||
```console
|
||||
rclone --interactive backend restore --include "*.txt" oos:bucket/path -o hours=72
|
||||
```
|
||||
|
||||
All the objects shown will be marked for restore, then:
|
||||
|
||||
```console
|
||||
rclone backend restore --include "*.txt" oos:bucket/path -o hours=72
|
||||
```
|
||||
|
||||
It returns a list of status dictionaries with Object Name and Status keys.
|
||||
The Status will be "RESTORED"" if it was successful or an error message if not.
|
||||
|
||||
```json
|
||||
[
|
||||
{
|
||||
"Object": "test.txt"
|
||||
"Status": "RESTORED",
|
||||
},
|
||||
{
|
||||
"Object": "test/file4.txt"
|
||||
"Status": "RESTORED",
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
Options:
|
||||
|
||||
- "hours": The number of hours for which this object will be restored.
|
||||
Default is 24 hrs.
|
||||
|
||||
<!-- autogenerated options stop -->
|
||||
|
||||
## Tutorials
|
||||
|
||||
### [Mounting Buckets](/oracleobjectstorage/tutorial_mount/)
|
||||
+503
@@ -0,0 +1,503 @@
|
||||
---
|
||||
title: "Oracle Object Storage Mount"
|
||||
description: "Oracle Object Storage mounting tutorial"
|
||||
---
|
||||
|
||||
> **Official documentation:** [https://rclone.org/oracleobjectstorage/](https://rclone.org/oracleobjectstorage/)
|
||||
# Mount Buckets and Expose via NFS Tutorial
|
||||
|
||||
This runbook shows how to [mount](/commands/rclone_mount/) *Oracle Object Storage*
|
||||
buckets as local file system in OCI compute Instance using rclone tool.
|
||||
|
||||
You will also learn how to export the rclone mounts as NFS mount, so that other
|
||||
NFS client can access them.
|
||||
|
||||
Usage Pattern:
|
||||
|
||||
NFS Client --> NFS Server --> RClone Mount --> OCI Object Storage
|
||||
|
||||
## Step 1 : Install Rclone
|
||||
|
||||
In oracle linux 8, Rclone can be installed from
|
||||
[OL8_Developer](https://yum.oracle.com/repo/OracleLinux/OL8/developer/x86_64/index.html)
|
||||
Yum Repo, Please enable the repo if not enabled already.
|
||||
|
||||
```console
|
||||
[opc@base-inst-boot ~]$ sudo yum-config-manager --enable ol8_developer
|
||||
[opc@base-inst-boot ~]$ sudo yum install -y rclone
|
||||
[opc@base-inst-boot ~]$ sudo yum install -y fuse
|
||||
# rclone will prefer fuse3 if available
|
||||
[opc@base-inst-boot ~]$ sudo yum install -y fuse3
|
||||
[opc@base-inst-boot ~]$ yum info rclone
|
||||
Last metadata expiration check: 0:01:58 ago on Fri 07 Apr 2023 05:53:43 PM GMT.
|
||||
Installed Packages
|
||||
Name : rclone
|
||||
Version : 1.62.2
|
||||
Release : 1.0.1.el8
|
||||
Architecture : x86_64
|
||||
Size : 67 M
|
||||
Source : rclone-1.62.2-1.0.1.el8.src.rpm
|
||||
Repository : @System
|
||||
From repo : ol8_developer
|
||||
Summary : rsync for cloud storage
|
||||
URL : http://rclone.org/
|
||||
License : MIT
|
||||
Description : Rclone is a command line program to sync files and directories to and from various cloud services.
|
||||
```
|
||||
|
||||
To run it as a mount helper you should symlink rclone binary to /sbin/mount.rclone
|
||||
and optionally /usr/bin/rclonefs, e.g. `ln -s /usr/bin/rclone /sbin/mount.rclone`.
|
||||
rclone will detect it and translate command-line arguments appropriately.
|
||||
|
||||
```console
|
||||
ln -s /usr/bin/rclone /sbin/mount.rclone
|
||||
```
|
||||
|
||||
## Step 2: Setup Rclone Configuration file
|
||||
|
||||
Let's assume you want to access 3 buckets from the oci compute instance using
|
||||
instance principal provider as means of authenticating with object storage service.
|
||||
|
||||
- namespace-a, bucket-a,
|
||||
- namespace-b, bucket-b,
|
||||
- namespace-c, bucket-c
|
||||
|
||||
Rclone configuration file needs to have 3 remote sections, one section of each
|
||||
of above 3 buckets. Create a configuration file in a accessible location that
|
||||
rclone program can read.
|
||||
|
||||
```console
|
||||
[opc@base-inst-boot ~]$ mkdir -p /etc/rclone
|
||||
[opc@base-inst-boot ~]$ sudo touch /etc/rclone/rclone.conf
|
||||
|
||||
|
||||
# add below contents to /etc/rclone/rclone.conf
|
||||
[opc@base-inst-boot ~]$ cat /etc/rclone/rclone.conf
|
||||
|
||||
|
||||
[ossa]
|
||||
type = oracleobjectstorage
|
||||
provider = instance_principal_auth
|
||||
namespace = namespace-a
|
||||
compartment = ocid1.compartment.oc1..aaaaaaaa...compartment-a
|
||||
region = us-ashburn-1
|
||||
|
||||
[ossb]
|
||||
type = oracleobjectstorage
|
||||
provider = instance_principal_auth
|
||||
namespace = namespace-b
|
||||
compartment = ocid1.compartment.oc1..aaaaaaaa...compartment-b
|
||||
region = us-ashburn-1
|
||||
|
||||
|
||||
[ossc]
|
||||
type = oracleobjectstorage
|
||||
provider = instance_principal_auth
|
||||
namespace = namespace-c
|
||||
compartment = ocid1.compartment.oc1..aaaaaaaa...compartment-c
|
||||
region = us-ashburn-1
|
||||
|
||||
# List remotes
|
||||
[opc@base-inst-boot ~]$ rclone --config /etc/rclone/rclone.conf listremotes
|
||||
ossa:
|
||||
ossb:
|
||||
ossc:
|
||||
|
||||
# Now please ensure you do not see below errors while listing the bucket,
|
||||
# i.e you should fix the settings to see if namespace, compartment, bucket name are all correct.
|
||||
# and you must have a dynamic group policy to allow the instance to use object-family in compartment.
|
||||
|
||||
[opc@base-inst-boot ~]$ rclone --config /etc/rclone/rclone.conf ls ossa:
|
||||
2023/04/07 19:09:21 Failed to ls: Error returned by ObjectStorage Service. Http Status Code: 404. Error Code: NamespaceNotFound. Opc request id: iad-1:kVVAb0knsVXDvu9aHUGHRs3gSNBOFO2_334B6co82LrPMWo2lM5PuBKNxJOTmZsS. Message: You do not have authorization to perform this request, or the requested resource could not be found.
|
||||
Operation Name: ListBuckets
|
||||
Timestamp: 2023-04-07 19:09:21 +0000 GMT
|
||||
Client Version: Oracle-GoSDK/65.32.0
|
||||
Request Endpoint: GET https://objectstorage.us-ashburn-1.oraclecloud.com/n/namespace-a/b?compartmentId=ocid1.compartment.oc1..aaaaaaaa...compartment-a
|
||||
Troubleshooting Tips: See https://docs.oracle.com/iaas/Content/API/References/apierrors.htm#apierrors_404__404_namespacenotfound for more information about resolving this error.
|
||||
Also see https://docs.oracle.com/iaas/api/#/en/objectstorage/20160918/Bucket/ListBuckets for details on this operation's requirements.
|
||||
To get more info on the failing request, you can set OCI_GO_SDK_DEBUG env var to info or higher level to log the request/response details.
|
||||
If you are unable to resolve this ObjectStorage issue, please contact Oracle support and provide them this full error message.
|
||||
[opc@base-inst-boot ~]$
|
||||
|
||||
```
|
||||
|
||||
## Step 3: Setup Dynamic Group and Add IAM Policy
|
||||
|
||||
Just like a human user has an identity identified by its USER-PRINCIPAL, every
|
||||
OCI compute instance is also a robotic user identified by its INSTANCE-PRINCIPAL.
|
||||
The instance principal key is automatically fetched by rclone/with-oci-sdk
|
||||
from instance-metadata to make calls to object storage.
|
||||
|
||||
Similar to [user-group](https://docs.oracle.com/en-us/iaas/Content/Identity/Tasks/managinggroups.htm),
|
||||
[instance groups](https://docs.oracle.com/en-us/iaas/Content/Identity/Tasks/managingdynamicgroups.htm)
|
||||
is known as dynamic-group in IAM.
|
||||
|
||||
Create a dynamic group say rclone-dynamic-group that the oci compute instance
|
||||
becomes a member of the below group says all instances belonging to compartment
|
||||
a...c is member of this dynamic-group.
|
||||
|
||||
```console
|
||||
any {instance.compartment.id = '<compartment_ocid_a>',
|
||||
instance.compartment.id = '<compartment_ocid_b>',
|
||||
instance.compartment.id = '<compartment_ocid_c>'
|
||||
}
|
||||
```
|
||||
|
||||
Now that you have a dynamic group, you need to add a policy allowing what
|
||||
permissions this dynamic-group has. In our case, we want this dynamic-group to
|
||||
access object-storage. So create a policy now.
|
||||
|
||||
```text
|
||||
allow dynamic-group rclone-dynamic-group to manage object-family in compartment compartment-a
|
||||
allow dynamic-group rclone-dynamic-group to manage object-family in compartment compartment-b
|
||||
allow dynamic-group rclone-dynamic-group to manage object-family in compartment compartment-c
|
||||
```
|
||||
|
||||
After you add the policy, now ensure the rclone can list files in your bucket,
|
||||
if not please troubleshoot any mistakes you did so far. Please note, identity
|
||||
can take upto a minute to ensure policy gets reflected.
|
||||
|
||||
## Step 4: Setup Mount Folders
|
||||
|
||||
Let's assume you have to mount 3 buckets, bucket-a, bucket-b, bucket-c at path
|
||||
/opt/mnt/bucket-a, /opt/mnt/bucket-b, /opt/mnt/bucket-c respectively.
|
||||
|
||||
Create the mount folder and set its ownership to desired user, group.
|
||||
|
||||
```console
|
||||
[opc@base-inst-boot ~]$ sudo mkdir /opt/mnt
|
||||
[opc@base-inst-boot ~]$ sudo chown -R opc:adm /opt/mnt
|
||||
```
|
||||
|
||||
Set chmod permissions to user, group, others as desired for each mount path
|
||||
|
||||
```console
|
||||
[opc@base-inst-boot ~]$ sudo chmod 764 /opt/mnt
|
||||
[opc@base-inst-boot ~]$ ls -al /opt/mnt/
|
||||
total 0
|
||||
drwxrw-r--. 2 opc adm 6 Apr 7 18:01 .
|
||||
drwxr-xr-x. 10 root root 179 Apr 7 18:01 ..
|
||||
|
||||
[opc@base-inst-boot ~]$ mkdir -p /opt/mnt/bucket-a
|
||||
[opc@base-inst-boot ~]$ mkdir -p /opt/mnt/bucket-b
|
||||
[opc@base-inst-boot ~]$ mkdir -p /opt/mnt/bucket-c
|
||||
|
||||
[opc@base-inst-boot ~]$ ls -al /opt/mnt
|
||||
total 0
|
||||
drwxrw-r--. 5 opc adm 54 Apr 7 18:17 .
|
||||
drwxr-xr-x. 10 root root 179 Apr 7 18:01 ..
|
||||
drwxrwxr-x. 2 opc opc 6 Apr 7 18:17 bucket-a
|
||||
drwxrwxr-x. 2 opc opc 6 Apr 7 18:17 bucket-b
|
||||
drwxrwxr-x. 2 opc opc 6 Apr 7 18:17 bucket-c
|
||||
```
|
||||
|
||||
## Step 5: Identify Rclone mount CLI configuration settings to use
|
||||
|
||||
Please read through this [rclone mount](https://rclone.org/commands/rclone_mount/)
|
||||
page completely to really understand the mount and its flags, what is rclone
|
||||
[virtual file system](https://rclone.org/commands/rclone_mount/#vfs-virtual-file-system)
|
||||
mode settings and how to effectively use them for desired Read/Write consistencies.
|
||||
|
||||
Local File systems expect things to be 100% reliable, whereas cloud storage
|
||||
systems are a long way from 100% reliable. Object storage can throw several
|
||||
errors like 429, 503, 404 etc. The rclone sync/copy commands cope with this
|
||||
with lots of retries. However rclone mount can't use retries in the same way
|
||||
without making local copies of the uploads. Please Look at the VFS File Caching
|
||||
for solutions to make mount more reliable.
|
||||
|
||||
First lets understand the rclone mount flags and some global flags for troubleshooting.
|
||||
|
||||
```console
|
||||
rclone mount \
|
||||
ossa:bucket-a \ # Remote:bucket-name
|
||||
/opt/mnt/bucket-a \ # Local mount folder
|
||||
--config /etc/rclone/rclone.conf \ # Path to rclone config file
|
||||
--allow-non-empty \ # Allow mounting over a non-empty directory
|
||||
--dir-perms 0770 \ # Directory permissions (default 0777)
|
||||
--file-perms 0660 \ # File permissions (default 0666)
|
||||
--allow-other \ # Allow access to other users
|
||||
--umask 0117 \ # sets (660) rw-rw---- as permissions for the mount using the umask
|
||||
--transfers 8 \ # default 4, can be set to adjust the number of parallel uploads of modified files to remote from the cache
|
||||
--tpslimit 50 \ # Limit HTTP transactions per second to this. A transaction is roughly defined as an API call;
|
||||
# its exact meaning will depend on the backend. For HTTP based backends it is an HTTP PUT/GET/POST/etc and its response
|
||||
--cache-dir /tmp/rclone/cache # Directory rclone will use for caching.
|
||||
--dir-cache-time 5m \ # Time to cache directory entries for (default 5m0s)
|
||||
--vfs-cache-mode writes \ # Cache mode off|minimal|writes|full (default off), writes gives the maximum compatibility like a local disk
|
||||
--vfs-cache-max-age 20m \ # Max age of objects in the cache (default 1h0m0s)
|
||||
--vfs-cache-max-size 10G \ # Max total size of objects in the cache (default off)
|
||||
--vfs-cache-poll-interval 1m \ # Interval to poll the cache for stale objects (default 1m0s)
|
||||
--vfs-write-back 5s \ # Time to writeback files after last use when using cache (default 5s).
|
||||
# Note that files are written back to the remote only when they are closed and
|
||||
# if they haven't been accessed for --vfs-write-back seconds. If rclone is quit or
|
||||
# dies with files that haven't been uploaded, these will be uploaded next time rclone is run with the same flags.
|
||||
--vfs-fast-fingerprint # Use fast (less accurate) fingerprints for change detection.
|
||||
--log-level ERROR \ # log level, can be DEBUG, INFO, ERROR
|
||||
--log-file /var/log/rclone/oosa-bucket-a.log # rclone application log
|
||||
```
|
||||
|
||||
### --vfs-cache-mode writes
|
||||
|
||||
In this mode files opened for read only are still read directly from the
|
||||
remote, write only and read/write files are buffered to disk first. This mode
|
||||
should support all normal file system operations. If an upload fails it will be
|
||||
retried at exponentially increasing intervals up to 1 minute.
|
||||
|
||||
VFS cache mode of writes is recommended, so that application can have maximum
|
||||
compatibility of using remote storage as a local disk, when write is finished,
|
||||
file is closed, it is uploaded to backend remote after vfs-write-back duration
|
||||
has elapsed. If rclone is quit or dies with files that haven't been uploaded,
|
||||
these will be uploaded next time rclone is run with the same flags.
|
||||
|
||||
### --tpslimit float
|
||||
|
||||
Limit transactions per second to this number. Default is 0 which is used to
|
||||
mean unlimited transactions per second.
|
||||
|
||||
A transaction is roughly defined as an API call; its exact meaning will depend
|
||||
on the backend. For HTTP based backends it is an HTTP PUT/GET/POST/etc and its
|
||||
response. For FTP/SFTP it is a round trip transaction over TCP.
|
||||
|
||||
For example, to limit rclone to 10 transactions per second use --tpslimit 10,
|
||||
or to 1 transaction every 2 seconds use --tpslimit 0.5.
|
||||
|
||||
Use this when the number of transactions per second from rclone is causing a
|
||||
problem with the cloud storage provider (e.g. getting you banned or rate
|
||||
limited or throttled).
|
||||
|
||||
This can be very useful for rclone mount to control the behaviour of
|
||||
applications using it. Let's guess and say Object storage allows roughly 100
|
||||
tps per tenant, so to be on safe side, it will be wise to set this at 50
|
||||
(tune it to actuals per region).
|
||||
|
||||
### --vfs-fast-fingerprint
|
||||
|
||||
If you use the --vfs-fast-fingerprint flag then rclone will not include the
|
||||
slow operations in the fingerprint. This makes the fingerprinting less accurate
|
||||
but much faster and will improve the opening time of cached files. If you are
|
||||
running a vfs cache over local, s3, object storage or swift backends then using
|
||||
this flag is recommended.
|
||||
|
||||
Various parts of the VFS use fingerprinting to see if a local file copy has
|
||||
changed relative to a remote file. Fingerprints are made from:
|
||||
|
||||
- size
|
||||
- modification time
|
||||
- hash
|
||||
where available on an object.
|
||||
|
||||
## Step 6: Mounting Options, Use Any one option
|
||||
|
||||
### Step 6a: Run as a Service Daemon: Configure FSTAB entry for Rclone mount
|
||||
|
||||
Add this entry in /etc/fstab:
|
||||
|
||||
```text
|
||||
ossa:bucket-a /opt/mnt/bucket-a rclone rw,umask=0117,nofail,_netdev,args2env,config=/etc/rclone/rclone.conf,uid=1000,gid=4,
|
||||
file_perms=0760,dir_perms=0760,allow_other,vfs_cache_mode=writes,cache_dir=/tmp/rclone/cache 0 0
|
||||
```
|
||||
|
||||
IMPORTANT: Please note in fstab entry arguments are specified as underscore
|
||||
instead of dash, example: vfs_cache_mode=writes instead of vfs-cache-mode=writes
|
||||
Rclone in the mount helper mode will split -o argument(s) by comma, replace `_`
|
||||
by `-` and prepend `--` to get the command-line flags. Options containing commas
|
||||
or spaces can be wrapped in single or double quotes. Any inner quotes inside outer
|
||||
quotes of the same type should be doubled.
|
||||
|
||||
Then run sudo mount -av
|
||||
|
||||
```console
|
||||
[opc@base-inst-boot ~]$ sudo mount -av
|
||||
/ : ignored
|
||||
/boot : already mounted
|
||||
/boot/efi : already mounted
|
||||
/var/oled : already mounted
|
||||
/dev/shm : already mounted
|
||||
none : ignored
|
||||
/opt/mnt/bucket-a : already mounted # This is the bucket mounted information, running mount -av again and again is idempotent.
|
||||
```
|
||||
|
||||
## Step 6b: Run as a Service Daemon: Configure systemd entry for Rclone mount
|
||||
|
||||
If you are familiar with configuring systemd unit files, you can also configure
|
||||
the each rclone mount into a systemd units file.
|
||||
various examples in git search: <https://github.com/search?l=Shell&q=rclone+unit&type=Code>
|
||||
|
||||
```console
|
||||
tee "/etc/systemd/system/rclonebucketa.service" > /dev/null <<EOF
|
||||
[Unit]
|
||||
Description=RCloneMounting
|
||||
After=multi-user.target
|
||||
[Service]
|
||||
Type=simple
|
||||
User=0
|
||||
Group=0
|
||||
ExecStart=/bin/bash /etc/rclone/scripts/bucket-a.sh
|
||||
ExecStop=/bin/fusermount -uz /opt/mnt/bucket-a
|
||||
TimeoutStopSec=20
|
||||
KillMode=process
|
||||
RemainAfterExit=yes
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
EOF
|
||||
```
|
||||
|
||||
## Step 7: Optional: Mount Nanny, for resiliency, recover from process crash
|
||||
|
||||
Sometimes, rclone process crashes and the mount points are left in dangling
|
||||
state where its mounted but the rclone mount process is gone. To clean up the
|
||||
mount point you can force unmount by running this command.
|
||||
|
||||
```console
|
||||
sudo fusermount -uz /opt/mnt/bucket-a
|
||||
```
|
||||
|
||||
One can also run a rclone_mount_nanny script, which detects and cleans up mount
|
||||
errors by unmounting and then auto-mounting.
|
||||
|
||||
Content of /etc/rclone/scripts/rclone_nanny_script.sh
|
||||
|
||||
```sh
|
||||
#!/usr/bin/env bash
|
||||
erroneous_list=$(df 2>&1 | grep -i 'Transport endpoint is not connected' | awk '{print ""$2"" }' | tr -d \:)
|
||||
rclone_list=$(findmnt -t fuse.rclone -n 2>&1 | awk '{print ""$1"" }' | tr -d \:)
|
||||
IFS=$'\n'; set -f
|
||||
intersection=$(comm -12 <(printf '%s\n' "$erroneous_list" | sort) <(printf '%s\n' "$rclone_list" | sort))
|
||||
for directory in $intersection
|
||||
do
|
||||
echo "$directory is being fixed."
|
||||
sudo fusermount -uz "$directory"
|
||||
done
|
||||
sudo mount -av
|
||||
```
|
||||
|
||||
Script to idempotently add a Cron job to babysit the mount paths every 5 minutes
|
||||
|
||||
```sh
|
||||
echo "Creating rclone nanny cron job."
|
||||
croncmd="/etc/rclone/scripts/rclone_nanny_script.sh"
|
||||
cronjob="*/5 * * * * $croncmd"
|
||||
# idempotency - adds rclone_nanny cronjob only if absent.
|
||||
( crontab -l | grep -v -F "$croncmd" || : ; echo "$cronjob" ) | crontab -
|
||||
echo "Finished creating rclone nanny cron job."
|
||||
```
|
||||
|
||||
Ensure the crontab is added, so that above nanny script runs every 5 minutes.
|
||||
|
||||
```console
|
||||
[opc@base-inst-boot ~]$ sudo crontab -l
|
||||
*/5 * * * * /etc/rclone/scripts/rclone_nanny_script.sh
|
||||
[opc@base-inst-boot ~]$
|
||||
```
|
||||
|
||||
## Step 8: Optional: Setup NFS server to access the mount points of rclone
|
||||
|
||||
Let's say you want to make the rclone mount path /opt/mnt/bucket-a available
|
||||
as a NFS server export so that other clients can access it by using a NFS client.
|
||||
|
||||
### Step 8a : Setup NFS server
|
||||
|
||||
Install NFS Utils
|
||||
|
||||
```console
|
||||
sudo yum install -y nfs-utils
|
||||
```
|
||||
|
||||
Export the desired directory via NFS Server in the same machine where rclone
|
||||
has mounted to, ensure NFS service has desired permissions to read the directory.
|
||||
If it runs as root, then it will have permissions for sure, but if it runs
|
||||
as separate user then ensure that user has necessary desired privileges.
|
||||
|
||||
```sh
|
||||
# this gives opc user and adm (administrators group) ownership to the path, so any user belonging to adm group will be able to access the files.
|
||||
[opc@tools ~]$ sudo chown -R opc:adm /opt/mnt/bucket-a/
|
||||
[opc@tools ~]$ sudo chmod 764 /opt/mnt/bucket-a/
|
||||
|
||||
# Not export the mount path of rclone for exposing via nfs server
|
||||
# There are various nfs export options that you should keep per desired usage.
|
||||
# Syntax is
|
||||
# <path> <allowed-ipaddr>(<option>)
|
||||
[opc@tools ~]$ cat /etc/exports
|
||||
/opt/mnt/bucket-a *(fsid=1,rw)
|
||||
|
||||
|
||||
# Restart NFS server
|
||||
[opc@tools ~]$ sudo systemctl restart nfs-server
|
||||
|
||||
|
||||
# Show Export paths
|
||||
[opc@tools ~]$ showmount -e
|
||||
Export list for tools:
|
||||
/opt/mnt/bucket-a *
|
||||
|
||||
# Know the port NFS server is running as, in this case it's listening on port 2049
|
||||
[opc@tools ~]$ sudo rpcinfo -p | grep nfs
|
||||
100003 3 tcp 2049 nfs
|
||||
100003 4 tcp 2049 nfs
|
||||
100227 3 tcp 2049 nfs_acl
|
||||
|
||||
# Allow NFS service via firewall
|
||||
[opc@tools ~]$ sudo firewall-cmd --add-service=nfs --permanent
|
||||
Warning: ALREADY_ENABLED: nfs
|
||||
success
|
||||
[opc@tools ~]$ sudo firewall-cmd --reload
|
||||
success
|
||||
[opc@tools ~]$
|
||||
|
||||
# Check status of NFS service
|
||||
[opc@tools ~]$ sudo systemctl status nfs-server.service
|
||||
● nfs-server.service - NFS server and services
|
||||
Loaded: loaded (/usr/lib/systemd/system/nfs-server.service; enabled; vendor preset: disabled)
|
||||
Active: active (exited) since Wed 2023-04-19 17:59:58 GMT; 13min ago
|
||||
Process: 2833741 ExecStopPost=/usr/sbin/exportfs -f (code=exited, status=0/SUCCESS)
|
||||
Process: 2833740 ExecStopPost=/usr/sbin/exportfs -au (code=exited, status=0/SUCCESS)
|
||||
Process: 2833737 ExecStop=/usr/sbin/rpc.nfsd 0 (code=exited, status=0/SUCCESS)
|
||||
Process: 2833766 ExecStart=/bin/sh -c if systemctl -q is-active gssproxy; then systemctl reload gssproxy ; fi (code=exit>
|
||||
Process: 2833756 ExecStart=/usr/sbin/rpc.nfsd (code=exited, status=0/SUCCESS)
|
||||
Process: 2833754 ExecStartPre=/usr/sbin/exportfs -r (code=exited, status=0/SUCCESS)
|
||||
Main PID: 2833766 (code=exited, status=0/SUCCESS)
|
||||
Tasks: 0 (limit: 48514)
|
||||
Memory: 0B
|
||||
CGroup: /system.slice/nfs-server.service
|
||||
|
||||
Apr 19 17:59:58 tools systemd[1]: Starting NFS server and services...
|
||||
Apr 19 17:59:58 tools systemd[1]: Started NFS server and services.
|
||||
```
|
||||
|
||||
### Step 8b : Setup NFS client
|
||||
|
||||
Now to connect to the NFS server from a different client machine, ensure the
|
||||
client machine can reach to nfs server machine over tcp port 2049, ensure your
|
||||
subnet network acls allow from desired source IP ranges to destination:2049 port.
|
||||
|
||||
In the client machine Mount the external NFS
|
||||
|
||||
```sh
|
||||
# Install nfs-utils
|
||||
[opc@base-inst-boot ~]$ sudo yum install -y nfs-utils
|
||||
|
||||
# In /etc/fstab, add the below entry
|
||||
[opc@base-inst-boot ~]$ cat /etc/fstab | grep nfs
|
||||
<ProvideYourIPAddress>:/opt/mnt/bucket-a /opt/mnt/buckert-a nfs rw 0 0
|
||||
|
||||
# remount so that newly added path gets mounted.
|
||||
[opc@base-inst-boot ~]$ sudo mount -av
|
||||
/ : ignored
|
||||
/boot : already mounted
|
||||
/boot/efi : already mounted
|
||||
/var/oled : already mounted
|
||||
/dev/shm : already mounted
|
||||
/home/opc/share_drive/bucketa: already mounted
|
||||
/opt/mnt/bucket-a: successfully mounted # this is the NFS mount
|
||||
```
|
||||
|
||||
### Step 8c : Test Connection
|
||||
|
||||
```sh
|
||||
# List files to test connection
|
||||
[opc@base-inst-boot ~]$ ls -al /opt/mnt/bucket-a
|
||||
total 1
|
||||
drw-rw----. 1 opc adm 0 Apr 18 17:28 .
|
||||
drwxrw-r--. 7 opc adm 85 Apr 18 17:36 ..
|
||||
drw-rw----. 1 opc adm 0 Apr 18 17:29 FILES
|
||||
-rw-rw----. 1 opc adm 15 Apr 18 18:13 nfs.txt
|
||||
```
|
||||
Reference in New Issue
Block a user