📦 deps(thirdparty): update snapshots
This commit is contained in:
ci[bot]
@@ -0,0 +1,355 @@
|
||||
---
|
||||
title: "Internet Archive"
|
||||
description: "Rclone docs for Internet Archive"
|
||||
versionIntroduced: "v1.59"
|
||||
---
|
||||
|
||||
> **Official documentation:** [https://rclone.org/internetarchive/](https://rclone.org/internetarchive/)
|
||||
# Internet Archive
|
||||
|
||||
The Internet Archive backend utilizes Items on [archive.org](https://archive.org/)
|
||||
|
||||
Refer to [IAS3 API documentation](https://archive.org/services/docs/api/ias3.html)
|
||||
for the API this backend uses.
|
||||
|
||||
Paths are specified as `remote:bucket` (or `remote:` for the `lsd`
|
||||
command.) You may put subdirectories in too, e.g. `remote:item/path/to/dir`.
|
||||
|
||||
Unlike S3, listing up all items uploaded by you isn't supported.
|
||||
|
||||
Once you have made a remote, you can use it like this:
|
||||
|
||||
Make a new item
|
||||
|
||||
```console
|
||||
rclone mkdir remote:item
|
||||
```
|
||||
|
||||
List the contents of a item
|
||||
|
||||
```console
|
||||
rclone ls remote:item
|
||||
```
|
||||
|
||||
Sync `/home/local/directory` to the remote item, deleting any excess
|
||||
files in the item.
|
||||
|
||||
```console
|
||||
rclone sync --interactive /home/local/directory remote:item
|
||||
```
|
||||
|
||||
## Notes
|
||||
|
||||
Because of Internet Archive's architecture, it enqueues write operations (and
|
||||
extra post-processings) in a per-item queue. You can check item's queue at
|
||||
<https://catalogd.archive.org/history/item-name-here>. Because of that, all
|
||||
uploads/deletes will not show up immediately and takes some time to be available.
|
||||
The per-item queue is enqueued to an another queue, Item Deriver Queue.
|
||||
[You can check the status of Item Deriver Queue here.](https://catalogd.archive.org/catalog.php?whereami=1)
|
||||
This queue has a limit, and it may block you from uploading, or even deleting.
|
||||
You should avoid uploading a lot of small files for better behavior.
|
||||
|
||||
You can optionally wait for the server's processing to finish, by setting
|
||||
non-zero value to `wait_archive` key. By making it wait, rclone can do normal
|
||||
file comparison. Make sure to set a large enough value (e.g. `30m0s` for smaller
|
||||
files) as it can take a long time depending on server's queue.
|
||||
|
||||
## About metadata
|
||||
|
||||
This backend supports setting, updating and reading metadata of each file.
|
||||
The metadata will appear as file metadata on Internet Archive.
|
||||
However, some fields are reserved by both Internet Archive and rclone.
|
||||
|
||||
The following are reserved by Internet Archive:
|
||||
|
||||
- `name`
|
||||
- `source`
|
||||
- `size`
|
||||
- `md5`
|
||||
- `crc32`
|
||||
- `sha1`
|
||||
- `format`
|
||||
- `old_version`
|
||||
- `viruscheck`
|
||||
- `summation`
|
||||
|
||||
Trying to set values to these keys is ignored with a warning.
|
||||
Only setting `mtime` is an exception. Doing so make it the identical
|
||||
behavior as setting ModTime.
|
||||
|
||||
rclone reserves all the keys starting with `rclone-`. Setting value for
|
||||
these keys will give you warnings, but values are set according to request.
|
||||
|
||||
If there are multiple values for a key, only the first one is returned.
|
||||
This is a limitation of rclone, that supports one value per one key.
|
||||
It can be triggered when you did a server-side copy.
|
||||
|
||||
Reading metadata will also provide custom (non-standard nor reserved) ones.
|
||||
|
||||
## Filtering auto generated files
|
||||
|
||||
The Internet Archive automatically creates metadata files after
|
||||
upload. These can cause problems when doing an `rclone sync` as rclone
|
||||
will try, and fail, to delete them. These metadata files are not
|
||||
changeable, as they are created by the Internet Archive automatically.
|
||||
|
||||
These auto-created files can be excluded from the sync using [metadata
|
||||
filtering](/filtering/#metadata).
|
||||
|
||||
```console
|
||||
rclone sync ... --metadata-exclude "source=metadata" --metadata-exclude "format=Metadata"
|
||||
```
|
||||
|
||||
Which excludes from the sync any files which have the
|
||||
`source=metadata` or `format=Metadata` flags which are added to
|
||||
Internet Archive auto-created files.
|
||||
|
||||
## Configuration
|
||||
|
||||
Here is an example of making an internetarchive configuration.
|
||||
Most applies to the other providers as well, any differences are described [below](#providers).
|
||||
|
||||
First run
|
||||
|
||||
```console
|
||||
rclone config
|
||||
```
|
||||
|
||||
This will guide you through an interactive setup process.
|
||||
|
||||
```text
|
||||
No remotes found, make a new one\?
|
||||
n) New remote
|
||||
s) Set configuration password
|
||||
q) Quit config
|
||||
n/s/q> n
|
||||
name> remote
|
||||
Option Storage.
|
||||
Type of storage to configure.
|
||||
Choose a number from below, or type in your own value.
|
||||
XX / InternetArchive Items
|
||||
\ (internetarchive)
|
||||
Storage> internetarchive
|
||||
Option access_key_id.
|
||||
IAS3 Access Key.
|
||||
Leave blank for anonymous access.
|
||||
You can find one here: https://archive.org/account/s3.php
|
||||
Enter a value. Press Enter to leave empty.
|
||||
access_key_id> XXXX
|
||||
Option secret_access_key.
|
||||
IAS3 Secret Key (password).
|
||||
Leave blank for anonymous access.
|
||||
Enter a value. Press Enter to leave empty.
|
||||
secret_access_key> XXXX
|
||||
Edit advanced config?
|
||||
y) Yes
|
||||
n) No (default)
|
||||
y/n> y
|
||||
Option endpoint.
|
||||
IAS3 Endpoint.
|
||||
Leave blank for default value.
|
||||
Enter a string value. Press Enter for the default (https://s3.us.archive.org).
|
||||
endpoint>
|
||||
Option front_endpoint.
|
||||
Host of InternetArchive Frontend.
|
||||
Leave blank for default value.
|
||||
Enter a string value. Press Enter for the default (https://archive.org).
|
||||
front_endpoint>
|
||||
Option disable_checksum.
|
||||
Don't store MD5 checksum with object metadata.
|
||||
Normally rclone will calculate the MD5 checksum of the input before
|
||||
uploading it so it can ask the server to check the object against checksum.
|
||||
This is great for data integrity checking but can cause long delays for
|
||||
large files to start uploading.
|
||||
Enter a boolean value (true or false). Press Enter for the default (true).
|
||||
disable_checksum> true
|
||||
Option encoding.
|
||||
The encoding for the backend.
|
||||
See the [encoding section in the overview](/overview/#encoding) for more info.
|
||||
Enter a encoder.MultiEncoder value. Press Enter for the default (Slash,Question,Hash,Percent,Del,Ctl,InvalidUtf8,Dot).
|
||||
encoding>
|
||||
Edit advanced config?
|
||||
y) Yes
|
||||
n) No (default)
|
||||
y/n> n
|
||||
Configuration complete.
|
||||
Options:
|
||||
- type: internetarchive
|
||||
- access_key_id: XXXX
|
||||
- secret_access_key: XXXX
|
||||
Keep this "remote" remote?
|
||||
y) Yes this is OK (default)
|
||||
e) Edit this remote
|
||||
d) Delete this remote
|
||||
y/e/d> y
|
||||
```
|
||||
|
||||
<!-- autogenerated options start - DO NOT EDIT - instead edit fs.RegInfo in backend/internetarchive/internetarchive.go and run make backenddocs to verify --> <!-- markdownlint-disable-line line-length -->
|
||||
### Standard options
|
||||
|
||||
Here are the Standard options specific to internetarchive (Internet Archive).
|
||||
|
||||
#### --internetarchive-access-key-id
|
||||
|
||||
IAS3 Access Key.
|
||||
|
||||
Leave blank for anonymous access.
|
||||
You can find one here: https://archive.org/account/s3.php
|
||||
|
||||
Properties:
|
||||
|
||||
- Config: access_key_id
|
||||
- Env Var: RCLONE_INTERNETARCHIVE_ACCESS_KEY_ID
|
||||
- Type: string
|
||||
- Required: false
|
||||
|
||||
#### --internetarchive-secret-access-key
|
||||
|
||||
IAS3 Secret Key (password).
|
||||
|
||||
Leave blank for anonymous access.
|
||||
|
||||
Properties:
|
||||
|
||||
- Config: secret_access_key
|
||||
- Env Var: RCLONE_INTERNETARCHIVE_SECRET_ACCESS_KEY
|
||||
- Type: string
|
||||
- Required: false
|
||||
|
||||
#### --internetarchive-item-derive
|
||||
|
||||
Whether to trigger derive on the IA item or not. If set to false, the item will not be derived by IA upon upload.
|
||||
The derive process produces a number of secondary files from an upload to make an upload more usable on the web.
|
||||
Setting this to false is useful for uploading files that are already in a format that IA can display or reduce burden on IA's infrastructure.
|
||||
|
||||
Properties:
|
||||
|
||||
- Config: item_derive
|
||||
- Env Var: RCLONE_INTERNETARCHIVE_ITEM_DERIVE
|
||||
- Type: bool
|
||||
- Default: true
|
||||
|
||||
### Advanced options
|
||||
|
||||
Here are the Advanced options specific to internetarchive (Internet Archive).
|
||||
|
||||
#### --internetarchive-endpoint
|
||||
|
||||
IAS3 Endpoint.
|
||||
|
||||
Leave blank for default value.
|
||||
|
||||
Properties:
|
||||
|
||||
- Config: endpoint
|
||||
- Env Var: RCLONE_INTERNETARCHIVE_ENDPOINT
|
||||
- Type: string
|
||||
- Default: "https://s3.us.archive.org"
|
||||
|
||||
#### --internetarchive-front-endpoint
|
||||
|
||||
Host of InternetArchive Frontend.
|
||||
|
||||
Leave blank for default value.
|
||||
|
||||
Properties:
|
||||
|
||||
- Config: front_endpoint
|
||||
- Env Var: RCLONE_INTERNETARCHIVE_FRONT_ENDPOINT
|
||||
- Type: string
|
||||
- Default: "https://archive.org"
|
||||
|
||||
#### --internetarchive-item-metadata
|
||||
|
||||
Metadata to be set on the IA item, this is different from file-level metadata that can be set using --metadata-set.
|
||||
Format is key=value and the 'x-archive-meta-' prefix is automatically added.
|
||||
|
||||
Properties:
|
||||
|
||||
- Config: item_metadata
|
||||
- Env Var: RCLONE_INTERNETARCHIVE_ITEM_METADATA
|
||||
- Type: stringArray
|
||||
- Default: []
|
||||
|
||||
#### --internetarchive-disable-checksum
|
||||
|
||||
Don't ask the server to test against MD5 checksum calculated by rclone.
|
||||
Normally rclone will calculate the MD5 checksum of the input before
|
||||
uploading it so it can ask the server to check the object against checksum.
|
||||
This is great for data integrity checking but can cause long delays for
|
||||
large files to start uploading.
|
||||
|
||||
Properties:
|
||||
|
||||
- Config: disable_checksum
|
||||
- Env Var: RCLONE_INTERNETARCHIVE_DISABLE_CHECKSUM
|
||||
- Type: bool
|
||||
- Default: true
|
||||
|
||||
#### --internetarchive-wait-archive
|
||||
|
||||
Timeout for waiting the server's processing tasks (specifically archive and book_op) to finish.
|
||||
Only enable if you need to be guaranteed to be reflected after write operations.
|
||||
0 to disable waiting. No errors to be thrown in case of timeout.
|
||||
|
||||
Properties:
|
||||
|
||||
- Config: wait_archive
|
||||
- Env Var: RCLONE_INTERNETARCHIVE_WAIT_ARCHIVE
|
||||
- Type: Duration
|
||||
- Default: 0s
|
||||
|
||||
#### --internetarchive-encoding
|
||||
|
||||
The encoding for the backend.
|
||||
|
||||
See the [encoding section in the overview](/overview/#encoding) for more info.
|
||||
|
||||
Properties:
|
||||
|
||||
- Config: encoding
|
||||
- Env Var: RCLONE_INTERNETARCHIVE_ENCODING
|
||||
- Type: Encoding
|
||||
- Default: Slash,LtGt,CrLf,Del,Ctl,InvalidUtf8,Dot
|
||||
|
||||
#### --internetarchive-description
|
||||
|
||||
Description of the remote.
|
||||
|
||||
Properties:
|
||||
|
||||
- Config: description
|
||||
- Env Var: RCLONE_INTERNETARCHIVE_DESCRIPTION
|
||||
- Type: string
|
||||
- Required: false
|
||||
|
||||
### Metadata
|
||||
|
||||
Metadata fields provided by Internet Archive.
|
||||
If there are multiple values for a key, only the first one is returned.
|
||||
This is a limitation of Rclone, that supports one value per one key.
|
||||
|
||||
Owner is able to add custom keys. Metadata feature grabs all the keys including them.
|
||||
|
||||
Here are the possible system metadata items for the internetarchive backend.
|
||||
|
||||
| Name | Help | Type | Example | Read Only |
|
||||
|------|------|------|---------|-----------|
|
||||
| crc32 | CRC32 calculated by Internet Archive | string | 01234567 | **Y** |
|
||||
| format | Name of format identified by Internet Archive | string | Comma-Separated Values | **Y** |
|
||||
| md5 | MD5 hash calculated by Internet Archive | string | 01234567012345670123456701234567 | **Y** |
|
||||
| mtime | Time of last modification, managed by Rclone | RFC 3339 | 2006-01-02T15:04:05.999999999Z | **Y** |
|
||||
| name | Full file path, without the bucket part | filename | backend/internetarchive/internetarchive.go | **Y** |
|
||||
| old_version | Whether the file was replaced and moved by keep-old-version flag | boolean | true | **Y** |
|
||||
| rclone-ia-mtime | Time of last modification, managed by Internet Archive | RFC 3339 | 2006-01-02T15:04:05.999999999Z | N |
|
||||
| rclone-mtime | Time of last modification, managed by Rclone | RFC 3339 | 2006-01-02T15:04:05.999999999Z | N |
|
||||
| rclone-update-track | Random value used by Rclone for tracking changes inside Internet Archive | string | aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa | N |
|
||||
| sha1 | SHA1 hash calculated by Internet Archive | string | 0123456701234567012345670123456701234567 | **Y** |
|
||||
| size | File size in bytes | decimal number | 123456 | **Y** |
|
||||
| source | The source of the file | string | original | **Y** |
|
||||
| summation | Check https://forum.rclone.org/t/31922 for how it is used | string | md5 | **Y** |
|
||||
| viruscheck | The last time viruscheck process was run for the file (?) | unixtime | 1654191352 | **Y** |
|
||||
|
||||
See the [metadata](/docs/#metadata) docs for more info.
|
||||
|
||||
<!-- autogenerated options stop -->
|
||||
Reference in New Issue
Block a user