mirror of
https://github.com/ForeverPyrite/r2client.git
synced 2025-12-10 01:38:07 +00:00
Made this whole thing somewhat publicly presentable.
Now that's not to say I cleaned up the source code, who knows what the heck is in there...
This commit is contained in:
2225
Cargo.lock
generated
Normal file
2225
Cargo.lock
generated
Normal file
File diff suppressed because it is too large
Load Diff
77
README.md
Normal file
77
README.md
Normal file
@@ -0,0 +1,77 @@
|
|||||||
|
# r2client
|
||||||
|
This is a Rust project ("cargo workspace" :nerd::point_up:) containing a few tools and utilities I created to
|
||||||
|
have an easier time interfacing with [Couldflare's R2 Bucket Storage](https://www.cloudflare.com/developer-platform/products/r2/).
|
||||||
|
Yeah.
|
||||||
|
Cool, right?
|
||||||
|
|
||||||
|
## r2client
|
||||||
|
A really really freaking simple API for uploading, downloading, and listing files from Couldflare R2 Buckets.
|
||||||
|
It's fast too, with minimal dependencies that WON'T add 40 seconds to your compile time!
|
||||||
|
|
||||||
|
|
||||||
|
Brief example of usage:
|
||||||
|
```rust
|
||||||
|
use r2client::{R2Bucket, R2Error};
|
||||||
|
|
||||||
|
#[tokio::main]
|
||||||
|
fn main() -> Result<(), R2Error> {
|
||||||
|
// Assuming you have the required environment variables (as outlined in the totally
|
||||||
|
// existent documentation) set or in .env...
|
||||||
|
let bucket = R2Bucket::new("my-bucket");
|
||||||
|
|
||||||
|
bucket.upload_file("example.png").await?
|
||||||
|
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
```
|
||||||
|
Would you look at that!
|
||||||
|
It's really that easy! Not to mention that is an asynchronous example too...
|
||||||
|
|
||||||
|
As of now it's absolutely usable.
|
||||||
|
There is room for some improvement on the backend, as well as the potential for various new features to be added.
|
||||||
|
Will absolutely be adding them.
|
||||||
|
Trust me bro.
|
||||||
|
|
||||||
|
A few notes:
|
||||||
|
- Since the content-type is determined by the [local file's extension](./r2client/src/mimetypes.rs)...
|
||||||
|
- When using tempfiles, you want them to preserve file extension
|
||||||
|
- If a mimetype isn't known based off of file extension, then it will default to `application/octect-stream`
|
||||||
|
- This is a mid quality list that I linked above, feel free to add to it or tell me I'm missing something
|
||||||
|
- If you want to forgo or alter the file extension for one reason another, this is useless to you for now
|
||||||
|
|
||||||
|
The content type tomfoolery is just about it though, for most general purposes, this will do. (I hope to iron that out eventually)
|
||||||
|
I also hope that I would be much faster and easier to use than AWS SDKs, but I'm not benchmarking that.
|
||||||
|
|
||||||
|
## r2cli
|
||||||
|
Exactly what it sounds like.
|
||||||
|
It's a rudimentary wrapper for the r2client library.
|
||||||
|
|
||||||
|
I used it for some testing, and I imagine someone else out there can use it for verifying their R2 Credentials too lol.
|
||||||
|
|
||||||
|
No streamlined way to install it yet, sorry.
|
||||||
|
I'll probably take up a spot on crates.io just for you, one person who both stumbles across this and can use it.
|
||||||
|
|
||||||
|
## r2python
|
||||||
|
***LITERALLY DOES NOT EXIST YET!!!***
|
||||||
|
|
||||||
|
I will come back around and port this library to have a Python interface just for the experience, hopefully (that's
|
||||||
|
pretty much what this whole workspace is for).
|
||||||
|
|
||||||
|
For the time being, I don't know how and I want to get the project that I wanted this R2Client for done first.
|
||||||
|
|
||||||
|
## aws_sigv4
|
||||||
|
This part of the library is a Rust implementation of signing requests using AWS's SigV4, since R2 is "S3 compatible".
|
||||||
|
Barely any dependencies here.
|
||||||
|
Take *THAT* AWS S3 SDK and your 400 freaking dependencies with 250 custom types for one request. Largo.
|
||||||
|
|
||||||
|
While I did make it it's own crate, just because I wanted to decouple it from the R2Client itself, it's still only
|
||||||
|
been tested using the S3 library, and it is not nearly as efficient or user-friendly as it could be.
|
||||||
|
|
||||||
|
This will only be useful for people who are using some API and need their requests signed with SigV4, either for
|
||||||
|
their own abstracted client or a specific one time use in a program or something atypical.
|
||||||
|
|
||||||
|
---
|
||||||
|
## Credit where credit's due
|
||||||
|
The libraries' APIs are inspired by [fayharinn's Python R2-Client](https://github.com/fayharinn/R2-Client), including their minimal dependency nature.
|
||||||
|
It's also where I blatantly stole the mimetypes from, but hey, it seemed AI generated!!!
|
||||||
|
<sub>oh yeah I'll commit my changes and submit a pull request for that one if I ever remember...although r2python should supersede it</sub>
|
||||||
@@ -2,6 +2,7 @@
|
|||||||
name = "aws_sigv4"
|
name = "aws_sigv4"
|
||||||
version = "0.1.0"
|
version = "0.1.0"
|
||||||
edition = "2024"
|
edition = "2024"
|
||||||
|
description = "Minimal-dependancy library to sign requests via AWS's SigV4."
|
||||||
|
|
||||||
[lib]
|
[lib]
|
||||||
|
|
||||||
|
|||||||
@@ -1,78 +1,12 @@
|
|||||||
# aws_signing library
|
# aws_sigv4
|
||||||
Yeah so this is a library for signing things with AWS SigV4.
|
Used by the r2client's R2Client to sign requests.
|
||||||
Pretty straightforward.
|
So cool, ikr?
|
||||||
In fact, there is are only 3 methods on the public API
|
|
||||||
1. to make the client
|
|
||||||
2. Change the client region (cause why not?)
|
|
||||||
3. To transform an unsigned request that will be sent to an AWS compatible server to one with proper SigV4 headers
|
|
||||||
(With this wording, should I take a mutable reference to headers instead of...hmmm, we will see if I get bored enough to improve it myself.)
|
|
||||||
|
|
||||||
If I'm being completely honest, this was part of my own `r2client` that I wrote, which is uses S3
|
|
||||||
So I don't imagine that I come back to this to actually make a proper AWS Signing library.
|
|
||||||
I could imagine it being useful to *someone* who also wants to make an AWS abstraction that doesn't have 2 **BILLION** dependences.
|
|
||||||
|
|
||||||
## Usage
|
|
||||||
|
|
||||||
wait that's what the doc comments and `cargo doc` command is for lmao,
|
|
||||||
guess I'll get rid of this block
|
|
||||||
|
|
||||||
### Unexpected Errors
|
|
||||||
If you're getting errors related to invalid signatures and you're thinking "It's probably that damn crate I'm using from that newbie!",
|
|
||||||
you're probably right.
|
|
||||||
If you got here because crates.io search brought you here, nice.
|
|
||||||
Didn't mean for that to happen, sorry!
|
|
||||||
There are much more developed options out there, like [reqsign](https://github.com/apache/opendal-reqsign), which will also handle more
|
|
||||||
of the service related things for you.
|
|
||||||
|
|
||||||
Alternatively, you can try to stick with this crate (mistake)
|
|
||||||
Using log level trace with whatever logging crate you use will print out all steps of the AWS signing process, so you can follow along
|
|
||||||
with your favorite documentation/examples and figure out where things go wrong.
|
|
||||||
With this you can either raise an issue, or review the code yourself to implement a working solution.
|
|
||||||
|
|
||||||
## Todo
|
## Todo
|
||||||
|
|
||||||
- [ ] Create unit tests
|
- [ ] Replace the http::Uri with a &str and parse it instead...duh (unless that doesn't exist, but it 100% does)
|
||||||
- [ ] Add option for session keys
|
- [ ] Use a mutable reference to a HeaderMap instead of that UGLY UGLY Vec<(String, String)> format
|
||||||
- [ ] Additional client or whatever for weirdos who want to use SigV4a
|
- Although I'm not sure how much better this will be with the whole sort in alphabetical order, but it'll be a hell of a lot cooler
|
||||||
- [ ] Perhaps drop `http` as a dependency?
|
- [ ] The unit tests lol
|
||||||
- [ ] Alternatively, have users pass a mutable reference to a HeaderMap instead of cloning and returning a new one.
|
- [ ] Introduce the option for session tokens
|
||||||
- This can kinda hurt ease of use...maybe, but I feel like the slight performance gain and easier management of custom headers makes it worth.
|
- [ ] SigV4a?
|
||||||
- That's also acting like the crate is mature enough to do anything but provide the minimal headers for SigV4
|
|
||||||
- [ ] SigV4a
|
|
||||||
- Is this even really used?
|
|
||||||
|
|
||||||
## Contributing
|
|
||||||
What.
|
|
||||||
|
|
||||||
This crate is supposed to be a really simple way to sign AWS request headers and nothing else.
|
|
||||||
Again, I created it for my Cloudflare R2 (AWS S3) client and I imagine there are a few gaps for other services,
|
|
||||||
as well as some exceptions to the general programmatic things I saw in the documentation.
|
|
||||||
|
|
||||||
If you'd like to help improve it, maybe to make sign requests for some other AWS service, then I'd
|
|
||||||
appreciate if as specific unit test was added to ensure functionality.
|
|
||||||
This should be along with the other unit tests remaining intact and passing (unless they are blatantly incompatible
|
|
||||||
with some AWS documentation)
|
|
||||||
|
|
||||||
The code base (one lib.rs file) is pretty straightforward, it very directly follows the outline that AWS's SigV4 documentation provides,
|
|
||||||
explicitly containing all prerequisite functions and having a function call for all 5 of the steps.
|
|
||||||
I did this since the other examples were hard to follow, just a chunk of code that doesn't explain itself at any point.
|
|
||||||
All prerequisite functions, along with functions that don't require any of the keys/credential fields (in SigV4Credentials)
|
|
||||||
are defined outside of the struct.
|
|
||||||
|
|
||||||
Gl;hf
|
|
||||||
|
|
||||||
# welp
|
|
||||||
figure I'd try my hand at writing some public stuff like this, even though I'm not planing on publishing this publicly, only
|
|
||||||
compiled in the r2client.
|
|
||||||
I feel...okay about it.
|
|
||||||
I still feel like this stuff will be so small no matter what that I can still put a little personality into it.
|
|
||||||
|
|
||||||
I am incredibly verbose, shocking I know, but I think that's a bad thing.
|
|
||||||
I should be more brief, and then more verbose when relevant within the code.
|
|
||||||
Outside of that though, I think this is...fine.
|
|
||||||
I just need to get better at less yapping.
|
|
||||||
|
|
||||||
I really think this crate is too intermingled with the R2 Client to be useful to anyone else, really, although I think this was
|
|
||||||
*fine* for organizational purposes.
|
|
||||||
And who knows.
|
|
||||||
Maybe I'll need it again.
|
|
||||||
|
|||||||
@@ -1,3 +1,10 @@
|
|||||||
|
oh hi
|
||||||
|
I haven't done the unit tests yet.
|
||||||
|
I thought it wouldn't work but then it just randomly worked after not working with no changes
|
||||||
|
which was really weird
|
||||||
|
I should probably still put the unit tests in there, cause they kinda just...fail...
|
||||||
|
|
||||||
|
|
||||||
Example: GET Object
|
Example: GET Object
|
||||||
The following example gets the first 10 bytes of an object (test.txt) from examplebucket. For more information about the API action, see GetObject.
|
The following example gets the first 10 bytes of an object (test.txt) from examplebucket. For more information about the API action, see GetObject.
|
||||||
|
|
||||||
|
|||||||
@@ -371,6 +371,13 @@ mod tests {
|
|||||||
bucket,
|
bucket,
|
||||||
file_key
|
file_key
|
||||||
);
|
);
|
||||||
|
// "Two copies for the same thing? That's stupid!" I agree.
|
||||||
|
// I want people to be able to use whatever HTTP Client they want, but reqwest doesn't like
|
||||||
|
// that idea.
|
||||||
|
// It dawns upon me that I can just accept a string, try to parse it to http::Uri, and just
|
||||||
|
// use it internally.
|
||||||
|
// damn.
|
||||||
|
// crazy.
|
||||||
let endpoint: http::Uri = url.parse().unwrap();
|
let endpoint: http::Uri = url.parse().unwrap();
|
||||||
|
|
||||||
let headers = vec![("host".to_string(), endpoint.host().unwrap().to_owned())];
|
let headers = vec![("host".to_string(), endpoint.host().unwrap().to_owned())];
|
||||||
|
|||||||
@@ -2,12 +2,19 @@
|
|||||||
name = "r2cli"
|
name = "r2cli"
|
||||||
version = "0.1.0"
|
version = "0.1.0"
|
||||||
edition = "2024"
|
edition = "2024"
|
||||||
authors = ["foreverpyrite <foreverpyrite+cratesio@gmail.com"]
|
authors = ["foreverpyrite <r2client@foreverpyrite.com"]
|
||||||
description = "CLI for Cloudflare R2's S3-compatible storage using r2client"
|
description = "CLI for Cloudflare R2's S3-compatible storage using r2client"
|
||||||
|
|
||||||
[dependencies]
|
[dependencies]
|
||||||
|
# Command Line Argument Parsing :exploding_head:
|
||||||
clap = { version = "4", features = ["derive"] }
|
clap = { version = "4", features = ["derive"] }
|
||||||
|
|
||||||
|
# That way this can be used for some basic troubleshooting
|
||||||
|
# ~~not rn though, the error handling is non-existent~~
|
||||||
dotenv = "0.15"
|
dotenv = "0.15"
|
||||||
|
|
||||||
|
# The R2Client for the R2 Command Line, crazy
|
||||||
|
# Since I'm not doing anything asyncronus for now, it's easier (lazier) to just use the blocking client
|
||||||
r2client = { path = "../r2client", default-features = false, features = [
|
r2client = { path = "../r2client", default-features = false, features = [
|
||||||
"sync",
|
"sync",
|
||||||
] }
|
] }
|
||||||
|
|||||||
@@ -28,8 +28,10 @@ r2cli list-folders
|
|||||||
```
|
```
|
||||||
|
|
||||||
## Requirements
|
## Requirements
|
||||||
- Rust
|
- Rust (WOAH (I'm too lazy to build the binary and put it elsewhere))
|
||||||
- Valid Cloudflare R2 credentials
|
- Valid Cloudflare R2 credentials
|
||||||
|
|
||||||
## Todo
|
## Todo
|
||||||
- [ ] If you REALLY feel goofy, a TUI would be pretty sick
|
- [ ] Allow multiple, parallel, file uploads under a specific key/folder
|
||||||
|
- [ ] Just more stuff like the above, download all the objects in a key/folder, ect
|
||||||
|
- [ ] If you REALLY feel goofy, a TUI would be pretty sick, however the r2client APIs would need extended quite a bit
|
||||||
|
|||||||
@@ -1,5 +0,0 @@
|
|||||||
howdy
|
|
||||||
my name is
|
|
||||||
WHAT
|
|
||||||
my name is
|
|
||||||
HUH
|
|
||||||
@@ -2,22 +2,35 @@
|
|||||||
name = "r2client"
|
name = "r2client"
|
||||||
version = "0.2.0"
|
version = "0.2.0"
|
||||||
edition = "2024"
|
edition = "2024"
|
||||||
|
authors = ["ForeverPyrite <r2@foreverpyrite.com"]
|
||||||
|
|
||||||
[lib]
|
[lib]
|
||||||
|
|
||||||
[dependencies]
|
[dependencies]
|
||||||
|
# Client to send the http requests
|
||||||
reqwest = "0.12.19"
|
reqwest = "0.12.19"
|
||||||
|
# To parse the information about objects within a bucket
|
||||||
xmltree = "0.11.0"
|
xmltree = "0.11.0"
|
||||||
thiserror = "2"
|
# Validates and manages methods, headers, and urls
|
||||||
http = "1.3.1"
|
http = "1.3.1"
|
||||||
|
# Signs the S3 requests with SigV4
|
||||||
aws_sigv4 = { path = "../aws_sigv4/" }
|
aws_sigv4 = { path = "../aws_sigv4/" }
|
||||||
|
|
||||||
|
# Logging
|
||||||
log = "0.4.28"
|
log = "0.4.28"
|
||||||
|
# Painless error creation (for me)
|
||||||
|
thiserror = "2"
|
||||||
|
|
||||||
[dev-dependencies]
|
[dev-dependencies]
|
||||||
|
# Runtime for the async requests
|
||||||
tokio = { version = "1", features = ["full", "macros", "rt-multi-thread"] }
|
tokio = { version = "1", features = ["full", "macros", "rt-multi-thread"] }
|
||||||
|
# Cause you ain't getting my env variables and I ain't setting them every time
|
||||||
dotenv = "0.15"
|
dotenv = "0.15"
|
||||||
|
|
||||||
[features]
|
[features]
|
||||||
async = []
|
|
||||||
default = ["async"]
|
default = ["async"]
|
||||||
|
# The asyncronous API
|
||||||
|
async = []
|
||||||
|
# The syncronous, blocking API
|
||||||
|
# yeah surprise, still uses reqwest.
|
||||||
sync = ["reqwest/blocking"]
|
sync = ["reqwest/blocking"]
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
## For release:
|
## For release:
|
||||||
- [ ] Create a crate::Result that is Result<u8, R2Error>, and have Ok(status_code)
|
- [ ] Create a crate::Result that is Result<u8, R2Error>, and have Ok(status_code)
|
||||||
- [ ] Consider dropping more dependencies, using hyper or some lower level stuff for async, and then http for blocking
|
- [ ] Allow users to use custom mimetypes instead of only inferring from file extension
|
||||||
- [ ] A way to view the file contents (UTF-8 valid) would be cool
|
- [ ] A way to view the file contents (UTF-8 valid) would be cool
|
||||||
- [ ] Add functions that will list files with their metadata (perhaps a simple R2File type?)
|
- [ ] Add functions that will list files with their metadata (perhaps a simple R2File type?)
|
||||||
- [ ] Clear out all all print statements and consider logging (this is a library, after all)
|
- [ ] Clear out all all print statements and consider logging (this is a library, after all)
|
||||||
|
|||||||
@@ -4,4 +4,4 @@ version = "0.1.0"
|
|||||||
edition = "2024"
|
edition = "2024"
|
||||||
|
|
||||||
[dependencies]
|
[dependencies]
|
||||||
pyo3 = "0.25.0"
|
pyo3 = "0.26.0"
|
||||||
|
|||||||
Reference in New Issue
Block a user