Made this whole thing somewhat publicly presentable.

Now that's not to say I cleaned up the source code, who knows what the
heck is in there...
This commit is contained in:
ForeverPyrite
2025-09-19 23:31:53 -04:00
parent 844c9eea6e
commit f92026698c
12 changed files with 2355 additions and 87 deletions

2225
Cargo.lock generated Normal file

File diff suppressed because it is too large Load Diff

77
README.md Normal file
View File

@@ -0,0 +1,77 @@
# r2client
This is a Rust project ("cargo workspace" :nerd::point_up:) containing a few tools and utilities I created to
have an easier time interfacing with [Couldflare's R2 Bucket Storage](https://www.cloudflare.com/developer-platform/products/r2/).
Yeah.
Cool, right?
## r2client
A really really freaking simple API for uploading, downloading, and listing files from Couldflare R2 Buckets.
It's fast too, with minimal dependencies that WON'T add 40 seconds to your compile time!
Brief example of usage:
```rust
use r2client::{R2Bucket, R2Error};
#[tokio::main]
fn main() -> Result<(), R2Error> {
// Assuming you have the required environment variables (as outlined in the totally
// existent documentation) set or in .env...
let bucket = R2Bucket::new("my-bucket");
bucket.upload_file("example.png").await?
Ok(())
}
```
Would you look at that!
It's really that easy! Not to mention that is an asynchronous example too...
As of now it's absolutely usable.
There is room for some improvement on the backend, as well as the potential for various new features to be added.
Will absolutely be adding them.
Trust me bro.
A few notes:
- Since the content-type is determined by the [local file's extension](./r2client/src/mimetypes.rs)...
- When using tempfiles, you want them to preserve file extension
- If a mimetype isn't known based off of file extension, then it will default to `application/octect-stream`
- This is a mid quality list that I linked above, feel free to add to it or tell me I'm missing something
- If you want to forgo or alter the file extension for one reason another, this is useless to you for now
The content type tomfoolery is just about it though, for most general purposes, this will do. (I hope to iron that out eventually)
I also hope that I would be much faster and easier to use than AWS SDKs, but I'm not benchmarking that.
## r2cli
Exactly what it sounds like.
It's a rudimentary wrapper for the r2client library.
I used it for some testing, and I imagine someone else out there can use it for verifying their R2 Credentials too lol.
No streamlined way to install it yet, sorry.
I'll probably take up a spot on crates.io just for you, one person who both stumbles across this and can use it.
## r2python
***LITERALLY DOES NOT EXIST YET!!!***
I will come back around and port this library to have a Python interface just for the experience, hopefully (that's
pretty much what this whole workspace is for).
For the time being, I don't know how and I want to get the project that I wanted this R2Client for done first.
## aws_sigv4
This part of the library is a Rust implementation of signing requests using AWS's SigV4, since R2 is "S3 compatible".
Barely any dependencies here.
Take *THAT* AWS S3 SDK and your 400 freaking dependencies with 250 custom types for one request. Largo.
While I did make it it's own crate, just because I wanted to decouple it from the R2Client itself, it's still only
been tested using the S3 library, and it is not nearly as efficient or user-friendly as it could be.
This will only be useful for people who are using some API and need their requests signed with SigV4, either for
their own abstracted client or a specific one time use in a program or something atypical.
---
## Credit where credit's due
The libraries' APIs are inspired by [fayharinn's Python R2-Client](https://github.com/fayharinn/R2-Client), including their minimal dependency nature.
It's also where I blatantly stole the mimetypes from, but hey, it seemed AI generated!!!
<sub>oh yeah I'll commit my changes and submit a pull request for that one if I ever remember...although r2python should supersede it</sub>

View File

@@ -2,6 +2,7 @@
name = "aws_sigv4" name = "aws_sigv4"
version = "0.1.0" version = "0.1.0"
edition = "2024" edition = "2024"
description = "Minimal-dependancy library to sign requests via AWS's SigV4."
[lib] [lib]

View File

@@ -1,78 +1,12 @@
# aws_signing library # aws_sigv4
Yeah so this is a library for signing things with AWS SigV4. Used by the r2client's R2Client to sign requests.
Pretty straightforward. So cool, ikr?
In fact, there is are only 3 methods on the public API
1. to make the client
2. Change the client region (cause why not?)
3. To transform an unsigned request that will be sent to an AWS compatible server to one with proper SigV4 headers
(With this wording, should I take a mutable reference to headers instead of...hmmm, we will see if I get bored enough to improve it myself.)
If I'm being completely honest, this was part of my own `r2client` that I wrote, which is uses S3
So I don't imagine that I come back to this to actually make a proper AWS Signing library.
I could imagine it being useful to *someone* who also wants to make an AWS abstraction that doesn't have 2 **BILLION** dependences.
## Usage
wait that's what the doc comments and `cargo doc` command is for lmao,
guess I'll get rid of this block
### Unexpected Errors
If you're getting errors related to invalid signatures and you're thinking "It's probably that damn crate I'm using from that newbie!",
you're probably right.
If you got here because crates.io search brought you here, nice.
Didn't mean for that to happen, sorry!
There are much more developed options out there, like [reqsign](https://github.com/apache/opendal-reqsign), which will also handle more
of the service related things for you.
Alternatively, you can try to stick with this crate (mistake)
Using log level trace with whatever logging crate you use will print out all steps of the AWS signing process, so you can follow along
with your favorite documentation/examples and figure out where things go wrong.
With this you can either raise an issue, or review the code yourself to implement a working solution.
## Todo ## Todo
- [ ] Create unit tests - [ ] Replace the http::Uri with a &str and parse it instead...duh (unless that doesn't exist, but it 100% does)
- [ ] Add option for session keys - [ ] Use a mutable reference to a HeaderMap instead of that UGLY UGLY Vec<(String, String)> format
- [ ] Additional client or whatever for weirdos who want to use SigV4a - Although I'm not sure how much better this will be with the whole sort in alphabetical order, but it'll be a hell of a lot cooler
- [ ] Perhaps drop `http` as a dependency? - [ ] The unit tests lol
- [ ] Alternatively, have users pass a mutable reference to a HeaderMap instead of cloning and returning a new one. - [ ] Introduce the option for session tokens
- This can kinda hurt ease of use...maybe, but I feel like the slight performance gain and easier management of custom headers makes it worth. - [ ] SigV4a?
- That's also acting like the crate is mature enough to do anything but provide the minimal headers for SigV4
- [ ] SigV4a
- Is this even really used?
## Contributing
What.
This crate is supposed to be a really simple way to sign AWS request headers and nothing else.
Again, I created it for my Cloudflare R2 (AWS S3) client and I imagine there are a few gaps for other services,
as well as some exceptions to the general programmatic things I saw in the documentation.
If you'd like to help improve it, maybe to make sign requests for some other AWS service, then I'd
appreciate if as specific unit test was added to ensure functionality.
This should be along with the other unit tests remaining intact and passing (unless they are blatantly incompatible
with some AWS documentation)
The code base (one lib.rs file) is pretty straightforward, it very directly follows the outline that AWS's SigV4 documentation provides,
explicitly containing all prerequisite functions and having a function call for all 5 of the steps.
I did this since the other examples were hard to follow, just a chunk of code that doesn't explain itself at any point.
All prerequisite functions, along with functions that don't require any of the keys/credential fields (in SigV4Credentials)
are defined outside of the struct.
Gl;hf
# welp
figure I'd try my hand at writing some public stuff like this, even though I'm not planing on publishing this publicly, only
compiled in the r2client.
I feel...okay about it.
I still feel like this stuff will be so small no matter what that I can still put a little personality into it.
I am incredibly verbose, shocking I know, but I think that's a bad thing.
I should be more brief, and then more verbose when relevant within the code.
Outside of that though, I think this is...fine.
I just need to get better at less yapping.
I really think this crate is too intermingled with the R2 Client to be useful to anyone else, really, although I think this was
*fine* for organizational purposes.
And who knows.
Maybe I'll need it again.

View File

@@ -1,3 +1,10 @@
oh hi
I haven't done the unit tests yet.
I thought it wouldn't work but then it just randomly worked after not working with no changes
which was really weird
I should probably still put the unit tests in there, cause they kinda just...fail...
Example: GET Object Example: GET Object
The following example gets the first 10 bytes of an object (test.txt) from examplebucket. For more information about the API action, see GetObject. The following example gets the first 10 bytes of an object (test.txt) from examplebucket. For more information about the API action, see GetObject.

View File

@@ -371,6 +371,13 @@ mod tests {
bucket, bucket,
file_key file_key
); );
// "Two copies for the same thing? That's stupid!" I agree.
// I want people to be able to use whatever HTTP Client they want, but reqwest doesn't like
// that idea.
// It dawns upon me that I can just accept a string, try to parse it to http::Uri, and just
// use it internally.
// damn.
// crazy.
let endpoint: http::Uri = url.parse().unwrap(); let endpoint: http::Uri = url.parse().unwrap();
let headers = vec![("host".to_string(), endpoint.host().unwrap().to_owned())]; let headers = vec![("host".to_string(), endpoint.host().unwrap().to_owned())];

View File

@@ -2,12 +2,19 @@
name = "r2cli" name = "r2cli"
version = "0.1.0" version = "0.1.0"
edition = "2024" edition = "2024"
authors = ["foreverpyrite <foreverpyrite+cratesio@gmail.com"] authors = ["foreverpyrite <r2client@foreverpyrite.com"]
description = "CLI for Cloudflare R2's S3-compatible storage using r2client" description = "CLI for Cloudflare R2's S3-compatible storage using r2client"
[dependencies] [dependencies]
# Command Line Argument Parsing :exploding_head:
clap = { version = "4", features = ["derive"] } clap = { version = "4", features = ["derive"] }
# That way this can be used for some basic troubleshooting
# ~~not rn though, the error handling is non-existent~~
dotenv = "0.15" dotenv = "0.15"
# The R2Client for the R2 Command Line, crazy
# Since I'm not doing anything asyncronus for now, it's easier (lazier) to just use the blocking client
r2client = { path = "../r2client", default-features = false, features = [ r2client = { path = "../r2client", default-features = false, features = [
"sync", "sync",
] } ] }

View File

@@ -28,8 +28,10 @@ r2cli list-folders
``` ```
## Requirements ## Requirements
- Rust - Rust (WOAH (I'm too lazy to build the binary and put it elsewhere))
- Valid Cloudflare R2 credentials - Valid Cloudflare R2 credentials
## Todo ## Todo
- [ ] If you REALLY feel goofy, a TUI would be pretty sick - [ ] Allow multiple, parallel, file uploads under a specific key/folder
- [ ] Just more stuff like the above, download all the objects in a key/folder, ect
- [ ] If you REALLY feel goofy, a TUI would be pretty sick, however the r2client APIs would need extended quite a bit

View File

@@ -1,5 +0,0 @@
howdy
my name is
WHAT
my name is
HUH

View File

@@ -2,22 +2,35 @@
name = "r2client" name = "r2client"
version = "0.2.0" version = "0.2.0"
edition = "2024" edition = "2024"
authors = ["ForeverPyrite <r2@foreverpyrite.com"]
[lib] [lib]
[dependencies] [dependencies]
# Client to send the http requests
reqwest = "0.12.19" reqwest = "0.12.19"
# To parse the information about objects within a bucket
xmltree = "0.11.0" xmltree = "0.11.0"
thiserror = "2" # Validates and manages methods, headers, and urls
http = "1.3.1" http = "1.3.1"
# Signs the S3 requests with SigV4
aws_sigv4 = { path = "../aws_sigv4/" } aws_sigv4 = { path = "../aws_sigv4/" }
# Logging
log = "0.4.28" log = "0.4.28"
# Painless error creation (for me)
thiserror = "2"
[dev-dependencies] [dev-dependencies]
# Runtime for the async requests
tokio = { version = "1", features = ["full", "macros", "rt-multi-thread"] } tokio = { version = "1", features = ["full", "macros", "rt-multi-thread"] }
# Cause you ain't getting my env variables and I ain't setting them every time
dotenv = "0.15" dotenv = "0.15"
[features] [features]
async = []
default = ["async"] default = ["async"]
# The asyncronous API
async = []
# The syncronous, blocking API
# yeah surprise, still uses reqwest.
sync = ["reqwest/blocking"] sync = ["reqwest/blocking"]

View File

@@ -1,6 +1,6 @@
## For release: ## For release:
- [ ] Create a crate::Result that is Result<u8, R2Error>, and have Ok(status_code) - [ ] Create a crate::Result that is Result<u8, R2Error>, and have Ok(status_code)
- [ ] Consider dropping more dependencies, using hyper or some lower level stuff for async, and then http for blocking - [ ] Allow users to use custom mimetypes instead of only inferring from file extension
- [ ] A way to view the file contents (UTF-8 valid) would be cool - [ ] A way to view the file contents (UTF-8 valid) would be cool
- [ ] Add functions that will list files with their metadata (perhaps a simple R2File type?) - [ ] Add functions that will list files with their metadata (perhaps a simple R2File type?)
- [ ] Clear out all all print statements and consider logging (this is a library, after all) - [ ] Clear out all all print statements and consider logging (this is a library, after all)

View File

@@ -4,4 +4,4 @@ version = "0.1.0"
edition = "2024" edition = "2024"
[dependencies] [dependencies]
pyo3 = "0.25.0" pyo3 = "0.26.0"