mirror of
https://github.com/ForeverPyrite/r2client.git
synced 2025-12-10 01:38:07 +00:00
Made this whole thing somewhat publicly presentable.
Now that's not to say I cleaned up the source code, who knows what the heck is in there...
This commit is contained in:
2225
Cargo.lock
generated
Normal file
2225
Cargo.lock
generated
Normal file
File diff suppressed because it is too large
Load Diff
77
README.md
Normal file
77
README.md
Normal file
@@ -0,0 +1,77 @@
|
||||
# r2client
|
||||
This is a Rust project ("cargo workspace" :nerd::point_up:) containing a few tools and utilities I created to
|
||||
have an easier time interfacing with [Couldflare's R2 Bucket Storage](https://www.cloudflare.com/developer-platform/products/r2/).
|
||||
Yeah.
|
||||
Cool, right?
|
||||
|
||||
## r2client
|
||||
A really really freaking simple API for uploading, downloading, and listing files from Couldflare R2 Buckets.
|
||||
It's fast too, with minimal dependencies that WON'T add 40 seconds to your compile time!
|
||||
|
||||
|
||||
Brief example of usage:
|
||||
```rust
|
||||
use r2client::{R2Bucket, R2Error};
|
||||
|
||||
#[tokio::main]
|
||||
fn main() -> Result<(), R2Error> {
|
||||
// Assuming you have the required environment variables (as outlined in the totally
|
||||
// existent documentation) set or in .env...
|
||||
let bucket = R2Bucket::new("my-bucket");
|
||||
|
||||
bucket.upload_file("example.png").await?
|
||||
|
||||
Ok(())
|
||||
}
|
||||
```
|
||||
Would you look at that!
|
||||
It's really that easy! Not to mention that is an asynchronous example too...
|
||||
|
||||
As of now it's absolutely usable.
|
||||
There is room for some improvement on the backend, as well as the potential for various new features to be added.
|
||||
Will absolutely be adding them.
|
||||
Trust me bro.
|
||||
|
||||
A few notes:
|
||||
- Since the content-type is determined by the [local file's extension](./r2client/src/mimetypes.rs)...
|
||||
- When using tempfiles, you want them to preserve file extension
|
||||
- If a mimetype isn't known based off of file extension, then it will default to `application/octect-stream`
|
||||
- This is a mid quality list that I linked above, feel free to add to it or tell me I'm missing something
|
||||
- If you want to forgo or alter the file extension for one reason another, this is useless to you for now
|
||||
|
||||
The content type tomfoolery is just about it though, for most general purposes, this will do. (I hope to iron that out eventually)
|
||||
I also hope that I would be much faster and easier to use than AWS SDKs, but I'm not benchmarking that.
|
||||
|
||||
## r2cli
|
||||
Exactly what it sounds like.
|
||||
It's a rudimentary wrapper for the r2client library.
|
||||
|
||||
I used it for some testing, and I imagine someone else out there can use it for verifying their R2 Credentials too lol.
|
||||
|
||||
No streamlined way to install it yet, sorry.
|
||||
I'll probably take up a spot on crates.io just for you, one person who both stumbles across this and can use it.
|
||||
|
||||
## r2python
|
||||
***LITERALLY DOES NOT EXIST YET!!!***
|
||||
|
||||
I will come back around and port this library to have a Python interface just for the experience, hopefully (that's
|
||||
pretty much what this whole workspace is for).
|
||||
|
||||
For the time being, I don't know how and I want to get the project that I wanted this R2Client for done first.
|
||||
|
||||
## aws_sigv4
|
||||
This part of the library is a Rust implementation of signing requests using AWS's SigV4, since R2 is "S3 compatible".
|
||||
Barely any dependencies here.
|
||||
Take *THAT* AWS S3 SDK and your 400 freaking dependencies with 250 custom types for one request. Largo.
|
||||
|
||||
While I did make it it's own crate, just because I wanted to decouple it from the R2Client itself, it's still only
|
||||
been tested using the S3 library, and it is not nearly as efficient or user-friendly as it could be.
|
||||
|
||||
This will only be useful for people who are using some API and need their requests signed with SigV4, either for
|
||||
their own abstracted client or a specific one time use in a program or something atypical.
|
||||
|
||||
---
|
||||
## Credit where credit's due
|
||||
The libraries' APIs are inspired by [fayharinn's Python R2-Client](https://github.com/fayharinn/R2-Client), including their minimal dependency nature.
|
||||
It's also where I blatantly stole the mimetypes from, but hey, it seemed AI generated!!!
|
||||
<sub>oh yeah I'll commit my changes and submit a pull request for that one if I ever remember...although r2python should supersede it</sub>
|
||||
@@ -2,6 +2,7 @@
|
||||
name = "aws_sigv4"
|
||||
version = "0.1.0"
|
||||
edition = "2024"
|
||||
description = "Minimal-dependancy library to sign requests via AWS's SigV4."
|
||||
|
||||
[lib]
|
||||
|
||||
|
||||
@@ -1,78 +1,12 @@
|
||||
# aws_signing library
|
||||
Yeah so this is a library for signing things with AWS SigV4.
|
||||
Pretty straightforward.
|
||||
In fact, there is are only 3 methods on the public API
|
||||
1. to make the client
|
||||
2. Change the client region (cause why not?)
|
||||
3. To transform an unsigned request that will be sent to an AWS compatible server to one with proper SigV4 headers
|
||||
(With this wording, should I take a mutable reference to headers instead of...hmmm, we will see if I get bored enough to improve it myself.)
|
||||
|
||||
If I'm being completely honest, this was part of my own `r2client` that I wrote, which is uses S3
|
||||
So I don't imagine that I come back to this to actually make a proper AWS Signing library.
|
||||
I could imagine it being useful to *someone* who also wants to make an AWS abstraction that doesn't have 2 **BILLION** dependences.
|
||||
|
||||
## Usage
|
||||
|
||||
wait that's what the doc comments and `cargo doc` command is for lmao,
|
||||
guess I'll get rid of this block
|
||||
|
||||
### Unexpected Errors
|
||||
If you're getting errors related to invalid signatures and you're thinking "It's probably that damn crate I'm using from that newbie!",
|
||||
you're probably right.
|
||||
If you got here because crates.io search brought you here, nice.
|
||||
Didn't mean for that to happen, sorry!
|
||||
There are much more developed options out there, like [reqsign](https://github.com/apache/opendal-reqsign), which will also handle more
|
||||
of the service related things for you.
|
||||
|
||||
Alternatively, you can try to stick with this crate (mistake)
|
||||
Using log level trace with whatever logging crate you use will print out all steps of the AWS signing process, so you can follow along
|
||||
with your favorite documentation/examples and figure out where things go wrong.
|
||||
With this you can either raise an issue, or review the code yourself to implement a working solution.
|
||||
# aws_sigv4
|
||||
Used by the r2client's R2Client to sign requests.
|
||||
So cool, ikr?
|
||||
|
||||
## Todo
|
||||
|
||||
- [ ] Create unit tests
|
||||
- [ ] Add option for session keys
|
||||
- [ ] Additional client or whatever for weirdos who want to use SigV4a
|
||||
- [ ] Perhaps drop `http` as a dependency?
|
||||
- [ ] Alternatively, have users pass a mutable reference to a HeaderMap instead of cloning and returning a new one.
|
||||
- This can kinda hurt ease of use...maybe, but I feel like the slight performance gain and easier management of custom headers makes it worth.
|
||||
- That's also acting like the crate is mature enough to do anything but provide the minimal headers for SigV4
|
||||
- [ ] SigV4a
|
||||
- Is this even really used?
|
||||
|
||||
## Contributing
|
||||
What.
|
||||
|
||||
This crate is supposed to be a really simple way to sign AWS request headers and nothing else.
|
||||
Again, I created it for my Cloudflare R2 (AWS S3) client and I imagine there are a few gaps for other services,
|
||||
as well as some exceptions to the general programmatic things I saw in the documentation.
|
||||
|
||||
If you'd like to help improve it, maybe to make sign requests for some other AWS service, then I'd
|
||||
appreciate if as specific unit test was added to ensure functionality.
|
||||
This should be along with the other unit tests remaining intact and passing (unless they are blatantly incompatible
|
||||
with some AWS documentation)
|
||||
|
||||
The code base (one lib.rs file) is pretty straightforward, it very directly follows the outline that AWS's SigV4 documentation provides,
|
||||
explicitly containing all prerequisite functions and having a function call for all 5 of the steps.
|
||||
I did this since the other examples were hard to follow, just a chunk of code that doesn't explain itself at any point.
|
||||
All prerequisite functions, along with functions that don't require any of the keys/credential fields (in SigV4Credentials)
|
||||
are defined outside of the struct.
|
||||
|
||||
Gl;hf
|
||||
|
||||
# welp
|
||||
figure I'd try my hand at writing some public stuff like this, even though I'm not planing on publishing this publicly, only
|
||||
compiled in the r2client.
|
||||
I feel...okay about it.
|
||||
I still feel like this stuff will be so small no matter what that I can still put a little personality into it.
|
||||
|
||||
I am incredibly verbose, shocking I know, but I think that's a bad thing.
|
||||
I should be more brief, and then more verbose when relevant within the code.
|
||||
Outside of that though, I think this is...fine.
|
||||
I just need to get better at less yapping.
|
||||
|
||||
I really think this crate is too intermingled with the R2 Client to be useful to anyone else, really, although I think this was
|
||||
*fine* for organizational purposes.
|
||||
And who knows.
|
||||
Maybe I'll need it again.
|
||||
- [ ] Replace the http::Uri with a &str and parse it instead...duh (unless that doesn't exist, but it 100% does)
|
||||
- [ ] Use a mutable reference to a HeaderMap instead of that UGLY UGLY Vec<(String, String)> format
|
||||
- Although I'm not sure how much better this will be with the whole sort in alphabetical order, but it'll be a hell of a lot cooler
|
||||
- [ ] The unit tests lol
|
||||
- [ ] Introduce the option for session tokens
|
||||
- [ ] SigV4a?
|
||||
|
||||
@@ -1,3 +1,10 @@
|
||||
oh hi
|
||||
I haven't done the unit tests yet.
|
||||
I thought it wouldn't work but then it just randomly worked after not working with no changes
|
||||
which was really weird
|
||||
I should probably still put the unit tests in there, cause they kinda just...fail...
|
||||
|
||||
|
||||
Example: GET Object
|
||||
The following example gets the first 10 bytes of an object (test.txt) from examplebucket. For more information about the API action, see GetObject.
|
||||
|
||||
|
||||
@@ -371,6 +371,13 @@ mod tests {
|
||||
bucket,
|
||||
file_key
|
||||
);
|
||||
// "Two copies for the same thing? That's stupid!" I agree.
|
||||
// I want people to be able to use whatever HTTP Client they want, but reqwest doesn't like
|
||||
// that idea.
|
||||
// It dawns upon me that I can just accept a string, try to parse it to http::Uri, and just
|
||||
// use it internally.
|
||||
// damn.
|
||||
// crazy.
|
||||
let endpoint: http::Uri = url.parse().unwrap();
|
||||
|
||||
let headers = vec![("host".to_string(), endpoint.host().unwrap().to_owned())];
|
||||
|
||||
@@ -2,12 +2,19 @@
|
||||
name = "r2cli"
|
||||
version = "0.1.0"
|
||||
edition = "2024"
|
||||
authors = ["foreverpyrite <foreverpyrite+cratesio@gmail.com"]
|
||||
authors = ["foreverpyrite <r2client@foreverpyrite.com"]
|
||||
description = "CLI for Cloudflare R2's S3-compatible storage using r2client"
|
||||
|
||||
[dependencies]
|
||||
# Command Line Argument Parsing :exploding_head:
|
||||
clap = { version = "4", features = ["derive"] }
|
||||
|
||||
# That way this can be used for some basic troubleshooting
|
||||
# ~~not rn though, the error handling is non-existent~~
|
||||
dotenv = "0.15"
|
||||
|
||||
# The R2Client for the R2 Command Line, crazy
|
||||
# Since I'm not doing anything asyncronus for now, it's easier (lazier) to just use the blocking client
|
||||
r2client = { path = "../r2client", default-features = false, features = [
|
||||
"sync",
|
||||
] }
|
||||
|
||||
@@ -28,8 +28,10 @@ r2cli list-folders
|
||||
```
|
||||
|
||||
## Requirements
|
||||
- Rust
|
||||
- Rust (WOAH (I'm too lazy to build the binary and put it elsewhere))
|
||||
- Valid Cloudflare R2 credentials
|
||||
|
||||
## Todo
|
||||
- [ ] If you REALLY feel goofy, a TUI would be pretty sick
|
||||
- [ ] Allow multiple, parallel, file uploads under a specific key/folder
|
||||
- [ ] Just more stuff like the above, download all the objects in a key/folder, ect
|
||||
- [ ] If you REALLY feel goofy, a TUI would be pretty sick, however the r2client APIs would need extended quite a bit
|
||||
|
||||
@@ -1,5 +0,0 @@
|
||||
howdy
|
||||
my name is
|
||||
WHAT
|
||||
my name is
|
||||
HUH
|
||||
@@ -2,22 +2,35 @@
|
||||
name = "r2client"
|
||||
version = "0.2.0"
|
||||
edition = "2024"
|
||||
authors = ["ForeverPyrite <r2@foreverpyrite.com"]
|
||||
|
||||
[lib]
|
||||
|
||||
[dependencies]
|
||||
# Client to send the http requests
|
||||
reqwest = "0.12.19"
|
||||
# To parse the information about objects within a bucket
|
||||
xmltree = "0.11.0"
|
||||
thiserror = "2"
|
||||
# Validates and manages methods, headers, and urls
|
||||
http = "1.3.1"
|
||||
# Signs the S3 requests with SigV4
|
||||
aws_sigv4 = { path = "../aws_sigv4/" }
|
||||
|
||||
# Logging
|
||||
log = "0.4.28"
|
||||
# Painless error creation (for me)
|
||||
thiserror = "2"
|
||||
|
||||
[dev-dependencies]
|
||||
# Runtime for the async requests
|
||||
tokio = { version = "1", features = ["full", "macros", "rt-multi-thread"] }
|
||||
# Cause you ain't getting my env variables and I ain't setting them every time
|
||||
dotenv = "0.15"
|
||||
|
||||
[features]
|
||||
async = []
|
||||
default = ["async"]
|
||||
# The asyncronous API
|
||||
async = []
|
||||
# The syncronous, blocking API
|
||||
# yeah surprise, still uses reqwest.
|
||||
sync = ["reqwest/blocking"]
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
## For release:
|
||||
- [ ] Create a crate::Result that is Result<u8, R2Error>, and have Ok(status_code)
|
||||
- [ ] Consider dropping more dependencies, using hyper or some lower level stuff for async, and then http for blocking
|
||||
- [ ] Allow users to use custom mimetypes instead of only inferring from file extension
|
||||
- [ ] A way to view the file contents (UTF-8 valid) would be cool
|
||||
- [ ] Add functions that will list files with their metadata (perhaps a simple R2File type?)
|
||||
- [ ] Clear out all all print statements and consider logging (this is a library, after all)
|
||||
|
||||
@@ -4,4 +4,4 @@ version = "0.1.0"
|
||||
edition = "2024"
|
||||
|
||||
[dependencies]
|
||||
pyo3 = "0.25.0"
|
||||
pyo3 = "0.26.0"
|
||||
|
||||
Reference in New Issue
Block a user