About


Pygate is a HackFS project. Its goal is to be a gateway to the Filecoin network and IPFS technology for Pythonistas. To date, most of the tooling has been geared towards Javascript and Golang developers.

The Pygate project has three deliverables:

  1. A Python gRPC client for the Textile.io Powergate Filecoin API.
  2. A Flask based API that uses the pygate gRCP client to provide a web socket interface to the Filecoin network. It is intended to support web apps with issues such as wallet management and the long-running tasks which are expected for transactions on the Filecoin network.
  3. A Flask web application that is meant to serve as a template from which to create more sophisticated Filecoin applications.

The Pygate components are currently in active development and not yet available for release. You can follow progress in our Github repositories. Below are the use cases which we hope the API and web application will eventually support.

Use cases



Use case A: Configure and fund a Filecoin wallet.

  1. The user generates a GNU Privacy Guard (GPG) private and public key to use for unlocking their Filecoin wallet as well as encrypting their stored content.
  2. The application assists the user with generating one or more Filecoin wallets using their GPG private key. The user transfers Filecoin cryptocurrency tokens (FIL) from an existing wallet or cryptocurrency exchange to help fund subsequent storage deals.
  3. The application provides status updates about the FIL token transfer, including a link to the transaction on a FIL block explorer and indication of a successful transfer after a minimum number of confirmations have been met.
  4. The application provides a current balance and transaction history for each of the FIL wallets that the user has configured.

Use case B: Configure the application for use with the Filecoin network.

Using a "Settings" page, the user configures their preferences for Filecoin deals. These are global settings that apply to each storage deal. They are updated when a user’s storage preferences have changed. They include:
  1. Maximum price to offer per Filecoin piece size
  2. Number of redundant copies
  3. The country code for each redundant copy
  4. Minimum length of storage deal
  5. Automatic storage deal renewal (defaults to true)
  6. Deal expiry notification frequency
  7. Filecoin wallet balance notification frequency
  8. Blocked miners list
  9. Allowed miners list
  10. Filecoin wallet address(es)

Use case C: Upload files to the Filecoin network.

  1. From their desktop or laptop device, the user navigates to the top directory of content that they want to transfer to Filecoin. This directory may be on their local filesystem, an external mounted device, or on a network path accessible to their desktop/laptop machine.
  2. The user clicks a button to initiate the upload of this selected directory, its sub-directories, and files to Filecoin.
  3. The user may also select the top directory from their host OS file explorer and drag it into a designated area within the application's UI to initiate the transfer of the directory, its subdirectories, and files to Filecoin.
  4. The contents are encrypted with the user’s GPG private key.
  5. The directory and its encrypted contents is chunked to create an Inter Planetary Linked Data (IPLD) Directed Acyclic Graph (DAG). This IPLD DAG is serialized and packaged into a Content Addressable Archive (CAR) format, suitable for storage on Filecoin.
  6. The sub-directory structure will be preserved over the course of serializing to CAR format and deserializing it upon retrieval at some point in the future.
  7. If the CAR is larger than the maximum 64 GiB sector size currently supported by Filecoin, it will be split into smaller packages.
  8. The application runs the content to be stored through a hash-based process to generate the piece commitment (CommP), which is used for confirmation in later phases.
  9. The user is presented with the Piece Content Identifier (CID) for the CAR package and is given the option in the application to associate it with a human-readable name (e.g. "Movies August 2020", “Transfer 2020-003”, “John Doe Papers”, etc.).
  10. The CAR package is timestamped.
  11. Based on the user’s Filecoin deal settings (see Use case B), the application will prepare one or more Deal Proposals for the CAR package, query the Filecoin blockchain for a list of registered miners, and send each miner a request for its storage asking price. The “storage ask” contains details such as price, duration, and available storage capacity. The application will match this information to the user’s preferences to select one or more miners.
  12. The application locks up sufficient funds in its wallet to pay for the deal(s).
  13. The application contacts the preferred miner(s) and proposes the storage deal. If the miner accepts, the deal proposal is counter-signed and published to the Filecoin blockchain.
  14. After the deal is published, the desktop application begins transferring data to the miner.
  15. Upon receiving all of the data, the miner packs the data into a Filecoin sector, seals it, and begins submitting proofs to the blockchain.
  16. For each deal made, the application checks the blockchain for the miner’s SealedSectorCID and confirms that the miner is storing a physically unique copy of the user's data.
  17. The storage deal metadata is stored in the application's persistence layer and can be browsed, sorted and searched. This includes:
    1. CAR CID
    2. size of the data
    3. human-readable package name
    4. CAR package timestamp
    5. configuration preferences applied to the storage deal
    6. storage deal state
    7. storage deal expiration date (calculated via duration in block height)
    8. dealCID (proposal)
    9. dealID
    10. minerID
    11. SealedSectorCID
    12. PieceID (aka CommP)

Use case D: Provide dashboard information about files on the Filecoin network.

The application provides an aggregate dashboard of information about content stored on the Filecoin network. It will include:
  1. Number of individual files stored in Filecoin
  2. Number of CAR packages stored in Filecoin
  3. Total storage size of unique CAR packages stored in Filecoin
  4. Total storage size of all copies of CAR packages stored in Filecoin
  5. FIL balance of wallet(s)
  6. Total FIL paid for storage to date
  7. Total FIL paid for retrieval to date
  8. Current and historical FIL to fiat exchange rates, including conversions for totals shown in #5-7 above
  9. Total number of each file format type stored in Filecoin (and how much storage space each file of this type is taking)
  10. Total number of different miners storing the user’s content
  11. Number of completed storage deals
  12. Number of in progress storage deals
  13. Total number of storage deals expiring within a given time range (e.g. next 7 days, 30 days, year, etc.)

Use case E: Provide integrity and authenticity checks for files on the Filecoin network.

  1. The application monitors the Filecoin blockchain for the random, periodic proofs-of-space-time that the miner(s) will submit for each storage deal. When it encounters a proof for content that it has submitted, the application will update the corresponding storage deal metadata and index it for browsing and searching purposes.
  2. The application allows the user to periodically initiate their own check of the fixity of one, some, or all of their stored files. This process will guarantee to the user that the file(s) stored on the Filecoin network have not been changed and has remained in storage for an uninterrupted period of time. This is the key determinant in establishing the file’s authenticity. The result of this check will be added to the corresponding storage deal’s metadata and indexed for browsing and searching purposes.

Use case F: Renegotiate storage deals for files already on the Filecoin network.

  1. The application will monitor storage deals for upcoming expiry dates and if the renewal configuration is set to true, it will re-negotiate storage deals using the default configuration settings.
  2. The application will negotiate the best price for the user using the maximum price and minimum deal length settings.
  3. It will also comply with the number of redundant copies and geographical distribution settings.

Use case G: Retrieve files from the Filecoin network.

  1. The application will provide the user with a file browser style user interface to serve as a dashboard for directories, sub-directories and files stored on the Filecoin network. The user may click on a directory or file from this interface to initiate a retrieval request from the Filecoin network.
  2. The application will also allow the user to request a file download from the deal metadata's browse and search interface.
  3. The desktop application will allow the user to select a location on their host machine or networked file locations where the retrieved file(s) will be downloaded.
  4. The application will search IPFS for the CID
  5. If the CID is not available on IPFS, the application will search the Filecoin network for all possible providers of the desired data (via the DHT, the chain, or out-of-band aggregators), compare deal terms, and chose the best retrieval option.
  6. Once a miner is selected:
    1. The application will send a RetrievalQuery (“Here’s the PayloadCID of the data I want.”)
    2. The miner sends a RetrievalQueryResponse (“I do have that data, and can send it to you for this price”.)
    3. The desktop application sends a RetrievalDealProposal (“Yes I like those terms, go ahead.”)
    4. The miner sends a RetrievalDealResponse (“I accept your deal.”)
    5. The miner unseals the data
    6. A payment channel is opened
    7. The miner sends a RetrievalDealResponse (“Here's part of your data, pay me before I send the next.”)
    8. The application confirms the data is correct, and sends the partial payment via the payment channel (“Here's the partial payment, keep going.”)
    9. Steps 7 and 8 are repeated until retrieval is complete
    10. The last payment for the remainder of bytes is made and the retrieval deal is complete.
    11. The application decrypts the files using the private key it has custody of.
    12. The application will recreate the original directory and sub-directory structure in the designated download space.

Team


Peter Van Garderen
Vancouver, Canada
Antreas Pogiatzis
London, UK
Leonard Ge
London, UK
Art Richards
Berlin, Germany
Chinonso Eze
Abuja, Nigeria