Data Collection
This is a tutorial ( I am trying to brand this series as Tru-torials) how to collect data from a API and store that data into a SQL database.
There is a video associated with this tutorial. It assumes zero coding experience, but will be a great launching point to begin your data science or coding journey…
Overview
The 30,000 foot view of this project is: We will be pulling data from a server and populating database and getting data for a specific horse
More specifically: We will be requesting horse and race data from a platform called ZED Run (An NFT based horse racing platform) and putting that data into a SQL database.
We will be using git, python, MySql to build this project. The end product will be a SQL database to perform queries on.
Note: QUERY- is just a fancy way of saying “Give me certain data that matches what I am looking for”. For example data on a specific horse in this tutorial.
This tutorial assumes no coding or CS experience so if you have some you can just grad the code and go. Github repo
Source: The original github repo that sparked this tutorial was from the user(with the unfortunate name) KshitizGit
Disclaimer
I am not a expert on Zed Run. I don’t even use the platform, this was initial made for a friend that wanted some horse data. I just like data and tech and Zed Run seems to be a way to get data. Zed Run is a gambling platform with NFT and crypto so be careful.
“Man, money ain’t got no owners. Only spenders.” - Omar Little (The Wire)
Project Steps
View the video for the walk through, but I will break it down here.
-
If you do not have visual studio code installed install it
- Install python 3.8.5
- Stick with specific version
- Make sure to add to PATH, this allows command in terminal
- pip install pipenv
- use command
pip install pipenv
- use command
- Clone repositor with git
- use command
git clone https://github.com/ttruty/ZedRunner.git
- use command
- Now should be able to run the command to see API call
- should be able to see api call
- Data is not stored anywhere because we haven’t made a MySQL server
- Download and install MySQL Community 8.0.25
- MySql is from Oracle- read about the hate for Oracle
- Can leave all defaults and install… BUT BE SURE TO MAKE A USER WITH THE CORRECT USERNAME AND PASSWORD
- This is not the root accounts, but a new user with username: zedrunner, and password: #ZedRunner@2021
- Install can take a little bit
- Run python codes again in this order wait for each one to complete… it will take some time:
- run command
pipenv run python zed.py --type horse --force true pipenv run python zed.py --type race --force truepipenv run python zed.py --type stable --force true
- run command
- Now you should be able to query the database to see the data.
- Can read about SQL
- Click the ‘create query button.
- Enter a query and click the lightning bolt to run.
- Get your horse data!
- Can make a query to get your horse data
- Now your data is in a database. This is only a local database on your system.
WHATS NEXT?
Ok so now you have a local MySQL server running with the snapshot of data… great, right? This is not great. The data from the Zed Run platform keeps changing. Ideally you would want to set up this code to run on a schedule. The would be a cron job or a scheduled task on windows. But even then, you still only have a local database with all the data sitting on your machine.
The next step would be to turn this system into a web based system to pull the data on a regular basis and structure queries to update a frontend application with the specific information you want.
If there is interest I will make the next TRUtorial to make a simple frontend website to use this data and database. That might be a little more involved because we will need to cover things like web hosting, server security, cloud basics as you would probably want to host on AWS, Azure, GCP, or a service like Digital Ocean.
