Visualising transfer market data

General gameplay questions can be posted here.
Bottybot
Posts: 11
Joined: Mon Dec 16, 2019 10:01 pm

Visualising transfer market data

Post by Bottybot » Mon Jan 13, 2020 1:15 am

I've just come back to the game after a few years away, and found myself interested in the virtual economy of the transfer market. Me, being the nerd I am, decided to investigate!

To generate the data, I created a bot to download and save data for all players going through the market. As of the time I'm writing this, about 667 unique players have been processed (though there are gaps in the timeline of data).

Some examples of what can be generated. Bating skill by age, with color representing the highest wage of a player with that skill/age:
download.png
The rating/age of players through the market. A higher density means there are more players of that rating/age that have been sold than a lower density:
download (2).png
Rating/age but colored by wage:
download (3).png
For anyone interested in playing with the data, I've hosted a version of the program at fromthepaviliontrends.xyz:8000/datavisualizer/<skill>/<colormode>/

Replace <skill> and <colormode> with an option from below and place the URL in your browser to view the results.

Skills: Bat. Bowl. Keep. Field. End. Tech. Power Wage Rating
Colormodes: bid wage density

For example, to see keeping skill colored by density: fromthepaviliontrends.xyz:8000/datavisualizer/Keep./density/

This is a fun little project that I'll be working on periodically. Would love to hear any feedback or ideas! If there's interest, I can clean up and release the source code.
You do not have the required permissions to view the files attached to this post.

MOD-Qadir
MOD
Posts: 462
Joined: Fri Oct 02, 2015 9:30 am
Location: Valles Marineris

Re: Visualising transfer market data

Post by MOD-Qadir » Mon Jan 13, 2020 5:21 am

Nice stuff there. My inner nerd is well pleased.

The dataset that would interest me would require a timeline of data of what players by skill, age and wage sold on what weeks for how much. Which would be data that only our GM and above would have.

There has always been a belief that during early rounds of the Cup there was demand for the oldies as filler to get you through to the next round and consequently the sell off as players dropped out of the Cup. Plus it would demonstrate how the value of great YP's diminished massively after the first few weeks of each season.
There is no substitute for experience. Though luck can help - just ask Bongo.

GM-crowfan65
CAPT
Posts: 19506
Joined: Fri Jan 04, 2008 6:17 am
Location: Look...up in the sky

Re: Visualising transfer market data

Post by GM-crowfan65 » Mon Jan 13, 2020 5:53 am

Well I have access to all the data obviously and you pose a good question Q
The problem at the moment is there is very little quality for sale in the 16yo market so hard to gauge
We will see more as things settle and squads grow a little more
Mean and median for each age group for each week will give us a fair guide
Master Crowfan of the Blessed Spreadsheet
Image Current Aust U20 Asst
GarageTM Foundation Member

GA-Jim
MOD
Posts: 6762
Joined: Fri Oct 26, 2012 1:41 am
Location: The Naki!

Re: Visualising transfer market data

Post by GA-Jim » Mon Jan 13, 2020 7:12 am

Crowy needs to make his stats with fancy graphs :P :wink:
Herosnzeros Titles: T20: S35
Crazy Horses Titles: YOD: S29. Pavilion cup S39
NZ U19s NM S26-29 AM 24-25. Scottish NM 33-34 37-, AM 30-32 35-36

Bottybot
Posts: 11
Joined: Mon Dec 16, 2019 10:01 pm

Re: Visualising transfer market data

Post by Bottybot » Mon Jan 13, 2020 7:44 am

MOD-Qadir wrote:
Mon Jan 13, 2020 5:21 am
The dataset that would interest me would require a timeline of data of what players by skill, age and wage sold on what weeks for how much.
That's all data I'm collecting now, but only as fast as realtime! Missed the first week or two of this season so nothing long term yet.
GM-crowfan65 wrote:
Mon Jan 13, 2020 5:53 am
Mean and median for each age group for each week will give us a fair guide
Is probably a great indicator, and data that is probably easy to find even for past seasons :think:

Hellbound81
Posts: 3904
Joined: Wed Jan 11, 2012 10:42 am

Re: Visualising transfer market data

Post by Hellbound81 » Mon Jan 13, 2020 7:53 am

On top of the lack of great 16yos currently, youth focused teams simply do not have the capacity for players with no 20yo promotion.

Teams that use to stack with 19yos to be competitive at youth level already have these players now as 20yos which creates little to no reason to buy.

There is also no seasonal 20yo sell off where senior and contingency focused teams buy these players since they already have them now as 21yos.

With these factors in mind the transfer market is a bit more quiet this season while the climate should be returning back to normal again from next season.
Image South Africa NAT Team (Assistant: Season 36-38)
Image South Africa U19 Team (Manager: Season 32-35, Assistant: Season 38-41)

GM-crowfan65
CAPT
Posts: 19506
Joined: Fri Jan 04, 2008 6:17 am
Location: Look...up in the sky

Re: Visualising transfer market data

Post by GM-crowfan65 » Mon Jan 13, 2020 9:00 am

GA-Jim wrote:
Mon Jan 13, 2020 7:12 am
Crowy needs to make his stats with fancy graphs :P :wink:
Here you go Jim

Image
Master Crowfan of the Blessed Spreadsheet
Image Current Aust U20 Asst
GarageTM Foundation Member

Bottybot
Posts: 11
Joined: Mon Dec 16, 2019 10:01 pm

Re: Visualising transfer market data

Post by Bottybot » Fri Jan 17, 2020 1:10 pm

It's far too late (on a friday night, no less) for this, but I've just shared the current code over PM so thought I'd post it here too. Warning: super uncommented!

https://pastebin.com/ddsx3i6h

Will post a cleaner version eventually...

Bottybot
Posts: 11
Joined: Mon Dec 16, 2019 10:01 pm

Re: Visualising transfer market data

Post by Bottybot » Sun Mar 08, 2020 10:16 am

Been a bit on and off with this, but an update is long overdue!

This will really see some action at the start of the next season. Until then, the main objectives are to make the transfer market downloader as reliable as possible, and to create keep creating tools to visualize / use the collected data substantively.

When this first started, I ran the transfer market downloader program from my personal desktop computer. This led to a lot of problems early on- whenever the computer restarted or the power/internet went out the program would stop until the fires could be extinguished. The easy solution to this was to run the program on an VPS - a cheap, rented server with a high uptime. The downloader has been running smoothly for a little over a month. The graph below shows how many players my program collected over the first few months of the year. Days with no players sold indicate the program was not running (i.e my computer had been turned off and I was away/too lazy to restart it).

In the surviving data, note the peaks every 7 days!
uptime.png
Of course this data has more application than visualizations... The next stage of the project is to use the data to create a price prediction model. That is, a program that gives it's best estimate for the sale price of a player that's being sold on the market based on the sales data that has already been collected.

There are a lot of ways to do this, but my first attempt at it has been using a linear regression model in sklearn. The inital results were quite woeful, because the model had problems understanding the player types (batter, bowler, bat/keep. etc.). For example, a simple model would give a low score to a player with a low batting skill, even if that player was primarily a bowler.

Grouping players by their type has given much, much better results. Filtering players to only include those with a final sales price of above 10 000 and a batting skill above average yields the next graph. Each dot represents a player sale- the x-axis shows the actual sale price, while the y-axis shows the price the model predicted. The blue dotted line shows a perfect prediction- dots above the dotted line are predictions that were below the sales price and dots above the dotted line are predictions below the sales price. The red trend line shows that on average the predicted price is higher than the actual price.

Sklearn's cross-validation test shows that the predicted sales prices are 95% accurate.
priceprediction.png
I'm rethinking how to manage the website. For now, here's graphs made with the current data: https://imgur.com/a/YRsJP4L

Finally, is there an offical word on using a price predictor + bot to automatically purchase players on the market? Even without a price predictor, an automatic bot with enough capital behind it could surely be used to influence the market. Would love to hear any further discussion into this!
You do not have the required permissions to view the files attached to this post.

GM-crowfan65
CAPT
Posts: 19506
Joined: Fri Jan 04, 2008 6:17 am
Location: Look...up in the sky

Re: Visualising transfer market data

Post by GM-crowfan65 » Sun Mar 08, 2020 10:28 am

I am guessing the every 7 days thing would be a Thursday when the week's YR sell.
As for using a bot to buy yourself players at the best price etc whilst it seems to go agst the nature of the game in the end I couldn't stop you or anyone else
Master Crowfan of the Blessed Spreadsheet
Image Current Aust U20 Asst
GarageTM Foundation Member

Hellbound81
Posts: 3904
Joined: Wed Jan 11, 2012 10:42 am

Re: Visualising transfer market data

Post by Hellbound81 » Sun Mar 08, 2020 10:32 am

This is great stuff, well done.
Image South Africa NAT Team (Assistant: Season 36-38)
Image South Africa U19 Team (Manager: Season 32-35, Assistant: Season 38-41)

Bottybot
Posts: 11
Joined: Mon Dec 16, 2019 10:01 pm

Re: Visualising transfer market data

Post by Bottybot » Sun Mar 08, 2020 10:38 am

GM-crowfan65 wrote:
Sun Mar 08, 2020 10:28 am
As for using a bot to buy yourself players at the best price etc whilst it seems to go agst the nature of the game in the end I couldn't stop you or anyone else
Yeah, I don't see myself using this on the market- I definitely appreciate the human to human nature of the game. At most I'll be calculating what I could have gotten if I had let it loose ;)

And while bots are probably hard to technically prevent without a lot of added effort on your (the developers) part, there are a lot of game economies that have been affected by players using bots [1], so an explict rule against them doesn't seem ludicrous. Of course, it's very well possible that neither myself or anyone else ever even uses a bot adversely.

[1] https://www.pcgamer.com/au/bots-are-thr ... re-fed-up/

Hellbound81
Posts: 3904
Joined: Wed Jan 11, 2012 10:42 am

Re: Visualising transfer market data

Post by Hellbound81 » Sun Mar 08, 2020 10:42 am

I was thinking of exactly that. Bots are an annoying bane to the market in EVE.
Image South Africa NAT Team (Assistant: Season 36-38)
Image South Africa U19 Team (Manager: Season 32-35, Assistant: Season 38-41)

GM-crowfan65
CAPT
Posts: 19506
Joined: Fri Jan 04, 2008 6:17 am
Location: Look...up in the sky

Re: Visualising transfer market data

Post by GM-crowfan65 » Sun Mar 08, 2020 10:44 am

I will raise it with the board but I think you are right, a rule against them is the best idea
Can be seen as an unfair advantage
Master Crowfan of the Blessed Spreadsheet
Image Current Aust U20 Asst
GarageTM Foundation Member

Hellbound81
Posts: 3904
Joined: Wed Jan 11, 2012 10:42 am

Re: Visualising transfer market data

Post by Hellbound81 » Sun Mar 08, 2020 12:00 pm

What we just need to bare in mind here unlike with most MMOs is that with FTP we are not exactly dealing with account hackers were money is transferred to a primary account which is then used to manipulate the market, making prices of desired items unobtainable for regular players and forcing them to resort to "gold-selling" sites with real money to manage it.

All teams, even farm teams, still have very limited budgets which prevents them from going completely ham on the market to have any significant impact.

I am not saying that some players as a form of RP do not troll the market and inflate prices by chasing sales, but it will be minimal in the greater scheme.

We already play the game where knowledgeable managers try to buy good quality bargains to then later sell for profit. I really do not see a hand full of possible bots changing any of this or having any significant impact.

When any manager is not happy with the price of a player, they simply move on to the next one, while bot controlled teams will always be limited to their bank balance.
Image South Africa NAT Team (Assistant: Season 36-38)
Image South Africa U19 Team (Manager: Season 32-35, Assistant: Season 38-41)

Post Reply