Home

You are not logged in.
Language: French

#1 2015-04-10 16:28:02

jmarc
webMaster/noobJumper

API

Developers, urtjumpers API is available.
Check your profile page > API

Offline

  • Top

#2 2015-04-10 18:11:39

ie7
Map maker

Re: API

Nice! I might create a little map recommendation applet that uses it. big_smile

Offline

  • Top

#3 2015-04-10 19:16:21

KroniK
Admin

Re: API

This looks really neat! Would be nice to implement with B3 to get data on maps.


Still missing your ex? Try using a scope!
---
http://steamsigs.com/steammicro.php?id=KroniK907&pngimg=background&tborder=0

Offline

  • Top

#4 2015-04-10 19:42:20

BadFurDay
Moderator

Re: API

Awww I made a web scraper for the maps page just a couple days ago.

Well thanks a lot, this is going to be useful for me for a personal project. I'll make it public once it's ready.



Is there a limit on how often I can query? Do I need to cache my queries or can I make as many as I want?

up

Offline

  • Top

#5 2015-04-10 20:41:57

ie7
Map maker

Re: API

Quick Python implementation: http://pastebin.com/E6PtkuRa

http://i.imgur.com/SQ2duqM.png

@KroniK: I could extend this into a B3 plugin if you'd like to request any specific features.

Offline

  • Top

#6 2015-04-10 22:59:18

jmarc
webMaster/noobJumper

Re: API

KroniK wrote:

This looks really neat! Would be nice to implement with B3 to get data on maps.

It already exists since september 2013 in jumper plugin by Fenix

Offline

  • Top

#7 2015-04-10 23:35:03

jmarc
webMaster/noobJumper

Re: API

BadFurDay wrote:

Awww I made a web scraper for the maps page just a couple days ago.

Well thanks a lot, this is going to be useful for me for a personal project. I'll make it public once it's ready.



Is there a limit on how often I can query? Do I need to cache my queries or can I make as many as I want?

up

For the moment, no limit.
But if too spam, I will be forced to preserve server performance

You can test, I'll tell you if it's acceptable

Offline

  • Top

#8 2015-04-23 05:57:22

KroniK
Admin

Re: API

jmarc,

Would you be willing to implement a query for mdate when calling for maps? I'd rather not have to iterate through all the maps every time.

Im currently looking to set up a server which only plays the most recently released maps, and auto update using the API. I would love to just query for that day's date to see if any new maps were released and simply poll once a day.

For now I am stuck pulling the entire thing and iterating through them all. Not the greatest solution but it will work for now.


Still missing your ex? Try using a scope!
---
http://steamsigs.com/steammicro.php?id=KroniK907&pngimg=background&tborder=0

Offline

  • Top

#9 2015-04-23 06:34:25

KroniK
Admin

Re: API

Also, Would it be possible to provide a download url for each? I would like to have this whole thing automated so that I don't have to touch it, and it just works.


Still missing your ex? Try using a scope!
---
http://steamsigs.com/steammicro.php?id=KroniK907&pngimg=background&tborder=0

Offline

  • Top

#10 2015-04-23 07:58:17

KroniK
Admin

Re: API

Well, I could download new maps automatically already if it wasnt for your super mysterious dlmap.php code. I cant download the maps with curl, lynx, elinks, wget or any other form of command line utility. I was really hoping it would be as simple as wget http://www.urtjumpers.com/dlmap.php?id=590 or similar... but NOOOO cant have that can we...

Ok im going to stop trying to figure this out and hope that you can provide me a way to more simply download a new map from my command line.

Thanks!

Also, you can disregard my first and second request. For whatever reason, the site will only allow me to edit my most recent post, so I couldn't go back and request a disregard.

Last edited by KroniK (2015-04-23 08:04:20)


Still missing your ex? Try using a scope!
---
http://steamsigs.com/steammicro.php?id=KroniK907&pngimg=background&tborder=0

Offline

  • Top

#11 2015-04-23 08:17:35

KroniK
Admin

Re: API

Welp, went to go edit my last post but it timed out...

Anyway, I was also thinking that maybe it would be cool if we could request an actual file download from the API. That way you have full control over stopping people from spamming it. I would really like to be able to download the maps from within my update script but also understand wanting to keep them away from easy access. (servers and data cost money).

Anyway, thanks for your help!


Still missing your ex? Try using a scope!
---
http://steamsigs.com/steammicro.php?id=KroniK907&pngimg=background&tborder=0

Offline

  • Top

#12 2015-04-23 16:26:32

jmarc
webMaster/noobJumper

Re: API

map download is allowed for members only. you can't dl as guest.
UJ is not a public repository. Like you said 'server costs', and we want avoid spam
I don't know the goal of your script... but we could find secure solution with authentication of uj website

Also, you can disregard my first and second request.

you mean you don't need anymore request on update date?

Offline

  • Top

#13 2015-04-23 19:26:04

KroniK
Admin

Re: API

request on update date would be nice but not essential. And I would love to try and find a secure way to download maps. I am simply trying to find a way to have my server automatically download any new maps, since I don't really have time to check and download and add them manually.

Any help would be appreciated.


Still missing your ex? Try using a scope!
---
http://steamsigs.com/steammicro.php?id=KroniK907&pngimg=background&tborder=0

Offline

  • Top

#14 2015-04-24 04:32:06

moon
Jumper

Re: API

jmarc wrote:

map download is allowed for members only. you can't dl as guest.
UJ is not a public repository. Like you said 'server costs', and we want avoid spam
I don't know the goal of your script... but we could find secure solution with authentication of uj website

Also, you can disregard my first and second request.

you mean you don't need anymore request on update date?

Jmarc any chance you can set up a mirror system so some of us can set up public repos that are know to be good, full mirrors? I'm sure many would be glad to host public mirrors. Linus says he doesn't back up the kernel because it exists on dozens of mirrors smile

Offline

  • Top

#15 2015-04-24 05:19:17

KroniK
Admin

Re: API

moon wrote:

Jmarc any chance you can set up a mirror system so some of us can set up public repos that are know to be good, full mirrors? I'm sure many would be glad to host public mirrors. Linus says he doesn't back up the kernel because it exists on dozens of mirrors smile

This would also be an excellent solution. I myself would be willing to host a mirror on my server, as was kinda the plan from the beginning.


Still missing your ex? Try using a scope!
---
http://steamsigs.com/steammicro.php?id=KroniK907&pngimg=background&tborder=0

Offline

  • Top

#16 2015-04-24 05:33:04

X-RaY
Moderator

Re: API

For automated downloading we could use a token per user and an ip-whitelist.
Which basically means:
http://www.urtjumpers.com/dlmap.php?id=590&token=[token]
In dlmap.php we'll check if the remote-address is a whitelisted IP of [token]
Seems secure enough, people need - however - keep their token secure and private.
Just like working with oauth2 (for those familiar working with social media api's)


“Don’t judge everyone else by your limited experience.” - Carl Sagan

Offline

  • Top

#17 2015-04-24 08:05:40

KroniK
Admin

Re: API

X-RaY wrote:

For automated downloading we could use a token per user and an ip-whitelist.
Which basically means:
http://www.urtjumpers.com/dlmap.php?id=590&token=[token]
In dlmap.php we'll check if the remote-address is a whitelisted IP of [token]
Seems secure enough, people need - however - keep their token secure and private.
Just like working with oauth2 (for those familiar working with social media api's)

Could we simply use our API key as the token or is that not securely generated? I mean it all should be secure since I would assume we would have to request for you to manually add our IP to the whitelist.


Still missing your ex? Try using a scope!
---
http://steamsigs.com/steammicro.php?id=KroniK907&pngimg=background&tborder=0

Offline

  • Top

#18 2015-04-24 15:25:04

X-RaY
Moderator

Re: API

Thought about the same thing when writing that.
It should be secure enough though, although I don't know how jmarc generates these tokens. But all in all... looks secure to me wink


“Don’t judge everyone else by your limited experience.” - Carl Sagan

Offline

  • Top

#19 2015-04-24 16:16:41

jmarc
webMaster/noobJumper

Re: API

API key is not secure, you can find how I generate it whistle
your API key is yours but could be public, it depends of your script... if you code a script, then you share it...

it would be better to use a secure token or create mirrors by community

Offline

  • Top

#20 2015-04-24 18:23:13

KroniK
Admin

Re: API

jmarc wrote:

but could be public, it depends of your script... if you code a script, then you share it...

Well, I guess so, if someone starts looking into the packet data. But the secure token isn't exactly going to be free from that either. The only difference is that the token would be free from figuring out how it was generated. If we use a whitelist method, then someone would have to not only know my token but also be pulling files from my server. And if something like that happens, then we have the weirdest hacker I have ever seen.

jmarc wrote:

or create mirrors by community

I would love to host a mirror, but manually updating every time a new map comes out is not something I want to do.

Here is another alternative. I could fairly easily set up a very secure directory on my server with very limited permissions, and a user to go along with it. You could generate a set of ssh keys and then just scp or maybe sftp the new map files directly to my server. That is probably the MOST secure way.


Still missing your ex? Try using a scope!
---
http://steamsigs.com/steammicro.php?id=KroniK907&pngimg=background&tborder=0

Offline

  • Top

#21 2015-04-24 22:05:53

X-RaY
Moderator

Re: API

jmarc wrote:

API key is not secure, you can find how I generate it whistle
[...]

Hehe... looked it up... not secure indeed pc


“Don’t judge everyone else by your limited experience.” - Carl Sagan

Offline

  • Top

#22 2015-04-26 14:46:08

moon
Jumper

Re: API

KroniK wrote:

jmarc wrote:

or create mirrors by community

I would love to host a mirror, but manually updating every time a new map comes out is not something I want to do.

Here is another alternative. I could fairly easily set up a very secure directory on my server with very limited permissions, and a user to go along with it. You could generate a set of ssh keys and then just scp or maybe sftp the new map files directly to my server. That is probably the MOST secure way.

Goodness no.  Mirrors should pull not be pushed, and everything should be automated with differential updates by timestamp so that only incremental transfers take place after the first full sync.  Can accomplish this easily in 2 ways with rsync:

1. Create a dummy, password-less shell account (/sbin/nologin) that accepts only ssh key authentication and distribute the private key to your mirrors.

2. Symlink your map(s) directories to an .htaccess protected folder on the webserver and enable Indexes and FollowSymLinks.  Distribute the url and account information to your mirrors.

I'd probably opt for #2, since it poses no additional security risks beyond the current system which already runs a webserver.

Offline

  • Top

#23 2015-04-26 18:46:58

moon
Jumper

Re: API

Full mirror of all 391 maps listed by the API key at:

http://mirrors.urtctf.com

Size is 4061MB.  Mirror will update automatically every night at midnight.
Feel free to clone or set sv_dlURL to http://mirrors.urtctf.com/www.urtjumpers.com

Perl code to walk through the API map list in case others find it useful.

Code:

use LWP::Simple;
use JSON;
use strict;

my $API_KEY = '...';
my $API_URL = 'http://api.urtjumpers.com/?key=' . $API_KEY . '&liste=maps&format=json';

my $html = LWP::Simple::get($API_URL) or die;

foreach my $map (@{from_json($html)}) {
my $filename = $map->{'pk3'} . ".pk3";
...
}

Not putting in the headless download code until jmarc has an official and authorized way for us to do that.

Last edited by moon (2015-04-26 18:49:27)

Offline

  • Top

#24 2015-04-26 21:17:17

KroniK
Admin

Re: API

I was planning on setting mine up in python and have most of the code built for it. I can get the full list currently and parse through the list of dictionaries that is created from the json.

I have set up my server for a mirror location but have not actually downloaded all 391 maps yet.

I am currently holding it at http://ptk-clan.club/q3ut4/


Still missing your ex? Try using a scope!
---
http://steamsigs.com/steammicro.php?id=KroniK907&pngimg=background&tborder=0

Offline

  • Top

#25 2015-04-27 14:02:19

jmarc
webMaster/noobJumper

Re: API

Search recent Maps from february 1st 2015 to today :

Code:

&field=mdate&query=2015-02-01

Profile page updated

Offline

  • Top

Board footer

Powered by FluxBB - Adaptation: jmarc
Design: landouille - Icons: famfamfam