Joplin & Python ( Static Map ) : How to create a Maps (JPEG) with REST API and data in Joplin ?

Step 0 : Install Joplin and activate the REST API ( https://joplin.cozic.net/api/ ) .

Step 1: Install staticmap with pip ( for more information see https://github.com/komoot/staticmap )

$ pip install staticmap
Collecting staticmap
  Downloading https://files.pythonhosted.org/packages/f9/9f/5a3843533eab037cba031486175c4db1b214614404a29516208ff228dead/staticmap-0.5.4.tar.gz
Collecting Pillow (from staticmap)
  Downloading https://files.pythonhosted.org/packages/c9/ed/27cc92e99b9ccaa0985a66133baeea7e8a3371d3c04cfa353aaa3b81aac1/Pillow-5.4.1-cp37-cp37m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl (3.7MB)
    100% |████████████████████████████████| 3.7MB 6.3MB/s 
Requirement already satisfied: requests in /usr/local/lib/python3.7/site-packages (from staticmap) (2.21.0)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.7/site-packages (from requests->staticmap) (3.0.4)
Requirement already satisfied: idna<2.9,>=2.5 in /usr/local/lib/python3.7/site-packages (from requests->staticmap) (2.8)
Requirement already satisfied: urllib3<1.25,>=1.21.1 in /usr/local/lib/python3.7/site-packages (from requests->staticmap) (1.24.1)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/site-packages (from requests->staticmap) (2018.11.29)
Building wheels for collected packages: staticmap
  Building wheel for staticmap (setup.py) ... done
  Stored in directory: /Users/..../Library/Caches/pip/wheels/fe/a6/a5/2acceb72471d85bd0498973aabd611e6ff1cdd48796790f047
Successfully built staticmap
Installing collected packages: Pillow, staticmap
Successfully installed Pillow-5.4.1 staticmap-0.5.4

The source code :

#
# Version 1
# for Python 3
# 
#   ARIAS Frederic
#   Sorry ... It's difficult for me the python :)
#

from time import gmtime, strftime
import time
import json
import requests
import os
import gmplot
from staticmap import StaticMap, CircleMarker

m = StaticMap(1600, 800, url_template='http://a.tile.osm.org/{z}/{x}/{y}.png')

marker = CircleMarker((10, 47), '#0036FF', 3)
m.add_marker(marker)

#Token
ip = "127.0.0.1"
port = "41184"

token = "Put your token here"
nb_request = 0
nb_plot = 0
headers = {'Content-type': 'application/json', 'Accept': 'text/plain'}
url_notes = (
    "http://"+ip+":"+port+"/notes?"
    "token="+token
)
try:
    resp = requests.get(url_notes, headers=headers)
    nb_request += 1
    resp.raise_for_status()
    resp_dict = resp.json()
    #print(resp_dict)
    for my_note in resp_dict:
        #print(my_note.get('id'))
        url_notes2 = (
    "http://"+ip+":"+port+"/notes/"+my_note.get('id')+"?fields=longitude,latitude&"
    "token="+token
)
        try:
           resp2 = requests.get(url_notes2, headers=headers)
           nb_request += 1
           resp2.raise_for_status()
           resp_dict2 = resp2.json()
           #print(resp_dict2)
           long = resp_dict2.get('longitude')
           lat = resp_dict2.get('latitude')
           if (long != '0.00000000'):
              nb_plot += 1
              marker = CircleMarker((float(long), float(lat)), '#0036FF', 12)
              m.add_marker(marker)
        except requests.exceptions.HTTPError as e:
             print("Bad HTTP status code:", e)
        except requests.exceptions.RequestException as e:
             print("Network error:", e)
except requests.exceptions.HTTPError as e:
    print("Bad HTTP status code:", e)
except requests.exceptions.RequestException as e:
    print("Network error:", e)

image = m.render(zoom=1)
image.save('mymap_zoom1.png')
image = m.render(zoom=2)
image.save('mymap_zoom2.png')
image = m.render(zoom=3)
image.save('mymap_zoom3.png')
image = m.render(zoom=4)
image.save('mymap_zoom4.png')
image = m.render(zoom=5)
image.save('mymap_zoom5.png')
image = m.render(zoom=6)
image.save('mymap_zoom6.png')

print ("Number total of request",nb_request)
print ("Number total of pin",nb_plot)

On same folder, you see mymap_zoom*.png .

To change the size of map, it’s here : ( by default 1600 x 800 )

m = StaticMap(1600, 800, url_template='http://a.tile.osm.org/{z}/{x}/{y}.png')

Time to build the pictures :

$ time python3 JoplinToStaticMap.py 
Number total of request 1927
Number total of pin 643

real	0m18.137s
user	0m7.936s
sys	0m0.876s

Joplin & Python ( gmplot Google Map) : How to create a Maps with REST API and data in Joplin ?

Step 0 : Install Joplin and activate the REST API ( https://joplin.cozic.net/api/ ) .

Step 1: Install gmplot with pip

$ pip install gmplot
Collecting gmplot
  Downloading https://files.pythonhosted.org/packages/e2/b1/e1429c31a40b3ef5840c16f78b506d03be9f27e517d3870a6fd0b356bd46/gmplot-1.2.0.tar.gz (115kB)
    100% |████████████████████████████████| 122kB 1.0MB/s 
Requirement already satisfied: requests in /usr/local/lib/python3.7/site-packages (from gmplot) (2.21.0)
Requirement already satisfied: urllib3<1.25,>=1.21.1 in /usr/local/lib/python3.7/site-packages (from requests->gmplot) (1.24.1)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/site-packages (from requests->gmplot) (2018.11.29)
Requirement already satisfied: idna<2.9,>=2.5 in /usr/local/lib/python3.7/site-packages (from requests->gmplot) (2.8)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.7/site-packages (from requests->gmplot) (3.0.4)
Building wheels for collected packages: gmplot
  Building wheel for gmplot (setup.py) ... done
  Stored in directory: /Users/...../Library/Caches/pip/wheels/81/6a/76/4dd6a7cc310ba765894159ee84871e8cd55221d82ef14b81a1
Successfully built gmplot
Installing collected packages: gmplot
Successfully installed gmplot-1.2.0

The source code : (change your token)

#
# Version 1 
# for Python 3
# 
#   ARIAS Frederic
#   Sorry ... It's difficult for me the python :)
#

from time import gmtime, strftime
import time
import json
import requests
import os
import gmplot

strftime("%Y-%m-%d %H:%M:%S", gmtime())
start = time.time()

#Token
ip = "127.0.0.1"
port = "41184"
token = "Put your token here"

nb_request = 0
nb_plot = 0
headers = {'Content-type': 'application/json', 'Accept': 'text/plain'}
url_notes = (
    "http://"+ip+":"+port+"/notes?"
    "token="+token
)
x=[] #longitudes
y=[] #latitudes
try:
    resp = requests.get(url_notes, headers=headers)
    nb_request += 1
    resp.raise_for_status()
    resp_dict = resp.json()
    #print(resp_dict)
    for my_note in resp_dict:
        #print(my_note.get('id'))
        url_notes2 = (
    "http://"+ip+":"+port+"/notes/"+my_note.get('id')+"?fields=longitude,latitude&"
    "token="+token
)
        try:
           resp2 = requests.get(url_notes2, headers=headers)
           nb_request += 1
           resp2.raise_for_status()
           resp_dict2 = resp2.json()
           #print(resp_dict2)
           long = resp_dict2.get('longitude')
           lat = resp_dict2.get('latitude')
           if (long != '0.00000000'):
              #print(long,lat)
              nb_plot += 1
              if (nb_plot == 1):
                 gmap1 = gmplot.GoogleMapPlotter(float(lat), float(long), 13 )
              x.append(float(long))
              y.append(float(lat))
        except requests.exceptions.HTTPError as e:
             print("Bad HTTP status code:", e)
        except requests.exceptions.RequestException as e:
             print("Network error:", e)
except requests.exceptions.HTTPError as e:
    print("Bad HTTP status code:", e)
except requests.exceptions.RequestException as e:
    print("Network error:", e)

gmap1.scatter( y, x, '#FF0000', size = 10, marker = False )
gmap1.draw("mymap.html")

print ("Number total of request",nb_request)
print ("Number total of pin",nb_plot)

After clic on mymap.html on same folder.

Note n°1 : JOPLIN is robust

 $ time python3 JoplinToGoogleMap.py 
Number total of request 1927
Number total of pin 643


real 0m8.401s
user 0m4.564s
sys 0m0.577s

In 9 seconds, 1927 requests !! JOPLIN it’s perfect.

Note n°2 : To increase the size of the PIN it’s here ( size = 10 ) :

gmap1.scatter( y, x, '#FF0000', size = 10, marker = False )

Installation de pip sur Mac OS

Voici ce que j’ai fait pour faire l’installation de pip sur Mac OS :

$ curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 1662k  100 1662k    0     0   560k      0  0:00:02  0:00:02 --:--:--  560k

$ python3 get-pip.py
Collecting pip
  Downloading https://files.pythonhosted.org/packages/d7/41/34dd96bd33958e52cb4da2f1bf0818e396514fd4f4725a79199564cd0c20/pip-19.0.2-py2.py3-none-any.whl (1.4MB)
    100% |████████████████████████████████| 1.4MB 154kB/s 
Installing collected packages: pip
  Found existing installation: pip 18.1
    Uninstalling pip-18.1:
      Successfully uninstalled pip-18.1
Successfully installed pip-19.0.2

$ pip install feedparser
Collecting feedparser
  Downloading https://files.pythonhosted.org/packages/91/d8/7d37fec71ff7c9dbcdd80d2b48bcdd86d6af502156fc93846fb0102cb2c4/feedparser-5.2.1.tar.bz2 (192kB)
    100% |████████████████████████████████| 194kB 500kB/s 
Building wheels for collected packages: feedparser
  Building wheel for feedparser (setup.py) ... done
  Stored in directory: ....
Successfully built feedparser
Installing collected packages: feedparser
Successfully installed feedparser-5.2.1

How to import data of Google+ to Joplin ?

Install JOPLIN : https://joplin.cozic.net ,  and start REST API.

Step 1 : Download all with https://takeout.google.com

Step 2 : Uncompress and put all on same folder.

Step 3 : Put this script in folder.

Step 4 : Edit the script and put your token

The script :

#
# Version 1 
# for Python 3
# 
#   ARIAS Frederic
#   Sorry ... It's difficult for me the python :)
#


from os import listdir
from pathlib import Path
import glob
import csv
import locale
import os
import time
from datetime import datetime
import json
import requests

nb_metadata = 0
nb_metadata_import = 0
def month_string_to_number(string):
    m = {
        'janv.': 1,
        'feb.': 2,
        'févr.': 2,
        'mar.': 3,
        'mars': 3,
        'apr.':4,
        'avr.':4,
         'may.':5,
         'mai':5,
         'juin':6,
         'juil.':7,
         'aug.':8,
         'août':8,
         'sept.':9,
         'oct.':10,
         'nov.':11,
         'déc.':12
        }
    s = string.strip()[:5].lower()

    try:
        out = m[s]
        return out
    except:
        raise ValueError('Not a month')

locale.setlocale(locale.LC_TIME, 'fr_FR.UTF-8')
#today = datetime.date.today()
#print(today.strftime('The date :%d %b. %Y à %H:%M:%S UTC'))
from time import strftime,localtime
print(localtime())
print(strftime("%H:%M:%S, %d %b. %Y",localtime()))
date = datetime.strptime('2017-05-04',"%Y-%m-%d")

#Token
ip = "127.0.0.1"
port = "41184"
token = "Put your token here"

nb_import = 0;
headers = {'Content-type': 'application/json', 'Accept': 'text/plain'}

url_notes = (
    "http://"+ip+":"+port+"/notes?"
    "token="+token
)
url_folders = (
    "http://"+ip+":"+port+"/folders?"
    "token="+token
)
url_tags = (
    "http://"+ip+":"+port+"/tags?"
    "token="+token
)
url_ressources = (
    "http://"+ip+":"+port+"/ressources?"
    "token="+token
)

#Init
GooglePlus_UID = "12345678901234567801234567890123"
UID = {}

payload = {
    "id":GooglePlus_UID,
    "title":"GooglePlus Import"
}

try:
    resp = requests.post(url_folders, data=json.dumps(payload, separators=(',',':')), headers=headers)
    resp.raise_for_status()
    resp_dict = resp.json()
    print(resp_dict)
    print("My ID")
    print(resp_dict['id'])
    GooglePlus_UID_real = resp_dict['id']
    save = str(resp_dict['id'])
    UID[GooglePlus_UID]= save
except requests.exceptions.HTTPError as e:
    print("Bad HTTP status code:", e)
except requests.exceptions.RequestException as e:
    print("Network error:", e)

for csvfilename in glob.iglob('Takeout*/**/*.metadata.csv', recursive=True):
  nb_metadata += 1
  print(nb_metadata," ",csvfilename)
  #print("Picture:"+os.path.basename(csvfilename))
  mybasename = os.path.basename(csvfilename)
  mylist = mybasename.split(".")
  myfilename = mylist[0] + "." + mylist[1]
  filename = os.path.dirname(csvfilename)+"/"+myfilename
  my_file = Path(filename)
  with open(csvfilename) as csvfile:
    reader = csv.DictReader(csvfile)
    for row in reader:
        if (len(row['description']) > 0):
            print(row['title'], row['description'], row['creation_time.formatted'], row['geo_data.latitude'], row['geo_data.longitude'])
            #date = datetime.strptime(row['creation_time.formatted'], "%d %b %Y à %H:%M:%S %Z").timetuple()
            #print(date)
            mylist2 = row['creation_time.formatted'].split(" ");
            mylist3 = mylist2[4].split(":");
            date = date.replace(hour=int(mylist3[0]), year=int(mylist2[2]), month=month_string_to_number(mylist2[1]), day=int(mylist2[0]))
            timestamp = time.mktime(date.timetuple())*1000
            print(timestamp)
            nb_metadata_import += 1
            mybody = row['description']
            if (len(row['geo_data.latitude']) > 2):
              payload_note = {
                "parent_id":GooglePlus_UID_real,
                "title":row['creation_time.formatted'],
                "source":myfilename,
                "source_url":row['url'],
                "order":nb_metadata_import,
                "body":mybody
                }
              payload_note_put = {
                "latitude":float(row['geo_data.latitude']),
                "longitude":float(row['geo_data.longitude']),
                "source":myfilename,
                "source_url":row['url'],
                "order":nb_metadata_import,
                "user_created_time":timestamp,
                "user_updated_time":timestamp,
                "author":"Google+"
                }
            else:
               payload_note = {
                "parent_id":GooglePlus_UID_real,
                "title":row['creation_time.formatted'],
                "source":myfilename,
                "source_url":row['url'],
                "order":nb_metadata_import,
                "user_created_time":timestamp,
                "user_updated_time":timestamp,
                "author":"Google+",
                "body":mybody
                }
               payload_note_put = {
                "source":myfilename,
                "order":nb_metadata_import,
                "source_url":row['url'],
                "user_created_time":timestamp,
                "user_updated_time":timestamp,
                "author":"Google+"
                }

            try:
                resp = requests.post(url_notes, json=payload_note)
                resp.raise_for_status()
                resp_dict = resp.json()
                print(resp_dict)
                print(resp_dict['id'])
                myuid= resp_dict['id']
            except requests.exceptions.HTTPError as e:
                print("Bad HTTP status code:", e)
            except requests.exceptions.RequestException as e:
                print("Network error:", e)

            url_notes_put = (
    "http://"+ip+":"+port+"/notes/"+myuid+"?"
    "token="+token
)

            try:
                resp = requests.put(url_notes_put, json=payload_note_put)
                resp.raise_for_status()
                resp_dict = resp.json()
                print(resp_dict)
            except requests.exceptions.HTTPError as e:
                print("Bad HTTP status code:", e)
            except requests.exceptions.RequestException as e:
                print("Network error:", e)
            
            if my_file.is_file():
               cmd = "curl -F 'data=@"+filename+"' -F 'props={\"title\":\""+myfilename+"\"}' http://"+ip+":"+port+"/resources?token="+token
               print("Command"+cmd)
               resp = os.popen(cmd).read()
               try:
                  respj = json.loads(resp)
                  print(respj['id'])
                  myuid_picture= respj['id']
               except:
                  print('bad json: ', resp)

               mybody = row['description'] + "\n  ![" + myfilename + "](:/" + myuid_picture + ")   \n";

               payload_note_put = {
                "body":mybody
                }

               try:
                  resp = requests.put(url_notes_put, json=payload_note_put)
                  resp.raise_for_status()
                  resp_dict = resp.json()
                  print(resp_dict)
               except requests.exceptions.HTTPError as e:
                  print("Bad HTTP status code:", e)
               except requests.exceptions.RequestException as e:
                  print("Network error:", e)

print(nb_metadata)
print(nb_metadata_import)