Python : How to clean ressources of Joplin not used ?

50 x served & 13 x viewed

Here the script in Python : https://github.com/CYBERNEURONES/Python/blob/master/JoplinCleanRessource.py

#
# Version 1 
# for Python 3
# 
#   ARIAS Frederic
#   Sorry ... It's difficult for me the python :)
#

from time import gmtime, strftime
import time
import json
import requests
import os
import sqlite3
import re

#conn = sqlite3.connect('my_db.db')
find_this = "\(:/"

#c = conn.cursor()
#c.execute('''DROP TABLE LINK''')
#conn.commit()
#c.execute('''CREATE TABLE LINK (ID_NOTE text, ID_RESOURCE text, CHECKSUM_MD5 text)''')
#conn.commit()

#IP
ip = "127.0.0.1"
port = "41184"
token = "Put the token here"
nb_request = 0
my_body = ""
headers = {'Content-type': 'application/json', 'Accept': 'text/plain'}
url_notes = (
    "http://"+ip+":"+port+"/notes?"
    "token="+token
)
nb_total_ressource = 0
nb_local_ressource = 0
ALL_ID = {}
try:
    resp = requests.get(url_notes, headers=headers)
    nb_request += 1
    resp.raise_for_status()
    resp_dict = resp.json()
    #print(resp_dict)
    for my_note in resp_dict:
        nb_local_ressource = 0
        my_body = my_note.get('body')
        my_ressource = [m.start() for m in re.finditer(find_this, my_body)]
        for my_ressource_x in my_ressource:
             nb_total_ressource += 1
             nb_local_ressource += 1
             my_ressource_id = my_body[my_ressource_x+3:my_ressource_x+32+3]
             print(nb_local_ressource,":",my_note.get('id'),":",my_ressource_id)
             ALL_ID[my_ressource_id]=my_note.get('id')
             
             #c.execute(sql_request)
             #conn.commit()
except requests.exceptions.HTTPError as e:
    print("Bad HTTP status code:", e)
except requests.exceptions.RequestException as e:
    print("Network error:", e)

nb_keep = 0
nb_remove = 0
url_resources = (
    "http://"+ip+":"+port+"/resources?"
    "token="+token
)
try:
    resp = requests.get(url_resources, headers=headers)
    nb_request += 1
    resp.raise_for_status()
    resp_dict = resp.json()
    #print(resp_dict)
    for my_resource in resp_dict:
        my_id = my_resource.get('id')
        if my_id in ALL_ID:
            print("Keep for notes",ALL_ID[my_id])
            nb_keep += 1
        else:
            print("Remove");
            nb_remove += 1
            url_resources_delete = (
    "http://"+ip+":"+port+"/resources/"+my_id+"?"
    "token="+token
)
            try:
                 resp2 = requests.delete(url_resources_delete, headers=headers)
                 resp.raise_for_status()
                 nb_request += 1
            except requests.exceptions.HTTPError as e:
                 print("Bad HTTP status code:", e)
            except requests.exceptions.RequestException as e:
                 print("Network error:", e)
except requests.exceptions.HTTPError as e:
    print("Bad HTTP status code:", e)
except requests.exceptions.RequestException as e:
    print("Network error:", e)

#conn.close()
print("nb_request",nb_request,"nb_total_ressource : ",nb_total_ressource," nb_local_ressource : ",nb_local_ressource)
print("nb_keep",nb_keep,"nb_remove",nb_remove);

Here the result :

$ grep "Total resources:" .config/joplin-desktop/log.txt | awk '{print $1 " " $5}' | uniq | tail -f
2019-02-25 10191"
2019-02-25 10194"
2019-02-25 10190"
2019-02-26 10190"
2019-02-27 10190"
2019-02-28 10190"
2019-02-28 10192"
2019-03-01 10192"
2019-03-01 2919"
2019-03-01 2814"

Joplin : Je suis heureux de recevoir le badge « New User of the Month »

État

50 x served & 6 x viewed

Il reste encore du travail à faire … mais cette application est vraiment très bien. J’ai fait énormément de test avec beaucoup de donnée, et je n’ai pas eu de problème.

Joplin : version v1.0.127

83 x served & 4 x viewed

La nouvelle version arrive avec quelques fix et des améliorations :

On peut même faire du KaTeX !

Joplin : Très robuste, et avec une API REST ! Mais que demande le peuple ? Plus d’open source

70 x served & 18 x viewed

J’ai voulu tester Joplin https://joplin.cozic.net entièrement, pas seulement la synchronisation de 2 ou 3 fichiers.

J’ai donc fait une base de 2465 notes, et 9787 images :

: "Total folders: 32"
: "Total notes: 2465"
: "Total resources: 9787"

Mon fichier WebDEV :

$ du -sh WebDAV/
2,7G	WebDAV/
$ ls -l WebDAV/*.md | wc -l
-bash: /bin/ls: Argument list too long
       0

Il y a tellement de fichier que la commande « ls » plante 🙂 , en fait il y a 13051 fichiers pour 2,7 Go. Le fichier le plus gros fait 13 Ko.

Pour remplir la base je me suis fait des scripts ( voir mes articles ) :

J’ai aussi fait des scripts afin de faire des tests de l’API REST :

Tous les scripts sont sur mon compte GitHub en licence Open Bar.

Je n’ai pas eu un seul plantage de Joplin sur Mac, Linux ou Android ! Je conseille donc vivement ce logiciel qui est stable et efficace dans la synchronisation.

J’espère que l’application Android va avoir des améliorations afin d’avoir sur chaque ligne : Date utilisateur (et pas date de création) / Titre / Première photo , et pas seulement Titre. Un peu comme la présentation de Diaro ( mais sans le tracking ). Ou dans ce style ( DAYBOOK ) :

Python : How to migrate data of Awesome Note 2 (bridworks.com) to Joplin ?

61 x served & 18 x viewed

Awesome Note 2, it’s very popular on iPad :

The new All-in-one Organizer, Awesome Note 2 is integrated with note and schedule management.
And now it’s available!!

WONDERFUL WRITING FEATURES
· It can be used not only for simple notes, but also rich and wonderful writing tool.
· Make notes even more powerful to add photos, voice recording and drawings.
· Easily create diary notes to display feeling, weather or road map information.

SIMPLE, FLEXIBLE, AND FRIENDLY
·  Broadly use as diary to record everyday life, travel notes to write anywhere, photo albums, shopping lists, and record for work or study in any theme.
 
NOTE AND SCHEDULING AS ONE
· Manage your iOS Calendar and Reminders in one.
· Check todo lists and manage all schedules with calendar
· Receive notifications for important events and easily manage anniversaries such as birthdays.
 
NEAT AND STYLISH DESIGN
· Create your own style with tastefully designed icons, folders and various note backgrounds.
 
Capture all the memorable moments, stories, and everything in you. 

Step 0 : Install Joplin and activate the REST API ( https://joplin.cozic.net/api/ ) .

Step 1 : Install Python.

Step 2 : Create a backup of Awesome Note. ( for exemple : aNote_13Folders_20170520_00_24_21_579Notes.anb )

Step 3 : Uncompress the backup (It’s zip).

Step 4 : Put the script and change Token and name of folder of Backup.

The script :

#
# Version 1 
# for Python 3
# 
#   ARIAS Frederic
#   Sorry ... It's difficult for me the python :)
#

import plistlib
import os
import glob

folder = "Put the name of folder"

import requests
import re
import json
from subprocess import Popen, PIPE
import xml.etree.ElementTree
import xml.etree.ElementTree as ET
from xml.dom import minidom
from bpylist import bplist
import base64

###

#IP
ip = "127.0.0.1"
#Port
port = "41184"
#Token
token = "Put the token"
nb_import = 0;
headers = {'Content-type': 'application/json', 'Accept': 'text/plain'}

url_notes = (
    "http://"+ip+":"+port+"/notes?"
    "token="+token
)
url_folders = (
    "http://"+ip+":"+port+"/folders?"
    "token="+token
)
url_tags = (
    "http://"+ip+":"+port+"/tags?"
    "token="+token
)
url_ressources = (
    "http://"+ip+":"+port+"/ressources?"
    "token="+token
)

#Init
Awesome_UID = "12345678901234567801234567890123"
Awesome_UID_real = ""

payload = {
    "id":Awesome_UID,
    "title":"Awesome Note Import"
}

try:
    resp = requests.post(url_folders, data=json.dumps(payload, separators=(',',':')), headers=headers)
    #time.sleep(1)
    resp.raise_for_status()
    resp_dict = resp.json()
    print(resp_dict)
    print("My ID")
    print(resp_dict['id'])
    Awesome_UID_real = resp_dict['id']
    save = str(resp_dict['id'])
except requests.exceptions.HTTPError as e:
    print("Bad HTTP status code:", e)
except requests.exceptions.RequestException as e:
    print("Network error:", e)

###
nb_picture = 0
order = 0

# Convert to XML
files = os.listdir(folder)
for name in files:
  if name.endswith('.anote'):
    #print(name)
    name2 = os.path.splitext(name)[0]
    my_file = folder+"/"+name2+".xml"
    name = name.replace(" ", "\\ ")
    name = name.replace("(", "\\(")
    name = name.replace(")", "\\)")
    name = name.replace("&", "\\&")
    name2 = name2.replace(" ", "\\ ")
    name2 = name2.replace("(", "\\(")
    name2 = name2.replace(")", "\\)")
    name2 = name2.replace("&", "\\&")
    #print (name2)
    commande = "plutil -convert xml1 "+folder+"/"+name+" -o "+folder+"/"+name2+".xml"
    os.system(commande)
    my_dict = {}
    nb_array = 0
    nb_integer = 0
    nb_real = 0
    nb_string = 0
    flag_ok = 0
    nb_data = 0
    liste_picture = []
    my_timestamp = 0
    my_title = ""
    my_body = ""
    my_id = ""
    my_picture_name = ""
    my_lat = ""
    my_long = ""
    my_address = ""
    for event, elem in ET.iterparse(my_file):
        assert event == 'end'
        if elem.tag == 'array':
            key = elem.text
            nb_array += 1;
        elif elem.tag == 'key':
            value = elem.text
            if (value == 'Version'):
               flag_ok = 1
               #print("ok")
               nb_array = 0
               nb_integer = 0
               nb_real = 0
               nb_string = 0
               nb_data = 0
        elif elem.tag == 'integer':
            nb_integer += 1
            value = elem.text
        elif elem.tag == 'real':
            nb_real += 1
            value = elem.text
            if (flag_ok == 1):
                if (nb_real == 1):
                   my_timestamp = float(value)*1000;
        elif elem.tag == 'string':
            nb_string += 1
            value = elem.text
            if (flag_ok == 1):
                if (nb_string == 2):
                   my_key = value
                   if (value is None):
                      nb_string += 0
                   else:
                      if (value.startswith("Apple")):
                         nb_string += 1
                      elif (value.startswith("Times")):
                         nb_string += 1
                      else:
                         split_list = value.split()
                         if len(split_list) == 3:
                              my_long, my_lat, my_address = value.split("|")
                              print("Address",my_long, my_lat, my_address)
                if (nb_string == 4):
                   my_key = value
                   if (value is None):
                      nb_string += 0
                   else:
                      if (value.startswith("Apple")):
                         nb_string -= 1
                      elif (value.startswith("Times")):
                         nb_string -= 1
                if (nb_string == 5):
                   my_key = value
                   if (value is None):
                      nb_string += 0
                   else:
                      if (value.startswith("Apple")):
                         nb_string -= 2 
                      elif (value.startswith("Times")):
                         nb_string -= 2 
                if (nb_string == 5):
                   my_body = value
                   if (my_body is None):
                       my_body = ""
                   else:
                       my_body = re.sub(r"<@b>", "**", my_body)
                       my_body = re.sub(r"</@b>", "**", my_body)
                       my_body = re.sub(r"<@u>", "*", my_body)
                       my_body = re.sub(r"</@u>", "*", my_body)
                if (nb_string == 6):
                   my_title = value
                   if (my_title is None):
                       my_title = ""
                   elif (len(my_title) == 36 and (' ' not in my_title)):
                       if (len(my_body) > 0):
                           my_title = my_body
                if (nb_string == 7):
                   my_id = value
                   my_id = my_id.replace("-", "")
                   my_id = my_id[0:31]
        elif elem.tag == 'entry':
            my_dict[key] = value
            key = value = None
        elif elem.tag == 'data':
            nb_data += 1
            value = elem.text
            value_clean = re.sub(r" ", "", value)
            value_clean = re.sub(r"\t", "", value_clean)
            value_clean = re.sub(r"\n", "", value_clean)
            nb_picture += 1
            my_picture_name = str(nb_picture)+".jpg"
            liste_picture.append(my_picture_name)
            jpg_recovered = base64.decodestring(value_clean.encode())
            g = open(my_picture_name, "wb")
            g.write(jpg_recovered)
            g.close()
        elem.clear()

    #print("Data:",my_timestamp,",",my_title,",",my_body)
    if (len(my_title) > 150):
       print("Error",my_file);
    print("Filename:",my_file,"Data:",my_timestamp,",",my_title,",",my_id)
    nb_import += 1
    payload_note = {
                "parent_id":Awesome_UID_real,
                "title":my_title,
                "source":my_file,
                "order":nb_import,
                "user_created_time":my_timestamp,
                "user_updated_time":my_timestamp,
                "author":"Awesome Note",
                "body":my_body
}
    if (len(my_address) > 0):
        payload_note_put = {
                "source":my_file,
                "longitude":float(my_long),
                "latitude":float(my_lat),
                "order":nb_import,
                "user_created_time":my_timestamp,
                "user_updated_time":my_timestamp,
                "author":"Awesome Note"
}
    else :
        payload_note_put = {
                "source":my_file,
                "order":nb_import,
                "user_created_time":my_timestamp,
                "user_updated_time":my_timestamp,
                "author":"Awesome Note"
}

    myuid = my_id

    try:
        resp = requests.post(url_notes, json=payload_note)
        resp.raise_for_status()
        resp_dict = resp.json()
        print(resp_dict)
        myuid= resp_dict['id']
        my_id = myuid
    except requests.exceptions.HTTPError as e:
        print("Bad HTTP status code:", e)
        print("payload_note:", payload_note)
    except requests.exceptions.RequestException as e:
        print("Network error:", e)

    url_notes_put = (
    "http://"+ip+":"+port+"/notes/"+myuid+"?"
    "token="+token
)

    try:
        resp = requests.put(url_notes_put, json=payload_note_put)
        resp.raise_for_status()
        resp_dict = resp.json()
        print(resp_dict)
    except requests.exceptions.HTTPError as e:
        print("Bad HTTP status code:", e)
        print("payload_note:", payload_note_put)
    except requests.exceptions.RequestException as e:
        print("Network error:", e)

    for my_picture_name in liste_picture:
               cmd = "curl -F 'data=@"+my_picture_name+"' -F 'props={\"title\":\""+my_picture_name+"\"}' http://"+ip+":"+port+"/resources?token="+token
               print("Command"+cmd)
               resp = os.popen(cmd).read()
               try:
                  respj = json.loads(resp)
                  print(respj['id'])
                  myuid_picture= respj['id']
               except:
                  print('bad json: ', resp)

               my_body = my_body + "\n  ![" + my_picture_name + "](:/" + myuid_picture + ")   \n";

               payload_note_put = {
                "body":my_body
                }

               try:
                  resp = requests.put(url_notes_put, json=payload_note_put)
                  resp.raise_for_status()
                  resp_dict = resp.json()
                  print(resp_dict)
               except requests.exceptions.HTTPError as e:
                  print("Bad HTTP status code:", e)
                  print("payload_note:", payload_note_put)
               except requests.exceptions.RequestException as e:
                  print("Network error:", e)


With this script you have : Title, Body, Picture, Location. ( No tags but it’s possible, and no folder ). It’s impossible to use plistlib …. sniff.