A Student Hacker's Life

Protip: don't use 'yarn install' on your CI pipeline

Fri, 14 Jun 2019 00:00:00 +0000

I recently discovered the --frozen-lockfile parameter to yarn install, while randomly reading yarn’s documentation (yes I’m weird):

# Don’t generate a yarn.lock lockfile and fail if an update is needed.
yarn install --frozen-lockfile

This is super useful on a CI pipeline to make sure we developers don’t forget to update the lockfile when we add a dependency. In theory, it should never happen but sometimes people forget to commit some parts of the changes - so using the --frozen-lockfile will save you a new commit or a new PR when it happens.

So whatever CI pipeline you are using, from CircleCI to a self-hosted Jenkins server, that’s one more way of catching quality issues before they get merged ;)

It’s also available in other languages:

If you are also using Ruby for some of your services like we do at Akido Labs, bundler has a similar option:

bundle install --frozen

Python’s pipenv doesn’t look like it has a similar option yet, though. Please let me know if other languages are offering the same feature, I’ll be happy to add them to this post.

Why you should not use test coverage as a KPI

Tue, 05 Mar 2019 00:00:00 +0000

TLDR: Do not use test coverage as a proxy for your application quality. While a low coverage is a sign of a badly tested product, a high coverage has no real meaning in terms of quality.

Recently, I was surprised to see “test coverage” being added to the list of OKRs of my engineering team. This meant test coverage would become a metric we would track and that would help us ship higher-quality applications.

Why it looks like a good idea

First of all, let’s define what it is: test coverage is the percentage of unique logical lines of code in your codebase that is executed during the tests, out of the total number of lines of the codebase. A test coverage of 95% then means that your tests executed almost all your codebase. How cool is that?

Example code coverage on the source code of HowFast Monitoring

Tests are used to make sure the code performs its intended duty, in a repeatable and automated way. More tests usually means that a bug introduced can caught be more easily, leading to improved application quality.

Finally, picking the right metrics for product quality is hard. It depends heavily on what your product does (response time? clarity of the language so that anyone can easily understand what’s going on? time or number of clicks required to book your train ticket?). Code coverage is a metric that applies to all applications as long as they are made of code, so it’s tempting to pick it up.

However, code coverage is not a good proxy for your product’s quality.

Why and how it could go wrong

When building GéoSchool back in the days, I wanted to reach 100% test coverage on all the Python code I was writing. Let’s consider the following test I once wrote:

def test_bullshit(self):
    """ This test should really be improved """
    response = self.client.get('/api/markers_meta.json')
    self.assertEqual(response.status_code, 200)
    # Let's call it a day!

It basically calls a route, and executes the underlying controller. Then it checks whether the HTTP status code of the response is 200 - meaning there was no server error. The good side is that it will detect if the server explodes on this request. The other good side is that it allowed me to reach my target of 100% code coverage. Awesome, now my app will have no bug with such a high test coverage! Of course I was dumb.

This test is not completely assertion-free, but we don’t check anything about the response. The controller could just return an empty JSON response and the test would still pass. This high-coverage test doesn’t actually do any business logic verification.

This is the perfect example of how test coverage can go wrong and not be a proxy for product quality.

Regarding GéoSchool, I then lowered the standard from 100% to 80% and told our CI platform to reject any build below this threshold, so that developers would have to write tests on the feature they were adding. Yes, I still hadn’t learned the lesson. This led to other issues, like developers writing assertion-free tests to be able to merge their code. One interesting issue was that PRs removing code would fail the build because they decreased the overall ratio (if you burn code that is covered by tests, and keep code that is not tested, the coverage falls).

I finally learned from this mistake and I don’t pay much attention to code coverage nowadays.

Also on a more philosophical note, I know we are supposed to get wiser as we age, but it seems to just be that we are more able to see why we were being really stupid only a few years back. I’m afraid of what I’m going to discover in the next year about all the stupid things I did this year. Still, I guess some people do call this wisdom.

What to do instead

One thing is clear: low coverage is a good indicator of low-quality testing. It does mean your tests don’t even execute - let alone test - a good chunk of your code. Having a minimum coverage to respect can be a goal, even though it won’t guarantee that these (few) tests will catch any bug. But while a high test coverage doesn’t mean the application is well tested, a low coverage is definitely a sign that it is not tested.

By running a company and working with other talented CTOs I also learned other interesting things:

Writing tests and maintaining them is hard and takes time (usually from 30% to 100% of the time required to code the feature).
You need to balance the quality of your product with the time you are ready to invest. If you can achieve 80% of the quality in only 20% of the time then do it, and spend the time on something more important. This is especially true in startups where the product will change quite often as you iterate on what your users / the market need. Just stick a big “beta” label on your product until you know the market wants it and then proceed to improve the quality and reliability.
As a result, I don’t recommend investing time in writing tests if you haven’t reached the Product-Market Fit, because you may need to throw away half of your product if the market doesn’t need it.
You can choose to write no tests and to instead manually test the critical path of your app before each deploy. Important bugs will be caught by this manual testing and small bugs would be caught by Sentry when your users hit them. Not ideal but if you have a lot of users, it could make sense to save time on the test maintenance and use it on manual testing. I call this the “Sentry-driven deployment”, but I don’t recommend it unless you really know what you’re doing ;)
One last note about testing and code coverage: it’s fine to not cover branches that should almost never be executed, unless there is a high importance accorded to this branch (“you won the lottery!”). You’re going to invest time/resource into testing something that is not important. The same is also true about language features: you want to focus on testing the business logic of your code, but not the language feature - after all it’s not your job to make sure that the language you’re using does its job.

Just to be clear, I do not advise against writing tests: sometimes they will actually help you ship the feature faster, for instance when it’s easier to test your code using a test than to test it manually by pushing buttons in your application. Test-Drive Development (TDD) is then a very powerful way to go, and will guarantee that your tests actually test the business logic. And the time you save now by not writing tests now is converted into tech debt, that will sooner or later bite you and slow you down. It’s all about balancing the two, and “when to write tests” is a whole piece and will maybe become another blog post.

Overall, if your goal is to measure the quality of your product, do not use code coverage as a metric. Instead, you can use the following checklist items, at the end of every sprint:

did we discover any regression (a previously working feature stopped working unexpectedly) after merging the code?
are the developers proud of the source code?
do we meet our standards (goals) in term of performance (you can use HowFast to measure this), uptime, SEO, - insert whatever your product quality means here -, etc.

In terms of quantitative metric, we still have the following:

number of issues reported by your users or by error reporting tools like Sentry (which can also report the number of regressions). The lower the better.
how much does the customer(s)/user(s) like the application? Rate 1-10.
any customer-focus result metric: time saved for the user, number of projects created inside your app, etc.

I hope this has been useful. Please let me know if you disagree or if you have comments about this, I’d be happy to jump in a chat on Twitter or by email (see the footer of this page). You can also comment on the Hacker News thread.

Linux users: running docker without sudo is dangerous

Mon, 09 Jul 2018 00:00:00 +0000

TL;DR: unless you are using a docker-machine (default on OSX and Windows), running docker commands equals being root on the host. If you added yourself in the docker group or changed the permission of the Docker socket, you (or a malicious program) can become root without needing the password.

As a technical mentor for startups, I often get to talk with CTOs. A few days ago, I discovered that one of them was using docker on his Linux laptop without having to use sudo. This is very dangerous and you should know the consequences.

Docker Engine

Docker is awesome. Replicating a MongoDB database locally, spinning up a Redis server or a Selenium node in seconds, versioning your app’s requirements, all this substantially speeds up our developer lives.

As we type sudo docker run, sudo docker-compose logs, etc. several times per day or per hour, we might be tempted to drop the “sudo” part to avoid typing a password every 20 minutes. The two major “classic” solutions are either:

adding yourself in a “docker” group
changing the permission on the docker socket

If this is your case or if you plan to use one of these technique (or any other, actually), you need to know something:

Accessing the Docker Engine equals having root access on the host

Why so? Let’s take a practical example:

$ docker run -v /:/hostOS -i -t ubuntu

root@7a064e899aa3:/# whoami
root
root@7a064e899aa3:/# cat /hostOS/etc/shadow
at:!:17339::::::
...

Yeah, I just read the content of a file only root can read, and which contains my (encrypted) password a few lines after. Obviously, I can also write to it to change the password, or create a new user with root privileges, etc.

What happened

Thanks to the -v option, we “mounted” the host root folder / on the /hostOS/ folder inside the container.

Running any container image with the root user (default for the majority of images) gives us root privileges inside the container, thus… root privileges over all the files on the host. Now, you should start to understand that it allows you to read (and write!) any file, including the list of system passwords (/etc/shadow), any file containing credentials, and finally executing any command.

One could even create a root-owned program with the setuid bit set, allowing anybody executing it to execute it as root. Such a program could be used without Docker, by virtually anybody on the system, and even after the docker permission issue is fixed.

On OSX and Windows, the Docker Engine cannot typically directly execute on the host, so you have to setup a Docker Machine, which is basically a Virtual Machine running a derivative of Linux, and in which the Docker Engine is executed. On these systems, you gain root privileges on the VM itself and not directly on the host. There are still shared volumes with the actual host, and VM evasion vulnerabilities, though.

Understanding the risk

Ok, but I am the only user on my machine, so why bother?

There is a large number of reasons why you don’t run everything as root on your machine, and two are particularly important:

users execute untrusted code (ever copy-pasted a curl https://.../script.sh | sh command?)
software you execute have vulnerabilities (something as widespread as the Atom editor recently let almost everybody execute anything on your computer: see https://statuscode.ch/2017/11/from-markdown-to-rce-in-atom/) - this is more frequent than you think!

On servers, there are typically multiple users (or there should be): one per developer or team, one for the continuous deployment, etc. This is even more dangerous, since servers usually take care of sensitive data (passwords, personal information, more credentials, etc).

What to do now?

Now that you know what removing the sudo prefix actually does, you have two solutions:

always use sudo: this ensures you are allowed to become root otherwise, and reminds you of the root powers you are gaining
accept the risk associated to opening a privilege escalation vulnerability on your system, and keep in mind that anybody or any (flawed or malicious) program using the docker command will gain root-like privileges

I obviously recommend the first solution ;)

Found this article useful? Share it with your colleagues or friends!

Have a question? Ask it here or tweet me at @msuixo.

GitHub Pages generated a (rogue?) TLS cert for my own domain!

Wed, 18 Apr 2018 00:00:00 +0000

TL;DR: This blog is hosted on GitHub Pages, with a custom domain and over HTTPS. GitHub/Fastly generated the certificate without me knowing, and while this is not supposed to happen.

EDIT: this is a (silent) gradual release of HTTPS support for custom-domain websites hosted on GitHub Pages. Read more at the end of this post. While it shows the certificate is not “rogue”, having certs generated without you knowing or expecting it (this is not an official feature) remains surprising and unpleasant.

This blog is hosted on GitHub Pages

This blog is hosted on GitHub Pages, through Fastly (the CDN used by GitHub).

$ host blog.securem.eu
blog.securem.eu is an alias for mickaelbergem.github.io.
mickaelbergem.github.io is an alias for sni.github.map.fastly.net.
sni.github.map.fastly.net has address 151.101.17.147
sni.github.map.fastly.net has IPv6 address 2a04:4e42:1d::403

I do have a custom domain (“blog.securem.eu”), ie not a *.github.io one.

I am not using CloudFlare, which is commonly used to add HTTPS on top of a GitHub Pages website.

GitHub Pages does not allow you to use HTTPS on custom domains

This is understandable as it would mean generating, storing and deploying a large number of certificates for domains not owned by GitHub. I created a test (GitHub Pages website) at test-github.securem.eu.

On the official documentation, it is clearly stated:

You can enforce HTTPS to add a layer of encryption for traffic to your GitHub Pages site if it has a github.io domain.

Trying to access this website through HTTPS will raise an security error:

The certificate presented is only valid for www.github.com, *.github.com, github.com, *.github.io, github.io, *.githubusercontent.com, githubusercontent.com

As of today, people have used CloudFlare to bring HTTPS to their custom-domain GitHub-Pages-hosted websites, which is not the case here.

WTF

We have:

This blog is hosted on GitHub Pages with a custom domain, and no CloudFlare proxy.
GitHub Pages does not serve content over HTTPS for custom-domain websites

Then, we should get an error when trying to access it over HTTPS:

$ curl https://blog.securem.eu -v
*   Trying 151.101.17.147...
* TCP_NODELAY set
* Connected to blog.securem.eu (151.101.17.147) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=blog.securem.eu
*  start date: Apr 15 15:48:38 2018 GMT
*  expire date: Jul 14 15:48:38 2018 GMT
*  subjectAltName: host "blog.securem.eu" matched cert's "blog.securem.eu"
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x56280a3da620)
> GET / HTTP/2
> Host: blog.securem.eu
> User-Agent: curl/7.59.0
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
< HTTP/2 200
< server: GitHub.com
< content-type: text/html; charset=utf-8
< last-modified: Tue, 17 Apr 2018 14:40:47 GMT
< access-control-allow-origin: *
< expires: Wed, 18 Apr 2018 10:24:28 GMT
< cache-control: max-age=600
< x-github-request-id: D89E:34CA:4DBEB7:6CC5E4:5AD71A84
< accept-ranges: bytes
< date: Wed, 18 Apr 2018 10:14:29 GMT
< via: 1.1 varnish
< age: 1
< x-served-by: cache-lcy19247-LCY
< x-cache: HIT
< x-cache-hits: 1
< x-timer: S1524046470.828003,VS0,VE0
< vary: Accept-Encoding
< x-fastly-request-id: 2e84e2cab76ec88bccbc696a0c4984618c7f2a5f
< content-length: 9260
<


<!DOCTYPE html>
[...]

Oopsie?

So, yes, this very website is served by GitHub Pages over HTTPS. With a Let’s Encrypt certificate generated a few days ago. Here is the certificate in the Certificate Transparency logs: https://crt.sh/?id=399789700.

I did not generate this certificate. GitHub (or Fastly, or someone else) did. And GitHub has the private key associated with the certificate, since they are serving the content over a trusted HTTPS connection.

Conclusion?

I still do not explain how this happened. The best guess is that GitHub is preparing for automatic Let’s Encrypt certificate generation for custom domains, and tested it on a small (?) subset of the repositories.

One way to confirm this would be to check in the Certificate Transparency logs what other certificates have been generated around the same date and time, to see if other custom-domain GitHub Pages websites got new certificates. I didn’t have time yet to investigate further.

Anyway, this looks weird since GitHub generated a certificate for a domain I own without notifying me or asking whether I was OK with it. I could have used DNS CAA, or alerts set on the Certificate Transparency logs to prevent this or be notified of this “rogue” certificate.

Let me know if you think of any other possible explanation, or if you know something about it.

What is most disturbing is the fact that HTTPS seems to work only on blog.securem.eu. Renaming to blog-test.securem.eu shows the usual behaviour (with no HTTPS), but switching back to blog.securem.eu enables HTTPS again.

PS: I am not mad at GitHub for doing it, just surprised since I wasn’t expecting it and wasn’t notified about it. In the end I am even happy to benefit from HTTPS, even if I feel less in control of what happens. The GitHub team does a really great job, and I am very happy to be able to use GitHub Pages for free (even without HTTPS).

Thanks on comments on the HackerNews thread, I discovered this thread that suggests that the feature is progressively rolled out by GitHub. There is also an “official confirmation” in this gist.

Hence, there is no evil attack taking place, but instead GitHub Pages silently doing DV (Domain Validation) to generate certificates, since a few weeks.

What started as an unpleasant surprise (WTF is this certificate doing here?) changed to a happy ending, since I got HTTPS deployed on this blog (by the GitHub team).

I still encourage you to set up Certificate Transparency alerts to be sure all certificates generated for your domains are legit. I am currently testing CertSpotter by SSLMate (free up to 5 domains, covers all the subdomains).

Versioned DNS records with Gandi's LiveDNS

Fri, 24 Nov 2017 00:00:00 +0000

TL;DR: I wrote a tool to manage DNS records through any GIT repository, hence versioning the zone records. It uses Gandi’s LiveDNS API.

When updating my DNS records, I need to use the web interface of my registrar (Gandi). The zone records are versioned, but I can’t easily display diffs, let users propose changes through Pull Requests, or comment on potential implications of changes before applying the change. The best source version control available today is git, so why not using it for DNS zones?

Being a supporter of Infrastructure-as-Code, I wrote a small Python tool to manage my DNS records from a versioned (GIT) repository. You need to be a user of Gandi’s LiveDNS, or you will have to adapt the tool to your registrar’s API.

Configuration

Fork the main repository: https://github.com/MickaelBergem/gandi-livedns-zone-manager
Clone it locally
Retrieve your LiveDNS API key: go to your Gandi account, under “Security”, “Generate your API key”.
Put your key in a api_key.txt file in the repository
Test your configuration by running ./livedns.py view

This should output your current zones:

$ ./livedns.py view
== Zone test-zone [580549ec-c25f-11e7-9d8f-00163e6dc886] ==
        No domain associated with this zone.
        1 records in this zone:

        CNAME   10800   test                access.mail.gandi.net.

Usage

# Pull the zones and write them on disk
$ ./livedns.py pull
== Zone test-zone [580549ec-c25f-11e7-9d8f-00163e6dc886] ==
        Writing... done (test-zone_580549ec-c25f-11e7-9d8f-00163e6dc886.txt)
$ ls -l zones
-rw-r--r-- 1 suixo users  133 24 nov.  00:06 test-zone_580549ec-c25f-11e7-9d8f-00163e6dc886.txt
# Now make some changes in your zone files
$ vim zones/test-zone_580549ec-c25f-11e7-9d8f-00163e6dc886.txt
# And push the changes to your registrar 
$ ./livedns.py push
Found zone test-zone (580549ec-c25f-11e7-9d8f-00163e6dc886)
        Uploading the zone... ok
        Server answered: DNS Zone Record Created
# The changes are immediately effective
$ dig CNAME test.test-domain.org
[...]
;; ANSWER SECTION:
test.test-domain.org. 10800    IN      CNAME   access.mail.gandi.net.
[...]

Benefits

you can manage the zone files exactly like usual source files, with your favorite Version Control System such as git
if you are using GitHub or a similar platform, your collaborators can submit Pull Requests to change the zone file
changes are effective immediately

I hope this little tool will help you, do not hesitate to share suggestions or to report bugs on the project page :)

RSS autodiscover on your blog

Mon, 20 Nov 2017 00:00:00 +0000

Last week I read an interesting article about why RSS was still so useful in 2017, and it reminded me how much I was using RSS feeds and aggregators a few years ago.

"RSS: there's nothing better" - or why old geeky things are still very useful for everyonehttps://t.co/2tW6998dg2
— Suixo (@msuixo) 11 novembre 2017

But last week, Mozilla also released Firefox Quantum with a built-in “Subscribe to this page” RSS button (I suspect it had been there for a very long time and I just didn’t see it before).

And then, horror! The RSS feed for my blog is not detected! … Oh ok I need to add something. Easy.

To signal to browsers such as Firefox that your blog or your website has a RSS feed they can subscribe to, you need to add the following code (in your <head>):

<link rel="alternate"
    type="application/rss+xml"
    title="My awesome blog's RSS feed"
    href="/rss.xml" />

And here you go, the button now allows your users to subscribe to your feed. Aggregator will also now “see” your content and make it discoverable to other readers.

You can also add multiple tags if you have multiple feeds.

That was a quick fix to a problem many website have today, so if you haven’t already, add this nice little chunk of code to your pages :)

Pipe tar through SSH to transfer large files

Sat, 14 Oct 2017 00:00:00 +0000

Using tar and ssh to efficiently transfer large files across hosts

Ever ran out of disk space to create a (compressed) archive of a folder you want to backup or transfer to another host?

Here is the solution: directly pipe the output of tar (used to create archives of files and folders, and also often used to compress this final archive) through ssh to retrieve the compressed archive on another host. This allow you to backup large volumes of data when the source host doesn’t have enough space to store the original data + a compressed archive of the data.

From the distant host (the host you want to copy the backup to):

# Execute this on the host that will receive the data from the source server
ssh user@source-server "tar czpf - /some/important/data /some/other/file/or/folder" | tar xzpf - -C /backup/destination/folder

This will copy the two files/folders into /backup/destination/folder on the destination host (the one executing the command).

If on the other hand, you want to send the data to another server and you are executing the SSH commands from the source server, use the following command:

# Execute this on the machine with the data available locally
tar cpf - /some/important/data /some/other/file/or/folder | ssh user@destination-server "tar xpf - -C /backup/destination/folder/"

DNS CAA - A (not so) new mechanism for a better PKI

Sun, 02 Jul 2017 00:00:00 +0000

TLDR: DNS CAA is a DNS record allowing domain owners to restrict which CA are allowed to issue certificates for their domain. While it is a useful additional security feature, it does not protect against unwilling CAs nor mis-issued certificate being used in the wild. CA must implement it before September 2017.

A few days ago, I discovered DNS CAA (which stands for DNS Certification Authority Authorization) and wanted to get it a try. Here is a short write-up about what I learned, and the steps if you want to protect your domain with DNS CAA too.

While quite old (2013), this standard will become mandatory in September and all CA will have to implement it or face sanctions.

A new DNS record

DNS CAA is actually a new DNS record (the “DNS CAA Resource Record” or DNS CAA RR for the folks who love acronyms). RFC 6844 explains it role:

The Certification Authority Authorization (CAA) DNS Resource Record allows a DNS domain name holder to specify one or more Certification Authorities (CAs) authorized to issue certificates for that domain. CAA Resource Records allow a public Certification Authority to implement additional controls to reduce the risk of unintended certificate mis-issue.

Okay, it allows us - domain owners - to specify what CAs we are allowing to deliver certificates for our domain.

Until now, any CA can deliver a certificate for any domain name. It is a known weakness of the traditional web PKI, and we have seen many examples of CA abusing their power to deliver counterfeit certificates (the most recent one being Symantec, but also Trustwave, among others). Some even didn’t issued these certificates on purpose (sic).

Please note that some CAs mis-issuing certificates had actually been compromised: Comodo and Diginotar, Verisign… The new DNS CAA Resource Record aims at preventing the first class of these issues: non-compromised CA delivering certificates to the wrong person. DNS CAA will have no effect if the CA is controlled by a malicious actor.

How it works

Aside note: did you know the first checks restricting a CA’s ability to deliver certificates were written after… the French National CyberSecurity Agency (ANSSI) kind of (the whole story is bit more complicated) delivered a certificate for “google.com”. Of course, it wasn’t officially allowed to do it. So now, if you search for “ANSSI” in Mozilla’s source code, you will find this diff explicitely stating that any certificate delivered by the ANSSI is only valid for *.fr domains, or a couple of other French-related TLDs. I find it pretty amazing to have a hardcoded constraint in a browser just for my country’s agency.

Unfortunately, this kind of event will still be possible with DNS CAA. More on this in the next parts, but first, what is this brand new DNS record?

The DNS CA RR

A DNS CA RR is composed of:

some flags (currently just a “critical” flag if 128, otherwise value is 0)
a tag name (issue, issuewild, iodef)
a tag value

Thus, the following record states that only Lets’Encrypt is allowed to deliver certificates for example.com:

example.com.  CAA 0 issue "letsencrypt.org"

It is possible to allow multiple CAs for a same domain:

example.com.  CAA 0 issue "comodoca.com"
example.com.  CAA 0 issue "letsencrypt.org"

issuewild says the CA is allowed to issue wildcard certs:

example.com.  CAA 0 issue "letsencrypt.org"
example.com.  CAA 0 issuewild "comodoca.com"

iodef is used to get notified of policy violations:

example.com.  CAA 0 iodef "mailto:example@example.com"

For now, I don’t know of any CA supporting this directive, but it is a good practice to use one so that you are prepared.

DNS CAA only affects the certificate generation process

From the RFC:

Before issuing a certificate, a compliant CA MUST check for publication of a relevant CAA Resource Record set. If such a record set exists, a CA MUST NOT issue a certificate unless the CA determines that either (1) the certificate request is consistent with the applicable CAA Resource Record set or (2) an exception specified in the relevant Certificate Policy or Certification Practices Statement applies.

So the RR is only checked at certificate issuance time, but not at “runtime” in the browser, when a TLS connection is established.

This is important: DNS CAA will not protect you from the malicious CAs issuing certificates for your domain (not like the NSA actually needed it). If you are looking for this kind of protection, look at DANE (binding a x509 certificate to the DNS records) and HPKP (HTTP Public Key Pinning, basicaly the same idea but through a HTTP header).

Do it yourself

Unlike HPKP that can be pretty tragic if misconfigured, you can safely reproduce the following DNS CAA steps at home. The only inconvenience you may experience is an administrative one while trying to renew your certificates, and updating your DNS records will get you safe while not affecting your uptime or your visitors.

Read and check DNS CAA records

Before adding our own records, let’s check out how Google is using the CAA RR:

$ dig +nocomments type257 google.com
; <<>> DiG 9.10.4-P5 <<>> +nocomments type257 google.com
;; global options: +cmd
;google.com.                    IN      CAA
google.com.             86399   IN      CAA     0 issue "pki.goog"
google.com.             86399   IN      CAA     0 issue "symantec.com"

dig does not (yet) support this CAA RR, so we have to use type257 instead. Here we learn that only pki.goog and Symantec are allowed to deliver certificates for google.com (and no one is allowed to deliver wildcard certs).

Fun fact: Google seems to be still experimenting with this new standard as domain names like “google.fr” are not (yet?) returning CAA records.

Add your own records

Now, let’s add you own records. Let’s say you are using Let’s Encrypt and you want to protect your domain blog.securem.eu. Only Let’s Encrypt should be allowed to deliver certificates for this domain. You also want to be notified if there is a policy violation (if someone tries to register blog.securem.eu on GoDaddy for instance), at your email elon.musk@gmail.com (lucky you!). Here are the records you need to add to your DNS zone file:

blog.securem.eu.	IN	CAA	0 issue "letsencrypt.org"
blog.securem.eu.	IN	CAA	0 iodef "mailto:elon.musk@gmail.com"

Beware: not all registrars currently support CAA records, you may try https://sslmate.com/labs/caa/ with the “legacy mode” to make it work.

Did you do well?

Tools such as test.ssl or the Qualys Server Test now check these records, and you should now get a green result, yipeee!

Take-away

While it is not yet supported by all the providers, this 4-year-old standard has recently become mandatory for CAs. Thus, all of them will have to implement CAA verification by September, so be ready and add your own records today!

Please also note that this standard is not bulletproof but will contribute to your overall security.

I will be more than happy to answer your questions, comments, and thoughts about this “new” mechanism! :)

Feel free to comment on the Hacker News conversation or in the Discuss below.

How I renew my TLS certificates automatically thanks to Let's Encrypt

Wed, 27 Apr 2016 00:00:00 +0000

TL;DR: this post explains how my VPS automatically renew its HTTPS certificates, by using Let’s Encrypt and a simple crontab.

You said Let’s Encrypt?

First of all, you may not have heard of Let’s Encrypt, which is a free, automated, and open certificate authority (CA), run for the public’s benefit. One of its goal is to make the use of a SSL/TLS certificate (that gives you the HTTPS) more widespread, mainly by providing users with free certificates and in a very simple way: all you have to do is to install their tool letsencrypt on your server and then to run one command to retrieve and install the certificate.

Well, this is the theory, and the script is still in beta so it may require a bit of tweaking to actually receive your certificate, but it is still a very easy task.

To actually give you the certificate you need and to prevent you from asking a certificate for google.com, Let’s Encrypt checks that the server running the script does receive traffic for the domain name you specified. If I want to get a certificate for lab.securem.eu, Let’s Encrypt will check that I do receive the HTTP(S) requests for lab.securem.eu.

In my case, I have an OpenVPN server running on port 443, and I configured it with the port-share parameter that allows the non-OpenVPN traffic to be forwarded to another address/port. My Apache is also listening on the port 444 so I forward all the non-OpenVPN traffic on port 443 to this alternative port. I am not quite sure this is the reason of Let’s Encrypt having problems but it most certainly plays a role.

Get your first certificate

To start, you can try the standalone mode that will run a little server locally, and listen on the port 80 or 443 for the challenges. Let’s choose the port 443, because we do not have yet any HTTPS-enabled website.

First, install the tool:

git clone https://github.com/letsencrypt/letsencrypt
cd letsencrypt
./letsencrypt-auto --help

Let’s retrieve your certificate! We will use the standalone mode, so you will have to stop all the services listening on port 443. In my case:

sudo service apache2 stop

Then, just run one command:

./letsencrypt-auto certonly --standalone --standalone-supported-challenge tls-sni-01

Use http-01 instead of tls-sni-01 if you prefer to bind to port 80.

Just enter the domain(s) the server owns and for which you want a certificate. Et voilà!, you got the appropriate certificate! Any additional domains will be contained in the certificate using the SNI extension (Server Name Indication) that allows one server and one certificate to respond to multiple HTTPS-enabled domains with no name clash.

Now, all you have to do is to configure your web server (or whatever will use the certicate) and point it to the /etc/letsencrypt/live/your.domain.here/fullchain.pem public certificate file. The private key is placed in the same folder, as privkey.pem.

This process is really fast and easy compared to other CA where you have to send documents by mail or at least to wait for a few days.

One thing you may have noticed is that the certificates are only valid for three months. This is done on purpose, for security reasons (in case of key compromise and mis-issuance) and to encourage automation. This means that in three months you will receive an email saying that your domains are close to the expiry date and that you should renew them as soon as possible.

Automatically renewing the certificates using the webroot method

After manually renewing you certificates the first two times, you will begin to say “Hey, I will definitely automate this process!”. Lucky you, Let’s Encrypt is really easy to automate.

If we automate it, we won’t be using the standalone server as we don’t want to stop and start the Apache daemon with no user interaction (and because we don’t like downtime). If you are lucky, you may be able to use the apache or even nginx plugins that automate everything for you. If you are not, like me, give a try to the webroot method. With this method, Let’s Encrypt will try to access a file inside a .well-known directory at the root of your website. In my case, it will try to access a file at http://kikoo.securem.eu/.well-known/letsencrypt-challenge-something. This method allows to keep your website running for every other endpoints on http://kikoo.securem.eu/.

First step: configure you virtualhost to redirect the requests on .well-known to /var/www/letsencrypt-webroot/.well-known. For Apache, all you need is an Alias directive (don’t forget to reload the configuration after that):

Alias /.well-known/ /var/www/letsencrypt-webroot/.well-known/

Second step: Run the command a first time to ensure everything works fine:

./letsencrypt-auto certonly -d your.expiring.domain.com --webroot --webroot-path /var/www/letsencrypt-webroot/ --non-interactive

I chose to put then renew command in a crontab, ran as root as it needs to access protected parts of the system. It is ran every month at 1am.

sudo crontab -e
# This should open an editor, so add the following line
0 1 1 * * /where/you/installed/letsencrypt/letsencrypt-auto certonly -d your.expiring.domain.com --webroot --webroot-path /var/www/letsencrypt-webroot/ --non-interactive && nginx -s reload

One important thing is that you need to reload your webserver (the && nginx -s reload part) to make it use the new certificate: without that it will continue using the old one. Thanks to Justin Finkelstein in the comments for pointing it out.

Now you can let your server manage its own certificates :)

I hope this article will be useful to you, let me know how YOU proceed to renew your certificates!

Creating and publishing a Python package

Mon, 29 Feb 2016 00:00:00 +0000

In this post I explain how to transform a working code into a python package that can be installed from PyPi using pip install yourmodule. I will suppose you are modern and use Python 3. For those in a hurry, three steps: clone a repository, update the setup.py file, and follow the instructions in the last part.

Quick reminder on Python modules and packages

Creating a Python module is easy:

Create a file with your code, call it myhellomodule.py (or whateveriwant.py):
```
def hello():
    print("I am a Python3 module!")
```
This a module: a file with things in it. You can import it or do whatever you want.
Make it a package:
```
touch __init__.py
```

And ta-daaa you have a Python package. If the directory containing the two files (__init__.py and myhellomodule.py) is called swaggymodule, then you can from swaggymodule.myhellomodule import hello (from the parent directory).

Nothing fancy for now. Let’s see how to publish such a wonderful package, so that your grandmother can install it with a simple pip install swaggymodule.

Packaging your project

First, clone the sample project and give it the name of your module:

git clone https://github.com/pypa/sampleproject swaggymodule

The important files:

setup.py allows you to specify your project configuration, and to run packaging commands: try python setup.py --help-commands
setup.cfg is an INI file containing option defaults for setup.py commands
README.rst describes the goal of the project, using reStructuredText

Copy your module(s) inside this new folder and remove the existing “sample” module:

swaggymodule
|- swaggymodule
  |- __init__.py
  |- myhellomodule.py
|- setup.py
|- setup.cfg
|- README.rst

Configure the name, version, description…

First, open the setup.py file, to enter a few details about the project. All the fields are self-explanatory but here are a few additional pieces of advice:

name: according to PEP0008:

Modules should have short, all-lowercase names. Underscores can be used in the module name if it improves readability. Python packages should also have short, all-lowercase names, although the use of underscores is discouraged.

Never use dashes (-) in your package name because importing it would become a very hard task.
license: in case of doubt (what should I use?), try http://choosealicense.com/: it’s super clear.
classifiers: check https://pypi.python.org/pypi?%3Aaction=list_classifiers for a full list.
entry_points: in case your application plugs in an existing system, you may specify an entry point. See this page for more details. But in most cases you should just set it to an empty dictionnary.

The actual packaging steps

So that you don’t run into the same issues as me:

pip install wheel twine setuptools --upgrade

First, create a Source Distribution. This kind of “distribution” (aka “Package”) requires a build step when installed by pip.

python setup.py sdist

But we want to install a “wheel” (a built package) which is faster to install than a source distribution:

python setup.py bdist_wheel

Uploading your package

You will need an account on PyPi. But also create one on the test platform we will use: testpypi.python.org.

The official documentation recommends to use twine instead of the default python setup.py upload because the latter uploads files and credentials over plain HTTP whereas twine uses TLS (so use twine).

To save the configuration data, create a file in ~/.pypirc:

[distutils]
index-servers=pypi

[pypi]
repository = https://testpypi.python.org/pypi
username = yourusername

You may also add a password line if you don’t want to type your password everytime. Then:

# Register your project
twine register dist/*.whl
# Upload the project
twine upload dist/*

And this is what you should get when you head to testpypi.python.org:

If you need to make updates: change your code, bump the version to 1.0.1 in the setup.py file, optionally commit your code to your repository, and do:

twine upload dist/swaggymodule-1.0.1*

This should have updated your project online and made the new version available. Now try it:

$ pip install -i https://testpypi.python.org/pypi swaggymodule
Downloading/unpacking swaggymodule
  Downloading swaggymodule-1.0.1-py3-none-any.whl
  ...

It works! But before letting people know about your new package, we should upload it to the actual PyPi (not the test one). The process is summed up in the next section:

In a nutshell

pip install wheel twine setuptools --upgrade
# Check and update the server to use (https://pypi.python.org/pypi) and your credentials
nano ~/.pypirc
# Build the distributions
python setup.py sdist
python setup.py bdist_wheel
# Register and upload your package
twine register dist/*.whl
twine upload dist/*

To go further

The official tutorial
The naming convention
A more complete article: Open Sourcing a Python Project the Right Way by Jeff Knup.

Following these steps, I was able to publish a small application we were using on a closed-source project, to share it but also to decrease the amount of code inside the project itself that was not focused on the business logic. This application is a Django application enabling users to generate a new secret key when deploying their Django project. In our project, a simple pip install django-generate-secret-key replaced a whole folder and saved us space, complexity, and increased the readability of the source code.

Recording a timelapse with a Raspberry Pi

Sun, 21 Feb 2016 00:00:00 +0000

TLDR: this post explains how to use your Raspberry Pi to record pictures and then assemble them into a video with ffmpeg to produce a timelapse video.

Yesterday I had a party at home, and I chose to record the fun of the party with my brand new Raspberry Pi. I had bought a camera with the Pi, the model without the infrared filter (that blocks infrared) called the NoIR Camera. This camera basically allows you to “see” the infrared light - that humans cannot see - used for TV remotes, Wiimotes, fingertracking or… night vision. Useless today, it just produced weird colors (blue / pink shades) on the pictures, so you may better use the standard camera (that blocks IR).

Setting up the Raspberry Pi and the camera

Several third-party tools allow you to do a timelapse with a Pi, with a lot of options and features, but I chose to stick with a more simple tool already present on the Pi: raspistill. Along with raspivid, these tools allow you to play with your camera.

First of all, plug your camera. You may need to activate the support for the camera with raspi-config if you never did it before. I had a permission issue because I was not using the default pi user and thus was not in the video group.

We can take a picture with raspistill (see the documentation if needed). Let’s try it now to check if the camera does see us. Maybe we will need to flip the image or move the camera.

raspistill -o cam.jpg

I am using a headless Pi (without graphical server) with SSH so I had to scp pi:cam.jpg . to retrieve the picture and check the result.

Start the capture

Then, now that your camera is placed well and you are ready to start the capture, all you have to do is save an image every N seconds:

raspistill -t 0 -tl 10000 -n -w 1920 -h 1080 -o image%04d.jpg

raspistill has a built-in feature to do the timelapse:

-tl T: a picture will be taken every T millisecond (here 10s)
-t 0: never timeout, keep shooting. You may want to set it to a high value like a few hours (value in ms)
-n: do not show any preview on the HDMI screen (anyway we have no screen)
-w 1920 -h 1080: HD is sufficient for a timelapse, you don’t want a final 4GB video! Default is 2592 x 1944 (5 megapixels)
-o image%04d.jpg: filename for saved files, here %04d means the number of the picture, on 4 digits

About the output format, you can replace %04d by a simple %d and add the -dt (--datetime) flag that will replace the frame number by the datetime (MonthDayHourMinSec), to produce test221172225.jpg (February 21st, 17:22:25). If needed, the -dt flag can be replaced by -ts (--timestamp) that will use the number of seconds since 1900.

Using the initial format, you will soon see a large number of files like image0012.jpg, that can take up a large disk space.

What delay for what storage space? I chose to take a picture (~3MB) every ~20s, which is ~9MB every minute, 520MB every hour. I got a nice 4,7GB for 9h of time. Plan carefully your timelapse if you do not have much space on your SD card.

Assembling the pictures into a video

First, move the pictures to your computer, to benefit from a higher processing power (needed for video edition). You can use scp, rsync, but I recommend to just insert your SD card into your computer (way more quick and easy).

Some tutorials suggest using mencoder but this package is not available on my Ubuntu. I thus used ffmpeg.

Assemble the pictures (this step is I/O intensive, you may want to use your SSD), at a rate of 25 pictures per second:
```
ffmpeg -r 25 -pattern_type glob -i '*.jpg' -c:v copy output.avi
```

Convert your video to a smaller format (this step is CPU intensive):

ffmpeg -i output.avi -c:v libx264 -preset slow -crf 15 output-final.mkv

And TADAAA your timelapse video is ready! :D

Now, let’s laugh a bit seeing the potato chips bowl magically emptying, the glasses moving, people dancing, playing games, kissing or falling asleep. And you can even create GIFs of your funny friends using the raw pictures!

Setting up an ownCloud Server in a Docker container using Docker Compose

Tue, 25 Aug 2015 00:00:00 +0000

TLDR: this post explains how to use Docker to setup an ownCloud server with a PostgreSQL database, persisting the data across reboots or image upgrades.

I finally reinstalled my entire VPS, using the provisioning tool Ansible, and I wanted to have a more isolated install of ownCloud. I decided to use a Docker container to run this application in a more isolated and portable way, and to avoid installing this application directly on the server.

So I used Docker

First question, what image to use? When typing owncloud on DockerHub, I immediately see there is a (recent) official image, simply named owncloud.

We can try to run it right now to check everything works fine:

docker run --rm -it --name owncloud -p 80:80 owncloud

That was quick. Now, you can exit/stop this container.

We will need a database

If you don’t already have a database up and running you want to use, you can either install one on the host, or spin up a second container for it. Being an adept of PostgreSQL, let’s use the associate container:

docker run --name owncloud-postgres -e POSTGRES_PASSWORD=mysecretpassword -d postgres
docker run --rm -it --link owncloud-postgres:owncloud-db --name owncloud -p 80:80 owncloud

Instead of a SQLite database, we use a PostgreSQL one, with user postgres, the password supplied when creating the container (here mysecretpassword), the database postgres, and the host owncloud-db.

You should be able to get a running instance by entering these settings in the startup page of ownCloud. If it doesn’t work you can see the debug log (created after the first request on owncloud), using:

docker exec -it owncloud tail -f /var/www/html/data/owncloud.log

Persisting the data across container destruction and upgrade

We will need to store the ownCloud data and configuration folders outside of the container, as upgrading the container would destroy everything.

This can be achieved by using Docker Volumes, and even docker data volumes. We will need to do the same for the PostgreSQL container (think “database backup”).

At this point, copy-pasting the commands to run these containers begins to be unreadable, and painful. What we want is to describe how containers should be started, linked, and configured, so let’s use Docker Compose.

Docker Compose

The concept is simple: use a YAML file to describe how your containers should be… composed together, and then let a docker-compose up start everything!

You can install it with pip install docker-compose.

Here is the docker-compose.yml file:

# Composition of the containers needed

owncloud:
  container_name: owncloud
  image: owncloud
  ports:
   - 80:80
  links:
   - postgres:owncloud-db

postgres:
  image: postgres
  environment:
    - POSTGRES_PASSWORD=mysecretpassword

All you have to do now is docker-compose up to start all the containers.

Persist the data

Let’s add some persistence by adding some data volumes: we want to create “data” containers for PostgreSQL (for the data itself) and ownCloud (configurations and files stored).

The folders we need to persist are:

/etc/postgresql for the database configuration
/var/lib/postgresql for the database content itself
/var/www/html/data for owncloud’s data
/var/www/html/config for owncloud’s configuration

This is achieved using the docker-compose.yml file I uploaded in this Gist, take a look at it.

To check all the data is persisted, you can star files (or remove/create files), and then remove the postgres and owncloud containers to check if the next docker-compose up does persist these modifications:

docker rm -f test_owncloud_1 test_postgres_1 # Replace "test" by the name of your folder
docker-compose up

Then go back in your browser:

Here you are! You have a fully functional ownCloud installation running locally!

Upgrade?

Please note that the owncloud-data_1 and postgres-data_1 containers should never be deleted. To upgrade the other non-data containers (postgres or owncloud) for example when a new image is available:

docker pull owncloud
docker pull postgres
docker-compose -d

The -d flag is to detach from the containers so they can start in the background :)

Last settings

To setup your own cloud, don’t forget to change the hostname and domainname lines in the compose file.

You may also want to enable HTTPS, this will be for another blog post.

Tracking events with Piwik

Mon, 09 Mar 2015 00:00:00 +0000

Piwik ?

Piwik is the leading open source web analytics platform, and offers many advantages compared to Google Analytics (that I have used for years before switching to Piwik):

free and open-source software: although Google Analytics is free too, Piwik is open-source and will always remain free and hackable
you can install it on your own server: no data is sent to a third party and you stay the master of the data, this is especially important for people (like me) thinking it is important not to share all your (and your customers’) data with Google or advertising companies
you have access to the data: you can use APIs, custom plugins or even manually execute SQL queries to build custom reports or bind a CRM to it
the community is big, and the development is fast, as shown by the Github activity graphs :
Piwik appears to be lighter and to load faster (haven’t done the test myself)

The main advantage I find in Piwik is you are in control, if you need to remember one thing. Note that features may be more developed and advanced in Google Analytics than in Piwik, this blog post compares both.

Tracking events

Now, what ?

Piwik and Google Analytics only track page views by default (who was where, when, how), but not what happened inside the page. And you may want to track more precisely the behavior of your visitors, for many different reasons :

who clicked on which button, and triggered what action inside the page ?
who read the text to the end (footer) ?
which photos are the most hovered on the “about / team” page of your website like this one ? This one is a very nice idea for curious people like me (who is the most charismatic in your team ?)
when you have dynamic content : who visited tabs, accordions ?

Every time you need to track what happened on the client side, Piwik can track it, and this is really easy. All you have to do is to call the following piece of Javascript :

_paq.push(['trackEvent', 'CategoryName', 'ActionName', 'ActionValue']);

For example, in the case of the team pictures :

$(function () {
    $('.team-member > img').on("mouseover", function () {
        _paq.push(['trackEvent', 'Behavior', 'HoverPhoto', this.id]);
    })
});

Here is the result :

Note that you can also pass an optional numeric value for the action :

_paq.push(['trackEvent', 'Posts', 'RatingPost', 'PostName', 8.5]);

Tracking events directly from you application or server

It is also possible, but I haven’t tried it yet, to trigger events from your backend, with the Tracking HTTP API : register when emails are sent, when an error occurs, or anytime something happens inside the backend !

And if you want to join the community, head to the Piwik Developer Zone ! :)

Flooding the Phisherman (with fake credentials)

Sun, 08 Mar 2015 00:00:00 +0000

This morning, another phishing email in my inbox. Short message, and a link to http://priveservicetemporaire.fr/ (targets French users).

I remembered reading this tweet from @averagesecguy :

Find a phishing site? Overwhelm it with fake credentials. https://t.co/nS56XNb6ek It's preconfigured for a phishing site I found today.
— Stephen Haywood (@averagesecguy) 23 Février 2015

I didn’t found the tweet and the link in the first place, so I built a small Python script to do the same. It just posts random credentials on the given URL and checks the response to be sure everything went fine.

Release the Kraken !

Then I began to flood the phishing website :

$ python3 phisherflooder.py
#0000   Envoi : (.abxlsniznaan@gmail.com, MHPTwHcJw)
#0001   Envoi : (ukyf.pksw8rnp_hxkimb2gena@laposte.net, RiwnRKQ6DVKiVOKM)
#0002   Envoi : (zc1fmuuzqrtsp8hxscpf@live.net, gThR4c)
#0003   Envoi : (rijnikhdj4pa4dtdjcmlc4nyvgwc@gmail.com, 1WP83dxul3O6ECx)
#0004   Envoi : (ezd9_92zbosrisy@orange.fr, AF8oKV3cEn)

[...]

#0075   Envoi : (l1x3e40vkjnua8zr@yahoo.com, nQWCCi)

After the 75^th post, the flood stopped working: the website was throwing an error (“Error sending the message”). My guess is that the form sends an email to the “phisherman” instead of storing the credentials on the server, and the hoster limits the number of sent email per hour (75 mail / hour ?). Trying a few hours after gave the same result.

Now the phisherman is receiving hundreds of fake credentials into his mailbox, each day. Nice, eh ?

Okay, now even if someone fills in their credentials, nothing will be sent. But even if I plan to let the script run until the website finally gets down, there is something more efficient to do: report the incident !

Reporting phishing incidents

Whois can give us the “abuse” registrar contact, and information about who and when the domain name has been registered.

whois priveservicetemporaire.fr -H

Sending an email to report the phishing website will hopefully result in the server being shut down.

I then looked at the email, especially the headers to see from where did the email come from :

Received: from webmail.no-log.org (webmail.no-log.org [80.67.172.39])
    (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits))
    (No client certificate requested)
    by lpn-prd-vrin014 (Postfix) with ESMTPS id 558E9180143;
    Sun,  8 Mar 2015 06:10:00 +0100 (CET)
Received: by webmail.no-log.org (Postfix, from userid 33)
    id 1802C4BDCC; Sun,  8 Mar 2015 06:10:00 +0100 (CET)
X-Squirrel-UserHash: QFEPDQEJKSAAAAAAABpdRlU=
X-Squirrel-FromHash: AgsCDl4DCio=
Message-ID: <cf9a229bb89f6718c704165a5462e8b3.squirrel@webmail.no-log.org>
Date: Sun, 8 Mar 2015 06:10:00 +0100
Subject: infos m.laposte
From: P.arametres-La-poste@no-log.org
User-Agent: SquirrelMail/1.4.21

So this email has been sent from a webmail webmail.no-log.org : their website shows an abuse email contact. I then sent them the full email (with headers) to them so they can block the corresponding account.

TL;DR

received a phishing email
built a tool to flood the fake website with fake credentials
reached a limit that prevents new credentials to be submitted within a period of time
reported the phishing campaign to the hoster and the SMTP provider

Python ProTip : caching the PyPi packages downloaded by PIP

Fri, 06 Feb 2015 00:00:00 +0000

WARNING: this method is deprecated since pip 6.0, as caching is now enabled by default

The How

Add the following lines in ~/.pip/pip.conf (which may not exist) :

The Why

A few weeks ago, I was still working (as an intern) at Theodo as an Agile Web Developer, deploying my source code several times a day on a staging server (and pushing to production at least once a week). This is continous deployment.

The deployment script, written with Ansible, was creating a whole new build of the project in a new folder (then linked the “current” folder to this “release” build):

downloading and building the assets for the frontend: Bootstrap, Font Awesome, etc. (with Bower) with npm install
installing the Python backend part by creating a new virtualenv inside the “release” folder, and then running pip install

At each and every deployment, all the python packages needed were downloaded, then installed, which is time-consuming. Caching allow us to gain time on the downloading part.

Note : To gain time on the installation part, it is also possible to use a common virtualenv for all the releases, but there is the risk of suppressing a vital package in the requirements.txt file without noticing (only builds on brand new machines will fail). I also experienced a similar issue with the frontend part (npm install downloads and install the JS packages in ./node_modules) where a package suddenly stopped working in the way we were using it : we noticed it immediately and fixed our code, instead of facing this bug during a production deployment.

Unfollower : minimal Python script to track who unfollowed you

Thu, 05 Feb 2015 00:00:00 +0000

You may have noticed that some Twitter accounts are massively following other people, adding several dozens of accounts each day, in order to be followed back and to increase their number of followers. In the beginning, I followed some of them as they were matching some of my interest fields (entrepreneurship, web, new technologies…), but I then discovered that as soon as I followed them, some of them were unfollowing me.

I guess this is to keep a “decent” number of “following” accounts (to not have 100x more accounts followed by you than following you).

I also guess some of my followers either don’t find me interesting, or don’t like me posting French content (which I understand), but this is another question and it won’t be solved by a script ;)

So I created this Python script, which basically stores in a SQLite file (but you can configure any SQLAlchemy-compatible backend) the list of you follower and poll every hour the Twitter API to check if somebody unfollowed you.

There are already many online tools which tracks unfollowers, with nice charts like unfollowerstats.com, and even paid plans : who.unfollowed.me. My script is not intended at all as a replacement of these web interfaces, but is a base to bind custom notifications or callbacks.

Check the code !

Technical details

SQLAlchemy allows you to use a lot of database backend if you need / want to. By default, SQLite is used for its simplicity.
the Python script is ~60 lines and is fully customizable, whenever you want to adapt the notification system
using the Twitter API is really easy thanks to the python-twitter package
in order to use the API, you will need to create tokens and consumer key

The installation is straightforward, just follow the instructions.

I recommend using python-twitter (instead of the twitter PyPi package), because I find it simpler and easier to use.

Demonstration

At the first start, Unfollower will detect your followers and add them into the database.

Then, whenever someone unfollow you, a line will be printed :

Reloading this blog on GitHub Pages with Jekyll

Sat, 13 Dec 2014 00:00:00 +0000

Hey there ! It has been a long time since I have posted anything on my old Dotclear blog. I used to write things for myself, to share the little knowledge I gained, interesting websites, and good security practices.

It was written in French, my mother tongue, and was left abandoned, but I felt that there would be a wide variety of topics I would love to talk about, and I wanted a new start. To be able to share with more people across the world, I also wanted to write the content in English. Sorry for the misspellings, mistakes, and other weird sentences, don’t hesitate to post a comment to correct me !

I wanted a fast (my previous hoster, Olympe Network , is doing a wonderful work to provide free, ad-free, and opensource hosting to ~50000 users, but the performances weren’t good enough), simple, and customizable blog.

Why Github Pages and Jekyll

First, Github Pages is a really simple (free) way to publish static code : just push code in the gh-pages branch of a repository and it will become available at http://yourusername.github.io/repositoryname/ ! GitHub pages are using Fastly CDN to improve loading times, which makes it fast.

But as only static (HTML for instance) pages can be committed, you will need Jekyll to manage the global layout, structure and formatting :

Ability to write content in markdown or textile in your favorite text-editor.
Ability to write and preview your content via localhost.
No internet connection required (to edit and preview the content).
Ability to publish via git.
Ability to host your blog on a static web-server, like GitHub Pages.
No database required.

How ?

I often do “bang-documentation”, meaning that I only provide the key steps to achieve the goal, assuming that you are able to find more detailed documentation by yourself if needed. If a part is unclear, though, please leave me a comment and I will fix it !

I used Jekyll Bootstrap, which use pre-built HTML Bootstrap pages instead of Liquid templating : directly editing HTML seems to me easier than learning a new templating language.

Here is the process I followed to build this blog, you can find the original quickstart guide here :

Fork https://github.com/plusjade/jekyll-bootstrap.git or clone/push it into a new public / private repository

 git clone https://github.com/plusjade/jekyll-bootstrap.git jekyll-blog
 cd jekyll-blog
 git remote set-url origin git@github.com:USERNAME/USERNAME.github.com.git
 git push origin master:gh-pages

Add a CNAME file and entry in your DNS zone

 echo "blog.securem.eu" > CNAME

 git add CNAME && git ci -m "Add CNAME file" && git push
 # You will also need to edit your DNS zone to point the right subdomain to your github pages (username.github.io)

Run gem install --user-install jekyll on your computer to install Jekyll
Configure /_config.yml to your needs (for example, the blog’s name or a Piwik / Google Analytics tracking ID)
Launch the built-in webserver with jekyll serve then head to http://127.0.0.1:4000/ (will autoreload the server when you create/edit content, but it doesn’t watch the /_config.yml file)
Create your first blog post with
```
 rake post title="My first blog post !"
```
Then edit the newly created file in /_posts/.

Choose a nice theme from http://themes.jekyllbootstrap.com/ and install it with

 rake theme:install git="https://github.com/jekyllbootstrap/theme-the-minimum.git"

Change the landing page in /index.md. Pages can be placed anywhere (root directory or subdirectory to class them) but must contain the YAML Front-matter header.
Commit, push, and enjoy !

My old blog has been archived

Sat, 13 Dec 2014 00:00:00 +0000

I was talking about my previous blog in my article about Jekyll and Github Pages, it has been archived today and will still be accessible at archive-blog.securem.eu.

I will still be talking about web development, new technologies, and network security here, and in English now. I often use my blog as a reminder for complex manipulations or settings, but I also love to share things I may find or reviews of things I may try. Maybe the next post will be talking about performance of DNS Tunnels ! ;)

Jekyll Rake cheatsheet

Sat, 13 Dec 2014 00:00:00 +0000

When you create blog posts or pages with Jekyll, here are the Rake commands to remember :

# Only the title parameter is mandatory
rake post title="Jekyll Rake cheatsheet" tags=['Publishing', 'Markdown'] category="Tips and tricks" [date="2014-12-13"]

rake page name="about.html"
rake page name="subcategory/niceformatting/index.html"